Because DEFLATE uses a small 32KB window, this task is just matching the last 32KB of one file with the first 32KB of another file for most redundancy. A tool can do those estimations to find the best matching file to put in the archive after the current file, without doing the huffman coding when doing it.
For cases like multiple checkouts of the same repo in a single archive, you want real long range compression like lrzip or zstd --long.
And in the end, using zstd would be a win over DEFLATE every time. Just stop using DEFLATE unless you have to for compatibility with something that can't do zstd.
The fact that most of us use DEFLATE without ever realizing that it's optimized for hardware specs from many decades ago, and especially this 32KB window part, is is pretty much the "grandma's cooking secret"[0] of compression, isn't it?
That is the beauty of deflate! It was fast on hardware many decades ago and it does well on today’s hardware as well. It has found a real sweet spot between execution time and memory consumption. You can pick up a few extra percent here and there but often requiring massive increases in one and/or the other, or specific knowledge of what your compressing (images, audio, assembker output, etc)
No, compression technology has advanced a lot since DEFLATE. Today's general purpose compressors beat DEFLATE at all points on the Pareto frontier (any network/disk speed, any compression ratio vs CPU time trade-off). Use zstd. If you need even more speed, use lz4. If you need even more compression, use LZMA (or if it's natural language text use ppmd).
Agree that zstd has performance but memory requirements are potentially orders larger, there is no browser support, and you can’t count on people having decompressor installed. It’s good if your compressing data for your use but it doesn’t have the reach of deflate. Maybe that will change in the future.
How many orders larger, and larger than what? The 32kb? A megabyte is two orders larger than that, a gigabyte five orders larger. We got quite a bit of leeway here..
zstd and lz4 can be run in small memory footprints and without heap memory allocation and I bet zstd would beat DEFLATE in 32KB too (just not as much).
It's not as popular yet, yes, but for internal use, including using HTTP content encoding if both your server and client support it, it's great.
For cases like multiple checkouts of the same repo in a single archive, you want real long range compression like lrzip or zstd --long.
And in the end, using zstd would be a win over DEFLATE every time. Just stop using DEFLATE unless you have to for compatibility with something that can't do zstd.