Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

This is an interesting take on things.

Decompressing using a different algorithm like .bz2 or .xz is noticeably slower, so sticking with gz and shuffling the files might split the difference.



This would come with a cost of longer compression times - either multiple attempts with random shuffling or pre-compression file ordering optimization process. For resources that are compressed once and then distributed and decompressed multiple times this would be quite interesting solution


I got curious and looked it up. On this page [1] it looks like uncompressing with gzip vs bzip2 vs xz is:

          gzip    bzip2   xz
  1       6.771   24.23   13.251
  2       6.581   24.101  12.407
  3       6.39    23.955  11.975
  4       6.313   24.204  11.801
  5       6.153   24.513  11.08
  6       6.078   24.768  10.911
  7       6.057   23.199  10.781
  8       6.033   25.426  10.676
  9       6.026   23.486  10.623
so gzip has the fastest decompression.

That said, xz is in the ballpark and can be significantly smaller.

[1] https://www.rootusers.com/gzip-vs-bzip2-vs-xz-performance-co...


zstd is faster and smaller. If you can choose the format, zstd beats deflate across the board, on every front except for compatibility with things that only understand deflate.

Also, if you need to use deflate for compatibility, use https://github.com/zlib-ng/zlib-ng , which is substantially faster than either zlib or gzip.


I've never heard of zstd (thanks!), but it seems it's right under my nose.

"Arch Linux added support for zstd as a package compression method in October 2019 with the release of the pacman 5.2 package manager, and in January 2020 switched from xz to zstd for the packages in the official repository. Arch uses zstd -c -T0 --ultra -20 -, the size of all compressed packages combined increased by 0.8% (compared to xz), the decompression speed is 1300% faster, decompression memory increased by 50 MiB when using multiple threads, compression memory increases but scales with the number of threads used."

https://en.wikipedia.org/wiki/Zstandard


Is it though? In https://quixdb.github.io/squash-benchmark/unstable/ deflate beats zstd at least in some tests.


Yeah, but zstd beats deflate in others. For example the latest MySQL 8 https://dev.mysql.com/worklog/task/?id=13442


It's universally faster if you compress threaded, and otherwise it's generally still a win.


Yeah, and it is IETF standard since 2018 https://tools.ietf.org/html/rfc8478


I wonder how zstd would compare.


this is my go-to comparison of different compression algorithms [0]

TL;DR zstd is faster, smaller and has a wider range than gzip. For maximum compression, plzip seems to be the best.

[0] https://community.centminmod.com/threads/round-4-compression...




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: