Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

bzip2 can take advantage of any number of CPU cores when compressing.


Bzip2 doesn't handle multiple cores as far as I'm aware, but tools such as pbzip2 can. I wrote about this some time ago: https://hackercodex.com/guide/parallel-bzip-compression/

That said, parallel XZ is even better: https://github.com/vasi/pixz



I don't believe that's true, though there are multiple projects that offer that feature. lbzip2.org and compression.ca/pbzip2 to name a couple.


Really? How? My bzip2 has no option for it and when tested it stuck to one CPU. xz on the other hand has

  -T, --threads=NUM   use at most NUM threads; the default is 1; set to 0
                      to use the number of processor cores


Side note: `xz` only got -T option in stable releases less than 12 months ago (5.2.0 in 2014-12-21), so it hasn't made it into every distro yet.


The xz installed on my system carries a rather promising -T option, but then this text below it

> Multithreaded compression and decompression are not implemented > yet, so this option has no effect for now.


I believe you're looking for pbzip2. Parallel bzip2 file compressor. I've replaced it as my go-to compression.


lbzip2 is pretty good


pbzip2 output isn't universally readable by third-party bz2 decompressors (Hadoop, for example).




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: