We are working on a pre-processing pipeline for a large neuroimaging dataset, and we want to be sure we are being judicious with our disk space usage. .nii Files are, conventionally, compressed with the program gzip (sometimes wrapped around a tape archive or tar file). Gzip is ubiquitously available, has a low memory footprint, and does an ok job. However, there are other perfectly mature, lossless compression formats available which get better results. If you are working with >100TB of data, this could matter a lot to your operating costs. Since compression performance is dependent on the type of data you had, I wanted to compare the efficiency of a number of algorithms and see what our options were.
Algorithms we are comparing.
Gzip and memcpy are included for comparison. Other compression tools were chosen based on their apparent popularity (from other compression tests published online or because of their inclusion in turbobench’s ‘standard lineups’) and to give a good range of datapoints from fast, minimally compressed to slow, highly compressed:
Blosc at level 11 was stopped manually after running for >12 hours.
Each tool was tested once on an HTPC instance with 1 processor and 8GB of memory. I additionally evaluated some methods with an instance with 4 processors and 32 GB of memory, but didn’t see large differences. Possibly Turbobench does not account for multithreading appropriately. I probably did not do this correctly – one thread is our target use case, so I did not spend a lot of time on multithreading.
Note that I am not positive the processors on the various HTPC servers used were identical, so there may be some noise in the timing data.
The tool I ended up using for most of the comparisons is called TurboBench, which has the advantages that it tests strictly in memory, has a lot of compression algorithms available, is flexible, and was easy for me to run on our HTPC cluster.
One thing Turbobench does not do is test gzip. Potentially one of the algorithms it offers is identical to gzip’s but I could not discern that, so I tested gzip using a separate script.
I was very curious about a library called blosc. Discussion on the github for NRRD suggested it might be ideal for this application. However, the lack of easily available command line tools for its use made me give up on it.
All these analyses were run at UW-Madison at CHTC using HTCondor. Code for analysis is available on the github repo.
The full data table for this analysis is in the github repository as ‘fulldata.Rds’. I’m only going to plot points that are optimal on some dimension, and I’ll exclude a few outliers.
In general, it is the compression benchmarks that seem to vary the most. Decompression is not much over 30 seconds even for the most time intensive method. flzma2 is a clear winner in these trials, with about 4% more compression than gzip. Flzma2 is not commonly available, and it would be best if we could use something less obscure. It is a fast implementation of LZMA, which is available in the package xz, so let’s compare those:
Lzma at level 6 is within 1.5% of flzma2 at level 9, and is faster and uses less memory. So that’s probably our winner. It’s also the default setting of xz. As a bonus, xz supports integrity checking as a built in, which is very nice.
Here’s a plot of all the ‘lzma’ methods:
Mind the scales – the compression ratios are not actually that different here.
‘Real World’ testing
So the above testing is using just memory to memory compression, which is not the environment where our compression will actually happen. What about when we do this with disk i/o?
$ /usr/bin/time -f 'time: \t%e realSeconds \t%M peakMem' xz -zk subject.tar time: 1525.79 realSeconds 97608 peakMem $ ls -l subject.* -rw-rw-r-- 1 peverill peverill 3045427200 Dec 16 09:37 subject.tar -rw-rw-r-- 1 peverill peverill 2386532328 Dec 16 09:37 subject.tar.xz
So xz (lzma level 6) takes 25.4166667 minutes to compress the data, achieves a compression ratio of 0.7836445, and uses 97.6 MB of memory. It also appears to embed a file integrity check automatically. Sounds good!
What about Blosc?
The promise of Blosc for this type of data is that by using a pre-filter, it can better take advantage of the fact that a nifti file is ultimately an array of 16bit numbers, and the most significant digits don’t change that much (most compression algorithms do not account for this, but blosc’s pre-filtering options do). Don’t quote me on that, I’m following this forum post.
I tried a few times to get this working with various tools, but could not realize gains (certainly not to the extent to justify using a less mature tool).
With the compress_file program packaged with c-blosc2:
$ /usr/bin/time -f 'time: \t%e realSeconds \t%M peakMem' ./c-blosc2-2.6.0/build/examples/compress_file subject.tar subject.tar.b2frame Blosc version info: 2.6.0 ($Date:: 2022-12-08 #$) Compression ratio: 2904.3 MB -> 2710.9 MB (1.1x) Compression time: 11.2 s, 260.3 MB/s time: 11.15 realSeconds 5344 peakMem
With bloscpack using default options:
$ /usr/bin/time -f 'time: \t%e realSeconds \t%M peakMem' \ python3 packages/bin/blpk -v -n 1 c subject.tar blpk: using 1 thread blpk: getting ready for compression blpk: input file is: 'subject.tar' blpk: output file is: 'subject.tar.blp' blpk: input file size: 2.84G (3045427200B) blpk: nchunks: 2905 blpk: chunk_size: 1.0M (1048576B) blpk: last_chunk_size: 354.0K (362496B) blpk: output file size: 2.49G (2668748652B) blpk: compression ratio: 1.141144 blpk: done time: 8.15 realSeconds 44392 peakMem
The same, but using the zstd algorithm:
$ /usr/bin/time -f 'time: \t%e realSeconds \t%M peakMem' python3 packages/bin/blpk -vn 1 c --codec zstd subject.tar blpk: using 1 thread blpk: getting ready for compression blpk: input file is: 'subject.tar' blpk: output file is: 'subject.tar.blp' blpk: input file size: 2.84G (3045427200B) blpk: nchunks: 2905 blpk: chunk_size: 1.0M (1048576B) blpk: last_chunk_size: 354.0K (362496B) blpk: output file size: 2.15G (2306001080B) blpk: compression ratio: 1.320653 blpk: done time: 134.08 realSeconds 51328 peakMem
Finally, to make sure that I was using bit-shuffling (which is supposedly where the magic happens), I wrote a custom version of the compress_file program. Assuming I did that right, here is the output:
$ /usr/bin/time -f 'time: \t%e realSeconds \t%M peakMem' c-blosc2-2.6.0/build/examples/compress_file subject.tar subject.tar.b2frame Blosc version info: 2.6.0 ($Date:: 2022-12-08 #$) Compression ratio: 2904.3 MB -> 2397.1 MB (1.2x) Compression time: 52.3 s, 55.5 MB/s time: 52.34 realSeconds 9084 peakMem
In fairness, the best version (zstd using bloscpack) compressed the file at 75.7% in just over two minutes, using 51MB of ram – much superior to lzma. Also, all of these tests used typesize=8, and possibly it should be 16. However, it’s not enough of a benefit to justify the additional complexity (and I ran out of time exploring it).