squashfs zstd vs lz4

[CardealRusso]

I emailed an update to squashfs but thought it more appropriate to open up the default compression used for discussion.

The content of the email I sent to tcesubmit@gmail.com I will leave here:

The reason for this topic was because, right after sending the email, I tested lz4 (which was disabled in the squashfs present in the tinycore repository, and I didn't even know lz4) and the result was as surprising as zstd.

While zstd generates the very small tcz file it increases the memory usage considerably, chromium made the system memory usage at almost 420mb while lz4 left the system memory usage at 370mb. chromium was not running, simply mounted on the system! the difference in favor of zstd was that while tcz lz4 was at 180mb, tcz zstd was at approximately 108mb.

[CardealRusso]

package compression/decompression is instantaneous with lz4.
lz4 was, confusingly, faster than using mksquashfs with -noI, -noF, -noD (disabled compression) (I assume it was a disk deficiency, as uncompressed chromium was nearly 800mb)

[CardealRusso]

I see no good reason to continue using gzip with these two excellent options available.
the question here is: exchange some mbs from the tinycore server for some free mbs in the client's memory or make the client's memory usage higher in exchange for saving some mbs from the tinycore server

i uploaded the update which switches the default to zstd but honestly after testing lz4 i prefer lz4.

[CardealRusso]

sorry, I'm having to split the message because the server refuses to accept long messages.

[Juanito]

If you search these forums, there was a lengthy discussion on this subject

[Rich]

Hi Juanito
Do you mean this one:
https://forum.tinycorelinux.net/index.php/topic,22854.0.html

[CardealRusso]

If you search these forums, there was a lengthy discussion on this subject

Most ignore lz4 as it is a relatively new algorithm. I don't think it's enough to take into account lagged discussions.

Hi Juanito
Do you mean this one:
https://forum.tinycorelinux.net/index.php/topic,22854.0.html

Lagged. zstd surpassed xz a long time ago.

[Juanito]

Do you mean this one:
https://forum.tinycorelinux.net/index.php/topic,22854.0.html

Yes - thanks   

[CardealRusso]

Do you mean this one:
https://forum.tinycorelinux.net/index.php/topic,22854.0.html

Yes - thanks   

They should have joined xz when it was discussed, zstd when it was discussed, and should join lz4 now.

Discussions are slow and new algorithms are released periodically. I don't think we should argue about the old ones provided, but rather look for some reason to not use lz4.

If it's an aimless discussion, this will be just another topic forgotten in time like the others you refer to.

I'll send an email with squashfs with lz4 as the default and hope it gets accepted.

[Juanito]

I'll send an email with squashfs with lz4 as the default and hope it gets accepted.

The tinycore developers are the maintainers of the squashfs-tools extensions, so it will not be accepted.

If there's an agreed change to block size or compression type it needs to follow on from testing on x86, x86_64, armv* and aarch64 and might be different on each architecture.

[GNUser]

Changes to said defaults would affect hundreds and eventually thousands of extensions, and would impact every TCL user. So it seems appropriate that the threshold for testing and accepting changes to squashfs-tools is very high. Thank you, Juanito.

[CNK]

While zstd generates the very small tcz file it increases the memory usage considerably, chromium made the system memory usage at almost 420mb while lz4 left the system memory usage at 370mb. chromium was not running, simply mounted on the system! the difference in favor of zstd was that while tcz lz4 was at 180mb, tcz zstd was at approximately 108mb.

So it's a RAM Vs file size trade-off and you value RAM more. You really should have noted the figures for Gzip compression too (the x86_64 chromium-browser.tcz extension in TC 14 is 87MB, so I guess you're not using that).

In late Feb I compared these compressors in TC13 on x86_64 for compressing mydata.tar (uncompressed mydata.tgz, 112.2MB): gzip/BusyBox: BusyBox v1.34.1, lz4: 1.9.2, zstd: 1.5.0, xz: 5.2.5, bzip2: 1.0.8

I noticed that LZ4 couldn't match the compression of Gzip. I mainly boot Tiny Core from SD cards (slow), so for me LZ4 compared poorly to Zstandard which came close to the same decomression speed of LZ4 (both under 1/3rd the decompression time with Gzip) yet could also compress to a smaller file size than Gzip.

I did some more brief testing recompressing firefox.tcz (generated by firefox_getLatest.sh), which is the largest extension that I use at (then) 102.3MB. At compression level 19 Zstd reduced this to 99.4MB, but I didn't think this was really significant though. Especially because, in spite of the mydata.tar tests, loading and starting Firefox took about the same time as for the Gzip-compressed extension. I concluded that there wasn't a significant enough advantage to using Zstandard instead of Gzip for extensions.

I did find that compression of a modified rootfs-13.1.0 file for PiCore was more significantly better with Zstandard than Gzip: 2.2MB with zstd -19, 2.8MB with gzip -9. Decompression time was also slightly better for Zstd than Gzip, so seeing as I was rebuilding it anyway I decided to compress with Zstandard because that's already supported in the PiCore's build of the Linux kernel (CONFIG_RD_ZSTD).

Linux also supports Zstandard compressed vmlinuz files since kernel version 5.9, but I think that needs the kernel to be rebuilt in order to switch the compression method that's built in to the start-up system.

Overall my conclusions (right or wrong) were:

• Zstandard compression of mydata.tar is probably worthwhile, at least for mine which is mainly Firefox profile textual data in ~/.mozilla
• Zstandard compression of extensions isn't worth it (not much smaller and somehow not much decrease in program start-up time on first run)
• Zstandard compression of the rootfs file/s should have a slight benefit for boot time
• Zstandard compression of vmlinuz has an unknown advantage and it probably isn't worth the trouble to build the kernel just to try and test it
• LZ4, XZ, BZIP2, are worse than Gzip - either much slower (XZ, BZIP2) or larger compressed files (LZ4) at all compression settings tested

I understand that Zstandard's main advantage over Gzip is for compressing text-based files that are at least a few megabytes in size, because then it can use its dictionary-based techniques for improving compression.

[jazzbiker]

The tinycore developers are the maintainers of the squashfs-tools extensions, so it will not be accepted.

So nice :-)

[Rich]

Hi Juanito

... If there's an agreed change to block size or compression type it needs to follow on from testing on x86, x86_64, armv* and aarch64 and might be different on each architecture.

I feel that keeping the compression parameters consistent across architectures and
TC versions is a good thing. About 3 years ago I had to fix a script in most of the
window managers in all of the architectures for TC10, TC11, and TC12:
https://forum.tinycorelinux.net/index.php/topic,22823.msg153740.html#msg153740

Thanks to this consistency, I was able unsquash, fix, and re-squash all of those
extensions from a machine running TC10 x86.

I did not need to boot into a 64 bit system to fix the 64 bit extensions.
I did not need to buy ARM systems to fix the ARM extensions.
There was no risk of me picking the wrong compression.
There was no risk of me picking the wrong block size.

Consistency is a good thing.

[CardealRusso]

Hello. Sorry for the confusion, but I withdraw my recommendation to use lz4 or zstd.

I made a mistake comparing the 128k block size zstd/lz4 with the tcz from the tinycore repositories which are 4k. I realized that the difference is not (much) in the compression algorithm but simply in the block size.

I redid the tests and, honestly, gzip seems to be enough, only the block size would be interesting to increase.