Tiny Core Base > TCB Talk
Sugestion: XZ compression in kernel + tcz; squashfs with xz + biger block size
nick65go:
thanks to andyj, the main thing to take home (for me) is that the actual compression gz (used in tcx64) is not the winner. zstd beating gz (for modest tc-users), both in size and decompresion speed, is a revelation :)
EDIT: i am talking about (relativly) big sized tcz (libreoffice, firefox), or eventualy personaly combined tcz into only one (remaster) - when HDD/USB size is not constrained, and flash USB/SSD speed helping ever better versus rotational HDD.
hiro:
--- Quote from: andyj on May 17, 2019, 06:02:27 PM ---Dictionaries are good for words, which is why text files compress so much better than binaries. Since the majority of extensions are stripped binaries, compression might be working against us except when the binaries have lots of embedded strings.
--- End quote ---
*compression* is easier with low entropy stuff like words.
dictionaries help smoothen the trade-offs with smaller block-sizes, in all relevant cases (including binary data).
andyj:
--- Quote from: andyj on May 17, 2019, 06:02:27 PM ---... compression might be working against us ...
--- End quote ---
What I was trying to say here is that the tradeoff between space saved and time spent decompressing depends a lot on how often something has to be decompressed. The kernel modules are already compressed before they are archived into the squashfs file. Compressing an already compressed file more often than not uses more space to support the overhead while not gaining on space saved. But the files are only decompressed once when insmod is run, so the tradeoff makes sense for consistency's sake. Programs which are run frequently would have to be decompressed every time, which will add up unless we put the extension in copy2fs.lst. Bottom line, compression can be beneficial but it's not always the case.
Rich:
Hi andyj
--- Quote from: andyj on May 19, 2019, 09:24:58 AM --- ... the tradeoff between space saved and time spent decompressing depends a lot on how often something has to be decompressed. ...
... Programs which are run frequently would have to be decompressed every time, which will add up unless we put the extension in copy2fs.lst. ...
--- End quote ---
That depends on the size of the programs and the amount of installed RAM. It also depends on what other activities were performed
on the computer since the last time you ran the program (read a large file into RAM or execute cache-clear for example).
If you have lots of RAM your program may still be in the cache from the last time you ran it. As long as it remains in the cache it
doesn't need to be decompressed again.
andyj:
--- Quote from: Rich on May 19, 2019, 09:56:46 AM ---If you have lots of RAM your program may still be in the cache from the last time you ran it. As long as it remains in the cache it doesn't need to be decompressed again.
--- End quote ---
Actually, rootfs is basically an ever present ramfs, which is basically cache mapped as a file system (as I understand it, because it would be rather inefficient to have a RAM cache for a RAM based file system), so unless you are using copy2fs (because you have lots of RAM) then the compressed file (the squashfs archive) will be the only file in RAM. Thus, every access to any file in an extension will require decompression.
Navigation
[0] Message Index
[#] Next page
[*] Previous page
Go to full version