Reducing TCZ extension at 68% of their original size

[Roberto A. Foglietta]

Hi all,

 I made some test using mksquashfs and I have tried some options. Included the compression alghorits.

 I have found very promising two of them lzma and xz. The last could be optimised for x86, arm, etc. ​I am not sure if the optimisation refers to the host or to the guest, I think the guest otherwise it would be automatically determined.

 However, these are the results:

90084 KB - ​original size
64780 KB - lzma size
64628 KB - ​xz size
61040 KB - ​xz x86 size
 
 How about the loading time? I took the boot time three times for each of them:

60.44, 56.80, 56.05 - original time
56.27, 54.60, 54.24 - xz time
55.66, 55.61, 55.05 - xz 86 time
failed to load extensions - lzma

 So the lzma should be discarded but xz preformed very well. I hope this helps. R-

[Roberto A. Foglietta]

I add the script that converts all tcz in a folder, only if recompression is required.

[Roberto A. Foglietta]

In the next release, zstd could be used instead of xz

To include tce based on zstd compression the libraries need to be added in rootfs.gz

This are a comparative benchmark of zstd

https://facebook.github.io/zstd/

Here the comparison between zstd vs xz

https://sysdfree.wordpress.com/2020/01/04/293/

I have tried with zstd compression:

96864 KB - original size
63800 KB - xz size
74544 KB - zstd size

but it does not seem to overbeat the xz which remain the best size-oriented


I hope this helps, -R

[curaga]

If you search the forum, you will find all these have already been considered. You will also find they require greatly more RAM, preventing their use on low-end 32-bit systems.

[Roberto A. Foglietta]

If you search the forum, you will find all these have already been considered. You will also find they require greatly more RAM, preventing their use on low-end 32-bit systems.

Searching in the forum I found this post that redirected me to Core Book

http://forum.tinycorelinux.net/index.php/topic,20545.msg128097.html#msg128097

It seems that xz will add 1MB of RAM for the dictionary for each extension, this is my system

- 55 extensions loaded
- 59 MB disk space for extensions
- 83 MB memory used (using xz)

Code: [Select]

roberto@vm-ubuntu18:~/tinycore-editor/tmp$ du -ks *
8 afterboot.sh
16857 boot
341 custom
3 extra
3 flags
31 patches
210 provides
5 sshkeys.pub
1 tcl-usb-boot-disable.gz
1 tcl-usb-boot-enable.gz
59114 tcz
roberto@vm-ubuntu18:~/tinycore-editor/tmp$ echo $[16857+59114]
75971
roberto@vm-ubuntu18:~/tinycore-editor/tmp$ ls -1 tcz/*.tcz | wc -l
55

root@box:~# free
              total        used        free      shared  buff/cache   available
Mem:         499276       83644      358316       21140       57316      381640
Swap:         97112           0       97112
root@box:~# df -h | grep tcloop | wc -l
55


[Roberto A. Foglietta]

This is my system using original gzip extensions:

- 55 extensions loaded
- 90 MB disk space for extensions
- 47 MB memory used (original)

It seems that xz extensions suck 36MB of RAM, on average 660KB for each extension.

Code: [Select]

roberto@vm-ubuntu18:~/tinycore-editor/tmp$ du -ks tcz
90418 tcz

froot@box:~# free
              total        used        free      shared  buff/cache   available
Mem:         499276       47344      396108       21136       55824      417944
Swap:        109172           0      109172


[nick65go]

@Roberto: The problem hard to over-come in Tinycore is that the design goal is for 32 bits, low/average CPU power (486) and low RAM (< 128 MB?)  <-- tiny!
 Of course THIS if good also for 64 Bits, average/powerful CPU, average/big RAM.

I think for "modern" machines, with >1 GB RAM and an i686 CPU (as are almost all NEW-BUY linux machine produced in the last 8 years) that optimized (special-purpose, home-made specific) scripts, could be produced for x86_64. But this involves different scripts for different architectures (arm / x86 / x86_64), which implies more maintenance.
FYI: 36 MB "wasted" for a machine with [1 - 2] GB RAM can be acceptable (xz compression in kernel), but not for old (second hand) machines still used in "3-rd" countries.

PS: most machines are designed to fail after few (max 6) years; with expensive  to replace (or hard to find) spare parts, so to force the "user" to buy new hardware.

EDIT: in tc ver. [8 -12] the Xorg.tcz drag with it near 102 other tcz. But combining all of them in a pseudo Xorg-full.tcz you get boot speed...(and not 1MB wasted on each Xorg dependency, if xz compressed tczs and kernel support to mount squshfs xz modules).

[Roberto A. Foglietta]

EDIT: in tc ver. [8 -12] the Xorg.tcz drag with it near 102 other tcz. But combining all of them in a pseudo Xorg-full.tcz you get boot speed...(and not 1MB wasted on each Xorg dependency, if xz compressed tczs and kernel support to mount squshfs xz modules).

Thanks for this suggestion, using unionfs-fuse to merge tcz I reduced my boot time from 55 seconds to 13 seconds! 

[curaga]

Also, we did okay the other compression algos for x86_64. It's not required to use gzip when submitting 64-bit extensions.

[Roberto A. Foglietta]

Also, we did okay the other compression algos for x86_64. It's not required to use gzip when submitting 64-bit extensions.

For x86_64 you might want use XZ for example. It takes RAM but x86_64 are plenty of that usually.

A the moment, I compressed 55 extensions in 4 meta packages: sshonly, usbkey, develop, git.

Very fast and and low RAM consuming: 20MB less than loading the original extensions separately!

Code: [Select]

root@box:~# free
              total        used        free      shared  buff/cache   available
Mem:         499276       26776      421200       20252       51300      439396
Swap:        102720           0      102720


[Roberto A. Foglietta]

When I merge the tcz in a single tcz the installation scripts goes in the same directory but they are not executed probably because the name of the extention does not match any of them. This could be a problem. Do you know how to address it?

Code: [Select]

roberto@vm-ubuntu18:~/tinycore-editor/tinycore/tcz/tmp/usr/local/tce.installed$
total 3
drwxrwxr-x 2 root    staff  91 ott  5  2015 ./
drwxr-xr-x 9 root    root  145 nov  2  2018 ../
-rwxr-xr-x 1 roberto staff  69 ott  5  2015 binutils*
-rwxr-xr-x 1 roberto staff 399 mag 25  2017 ca-certificates*
-rwxr-xr-x 1 roberto staff 219 gen  3  2021 libtirpc*
-rwxr-xr-x 1 roberto staff 527 apr 28  2019 openssl-1.1.1*


[Roberto A. Foglietta]

When I merge the tcz in a single tcz the installation scripts goes in the same directory but they are not executed probably because the name of the extention does not match any of them. This could be a problem. Do you know how to address it?

I have solved the problem improving the script which makes the merge of TCZs

https://github.com/robang74/tinycore-editor/blob/main/tinycore/tczmetamerge.sh

[Roberto A. Foglietta]

Also, we did okay the other compression algos for x86_64. It's not required to use gzip when submitting 64-bit extensions.

Hi Curaga, in my TinyCore Editor (Suite) the XZ compression takes places only if the architecture is x86_64

Thank you for the suggestion, -R