Tiny Core Base > TCB Tips & Tricks

advdef ver.2.1 versus 7z ver.26

<< < (2/3) > >>

Rich:
Hi nick65go

--- Quote from: nick65go on March 13, 2026, 08:52:22 AM --- ... 2. The decompressed size in RAM is/must/should be the same because is LossLESS algorithm, but size on disk/ storage matters also (a little?) because is the boot-loader speed to read+load rootfs.gz in memory FROM HDD/USB/CDROM. ...
--- End quote ---
Boot loader speed will also be influenced by the storage device:
CDROM  Slowest
USB    Faster
HDD    Much faster
SSD    Fastest
Loading time will also be influenced by compression level.
As the compression level increases, the CPU has to work harder
to decompress the file and take longer to do so.

An interesting experiment would be to boot with an uncompressed CPIO.
Then see if the amount of required RAM decreases.
And measure the impact on boot time.

nick65go:
Hi @Rich, I try share my experiments with TC, to be constructive, and learn (again) things that I forgot or that there were never answered on time so I move out.
My experiments are usually in virtual machines or seldom in chroot from trusted/signed distros (Arch, Alpine). I mean more eyes to check things, faster patches, updates (the usual bull-sheet motives that there are so many linux distro). Also my current machine is more powerful than I really need.

--- Code: ---> inxi
CPU: 10-core (2-mt/8-st) 13th Gen Intel Core i5-1335U (-MST AMCP-)
speed/min/max: 400/400/4600:3400 MHz Kernel: 6.19.6-2-cachyos x86_64 Up: 2m
Mem: 2.6/31.06 GiB (8.4%) Storage: 476.94 GiB (6.4% used) Procs: 276
Shell: fish inxi: 3.3.40
--- End code ---

This means that in VM the speed is slower than in real life; or on host the speed is too fast to matters for me (cold boot in 5-7 seconds into firefox). I am interested in size and speed for NOMAD / RESCUE minimal distro CONCEPTS and proved facts. Like prepare for the war if you want peace. Such as when I will not be able to afford to buy another computer because cost, embargo, surveillance, or I do not like the new CPU manufacture spies.

Paul_123:

--- Quote from: nick65go on March 13, 2026, 08:52:22 AM ---1. Really? this is "disturbing", that using same original compression program offered by TC and the same input original TC data, for a single-threaded advdef you said I can not obtain the same result.

--- End quote ---

Just unpacking the cpio archive, then repacking it changes it.  Even if you do not change the content.   Take compression out of the equation.

Decompress the initrd, now you have an uncompressed cpio archive.  Now run a sha256sum for that initrd.

extract the archive through cpio......then recreate it, but don't pass it through gzip.   Now run the sha256sum again.   I would expect different checksums.

nick65go:
Lets do not confuse sha256sum with file size in Bytes. Reproducibility depends also of file calendar data (EPOCH). So please publish your results of using better tools (pipz) to make rootfs.cpio.gz.

Now back in bussiness/ topic of advdef: After I read wiki about deflate64 (advdef) versus deflate (gzip), it seams is about dictionary (32KB -> 64 KB) but also about length of strings in the window/range chunks to compress. So I have the idea to pre-sort (the extracted rootfs.gz), files with smaller size are first in the list send to cpio. My results below, using 7zip on sort(x0.cpio) and not-sorted cpio (123.gz), then advdef on x1.cpio (pre-sorted) or x2.cpio (not-sorted)
--- Code: --- sudo find -type f -printf '%s %p\n' | sort -n | cut -f2 -d" " | sudo cpio -o -H newc > ../x1.cpio
18057 blocks 
#normal un-sorted cpio
 sudo find | sudo cpio -o -H newc > ../x2.cpio 
18182 blocks
> ls -alB ../x*
.rw-r--r-- 9.245.184 abc 13 Mär 19:16  ../x1.cpio
.rw-r--r-- 9.309.184 abc 13 Mär 19:16  ../x2.cpio
--- End code ---

--- Code: ---> gzip -1 x1.cpio
> gzip -1 x2.cpio
> ls -alB ./x*.gz
.rw-r--r-- 3.677.872 abc 13 Mär 19:20  ./x0.gz
.rw-r--r-- 4.218.542 abc 13 Mär 19:16  ./x1.cpio.gz
.rw-r--r-- 4.230.975 abc 13 Mär 19:16  ./x2.cpio.gz
--- End code ---

--- Code: ---> ./advdef -z4 x1.cpio.gz
     4218542     3695808  87% x1.cpio.gz
     4218542     3695808  87%

> ./advdef -z4 x2.cpio.gz
     4230975     3701897  87% x2.cpio.gz
     4230975     3701897  87%

> ls -alB ./x*.gz
.rw-r--r-- 3.677.872 abc 13 Mär 19:20  ./x0.gz       ; 7zip of sorted X0.cpio
.rw-r--r-- 3.695.808 abc 13 Mär 19:25  ./x1.cpio.gz  ; advdef of x1.cpio (pre-sorted)
.rw-r--r-- 3.701.897 abc 13 Mär 19:26  ./x2.cpio.gz  ; advdef of x2.cpio not-sorted
--- End code ---
so with pre-sorted x0.cpio I obtained with 7z even a better rootfs.gz , than initial reported
--- Code: ---.rw-r--r--  3.684.655 abc 13 Mär 11:55  123.gz
--- End code ---
summary: best in class solution (until now): 7z ver.26 with per-sorted cpio archive.

Paul_123:
One of your stated goals was trying to exactly produce the distributed initrd.gz file, which I was trying to help you understand was impossible.

If your definition of better is the compressed size of the initrd, then why not use xz or zstd to compress the image.
If your definition of better is boot time, did you try a uncompressed initrd, or just a gzip -1 compressed file.

Navigation

[0] Message Index

[#] Next page

[*] Previous page