advdef ver.2.1 versus 7z ver.26

[nick65go]

I tried but I can not reproduce the rootfs.gz of TC17 32 bits. I use an OS 64 bits. advdef (of TC17-64bits) gave me a bigger than original size,  but 7z (from x86-64-v3) gave me a smaller size than original size. gzip parameters (-1 until -9) did not help me with TC advdef.
It means that latest 7zip version 26 has a better Deflate64 encoder than for Advdef (still using deflate64 - as wikipedia said).

Code: [Select]

> sudo find | sudo cpio -o -H newc | sudo gzip -1 > ../456.gz
18182 blocks

> ./advdef -z4 456.gz
     4230967     3701889  87% 456.gz
     4230967     3701889  87%


Code: [Select]

> sudo find | sudo cpio -o -H newc | sudo gzip -9 > ../456.gz
18182 blocks
> ./advdef -z4 456.gz
     3846705     3701889  96% 456.gz
     3846705     3701889  96%

> ./advdef -V
advancecomp v2.1 by Andrea Mazzoleni, http://www.advancemame.it

Code: [Select]

> sudo find | sudo cpio -o -H newc ../123.cpio

> 7z a -tgzip -mx9 123.gz 123.cpio

7-Zip 26.00 (x64) : Copyright (c) 1999-2026 Igor Pavlov : 2026-02-12
 64-bit locale=en_US.UTF-8 Threads:12 OPEN_MAX:1024, ASM

Scanning the drive:
1 file, 9309184 bytes (9091 KiB)
Creating archive: 123.gz
Add new data to archive: 1 file, 9309184 bytes (9091 KiB)
Files read from disk: 1
Archive size: 3684655 bytes (3599 KiB)
Everything is Ok

Code: [Select]

> ls -alB
drwxr-xr-x          - abc 13 Mär 12:18  .
drwxr-xr-x@         - abc 13 Mär 12:12  ..
drwxr-xr-x          - abc 13 Mär 12:10  tmp
.rw-r--r--  9.309.184 abc 13 Mär 10:49  123.cpio
.rw-r--r--  3.684.655 abc 13 Mär 11:55  123.gz
.rw-r--r--  3.701.889 abc 13 Mär 12:18  456.gz
.rwxr-xr-x     97.456 abc  3 Nov  2018 󰡯 advdef
.rw-r--r--  3.697.459 abc  8 Mär 15:22  rootfs.gz

[Paul_123]

Advdef and gzip are single threaded programs.  Not sure what 7zip is doing but I use pigz, and you can definitely smash gzip and advdef files smaller by throwing multiple threads iterating on it.   But its a case of how much CPU power do you want to throw at it.

You examples above are less than 1% difference in size.  Its just for storage anyway, the initrd is going to be decompressed at boot anyway.

[nick65go]

@Paul_123 thanks for the info. Could you please post your similar results for rootfs.gz of TC17 32 bits, using pigz (or what ever compress better than 7zip) ?

1. My first complain is that I can NOT reproduce original TC work / size, which does not look good for me; but hey I am minority concerned by reproducibility of software ...

2. My (temporary) goal is to obtain MINIMM size for a rootfs.gz, to boot in a less than 64MB RAM (unfortunately I do not trust KolibryOS). Deftate structure allows ANY encoding algorithm (even DEFLATE64 form 7zip, Advcomp/advdef etc) and then using ANY compatible zlib/ gzip decoder (Wikipedia citation). So as long as compression is one time only (to create the rootfs.gz) and has (near) the SAME speed to decode many times, the tool did not matter, just the result. 
PS: from https://documentation.help/7-Zip/method.htm , i did not use switch -mmt12 (aka Sets multithreading mode), but maybe there were some defaults in 7z. I tried later with no improvement even with large page (-slp : set Large Pages mode), etc..

[Paul_123]

The rootfs is decompressed into RAM, so the size on the disk is not important.

Reproducing compression is nearly impossible….even though it is lossless.

[nick65go]

The rootfs is decompressed into RAM, so the size on the disk is not important.
Reproducing compression is nearly impossible….even though it is lossless.

1. Really? this is "disturbing", that using same original compression program offered by TC and the same input original TC data, for a single-threaded advdef you said I can not obtain the same result.

2. The decompressed size in RAM is/must/should be the same because is LossLESS algorithm, but size on disk/ storage matters also (a little?) because is the boot-loader speed to read+load rootfs.gz in memory FROM HDD/USB/CDROM. YMMV, same as your goal.

FYI: I do not intend to be pedantic/annoying, but for me the small RAM request is high priority versus CPU speed. Like with a slow CPU but enough RAM I wait 10 minute (exaggerate!) but run and solve the task/problem. Vice-versa, with fast CPU but not enough RAM I can not run, so is useless as a stealth nomad in wild environment.

[Rich]

Hi nick65go

... 2. The decompressed size in RAM is/must/should be the same because is LossLESS algorithm, but size on disk/ storage matters also (a little?) because is the boot-loader speed to read+load rootfs.gz in memory FROM HDD/USB/CDROM. ...

Boot loader speed will also be influenced by the storage device:

CDROM  Slowest
USB    Faster
HDD    Much faster
SSD    Fastest

Loading time will also be influenced by compression level.
As the compression level increases, the CPU has to work harder
to decompress the file and take longer to do so.

An interesting experiment would be to boot with an uncompressed CPIO.
Then see if the amount of required RAM decreases.
And measure the impact on boot time.

[nick65go]

Hi @Rich, I try share my experiments with TC, to be constructive, and learn (again) things that I forgot or that there were never answered on time so I move out.
My experiments are usually in virtual machines or seldom in chroot from trusted/signed distros (Arch, Alpine). I mean more eyes to check things, faster patches, updates (the usual bull-sheet motives that there are so many linux distro). Also my current machine is more powerful than I really need.

Code: [Select]

> inxi
CPU: 10-core (2-mt/8-st) 13th Gen Intel Core i5-1335U (-MST AMCP-)
speed/min/max: 400/400/4600:3400 MHz Kernel: 6.19.6-2-cachyos x86_64 Up: 2m
Mem: 2.6/31.06 GiB (8.4%) Storage: 476.94 GiB (6.4% used) Procs: 276
Shell: fish inxi: 3.3.40
This means that in VM the speed is slower than in real life; or on host the speed is too fast to matters for me (cold boot in 5-7 seconds into firefox). I am interested in size and speed for NOMAD / RESCUE minimal distro CONCEPTS and proved facts. Like prepare for the war if you want peace. Such as when I will not be able to afford to buy another computer because cost, embargo, surveillance, or I do not like the new CPU manufacture spies.

[Paul_123]

1. Really? this is "disturbing", that using same original compression program offered by TC and the same input original TC data, for a single-threaded advdef you said I can not obtain the same result.

Just unpacking the cpio archive, then repacking it changes it.  Even if you do not change the content.   Take compression out of the equation.

Decompress the initrd, now you have an uncompressed cpio archive.  Now run a sha256sum for that initrd.

extract the archive through cpio......then recreate it, but don't pass it through gzip.   Now run the sha256sum again.   I would expect different checksums.

[nick65go]

Lets do not confuse sha256sum with file size in Bytes. Reproducibility depends also of file calendar data (EPOCH). So please publish your results of using better tools (pipz) to make rootfs.cpio.gz.

Now back in bussiness/ topic of advdef: After I read wiki about deflate64 (advdef) versus deflate (gzip), it seams is about dictionary (32KB -> 64 KB) but also about length of strings in the window/range chunks to compress. So I have the idea to pre-sort (the extracted rootfs.gz), files with smaller size are first in the list send to cpio. My results below, using 7zip on sort(x0.cpio) and not-sorted cpio (123.gz), then advdef on x1.cpio (pre-sorted) or x2.cpio (not-sorted)

Code: [Select]

sudo find -type f -printf '%s %p\n' | sort -n | cut -f2 -d" " | sudo cpio -o -H newc > ../x1.cpio
18057 blocks 
#normal un-sorted cpio
 sudo find | sudo cpio -o -H newc > ../x2.cpio 
18182 blocks
> ls -alB ../x*
.rw-r--r-- 9.245.184 abc 13 Mär 19:16  ../x1.cpio
.rw-r--r-- 9.309.184 abc 13 Mär 19:16  ../x2.cpio

Code: [Select]

> gzip -1 x1.cpio
> gzip -1 x2.cpio
> ls -alB ./x*.gz
.rw-r--r-- 3.677.872 abc 13 Mär 19:20  ./x0.gz
.rw-r--r-- 4.218.542 abc 13 Mär 19:16  ./x1.cpio.gz
.rw-r--r-- 4.230.975 abc 13 Mär 19:16  ./x2.cpio.gz

Code: [Select]

> ./advdef -z4 x1.cpio.gz
     4218542     3695808  87% x1.cpio.gz
     4218542     3695808  87%

> ./advdef -z4 x2.cpio.gz
     4230975     3701897  87% x2.cpio.gz
     4230975     3701897  87%

> ls -alB ./x*.gz
.rw-r--r-- 3.677.872 abc 13 Mär 19:20  ./x0.gz       ; 7zip of sorted X0.cpio
.rw-r--r-- 3.695.808 abc 13 Mär 19:25  ./x1.cpio.gz  ; advdef of x1.cpio (pre-sorted)
.rw-r--r-- 3.701.897 abc 13 Mär 19:26  ./x2.cpio.gz  ; advdef of x2.cpio not-sortedso with pre-sorted x0.cpio I obtained with 7z even a better rootfs.gz , than initial reported

Code: [Select]

.rw-r--r--  3.684.655 abc 13 Mär 11:55  123.gzsummary: best in class solution (until now): 7z ver.26 with per-sorted cpio archive.

[Paul_123]

One of your stated goals was trying to exactly produce the distributed initrd.gz file, which I was trying to help you understand was impossible.

If your definition of better is the compressed size of the initrd, then why not use xz or zstd to compress the image.
If your definition of better is boot time, did you try a uncompressed initrd, or just a gzip -1 compressed file.

[nick65go]

1 . One of your stated goals was trying to exactly produce the distributed initrd.gz file, which I was trying to help you understand was impossible.
2. If your definition of better is the compressed size of the initrd, then why not use xz or zstd to compress the image.
3. If your definition of better is boot time, did you try a uncompressed initrd, or just a gzip -1 compressed file.

1. I try to understand first, then maybe to improve TC17. There are NO clear instructions how to reproduce rootfs.gz. If it can not be reproduced then so be it. Survive but not happy.
2. I try to use tools from TC17 to re-build TC17 (learning it). Maybe re-master it to boot in 40 MB RAM just for fun. The smaller is + on demand then the better, for me. Or else there are TC3,x until TC10, plenty to choose from. My current linux has XZ in kernel + packages (is Archlinux derivative named CachyOS x86-64-v3).
3. Boot time is secondary, 3-5 seconds wasted to boot did not matter in a full day.
PS: I assume you understand that all is play/ fun, pushing the limits. Someone can grab a linux (Fedora, OpenSuse, Debian, Ubuntu, whatever) in minutes! and install it also in few minutes with default settings. And a PC (even a Chrome OS) someone can buy new (lasting 3-8 years), with money earned from few days of work (I hope) ...

[Paul_123]

I know its for fun..  So I'll play the game with you

This is a different rootfs than you, but the results you can see.

Code: [Select]

-rw-r--r-- 1 paul paul 12633088 Mar 13 19:46 rootfs-piCore64-17.0.uncompressed
-rw-r--r-- 1 paul paul  4466798 Mar 13 19:42 rootfs-piCore64-17.0.gzip9
-rw-rw-r-- 1 paul paul  4246873 Mar 13 19:49 rootfs-piCore64-17.0.gz.advdevz4
-rw-rw-r-- 1 paul paul  4235095 Mar 13 19:50 rootfs-piCore64-17.0.gz.7zmx9
-rw-r--r-- 1 paul paul  4206895 Mar 13 19:42 rootfs-piCore64-17.0.pigz.zopfli.noiterations
-rw-r--r-- 1 paul paul  4205410 Mar 13 19:42 rootfs-piCore64-17.0.pigz.zopfli.I100
-rw-r--r-- 1 paul paul  4006378 Mar 13 19:42 rootfs-piCore64-17.0.zst-10
-rw-r--r-- 1 paul paul  3458360 Mar 13 19:42 rootfs-piCore64-17.0.zst-19
-rw-r--r-- 1 paul paul  3030460 Mar 13 19:42 rootfs-piCore64-17.0.xz-6

decompression

Code: [Select]

gzip9:   mesreal 0m0.078s
advdef:      real 0m0.075s
7zmx9:      real 0m0.085s
pigzzopfli:  real 0m0.075s
pigzzopfli100: real 0m0.091s
zstd-10:     real 0m0.035s
zstd-19:     real 0m0.028s
xz:             real 0m0.154s
So XZ gets it the smallest, but does take more to decompress it.  zstd is probably the winner.   But lets see.

This is the boot of a Raspi5,

I pulled some relevant lines from dmesg.
this rootfs was produced with pigz

Code: [Select]

[    0.031569] Trying to unpack rootfs image as initramfs...
[    0.102147] Freeing initrd memory: 5664K
[    0.630088] Run /init as init process

This is the zstd compressed rootfs

Code: [Select]

[    0.038710] Trying to unpack rootfs image as initramfs...
[    0.080141] Freeing initrd memory: 5424K
[    0.633339] Run /init as init process


hmmmm, I might switch piCore to zstd.  Need to do more testing.