remaster core.cpio with xz (LZMA2) or zpaq (PAQ8P) compression

[hcayero]

Hi,
I am trying to remaster a Tiny Core Linux with Citrix Receiver to deploy from PXE to 600 users on their refurbished PCs.

Therefore, network usage is more important than processor usage, also their computers are not so old.
Size is so important that I need to use other compression algorithms better than gz.

I did this compression tests:

   core.cpio 130,8Mb
   core.gz 56.1Mb 42%
   core.xz 44.1Mb 34%
   core.zpaq 40,4Mb 31%

Of course, the ZPAQ algorithm took much longer to compress and decompress.

The thing is, how can I make it readable for the kernel?
Thanks,
Héctor

[Juanito]

I believe you would need to recompile the kernel with xz compression enabled - you could also then disable unneeded built-in drivers to save more space.

see: http://forum.tinycorelinux.net/index.php?topic=15236.0

[tinypoodle]

*paq is not supported by the kernel.

[hcayero]

I believe you would need to recompile the kernel with xz compression enabled - you could also then disable unneeded built-in drivers to save more space.

see: http://forum.tinycorelinux.net/index.php?topic=15236.0

Hi, is still time to enable xz compression in the the kernel for the 6.x release?

[Juanito]

the problem is that it takes too much precessing effort for i486 and i586 machines...

..and yes, it is too late 

[CentralWare]

This MAY help to some degree:

When building core < cpio

Extract core.gz to /tmp/coregz and remaster as needed

Code: [Select]

cd /tmp/coregz
sudo find | sudo cpio -o -H newc | gzip -9 > ../core.gz
cd /tmp
advdef -z9 core.gz
This will take a while to accomplish (two-pass compression) but might be prudent to get the smallest size from LZ based compression.

Another direction you could take (if the 600 machines are already running TC) is the opposite approach.  Instead of trying to upgrade everyone "on the spot" have a small script launch which slowly downloads the new core in the background, replace the existing core once finished and once rebooted, the new core comes to life.

Good luck!


...another thought just hit me.  Instead of upgrading the CORE, why not just build a differential copy (files which are modified or not existing within the stock core on those machines) and have that, much smaller file get sent to the machines.  Remastering could be done on the machine itself very easily, especially if they're not "all that old."
core.gz > /tmp/coregz
merge the newly downloaded file
/tmp/coregz < newcore.gz
move newcore.gz to tce/boot/core.gz
viola'


Lol...  too many last minute thoughts...
final suggestion would be to create a separate file such as support.gz containing all of the non- core.gz files (or modified ones) and by simply sending that file to the machine and a simple addition to the boot code (initrd=core.gz,support.gz) they're technically merged upon boot - AND easily separated by removing the boot code

[hcayero]

the problem is that it takes too much precessing effort for i486 and i586 machines...

..and yes, it is too late 

Hi, wouldn't it be nice to have it, at least, as a non-default option?

Perhaps at TCL 6.1?

[hcayero]

This MAY help to some degree:

When building core < cpio

Extract core.gz to /tmp/coregz and remaster as needed

Code: [Select]

cd /tmp/coregz
sudo find | sudo cpio -o -H newc | gzip -9 > ../core.gz
cd /tmp
advdef -z9 core.gz
This will take a while to accomplish (two-pass compression) but might be prudent to get the smallest size from LZ based compression.

Another direction you could take (if the 600 machines are already running TC) is the opposite approach.  Instead of trying to upgrade everyone "on the spot" have a small script launch which slowly downloads the new core in the background, replace the existing core once finished and once rebooted, the new core comes to life.

Good luck!


...another thought just hit me.  Instead of upgrading the CORE, why not just build a differential copy (files which are modified or not existing within the stock core on those machines) and have that, much smaller file get sent to the machines.  Remastering could be done on the machine itself very easily, especially if they're not "all that old."
core.gz > /tmp/coregz
merge the newly downloaded file
/tmp/coregz < newcore.gz
move newcore.gz to tce/boot/core.gz
viola'


Lol...  too many last minute thoughts...
final suggestion would be to create a separate file such as support.gz containing all of the non- core.gz files (or modified ones) and by simply sending that file to the machine and a simple addition to the boot code (initrd=core.gz,support.gz) they're technically merged upon boot - AND easily separated by removing the boot code

Thanks for the suggestions. Dumb me, I did not try gz with ultra high compression.
I reduced it from 58 875 951 bytes to 58 314 285 bytes. Not much.

The 600 computers had Windows XP. They were suppoused to boot from PXE without touching the local drive.

[bmarkus]

the problem is that it takes too much precessing effort for i486 and i586 machines...

..and yes, it is too late 

Hi, wouldn't it be nice to have it, at least, as a non-default option?

Perhaps at TCL 6.1?

Why don't you make a custom kernel and try it? It is not so hard to rebuild the kernel specially for x86 where you can do it native, no crosscompilation. You can keep existing settings, except supported kermal image formats and remaster initrd. It is wll documented.

[curaga]

You don't really need XZ in the kernel, and your figures show you get more benefit out of zpaq. You can deploy a two-step solution to use any userspace compressor (zpaq, lrzip, etc.):
1. The initrd only includes TC base + your decompressor binary + command in bootsync.sh uncompressed.
2. Your additional files are in a tarball/lrzip file in the initrd, for example as /tmp/data.tar.zpaq.

In bootsync, you would extract that highly compressed data file over /, then delete it.

This gives you almost equal size to a fully zpaq-compressed initrd without requiring kernel support. Additional benefits include being able to implement a progress bar (kernel decompression is silent until success or failure). Boot speed is comparable to a fully compressed initrd, or perhaps even faster.