Tiny Core Linux

Tiny Core Base => TCB Q&A Forum => Topic started by: hcayero on February 13, 2014, 08:03:13 AM

Title: remaster core.cpio with xz (LZMA2) or zpaq (PAQ8P) compression
Post by: hcayero on February 13, 2014, 08:03:13 AM
Hi,
I am trying to remaster a Tiny Core Linux with Citrix Receiver to deploy from PXE to 600 users on their refurbished PCs.

Therefore, network usage is more important than processor usage, also their computers are not so old.
Size is so important that I need to use other compression algorithms better than gz.

I did this compression tests:

   core.cpio 130,8Mb
   core.gz 56.1Mb 42%
   core.xz 44.1Mb 34%
   core.zpaq 40,4Mb 31%

Of course, the ZPAQ algorithm took much longer to compress and decompress.

The thing is, how can I make it readable for the kernel?
Thanks,
Héctor
Title: Re: remaster core.cpio with xz (LZMA2) or zpaq (PAQ8P) compression
Post by: Juanito on February 13, 2014, 08:08:36 AM
I believe you would need to recompile the kernel with xz compression enabled - you could also then disable unneeded built-in drivers to save more space.

see: http://forum.tinycorelinux.net/index.php?topic=15236.0
Title: Re: remaster core.cpio with xz (LZMA2) or zpaq (PAQ8P) compression
Post by: tinypoodle on February 13, 2014, 09:04:58 AM
*paq is not supported by the kernel.
Title: Re: remaster core.cpio with xz (LZMA2) or zpaq (PAQ8P) compression
Post by: hcayero on December 09, 2014, 06:09:08 AM
I believe you would need to recompile the kernel with xz compression enabled - you could also then disable unneeded built-in drivers to save more space.

see: http://forum.tinycorelinux.net/index.php?topic=15236.0

Hi, is still time to enable xz compression in the the kernel for the 6.x release?
Title: Re: remaster core.cpio with xz (LZMA2) or zpaq (PAQ8P) compression
Post by: Juanito on December 09, 2014, 06:43:02 AM
the problem is that it takes too much precessing effort for i486 and i586 machines...

..and yes, it is too late  :)
Title: Re: remaster core.cpio with xz (LZMA2) or zpaq (PAQ8P) compression
Post by: CentralWare on December 09, 2014, 11:17:28 AM
This MAY help to some degree:

When building core < cpio

Extract core.gz to /tmp/coregz and remaster as needed
Code: [Select]
cd /tmp/coregz
sudo find | sudo cpio -o -H newc | gzip -9 > ../core.gz
cd /tmp
advdef -z9 core.gz
This will take a while to accomplish (two-pass compression) but might be prudent to get the smallest size from LZ based compression.

Another direction you could take (if the 600 machines are already running TC) is the opposite approach.  Instead of trying to upgrade everyone "on the spot" have a small script launch which slowly downloads the new core in the background, replace the existing core once finished and once rebooted, the new core comes to life.

Good luck!


...another thought just hit me.  Instead of upgrading the CORE, why not just build a differential copy (files which are modified or not existing within the stock core on those machines) and have that, much smaller file get sent to the machines.  Remastering could be done on the machine itself very easily, especially if they're not "all that old."
core.gz > /tmp/coregz
merge the newly downloaded file
/tmp/coregz < newcore.gz
move newcore.gz to tce/boot/core.gz
viola'


Lol...  too many last minute thoughts...
final suggestion would be to create a separate file such as support.gz containing all of the non- core.gz files (or modified ones) and by simply sending that file to the machine and a simple addition to the boot code (initrd=core.gz,support.gz) they're technically merged upon boot - AND easily separated by removing the boot code
Title: Re: remaster core.cpio with xz (LZMA2) or zpaq (PAQ8P) compression
Post by: hcayero on December 17, 2014, 08:54:38 AM
the problem is that it takes too much precessing effort for i486 and i586 machines...

..and yes, it is too late  :)

Hi, wouldn't it be nice to have it, at least, as a non-default option?

Perhaps at TCL 6.1?
Title: Re: remaster core.cpio with xz (LZMA2) or zpaq (PAQ8P) compression
Post by: hcayero on December 17, 2014, 09:11:16 AM
This MAY help to some degree:

When building core < cpio

Extract core.gz to /tmp/coregz and remaster as needed
Code: [Select]
cd /tmp/coregz
sudo find | sudo cpio -o -H newc | gzip -9 > ../core.gz
cd /tmp
advdef -z9 core.gz
This will take a while to accomplish (two-pass compression) but might be prudent to get the smallest size from LZ based compression.

Another direction you could take (if the 600 machines are already running TC) is the opposite approach.  Instead of trying to upgrade everyone "on the spot" have a small script launch which slowly downloads the new core in the background, replace the existing core once finished and once rebooted, the new core comes to life.

Good luck!


...another thought just hit me.  Instead of upgrading the CORE, why not just build a differential copy (files which are modified or not existing within the stock core on those machines) and have that, much smaller file get sent to the machines.  Remastering could be done on the machine itself very easily, especially if they're not "all that old."
core.gz > /tmp/coregz
merge the newly downloaded file
/tmp/coregz < newcore.gz
move newcore.gz to tce/boot/core.gz
viola'


Lol...  too many last minute thoughts...
final suggestion would be to create a separate file such as support.gz containing all of the non- core.gz files (or modified ones) and by simply sending that file to the machine and a simple addition to the boot code (initrd=core.gz,support.gz) they're technically merged upon boot - AND easily separated by removing the boot code

Thanks for the suggestions. Dumb me, I did not try gz with ultra high compression.
I reduced it from 58 875 951 bytes to 58 314 285 bytes. Not much.

The 600 computers had Windows XP. They were suppoused to boot from PXE without touching the local drive.

Title: Re: remaster core.cpio with xz (LZMA2) or zpaq (PAQ8P) compression
Post by: bmarkus on December 17, 2014, 09:13:24 AM
the problem is that it takes too much precessing effort for i486 and i586 machines...

..and yes, it is too late  :)

Hi, wouldn't it be nice to have it, at least, as a non-default option?

Perhaps at TCL 6.1?

Why don't you make a custom kernel and try it? It is not so hard to rebuild the kernel specially for x86 where you can do it native, no crosscompilation. You can keep existing settings, except supported kermal image formats and remaster initrd. It is wll documented.
Title: Re: remaster core.cpio with xz (LZMA2) or zpaq (PAQ8P) compression
Post by: curaga on December 17, 2014, 01:37:42 PM
You don't really need XZ in the kernel, and your figures show you get more benefit out of zpaq. You can deploy a two-step solution to use any userspace compressor (zpaq, lrzip, etc.):
1. The initrd only includes TC base + your decompressor binary + command in bootsync.sh uncompressed.
2. Your additional files are in a tarball/lrzip file in the initrd, for example as /tmp/data.tar.zpaq.

In bootsync, you would extract that highly compressed data file over /, then delete it.

This gives you almost equal size to a fully zpaq-compressed initrd without requiring kernel support. Additional benefits include being able to implement a progress bar (kernel decompression is silent until success or failure). Boot speed is comparable to a fully compressed initrd, or perhaps even faster.