WelcomeWelcome | FAQFAQ | DownloadsDownloads | WikiWiki

Author Topic: LZMA Squash FS in kernel?  (Read 7253 times)

Offline lordtangent

  • Newbie
  • *
  • Posts: 10
LZMA Squash FS in kernel?
« on: March 17, 2009, 10:31:05 PM »
I'm wondering if SquashFS-LZMA  is still in the Tiny Core kernel (if it ever was).  I'm also wondering if the "TCZ" format supports SquashFS-LZMA or if I'd have to just set it up the normal way.

Offline Jason W

  • Retired Admins
  • Hero Member
  • *****
  • Posts: 9730
Re: LZMA Squash FS in kernel?
« Reply #1 on: March 17, 2009, 11:06:43 PM »
From what I hear squashfs is mainlined in the 2.6.29 kernel.  LZMA is so far a different story.

But TCZ's can be zisofs, cramfs, squashfs, or whatever mountable images are supported by the system.  Right now zisofs and cramfs are supported in base TC.  I thought about making a squashfs extension for those who wanted to compress their own tcz's even further but of course they would have to convert them from tce's since squashfs is not supported in base TC. 

Offline curaga

  • Administrator
  • Hero Member
  • *****
  • Posts: 11044
Re: LZMA Squash FS in kernel?
« Reply #2 on: March 18, 2009, 01:03:14 PM »
Squashfs never so far was in TC. The tcz format does not specify any compression format, it can be anything loop-mounted, even ext2 for the matter. So if you compile the squashfs[-lzma] modules, and include them in the base, your remaster can thus use squashfs-compressed tcz-extensions.
The only barriers that can stop you are the ones you create yourself.

Offline lordtangent

  • Newbie
  • *
  • Posts: 10
Re: LZMA Squash FS in kernel?
« Reply #3 on: March 18, 2009, 01:35:48 PM »
Cool! From what I gathered from the squashfs site, the focus right now is to get squashfs in the mainline kernel. LZMA is a separate effort. Once both are in the mainline kernel, squashfs-LZMA will be a less "exotic" solution.     

I'm still not 100% familiar with the Tiny Core development philosophy, so I'm not sure how you guys feel about all the patching required to get a kernel and squashfs tools to work with LZMA. It might just be easier to wait for it to show up in the main line versions of both.

However, I want to register my interest in such a combo. It would be great for packing down loopback images that live on a thumb drive where the speed hit of LZMA wouldn't be so bad relative to the speed of the thumb drive. It would also be good for when someone wants to load up a bunch of TCZs and isn't as concerned about speed so as much as memory savings. (That is my particular interest)  I'm on the verge of doing it myself  and I'd be happy to contribute the results back if the project is interested.  I just need to get more  familiar with your requirements.

Offline Jason W

  • Retired Admins
  • Hero Member
  • *****
  • Posts: 9730
Re: LZMA Squash FS in kernel?
« Reply #4 on: March 18, 2009, 03:51:41 PM »
I plan on making a squashfs module extension when we upgrade our kernel and also squashfs-tools.  It would be made as a tce so it would load before any tczs thus enabling the use of squashfs tczs.  LZMA compression is of course slower than gzip so going that route would achieve smaller tczs at the price of performance. 

Having a the choice of squashfs as well as the choice of compression to use with it is a good thing.  Module extensions of each as well as their tools will emerge with the next kernel update along with a conversion script to convert existing tcz's into your desired format - cramfs, zisofs, squash-gzip, squash-lzma.  Cramfs and zisofs would be the official formats available for download and supported by the base sytem while squashfs and it's stuff would remain as extensions.

Of course, I would be glad if you or anyone else wants to make the extensions.  Our next kernel update is probably not far away, so it would save effort to wait on it before proceeding.


Offline Jason W

  • Retired Admins
  • Hero Member
  • *****
  • Posts: 9730
Re: LZMA Squash FS in kernel?
« Reply #5 on: March 18, 2009, 11:30:03 PM »
I posted a "teaser" squashfs module as well as tools to play with for now using only gzip, no LZMA.  But results still may make it worth messing with if you are tight on space.  Here is the size of boost.tczl before and after making into a squashfs image:

Before:
boost.tczl       size:  17,170,432 bytes

After:
boost.tczl       size:  10,121,216 bytes

That is a sizable difference, and only using native gzip.  Here is a script that will convert your tcz directory to squashfs if you decide to play with it.  Of course, have the squash module and tools installed.  Run this script from inside the same directory as your tczs.  And I would do this only if the tcz's are not mounted and in use as they are deleted upon creation of the squashfs tcz.  If this is done in your tce directory, as long as you have the squashfs-2.6.26.tcem in there you can boot with the squashfs tczs.

Code: [Select]
#!/bin/sh


mkdir pkgtmp
mktcz() {
mount -o loop "$FILE" pkgtmp
mksquashfs pkgtmp "$FILE".new
umount -d pkgtmp
rm "$FILE"
mv "$FILE".new "$FILE"
md5sum "$FILE" > "$FILE".md5.txt
}

for file in `ls` ; do
    case $file in
        *.tcz)
       FILE=$file
               mktcz;;
        *.tczl)
FILE=$file
               mktcz;;
*.tczm)
FILE=$file
       mktcz;;
        *.tczlm)
FILE=$file
       mktcz;;
        *.tczml)
FILE=$file
       mktcz;;
    esac
done
« Last Edit: March 19, 2009, 12:42:23 PM by Jason W »

Offline tobiaus

  • Suspended
  • Hero Member
  • *****
  • Posts: 599
Re: LZMA Squash FS in kernel?
« Reply #6 on: March 18, 2009, 11:53:49 PM »
this is a good idea for when people have very little in the way of space... but if we'd like tc to be as fast as it is on boot, we shouldn't use this often, otherwise in the future we'll be telling people to avoid squash tcz's to have faster performance. i'm hoping people avoid them as a rule anyway, and only use them as an exception.

Offline Jason W

  • Retired Admins
  • Hero Member
  • *****
  • Posts: 9730
Re: LZMA Squash FS in kernel?
« Reply #7 on: March 19, 2009, 12:01:55 AM »
Don't get me wrong, as I value the tried and true (cramfs, zisofs) over the new fangled (squashfs,lzma) but the performance numbers really aren't that bad for stock squashfs using gzip:

http://kerneltrap.org/files/PERFORMANCE.README.txt

Just saying that an option for squashfs/gzip may not cost too much in terms of performance.

EDIT: For clarification, squashfs and related tools are now and will remain extensions.  Therefore their use is completely optional and they exist to provide choice.  Having choices you can choose from is one of the beauty of extensions.
Just to make it completely safe, I will run some tests and if there is a noticable performance impact from using squashfs then I will issue a warning in the info file that the smaller image size is at a cost to performance if compressing tcz's for personal use.
« Last Edit: March 19, 2009, 10:30:35 AM by Jason W »

Offline lordtangent

  • Newbie
  • *
  • Posts: 10
Re: LZMA Squash FS in kernel?
« Reply #8 on: March 19, 2009, 11:59:36 AM »
I posted a "teaser" squashfs module as well as tools to play with for now using only gzip, no LZMA. ...
Before:
boost.tczl       size:  17,170,432 bytes

After:
boost.tczl       size:  10,121,216 bytes

Wow.I imagine LZMA would pack  it down even more. (Only sadly at the expense of performance.)  The speed numbers on the gzip version look really good though!