TC hybrid : Theories requested

[CentralWare]

Hello all!

I'm a week or so away from starting this next phase so if anything, now is the time to send out requests for brain-storming.

I've read more within the TC wiki and forums than I likely ever read in college psychology.  On a plus note...  TC is more rewarding and many times, more entertaining!  That said, I still don't know everything there is to know, thus these little projects to test the limits!

Envision if you will...  a low end machine, probably i586 as a  testing platform, say 512 RAM, no HDD, and a 4GB flash drive through USB.

Basic extensions: acpid, dropbear, dnsmasq, bb-http and a few other back-end services for hardware/network/power management and remote access.
Desktop: Xvesa/Xorg, Xprogs, flwm and similar.  At this point, everything is done standard (usb/frugal) and all is well.

Let's say I wanted to load up midnight commander (mc).  This would be an app that's rarely used, has a handful of dependencies making it that much larger, but compared to ondemand, it seems (to me) a bit overkill to keep it all in memory on "stand-by" around the clock when those resources could be used elsewhere.  (Mind you, I haven't completely researched on-demand scripts to see how much stays in /tce and how much is loaded into RAM, but the following gives reason for this not to be overly important.)

What I'm thinking (and feel free to object and/or suggest alternatives) is having a directory (/tce/hybrid) to store an overlapped rootfs.  When TC finishes booting it's normal operations, this second location would be linked onto the primary root, thus in my mind, only the link itself would technically be in memory, whereas the uncompressed files would remain on storage for if and when they're actually needed.  The storage apps would obviously take a bit longer to launch compared to reading from RAM, but even then once the app is no longer in use, that memory is released.

I'm expecting there's some apps that likely won't appreciate being linked and in other instances, this theory would fail for nfs exports/shares in many cases, but the end result "should" prove valid.

To pull this off (again, in theory) we'd simply scan the hybrid location.  For directories that already exist in the main root file system, we'd traverse into those directories and just sym-link the files and/or directories within it.  For files that already exist, rules would have to be implemented as to whether or not to overwrite, ignore or merge.  Since the main root file system is in memory and the hybrid is linked into it, it then suggests that cache files, temp files, etc. would all be written to the memory-based file system, thus writes to the flash should be minimal.

What got me thinking about this: Take OpenOffice for example...  the extension is huge.  In memory, that's more than 150MB of wasted space while it sits there and does nothing.  When using a single application within the package, you're still only utilizing a fraction of what the entire package is spending.  The alternative here was to strip the package and break it into a "common" or "shared" pack (libraries, shared images, etc.) and then package each app within it separately (writer, calc, and so on.)  If you installed all of the modules, you're still looking at 150+MB of wasted space when ever you're not using it.  Now, extract it and have it linked into the main file system...  my thoughts are that only the links themselves would take up any active resources and when launching Writer (for example) only those files used by the app would take up resources -- and released once the app closes, for the most part.  That's the theory, at least.

Does this go against the TCL principles...  yes and no.  Considering the fact that the core operating system is still entirely tcz based, we still have a pristine operating system at boot time.  Can OoO or other "hybrid" like applications become corrupt?  Certainly...  no differently than having /opt persistent and something happening to it.  The principle is still modular...  only how it launches and from where is what becomes non-standard.

Ideas welcome!  Constructive criticisms are also very welcome!

NOTE: Another reason this became a concern/idea was based on the maximum number of loop devices.  I haven't had this happen yet, but I'd imagine with all of the tiny libraries that get loaded (one loop each) this may eventually become a problem for TC setups with a large number of extensions installed.

[curaga]

So far it seems like a manual way of the default mount mode loading. A TCZ that is symlinked does not take any more RAM than what the symlinks take, which would be the exact same amount as in your scenario?

Ie, loading OpenOffice in the default mode will not take 150mb, but N*few bytes.

There is no limit to the number of loop devices, the old limit was abolished way back in the 2.6 kernel series.

[CentralWare]

@Curaga: Thank you for your response.

Regarding the loop max, this I did not know.  Each time I came across a reference in the forums and/or wiki I kept finding it worded as "...you no longer need to set this as a boot code..." or similar, which gave me the impression it was still an eight-bit pointer, but no longer needed as a boot variable.

Mounted versus extracted: If what you say is true, then my initial tests with TC must have been with one or more extensions which loaded a large amount of "stuff" into memory (I'd assume from tce.installed scripts) leaving the impression the hard-limits were storage and memory alike.  (The test device was a Geode based kiosk with 256M DoM and 256M DDR.  I loaded a number of extensions, the exact ones at the moment I'm not sure - it should be in my notes, but the end result showed a pretty close match between storage and memory usage - around 96MB spent on storage and just shy of 110MB of memory, so I've been building around that misconception (on paper) since.  It wasn't running a desktop or such...  it shouldn't have been "running" anything at all for that matter save for kernel modules.)  If so, then my annotation would not save much memory...  and only a tiny bit of time/speed (with tcz's compression) so it probably won't be a worthwhile venture.  It may sound like I'm writing a paper on Linux or something...  I'm not; what I'm putting on the white-board just takes a great number of trials before I start putting an actual prototype together, so older and newer test machines are my guinea pigs and my notes simply track the experiences.

Well, that being said, it looks as though I'll be moving onto the project that's slated after this one...  automation for rpm and/or deb repo support for missing/unsupported extensions.  This should be fun!  I haven't built an RPM in years...  my "specs" are likely very rusty!

Thanks C!

[curaga]

It's likely your measurement included cache, which can be discarded under memory pressure. Use the provided cache-clear script (sudo cache-clear) when doing such measurements.

[CentralWare]

@Curaga: Yes, that may in fact be true.  Thank you for the tip on the script.

Could you do a favor for me if it's not too much trouble?  (A bit off the thread topic, though mentioned.)
Just for "proof of concept" I went to try the rpm2cpio function and I'm getting unexpected results.

Test scenario:
test.sh

Code: [Select]

#!/bin/sh
mkdir extract; cd extract
wget -q -O webmin.rpm http://prdownloads.sourceforge.net/webadmin/webmin-1.720-1.noarch.rpm
rpm2cpio webmin.rpm | cpio -i -H newc -d
rm -f webmin.rpm
find | cpio -o -H newc | gzip -2 > ../webmin.tcz
cd ..
mkdir mnt
mount -o loop webmin.tcz mnt


Code: [Select]

mount: mounting /dev/loop121 on mnt failed: Invalid argument

I was only trying to experiment with the repackaging process; I have not yet tested the actual software thus it doesn't matter to me if the package WORKS software wise...  I cannot get the package to loop mount after it is built, so either I'm missing a concept or maybe even a step somewhere I'm not seeing?
TIA!

[gerald_clark]

A tcz is not a compressed cpio archive.
It is a squashfs file.  Please read the wiki article on creating extensions.

[CentralWare]

@gerald: Yes, thanks...  I felt stupid after re-reading the script and had hoped to get back here (sheepishly) before anyone had a chance to post/read this.  (One too many remasters this past month...  sorry, I wasn't thinking.  Too many hours, not enough coffee!)