patching tce-load -i to optionally use ram

[caminati]

Referring to
http://forum.tinycorelinux.net/index.php/topic,8489.0.html,
I have written a very simple patch to tce-load.
You patch tce-load with the included attachment, then do

Code: [Select]

cd /usr/bin
sudo ln -s tce-load Tce-load

Now, you can run

Code: [Select]

Tce-load -i app.tczas an alternative to

Code: [Select]

tce-load -i app.tcz.

The difference is that in the first case, the extension and all its dependencies get copied in /tmp/tce/optional before being mounted.
Note that the last directory must already exist.
This is a very preliminary test, feedback is welcome.
I have tried to keep as little obtrusive as possible: invoking tce-load should work as usual.

PS: I guess you have to be logged in to see the attached patch file.

[vinnie]

although I'm not sure how to install, is an interesting thing, if I have time I'll try

[hiro]

You should create an other option flag in tce-load rather than using this strange upper case symlink.
What's this useful for, does it save RAM in practice?

[caminati]

You should create an other option flag in tce-load rather than using this strange upper case symlink.
What's this useful for, does it save RAM in practice?

No, it wastes lots of ram indeed.
You should have read more carefully the motivation in the former thread I linked.
I just happen to have a diskless mini-itx with plenty of ram, and I want to boot it with some packages, then remove my usb stick while keeping it running.
I don't like copy mode because it is less reversible than tcz mounting (see my script untcz with a thread on this forum); moreover, mounted extensions not being writable is safe, I want to keep this aspect.
So I ended up with this solution.
A more elegant one would have been to set up a ftp/http server on my usb stick, but tinycore not supporting multiple simultaneous mirrors makes this impracticable.
About the issue of the symlink rather than dedicated switch: this solution minimizes the changes on the source of tce-load.
This is not a mere matter of laziness: since tce-load is a core script, I feel it is better to alter it as little as possible.
By the way, I have been profitably using Tce-load for two days, and it does its job; I shared it also to have more feedback.

[caminati]

although I'm not sure how to install, is an interesting thing, if I have time I'll try

It should suffice to do:

Code: [Select]

sudo busybox patch /usr/bin/tce-load < tce-load.patch
cd /usr/bin
sudo ln -s tce-load Tce-load

Alternatively, and equivalently, you can download the already-patched tce-load I attach to the present message, replace /usr/bin/tce-load with it, and create the symlink with the last two lines above.

PS: I am using microcore latest stable (4.1)

[hiro]

It makes sense to me if you say you unload extensions (which I also do, but not very often).

But do you have any figures about RAM usage for some common extensions?
I would naively assume being in RAM multiple times (the compressed .tcz + squashfs caches) is worse than once (only in rootfs) but uncompressed.

Edit: I misread, I thought you said it would save RAM, sorry (which perhaps is the case if executables/libraries are small, but docs and other mostly unused text files are big in contrast).

[hiro]

moreover, mounted extensions not being writable is safe, I want to keep this aspect.
[...]
About the issue of the symlink rather than dedicated switch: this solution minimizes the changes on the source of tce-load.

If you want it perfectly safe you'd also have to make the rootfs read only, don't make any wrong assumptions here...
If you have a script for unmounting and deleting symlinks, you can just as well write one for removing the copied files instead. It would even be one less operation.

Don't be shy to include any of your ideas into tinycore base...

[caminati]

But do you have any figures about RAM usage for some common extensions?
I would naively assume being in RAM multiple times (the compressed .tcz + squashfs caches) is worse than once (only in rootfs) but uncompressed.

Heck, I wishfully thought that mount mode would save ram with respect to copy-to-rootfs mode, but now that you say the opposite, I realize I never did any quantitative test, and at the moment I can't think of a way to do that.
What is a reasonable method to assess this?

[curaga]

sync; sudo cache-clear; free

[vinnie]

After replace your tce-load bin (i also use 4.1) and given symlink for Tce-load i test your program:

Code: [Select]

tc@box:~$ Tce-load -i inkscape.tcz
gettext.tcz: OK
perl5.tcz: OK
perl_xml_parser.tcz: OK
intltool.tcz: OK
gsl.tcz: OK
libxslt.tcz: OK
popt.tcz: OK
libltdl.tcz: OK
djvulibre.tcz: OK
libwmf.tcz: OK
libcroco.tcz: OK
libgsf.tcz: OK
librsvg.tcz: OK
gcc_libs.tcz: OK
ImageMagick.tcz: OK
openjpeg.tcz: OK
poppler.tcz: OK
libsigc++.tcz: OK
glibmm.tcz: OK
cairomm.tcz: OK
atkmm.tcz: OK
pangomm.tcz: OK
gtkmm.tcz: OK
libgc.tcz: OK
inkscape.tcz: OK
tc@box:~$ ls /tmp/tce
tc@box:~$Maybe I'm forced to use the patch

P.s. my tce-load during test http://nopaste.info/dbee9416ef.html

[caminati]

After replace your tce-load bin (i also use 4.1) and given symlink for Tce-load i test your program:

Code: [Select]

tc@box:~$ Tce-load -i inkscape.tcz
gettext.tcz: OK
...
inkscape.tcz: OK
tc@box:~$ ls /tmp/tce
tc@box:~$

You only need to

Code: [Select]

mkdir /tmp/tce/optional
as a first step.
Then it should work.
Sorry, forgot to mention that.

By the way, anyone knows if it would be acceptable to use /tmp/tce as a repository rather than /tmp/tce/optional?
Equivalently: what is the purpose of the further directory "optional" if every extension is stored in "optional"?

[curaga]

Some is history, when tce/* were all autoloaded. Nowadays it's just cleaner to have supporting files in tce/ and the ton of extensions under a separate dir.

[caminati]

sync; sudo cache-clear; free

I have a typescript of such a test on a 140Mb tcz:

Code: [Select]


tc@box:~$ sudo cache-clear ; free
             total         used         free       shared      buffers
Mem:       2049468        22092      2027376            0           12
-/+ buffers:              22080      2027388
Swap:       512360            0       512360
tc@box:~$ ls /mnt/sda/tcz/texlive.tcz -lh
-rwxrwxrwx    1 root     root      140.0M Feb 17  2011 /mnt/sda/tcz/texlive.tcz
tc@box:~$ time cp /mnt/sda/tcz/texlive.tcz /tmp/tce/
real    0m 4.58s
user    0m 0.02s
sys     0m 0.66s
tc@box:~$ time tce-load -i /tmp/tce/texlive.tcz
/tmp/tce/texlive.tcz: OK
real    0m 1.40s
user    0m 0.18s
sys     0m 1.17s
tc@box:~$ sudo cache-clear ; free
             total         used         free       shared      buffers
Mem:       2049468       241572      1807896            0           28
-/+ buffers:             241544      1807924
Swap:       512360            0       512360
tc@box:~$ ~/untcz texlive
tc@box:~$ echo $?
0
tc@box:~$ sudo cache-clear ; free
             total         used         free       shared      buffers
Mem:       2049468       167404      1882064            0            0
-/+ buffers:             167404      1882064
Swap:       512360            0       512360
tc@box:~$ rm /tmp/tce/texlive.tcz
rm: remove '/tmp/tce/texlive.tcz'? y
tc@box:~$ sudo cache-clear ; free
             total         used         free       shared      buffers
Mem:       2049468        23076      2026392            0            0
-/+ buffers:              23076      2026392
Swap:       512360            0       512360
tc@box:~$ time tce-load -ic /mnt/sda/tcz/texlive.tcz
/mnt/sda/tcz/texlive.tcz: OK
real    0m 10.83s
user    0m 0.37s
sys     0m 8.93s
tc@box:~$ sudo cache-clear ; free
             total         used         free       shared      buffers
Mem:       2049468       329720      1719748            0            8
-/+ buffers:             329712      1719756
Swap:       512360            0       512360
tc@box:~$

Comparing 1719748 with 1807896, I shall conclude that mount mode takes less memory than copy mode. It looks faster, too: 10.83s versus roughly 6s.

[hiro]

untcz umounts the tcz?

[caminati]

untcz umounts the tcz?

Yes, it is exactly the script found at
http://forum.tinycorelinux.net/index.php/topic,5012.msg60561.html#msg60561

It unsymlinks and umounts the extension. It also cleans tce.installed directory.