Tiny Core Linux

General TC => Programming & Scripting - Unofficial => Topic started by: caminati on December 04, 2011, 05:18:36 PM

Title: patching tce-load -i to optionally use ram
Post by: caminati on December 04, 2011, 05:18:36 PM
Referring to
http://forum.tinycorelinux.net/index.php/topic,8489.0.html (http://forum.tinycorelinux.net/index.php/topic,8489.0.html),
I have written a very simple patch to tce-load.
You patch tce-load with the included attachment, then do
Code: [Select]
cd /usr/bin
sudo ln -s tce-load Tce-load

Now, you can run
Code: [Select]
Tce-load -i app.tczas an alternative to
Code: [Select]
tce-load -i app.tcz.

The difference is that in the first case, the extension and all its dependencies get copied in /tmp/tce/optional before being mounted.
Note that the last directory must already exist.
This is a very preliminary test, feedback is welcome.
I have tried to keep as little obtrusive as possible: invoking tce-load should work as usual.

PS: I guess you have to be logged in to see the attached patch file.
Title: Re: patching tce-load -i to optionally use ram
Post by: vinnie on December 05, 2011, 12:21:15 AM
although I'm not sure how to install, is an interesting thing, if I have time I'll try :)
Title: Re: patching tce-load -i to optionally use ram
Post by: hiro on December 05, 2011, 12:24:17 AM
You should create an other option flag in tce-load rather than using this strange upper case symlink.
What's this useful for, does it save RAM in practice?
Title: Re: patching tce-load -i to optionally use ram
Post by: caminati on December 05, 2011, 02:44:21 AM
You should create an other option flag in tce-load rather than using this strange upper case symlink.
What's this useful for, does it save RAM in practice?

No, it wastes lots of ram indeed.
You should have read more carefully the motivation in the former thread I linked.
I just happen to have a diskless mini-itx with plenty of ram, and I want to boot it with some packages, then remove my usb stick while keeping it running.
I don't like copy mode because it is less reversible than tcz mounting (see my script untcz with a thread on this forum); moreover, mounted extensions not being writable is safe, I want to keep this aspect.
So I ended up with this solution.
A more elegant one would have been to set up a ftp/http server on my usb stick, but tinycore not supporting multiple simultaneous mirrors makes this impracticable.
About the issue of the symlink rather than dedicated switch: this solution minimizes the changes on the source of tce-load.
This is not a mere matter of laziness: since tce-load is a core script, I feel it is better to alter it as little as possible.
By the way, I have been profitably using Tce-load for two days, and it does its job; I shared it also to have more feedback.
Title: Re: patching tce-load -i to optionally use ram
Post by: caminati on December 05, 2011, 03:13:20 AM
although I'm not sure how to install, is an interesting thing, if I have time I'll try :)

It should suffice to do:
Code: [Select]
sudo busybox patch /usr/bin/tce-load < tce-load.patch
cd /usr/bin
sudo ln -s tce-load Tce-load

Alternatively, and equivalently, you can download the already-patched tce-load I attach to the present message, replace /usr/bin/tce-load with it, and create the symlink with the last two lines above.

PS: I am using microcore latest stable (4.1)
Title: Re: patching tce-load -i to optionally use ram
Post by: hiro on December 05, 2011, 09:50:26 AM
It makes sense to me if you say you unload extensions (which I also do, but not very often).

But do you have any figures about RAM usage for some common extensions?
I would naively assume being in RAM multiple times (the compressed .tcz + squashfs caches) is worse than once (only in rootfs) but uncompressed.

Edit: I misread, I thought you said it would save RAM, sorry (which perhaps is the case if executables/libraries are small, but docs and other mostly unused text files are big in contrast).
Title: Re: patching tce-load -i to optionally use ram
Post by: hiro on December 05, 2011, 10:02:06 AM
moreover, mounted extensions not being writable is safe, I want to keep this aspect.
[...]
About the issue of the symlink rather than dedicated switch: this solution minimizes the changes on the source of tce-load.

If you want it perfectly safe you'd also have to make the rootfs read only, don't make any wrong assumptions here...
If you have a script for unmounting and deleting symlinks, you can just as well write one for removing the copied files instead. It would even be one less operation.

Don't be shy to include any of your ideas into tinycore base...
Title: Re: patching tce-load -i to optionally use ram
Post by: caminati on December 05, 2011, 10:03:29 AM

But do you have any figures about RAM usage for some common extensions?
I would naively assume being in RAM multiple times (the compressed .tcz + squashfs caches) is worse than once (only in rootfs) but uncompressed.

Heck, I wishfully thought that mount mode would save ram with respect to copy-to-rootfs mode, but now that you say the opposite, I realize I never did any quantitative test, and at the moment I can't think of a way to do that.
What is a reasonable method to assess this?
Title: Re: patching tce-load -i to optionally use ram
Post by: curaga on December 05, 2011, 10:12:58 AM
sync; sudo cache-clear; free
Title: Re: patching tce-load -i to optionally use ram
Post by: vinnie on December 06, 2011, 03:04:52 AM
After replace your tce-load bin (i also use 4.1) and given symlink for Tce-load i test your program:
Code: [Select]
tc@box:~$ Tce-load -i inkscape.tcz
gettext.tcz: OK
perl5.tcz: OK
perl_xml_parser.tcz: OK
intltool.tcz: OK
gsl.tcz: OK
libxslt.tcz: OK
popt.tcz: OK
libltdl.tcz: OK
djvulibre.tcz: OK
libwmf.tcz: OK
libcroco.tcz: OK
libgsf.tcz: OK
librsvg.tcz: OK
gcc_libs.tcz: OK
ImageMagick.tcz: OK
openjpeg.tcz: OK
poppler.tcz: OK
libsigc++.tcz: OK
glibmm.tcz: OK
cairomm.tcz: OK
atkmm.tcz: OK
pangomm.tcz: OK
gtkmm.tcz: OK
libgc.tcz: OK
inkscape.tcz: OK
tc@box:~$ ls /tmp/tce
tc@box:~$
Maybe I'm forced to use the patch

P.s. my tce-load during test http://nopaste.info/dbee9416ef.html (http://nopaste.info/dbee9416ef.html)
Title: Re: patching tce-load -i to optionally use ram
Post by: caminati on December 06, 2011, 03:46:13 AM
After replace your tce-load bin (i also use 4.1) and given symlink for Tce-load i test your program:
Code: [Select]
tc@box:~$ Tce-load -i inkscape.tcz
gettext.tcz: OK
...
inkscape.tcz: OK
tc@box:~$ ls /tmp/tce
tc@box:~$

You only need to
Code: [Select]
mkdir /tmp/tce/optional
as a first step.
Then it should work.
Sorry, forgot to mention that.

By the way, anyone knows if it would be acceptable to use /tmp/tce as a repository rather than /tmp/tce/optional?
Equivalently: what is the purpose of the further directory "optional" if every extension is stored in "optional"?
Title: Re: patching tce-load -i to optionally use ram
Post by: curaga on December 06, 2011, 04:07:48 AM
Some is history, when tce/* were all autoloaded. Nowadays it's just cleaner to have supporting files in tce/ and the ton of extensions under a separate dir.
Title: Re: patching tce-load -i to optionally use ram
Post by: caminati on December 06, 2011, 04:23:14 AM
sync; sudo cache-clear; free

I have a typescript of such a test on a 140Mb tcz:
Code: [Select]

tc@box:~$ sudo cache-clear ; free
             total         used         free       shared      buffers
Mem:       2049468        22092      2027376            0           12
-/+ buffers:              22080      2027388
Swap:       512360            0       512360
tc@box:~$ ls /mnt/sda/tcz/texlive.tcz -lh
-rwxrwxrwx    1 root     root      140.0M Feb 17  2011 /mnt/sda/tcz/texlive.tcz
tc@box:~$ time cp /mnt/sda/tcz/texlive.tcz /tmp/tce/
real    0m 4.58s
user    0m 0.02s
sys     0m 0.66s
tc@box:~$ time tce-load -i /tmp/tce/texlive.tcz
/tmp/tce/texlive.tcz: OK
real    0m 1.40s
user    0m 0.18s
sys     0m 1.17s
tc@box:~$ sudo cache-clear ; free
             total         used         free       shared      buffers
Mem:       2049468       241572      1807896            0           28
-/+ buffers:             241544      1807924
Swap:       512360            0       512360
tc@box:~$ ~/untcz texlive
tc@box:~$ echo $?
0
tc@box:~$ sudo cache-clear ; free
             total         used         free       shared      buffers
Mem:       2049468       167404      1882064            0            0
-/+ buffers:             167404      1882064
Swap:       512360            0       512360
tc@box:~$ rm /tmp/tce/texlive.tcz
rm: remove '/tmp/tce/texlive.tcz'? y
tc@box:~$ sudo cache-clear ; free
             total         used         free       shared      buffers
Mem:       2049468        23076      2026392            0            0
-/+ buffers:              23076      2026392
Swap:       512360            0       512360
tc@box:~$ time tce-load -ic /mnt/sda/tcz/texlive.tcz
/mnt/sda/tcz/texlive.tcz: OK
real    0m 10.83s
user    0m 0.37s
sys     0m 8.93s
tc@box:~$ sudo cache-clear ; free
             total         used         free       shared      buffers
Mem:       2049468       329720      1719748            0            8
-/+ buffers:             329712      1719756
Swap:       512360            0       512360
tc@box:~$

Comparing 1719748 with 1807896, I shall conclude that mount mode takes less memory than copy mode. It looks faster, too: 10.83s versus roughly 6s.
Title: Re: patching tce-load -i to optionally use ram
Post by: hiro on December 06, 2011, 07:58:56 AM
untcz umounts the tcz?
Title: Re: patching tce-load -i to optionally use ram
Post by: caminati on December 06, 2011, 02:12:41 PM
untcz umounts the tcz?

Yes, it is exactly the script found at
http://forum.tinycorelinux.net/index.php/topic,5012.msg60561.html#msg60561 (http://forum.tinycorelinux.net/index.php/topic,5012.msg60561.html#msg60561)

It unsymlinks and umounts the extension. It also cleans tce.installed directory.
Title: Re: patching tce-load -i to optionally use ram
Post by: caminati on December 10, 2011, 10:48:20 AM
If you want it perfectly safe you'd also have to make the rootfs read only, don't make any wrong assumptions here...
Sorry, I just don't follow you here.

Quote
If you have a script for unmounting and deleting symlinks, you can just as well write one for removing the copied files instead. It would even be one less operation.
No. That's exactly the point: in copy mode the information binding copied files to the extension they belong is lost (I was talking about reversibility in a previous post right to mean this). Otherwise, please explain how.

Quote
Don't be shy to include any of your ideas into tinycore base...
Again, I can't understand what you mean here: how?
I am not in TC team, and there is no public cvs I know of.
I am just posting here what I find useful for myself.
Title: Re: patching tce-load -i to optionally use ram
Post by: hiro on December 14, 2011, 03:07:23 PM
What I meant to say is: If your idea is good it can be included.
But removing symlinks or files they point to is not a big difference from a "security" point of view.
You can rewrite and remove symlinks and then also lose "information".

Perhaps I just didn't get what you meant with "security". Also you can use squashfs tools to list complete package contents (including their paths relative to root) or download the same listing as a file from the repo if you want to know where an extension puts it's files.
Title: Re: patching tce-load -i to optionally use ram
Post by: caminati on December 15, 2011, 01:51:29 AM
You can rewrite and remove symlinks and then also lose "information".
Perhaps I just didn't get what you meant with "security".
I meant that copied files are writable, while links to files inside a squashfs mount are not.

Quote
Also you can use squashfs tools to list complete package contents (including their paths relative to root) or download the same listing as a file from the repo if you want to know where an extension puts it's files.
This is true; however, there are performance problems with your approaches (I tried the squashfs tools way when working on my untcz script).
Moreover, your approaches are less robust: suppose you, for some reason, have no longer access to the original tcz file...
In such cases, if you used copy mode, you have no way of knowing what files belong to what extension (in this sense I talked about losing information), while if you mounted the extension you still can.