WelcomeWelcome | FAQFAQ | DownloadsDownloads | WikiWiki

Author Topic: patching tce-load -i to optionally use ram  (Read 7135 times)

Offline caminati

  • Full Member
  • ***
  • Posts: 184
    • Homepage
patching tce-load -i to optionally use ram
« on: December 04, 2011, 05:18:36 PM »
Referring to
http://forum.tinycorelinux.net/index.php/topic,8489.0.html,
I have written a very simple patch to tce-load.
You patch tce-load with the included attachment, then do
Code: [Select]
cd /usr/bin
sudo ln -s tce-load Tce-load

Now, you can run
Code: [Select]
Tce-load -i app.tczas an alternative to
Code: [Select]
tce-load -i app.tcz.

The difference is that in the first case, the extension and all its dependencies get copied in /tmp/tce/optional before being mounted.
Note that the last directory must already exist.
This is a very preliminary test, feedback is welcome.
I have tried to keep as little obtrusive as possible: invoking tce-load should work as usual.

PS: I guess you have to be logged in to see the attached patch file.

Offline vinnie

  • Hero Member
  • *****
  • Posts: 1187
  • HandMace informatic works
Re: patching tce-load -i to optionally use ram
« Reply #1 on: December 05, 2011, 12:21:15 AM »
although I'm not sure how to install, is an interesting thing, if I have time I'll try :)

Offline hiro

  • Hero Member
  • *****
  • Posts: 1180
Re: patching tce-load -i to optionally use ram
« Reply #2 on: December 05, 2011, 12:24:17 AM »
You should create an other option flag in tce-load rather than using this strange upper case symlink.
What's this useful for, does it save RAM in practice?

Offline caminati

  • Full Member
  • ***
  • Posts: 184
    • Homepage
Re: patching tce-load -i to optionally use ram
« Reply #3 on: December 05, 2011, 02:44:21 AM »
You should create an other option flag in tce-load rather than using this strange upper case symlink.
What's this useful for, does it save RAM in practice?

No, it wastes lots of ram indeed.
You should have read more carefully the motivation in the former thread I linked.
I just happen to have a diskless mini-itx with plenty of ram, and I want to boot it with some packages, then remove my usb stick while keeping it running.
I don't like copy mode because it is less reversible than tcz mounting (see my script untcz with a thread on this forum); moreover, mounted extensions not being writable is safe, I want to keep this aspect.
So I ended up with this solution.
A more elegant one would have been to set up a ftp/http server on my usb stick, but tinycore not supporting multiple simultaneous mirrors makes this impracticable.
About the issue of the symlink rather than dedicated switch: this solution minimizes the changes on the source of tce-load.
This is not a mere matter of laziness: since tce-load is a core script, I feel it is better to alter it as little as possible.
By the way, I have been profitably using Tce-load for two days, and it does its job; I shared it also to have more feedback.

Offline caminati

  • Full Member
  • ***
  • Posts: 184
    • Homepage
Re: patching tce-load -i to optionally use ram
« Reply #4 on: December 05, 2011, 03:13:20 AM »
although I'm not sure how to install, is an interesting thing, if I have time I'll try :)

It should suffice to do:
Code: [Select]
sudo busybox patch /usr/bin/tce-load < tce-load.patch
cd /usr/bin
sudo ln -s tce-load Tce-load

Alternatively, and equivalently, you can download the already-patched tce-load I attach to the present message, replace /usr/bin/tce-load with it, and create the symlink with the last two lines above.

PS: I am using microcore latest stable (4.1)

Offline hiro

  • Hero Member
  • *****
  • Posts: 1180
Re: patching tce-load -i to optionally use ram
« Reply #5 on: December 05, 2011, 09:50:26 AM »
It makes sense to me if you say you unload extensions (which I also do, but not very often).

But do you have any figures about RAM usage for some common extensions?
I would naively assume being in RAM multiple times (the compressed .tcz + squashfs caches) is worse than once (only in rootfs) but uncompressed.

Edit: I misread, I thought you said it would save RAM, sorry (which perhaps is the case if executables/libraries are small, but docs and other mostly unused text files are big in contrast).
« Last Edit: December 05, 2011, 09:58:06 AM by hiro »

Offline hiro

  • Hero Member
  • *****
  • Posts: 1180
Re: patching tce-load -i to optionally use ram
« Reply #6 on: December 05, 2011, 10:02:06 AM »
moreover, mounted extensions not being writable is safe, I want to keep this aspect.
[...]
About the issue of the symlink rather than dedicated switch: this solution minimizes the changes on the source of tce-load.

If you want it perfectly safe you'd also have to make the rootfs read only, don't make any wrong assumptions here...
If you have a script for unmounting and deleting symlinks, you can just as well write one for removing the copied files instead. It would even be one less operation.

Don't be shy to include any of your ideas into tinycore base...

Offline caminati

  • Full Member
  • ***
  • Posts: 184
    • Homepage
Re: patching tce-load -i to optionally use ram
« Reply #7 on: December 05, 2011, 10:03:29 AM »

But do you have any figures about RAM usage for some common extensions?
I would naively assume being in RAM multiple times (the compressed .tcz + squashfs caches) is worse than once (only in rootfs) but uncompressed.

Heck, I wishfully thought that mount mode would save ram with respect to copy-to-rootfs mode, but now that you say the opposite, I realize I never did any quantitative test, and at the moment I can't think of a way to do that.
What is a reasonable method to assess this?

Offline curaga

  • Administrator
  • Hero Member
  • *****
  • Posts: 10693
Re: patching tce-load -i to optionally use ram
« Reply #8 on: December 05, 2011, 10:12:58 AM »
sync; sudo cache-clear; free
The only barriers that can stop you are the ones you create yourself.

Offline vinnie

  • Hero Member
  • *****
  • Posts: 1187
  • HandMace informatic works
Re: patching tce-load -i to optionally use ram
« Reply #9 on: December 06, 2011, 03:04:52 AM »
After replace your tce-load bin (i also use 4.1) and given symlink for Tce-load i test your program:
Code: [Select]
tc@box:~$ Tce-load -i inkscape.tcz
gettext.tcz: OK
perl5.tcz: OK
perl_xml_parser.tcz: OK
intltool.tcz: OK
gsl.tcz: OK
libxslt.tcz: OK
popt.tcz: OK
libltdl.tcz: OK
djvulibre.tcz: OK
libwmf.tcz: OK
libcroco.tcz: OK
libgsf.tcz: OK
librsvg.tcz: OK
gcc_libs.tcz: OK
ImageMagick.tcz: OK
openjpeg.tcz: OK
poppler.tcz: OK
libsigc++.tcz: OK
glibmm.tcz: OK
cairomm.tcz: OK
atkmm.tcz: OK
pangomm.tcz: OK
gtkmm.tcz: OK
libgc.tcz: OK
inkscape.tcz: OK
tc@box:~$ ls /tmp/tce
tc@box:~$
Maybe I'm forced to use the patch

P.s. my tce-load during test http://nopaste.info/dbee9416ef.html

Offline caminati

  • Full Member
  • ***
  • Posts: 184
    • Homepage
Re: patching tce-load -i to optionally use ram
« Reply #10 on: December 06, 2011, 03:46:13 AM »
After replace your tce-load bin (i also use 4.1) and given symlink for Tce-load i test your program:
Code: [Select]
tc@box:~$ Tce-load -i inkscape.tcz
gettext.tcz: OK
...
inkscape.tcz: OK
tc@box:~$ ls /tmp/tce
tc@box:~$

You only need to
Code: [Select]
mkdir /tmp/tce/optional
as a first step.
Then it should work.
Sorry, forgot to mention that.

By the way, anyone knows if it would be acceptable to use /tmp/tce as a repository rather than /tmp/tce/optional?
Equivalently: what is the purpose of the further directory "optional" if every extension is stored in "optional"?
« Last Edit: December 06, 2011, 06:20:28 AM by caminati »

Offline curaga

  • Administrator
  • Hero Member
  • *****
  • Posts: 10693
Re: patching tce-load -i to optionally use ram
« Reply #11 on: December 06, 2011, 04:07:48 AM »
Some is history, when tce/* were all autoloaded. Nowadays it's just cleaner to have supporting files in tce/ and the ton of extensions under a separate dir.
The only barriers that can stop you are the ones you create yourself.

Offline caminati

  • Full Member
  • ***
  • Posts: 184
    • Homepage
Re: patching tce-load -i to optionally use ram
« Reply #12 on: December 06, 2011, 04:23:14 AM »
sync; sudo cache-clear; free

I have a typescript of such a test on a 140Mb tcz:
Code: [Select]

tc@box:~$ sudo cache-clear ; free
             total         used         free       shared      buffers
Mem:       2049468        22092      2027376            0           12
-/+ buffers:              22080      2027388
Swap:       512360            0       512360
tc@box:~$ ls /mnt/sda/tcz/texlive.tcz -lh
-rwxrwxrwx    1 root     root      140.0M Feb 17  2011 /mnt/sda/tcz/texlive.tcz
tc@box:~$ time cp /mnt/sda/tcz/texlive.tcz /tmp/tce/
real    0m 4.58s
user    0m 0.02s
sys     0m 0.66s
tc@box:~$ time tce-load -i /tmp/tce/texlive.tcz
/tmp/tce/texlive.tcz: OK
real    0m 1.40s
user    0m 0.18s
sys     0m 1.17s
tc@box:~$ sudo cache-clear ; free
             total         used         free       shared      buffers
Mem:       2049468       241572      1807896            0           28
-/+ buffers:             241544      1807924
Swap:       512360            0       512360
tc@box:~$ ~/untcz texlive
tc@box:~$ echo $?
0
tc@box:~$ sudo cache-clear ; free
             total         used         free       shared      buffers
Mem:       2049468       167404      1882064            0            0
-/+ buffers:             167404      1882064
Swap:       512360            0       512360
tc@box:~$ rm /tmp/tce/texlive.tcz
rm: remove '/tmp/tce/texlive.tcz'? y
tc@box:~$ sudo cache-clear ; free
             total         used         free       shared      buffers
Mem:       2049468        23076      2026392            0            0
-/+ buffers:              23076      2026392
Swap:       512360            0       512360
tc@box:~$ time tce-load -ic /mnt/sda/tcz/texlive.tcz
/mnt/sda/tcz/texlive.tcz: OK
real    0m 10.83s
user    0m 0.37s
sys     0m 8.93s
tc@box:~$ sudo cache-clear ; free
             total         used         free       shared      buffers
Mem:       2049468       329720      1719748            0            8
-/+ buffers:             329712      1719756
Swap:       512360            0       512360
tc@box:~$

Comparing 1719748 with 1807896, I shall conclude that mount mode takes less memory than copy mode. It looks faster, too: 10.83s versus roughly 6s.

Offline hiro

  • Hero Member
  • *****
  • Posts: 1180
Re: patching tce-load -i to optionally use ram
« Reply #13 on: December 06, 2011, 07:58:56 AM »
untcz umounts the tcz?

Offline caminati

  • Full Member
  • ***
  • Posts: 184
    • Homepage
Re: patching tce-load -i to optionally use ram
« Reply #14 on: December 06, 2011, 02:12:41 PM »
untcz umounts the tcz?

Yes, it is exactly the script found at
http://forum.tinycorelinux.net/index.php/topic,5012.msg60561.html#msg60561

It unsymlinks and umounts the extension. It also cleans tce.installed directory.
« Last Edit: December 08, 2011, 12:13:42 AM by caminati »