Suggested tce-load KERNEL handling

[robc]

Is there a 64-bit kernel?  Only found the 32-bit vmlinuz in the iso.  microcore64.gz halted on boot with "modprobe: chdir (3.0.3-tinycore) not found" which is expected since the kernel is not 64-bit.

I noticed that the extension deps for kernel modules will be handled the same as in 3.x.  I would like to propose that the "-KERNEL" text is kept in all dep files that refer to kernel module extensions and have tce-load install the correct extension based on the current running kernel.  This will allow a single local repository of extensions to serve multiple different kernel environments (ie 32-bit vs 64-bit).  This would be very useful for extension development for switching between kernels to support multiple kernel module types. For example: alsa.tcz.dep would look like this in the tce/ directory:

alsa-modules-KERNEL.tcz
libasound.tcz
pci-utils.tcz
ncurses.tcz

When tinycore.gz is used, alsa would install alsa-modules-3.0.3-tinycore.tcz. 
When microcore64.gz is used, alsa would install alsa-modules-3.0.3-tinycore64.tcz.


Edit:
Here is a way to implement with tce-load:
remove dep_vars() and all calls to it
change pop_dep() to:

Code: [Select]

pop_dep(){
F=`echo ${DEPLIST%%/*} | sed "s@KERNEL@$KERNELVER@"`
DEPLIST="${DEPLIST#*/}"
}
This should replace KERNEL with KERNELVER (`uname -r`) in the script's local variable and not in the dep file.

[maro]

My thanks to robc for the excellent suggestion in reply #9.

I've been on the record a few times calling this feature of "fudging" dep files a "flaw". But then at some stage I kind of worked around it and did not spend any further effort in improving the situation.

First up my apologies to curaga, who quite some time ago send me a PM with a candidate to test. At the time I never got around to undertake such a test. But I'm guilty that I not even had the decency to send a simple reply to that effect.

Triggered by the suggested change I've now done a quick test (using a TC v3.8.4 VM which I had already running) and done a few more little tweaks to:

• eliminate the use of 'dep_vars()',
  
• add another two "KERNEL" replacement commands to allow also the use of 'tce-load -w i2c-KERNEL',
• put in some anchors around 'KERNEL' to limit it essentially to '-KERNEL(|\.tcz)$' (as I'm a bit pedantic WRT the use of "tight" RE), and
• change of the RE limiting character from '@' to ':" (as '$@' which would be a consequence of the above causes really nasty side effects)

That means the diff now looks like:

Code: (bash) [Select]

--- /usr/bin/tce-load
+++ tce-load
@@ -64,4 +64,5 @@
 }
+
 pop_dep(){
-       F="${DEPLIST%%/*}"
+       F=$( echo ${DEPLIST%%/*} | sed "s:-KERNEL\.tcz$:-${KERNELVER}.tcz:g" )
        DEPLIST="${DEPLIST#*/}"
@@ -69,6 +70,2 @@

-dep_vars(){
-       sed -i "s@KERNEL@$KERNELVER@" $1
-}
-
 copyInstall() {
@@ -168,3 +165,2 @@
                        if [ "$?" == 0 ]; then
-                               dep_vars ${d}.dep
                                [ "$SUPPRESS" ] || echo ${d}.dep OK
@@ -192,2 +188,4 @@
 if [ ${TARGETAPP} == ${APPNAME} ]; then TARGETAPP=${TARGETAPP}.tcz; fi
+APPNAME=$( echo ${APPNAME} | sed "s:-KERNEL$:-${KERNELVER}:" )
+TARGETAPP=$( echo ${TARGETAPP} | sed "s:-KERNEL\.tcz$:-${KERNELVER}.tcz:" )

@@ -212,3 +210,2 @@
        if [ "$?" == 0 ]; then
-               dep_vars $DEPLIST
                DEPLIST=""

EDIT 1: Removed a single surplus character from the diff.
EDIT 2: Turns out there was a was at least a global replacement required in 'pop_dep()', but I've now realized that there is another functional regression somewhere else. Clearly I've not done any serious testing when I wrote this posting. So apart from addressing the (potential) performance regression more analysis will be required.

[roberts]

My thanks to robc for the excellent suggestion in reply #9.

I've been on the record a few times calling this feature of "fudging" dep files a "flaw". But then at some stage I kind of worked around it and did not spend any further effort in improving the situation.

First up my apologies to curaga, who quite some time ago send me a PM with a candidate to test. At the time I never got around to undertake such a test. But I'm guilty that I not even had the decency to send a simple reply to that effect.

Triggered by the suggested change I've now done a quick test (using a TC v3.8.4 VM which I had already running) and done a few more little tweaks to:

• eliminate the use of 'dep_vars()',
  
• add another two "KERNEL" replacement commands to allow also the use of 'tce-load -w i2c-KERNEL',
• put in some anchors around 'KERNEL' to limit it essentially to '-KERNEL(|\.tcz)$' (as I'm a bit pedantic WRT the use of "tight" RE), and
• change of the RE limiting character from '@' to ':" (as '$@' which would be a consequence of the above causes really nasty side effects)

That means the diff now looks like:

Code: (bash) [Select]

--- /usr/bin/tce-load
+++ tce-load
@@ -64,4 +64,5 @@
 }
+
 pop_dep(){
-       F="${DEPLIST%%/*}"
+       F=$( echo ${DEPLIST%%/*} | sed "s:-KERNEL\.tcz$:-${KERNELVER}.tcz:" )
        DEPLIST="${DEPLIST#*/}"
@@ -69,6 +70,2 @@

-dep_vars(){
-       sed -i "s@KERNEL@$KERNELVER@" $1
-}
-
 copyInstall() {
@@ -168,3 +165,2 @@
                        if [ "$?" == 0 ]; then
-                               dep_vars ${d}.dep
                                [ "$SUPPRESS" ] || echo ${d}.dep OK
@@ -192,2 +188,4 @@
 if [ ${TARGETAPP} == ${APPNAME} ]; then TARGETAPP=${TARGETAPP}.tcz; fi
+APPNAME=$( echo ${APPNAME} | sed "s:-KERNEL$:-${KERNELVER}:" )
+TARGETAPP=$( echo ${TARGETAPP} | sed "s:-KERNEL\.tcz$:-${KERNELVER}.tcz:" )

@@ -212,3 +210,2 @@
        if [ "$?" == 0 ]; then
-               dep_vars $DEPLIST
                DEPLIST=""

EDIT: Removed a single surplus character from the diff.

This strikes me as benefiting the few whilst penalizing the many.

Few would have both 64 and 32 machines. Processing the KERNEL substitution once during download is far more efficient than repeatedly processing every time over and over upon every boot, every load of extensions that require modules.

It penalizes the many by the repeatedly processing the KERNEL substitution which results in a slower boot.

[robc]

This strikes me as benefiting the few whilst penalizing the many.

Few would have both 64 and 32 machines. Processing the KERNEL substitution once during download is far more efficient than repeatedly processing every time over and over upon every boot, every load of extensions that require modules.

It penalizes the many by the repeatedly processing the KERNEL substitution which results in a slower boot.

How about instead of processing the KERNEL substitution for every dep file, it is done only on a failure?  The tce-load would operate the same but if the extension is not found (ie i2c-KERNEL.tcz) it would then do the substitution for that extension, if it is not found after the substitution then it would display the file not found error.

I don't have time right now to test this or run benchmarks on the previous proposal to see if there is any significant performance hits.  I might have some time tomorrow though.

Since this is a new version cycle it would be the best place to add more support for different kernels.  This doesn't just affect 32 vs 64 but also custom kernels.

[robc]

This post is a followup on a previous post for alpha3:
http://forum.tinycorelinux.net/index.php/topic,11351.msg59956.html#msg59956

I was able to run tests for 3 different scenarios for tce-load.  Each scenerio loaded 1000 extensions that each depended upon a kernel module extension.  So 1000 extensions and 1000 kernel module extensions were loaded.  All of these extensions only had a single unique file in /usr/local/tce.installed. This command was used to run each scenerio:

Code: [Select]

time -v tce-load -i `cat mylist.txt` &> tce-load.outThe mylist.txt file just listed the 1000 extensions to load.

Scenerio 1:
Use the current tce-load script to load all of these exensions. Output:

   User time (seconds): 22.57
   System time (seconds): 75.77
   Percent of CPU this job got: 63%
   Elapsed (wall clock) time (h:mm:ss or m:ss): 2m 34.84s
   Average shared text size (kbytes): 0
   Average unshared data size (kbytes): 0
   Average stack size (kbytes): 0
   Average total size (kbytes): 0
   Maximum resident set size (kbytes): 3504
   Average resident set size (kbytes): 0
   Major (requiring I/O) page faults: 0
   Minor (reclaiming a frame) page faults: 9921980
   Voluntary context switches: 117237
   Involuntary context switches: 40697
   Swaps: 0
   File system inputs: 0
   File system outputs: 0
   Socket messages sent: 0
   Socket messages received: 0
   Signals delivered: 0
   Page size (bytes): 4096
   Exit status: 0

The elapsed time in seconds for this scenerio is 154.84.  That is 77.42 ms per extension.


Scenerio 2:
Use the a tce-load script that has the modifications from maro in the linked thread above to load all of these exensions. Output:

   User time (seconds): 21.76
   System time (seconds): 76.76
   Percent of CPU this job got: 60%
   Elapsed (wall clock) time (h:mm:ss or m:ss): 2m 42.86s
   Average shared text size (kbytes): 0
   Average unshared data size (kbytes): 0
   Average stack size (kbytes): 0
   Average total size (kbytes): 0
   Maximum resident set size (kbytes): 3520
   Average resident set size (kbytes): 0
   Major (requiring I/O) page faults: 0
   Minor (reclaiming a frame) page faults: 11201235
   Voluntary context switches: 138500
   Involuntary context switches: 44337
   Swaps: 0
   File system inputs: 0
   File system outputs: 0
   Socket messages sent: 0
   Socket messages received: 0
   Signals delivered: 0
   Page size (bytes): 4096
   Exit status: 0

The elapsed time in seconds for this scenerio is 162.86.  That is 81.43 ms per extension.  Since this solution affects all loaded extension, every extension loaded will have 4.01 ms of additional time to load on average.  Approximately a 5% increase in load time.  That seems a little high to implement for all users since it affects every extension loaded.


Scenerio 3:
Use the a tce-load script with modifications to only substitute for KERNEL on a failure to load all of these exensions. Output:

   User time (seconds): 21.33
   System time (seconds): 76.11
   Percent of CPU this job got: 61%
   Elapsed (wall clock) time (h:mm:ss or m:ss): 2m 37.64s
   Average shared text size (kbytes): 0
   Average unshared data size (kbytes): 0
   Average stack size (kbytes): 0
   Average total size (kbytes): 0
   Maximum resident set size (kbytes): 3520
   Average resident set size (kbytes): 0
   Major (requiring I/O) page faults: 0
   Minor (reclaiming a frame) page faults: 10379481
   Voluntary context switches: 125365
   Involuntary context switches: 41818
   Swaps: 0
   File system inputs: 0
   File system outputs: 0
   Socket messages sent: 0
   Socket messages received: 0
   Signals delivered: 0
   Page size (bytes): 4096
   Exit status: 0

The elapsed time in seconds for this scenerio is 157.64.  That is 78.82 ms per extension.  Since this solution only affects those extensions that had a failure, only 1000 of the 2000 loaded extensions were affected.  This would add 2.8 ms per kernel module extension, approximately 3.6%.  Kernel module extensions are few compared to all the other extensions.  I believe a less then 5% performance decrease for a small number of extensions is acceptable for this feature.

Here is the diff for scenerio 3:

Code: [Select]

@@ -67,10 +67,6 @@
  DEPLIST="${DEPLIST#*/}"
 }
 
-dep_vars(){
- sed -i "s@KERNEL@$KERNELVER@" $1
-}
-
 copyInstall() {
  [ -d /mnt/test ] || sudo mkdir -p /mnt/test
  sudo mount $1 /mnt/test -t squashfs -o loop,ro,bs=4096
@@ -166,7 +162,6 @@
  if [ ! -f ${d}.dep ]; then
  wget -c -P "$FROMWHERE" "$MIRROR"/"$d".dep 2>/dev/null
  if [ "$?" == 0 ]; then
- dep_vars ${d}.dep
  [ "$SUPPRESS" ] || echo ${d}.dep OK
  remote_recursive_scan $d
  fi
@@ -210,7 +205,6 @@
  DEPLIST="$EXTENSION".dep
  wget -cq "$MIRROR"/"$DEPLIST" 2>/dev/null
  if [ "$?" == 0 ]; then
- dep_vars $DEPLIST
  DEPLIST=""
  remote_recursive_scan ${EXTENSION}
  pop_dep
@@ -254,8 +248,11 @@
  if [ -f "$TCEDIR"/optional/"$TARGETAPP" ]; then
  TARGETAPP="$TCEDIR"/optional/"$TARGETAPP"
  else
- echo "$TARGETAPP not found!"
- abort_to_saved_dir
+ TARGETAPP=$( echo ${TARGETAPP%%/*} | sed "s:-KERNEL\.tcz$:-${KERNELVER}.tcz:g" )
+ if [ -f "$TCEDIR"/optional/"$TARGETAPP" ]; then
+ echo "$TARGETAPP not found!"
+ abort_to_saved_dir
+ fi
  fi
  fi
  FROMWHERE=`dirname "$TARGETAPP"` && cd "$FROMWHERE"
@@ -272,8 +269,14 @@
  install "$DEPFILE"
  [ "$SUPPRESS" ] || echo "$DEPFILE: OK"
  else
- echo "$DEPFILE not found!"
- abort_to_saved_dir
+ DEPFILE=$( echo ${DEPFILE%%/*} | sed "s:-KERNEL\.tcz$:-${KERNELVER}.tcz:g" )
+ if [ -f "$DEPFILE" ]; then
+ install "$DEPFILE"
+ [ "$SUPPRESS" ] || echo "$DEPFILE: OK"
+ else
+ echo "$DEPFILE not found!"
+ abort_to_saved_dir
+ fi
  fi
  fi
  pop_dep


Another option would be to include this feature along with the dep_vars substitution.  That would yield the faster loading results for normal users but developers and whoever else can change their dep files to use this feature.  Perhaps including a script in compiletc or submitqc that can change the dep files for dev use.

[maro]

robc (and everybody else interested in this subject): As I had indicated in my 'EDIT 2' of the alpha3 thread I had eventually observed a functional regression (i.e. under some circumstances some missing dependencies occurred).

I have since made some more changes which appeared to have corrected this issue, but I have not done enough testing to be satisfied that all is good. I've just come back from my dentist, so I'm not sure how long it will take me to post the "better" diff. But if possible could we please all defer any further discussion about the merits of that potential change to 'tce-load' for let's say 24h. I would then need to ask you to re-run your scenario 2 (and 3) testing again, but I hope you've got a setup that should allow for this without too much difficulty.

BTW, I was wondering whether this 4.0rc1 thread here is the proper place for this or whether those few posts specific to this question should rather all be merged into a separate thread. I guess that's up to Robert to do or not ...

[uggla]

I'm running a dual boot with mc32 and mc64 using the same tce-dir. I managed to do this using links and subdirectories. It's not perfect, and as much as I would love to have this issue fixed in base I don't think its worth slower loading of extensions (even if the slowdown is minimal).

(my main reason for booting mc32 is to run Virtualbox)

[hiro]

Has tinycore's main aim become fast boot up now?

[roberts]

Your post would seem to suggest a change in policy.

Have you not noticed the boot charts?
Have you not noticed the rejection of the use of lzma from the beginning?

It really comes down to implementing something that benefits everyone versus a few.

Size, Speed, benefits (everyone) are typically the criteria for accepting suggestions.

[robc]

I know there is a performance decrease with this feature.  That is why I suggested the following at the end of my last post:

Another option would be to include this feature along with the dep_vars substitution.  That would yield the faster loading results for normal users but developers and whoever else can change their dep files to use this feature.  Perhaps including a script in compiletc or submitqc that can change the dep files for dev use.

This would keep the current load speed of the extensions and give users the option to use this feature.

Though if your concern is speed then why use a script to load extensions and not a C binary?  Wouldn't it be faster to load extensions directly through the use of libraries rather then relying on a shell and multiple commands?

[roberts]

I knew this would come up. This project was never intended to be a binary only distribution. It was to see how far one could go with using only ash & awk from busybox.  So that others could easily read and modify. I have purposely keep C and even C++ to a minimum.  It would also increase overall size and likely make it less portable. Where ever busybox is supported the infrastructure of Tiny Core could be easily ported, e.g. microcore on arm WM8505 using an android kernel.  But again, now off topic, as this suggestion only benefits a select few.

[hiro]

I always thought policy was about simplicity, and size and speed just a lucky byproduct.
Size and speed also works for me, as a good sign it still seems to be interchangeable with simplicity on tinycore

[maro]

A bit more then 24h later, but I'm now happier with the results of my own testing. I ended up replacing the 'sed' commands with the pattern replacement variation of the parameter expansion (which is a feature of the shell itself). This way no costly "outside" call to 'sed' was required. Another significant change was to bring the replacement operations "inside" of the 'local_recursive_scan()' and 'remote_recursive_scan()' functions, as otherwise a functional regression for nested kernel module dependencies was observed (e.g. for test case: 'wxcam.tcz').

Code: (bash) [Select]

--- /usr/bin/tce-load
+++ tce-load
@@ -62,4 +62,5 @@
 push_dep(){
-       DEPLIST="$1""/""$DEPLIST"
+       DEPLIST="${1}/${DEPLIST}"
 }
+
 pop_dep(){
@@ -69,6 +70,2 @@

-dep_vars(){
-       sed -i "s@KERNEL@$KERNELVER@" $1
-}
-
 copyInstall() {
@@ -149,6 +146,7 @@
 local_recursive_scan() {
-       push_dep ${1}
+       EXT="${1//-KERNEL.tcz/-${KERNELVER}.tcz}"
+       push_dep ${EXT}
        deps=""
-       if [ -f ${1}.dep ]; then
-               deps=`cat ${1}.dep`
+       if [ -f ${EXT}.dep ]; then
+               deps=$( cat ${EXT}.dep )
                for d in $deps; do
@@ -160,6 +158,8 @@
 remote_recursive_scan() {
-       push_dep ${1}
+       EXT="${1//-KERNEL.tcz/-${KERNELVER}.tcz}"
+       push_dep ${EXT}
        deps=""
-       deps=`cat "$FROMWHERE"/${1}.dep`
+       deps=$( cat "${FROMWHERE}/${EXT}.dep" )
        for d in $deps; do
+               d="${d//-KERNEL.tcz/-${KERNELVER}.tcz}"
                push_dep ${d}
@@ -168,3 +168,2 @@
                        if [ "$?" == 0 ]; then
-                               dep_vars ${d}.dep
                                [ "$SUPPRESS" ] || echo ${d}.dep OK
@@ -192,2 +191,4 @@
 if [ ${TARGETAPP} == ${APPNAME} ]; then TARGETAPP=${TARGETAPP}.tcz; fi
+APPNAME="${APPNAME/-KERNEL/-${KERNELVER}}"
+TARGETAPP="${TARGETAPP/-KERNEL.tcz/-${KERNELVER}.tcz}"

@@ -212,3 +213,2 @@
        if [ "$?" == 0 ]; then
-               dep_vars $DEPLIST
                DEPLIST=""
I've done my own (limited) testing where I measured the installation time of a single extension (e.g. 'compiletc.tcz') multiple times and averaged out the measured results. Comparing the TC 3.8.4 version with my patched one showed that my version appeared to be in my tests always a tiny bit faster then the Core one. Maybe the elimination of the 'sed' call in 'dep_vars()' gained enough to "pay" for the additional multiple pattern replacement operations.

Clearly it is vital that these findings of my limited testing can be substantiated trough the testing of others (e.g. robc). If the result can be replicated then maybe, just maybe, what was meant to be an improvement for the few could become something to the benefit of the many.

[robc]

Scenario 4:
I ran a new scenario using the changes in maro's diff in the previous post.  Here are the results:

   User time (seconds): 21.64
   System time (seconds): 76.25
   Percent of CPU this job got: 62%
   Elapsed (wall clock) time (h:mm:ss or m:ss): 2m 35.42s
   Average shared text size (kbytes): 0
   Average unshared data size (kbytes): 0
   Average stack size (kbytes): 0
   Average total size (kbytes): 0
   Maximum resident set size (kbytes): 3536
   Average resident set size (kbytes): 0
   Major (requiring I/O) page faults: 0
   Minor (reclaiming a frame) page faults: 9894749
   Voluntary context switches: 118515
   Involuntary context switches: 40793
   Swaps: 0
   File system inputs: 0
   File system outputs: 0
   Socket messages sent: 0
   Socket messages received: 0
   Signals delivered: 0
   Page size (bytes): 4096
   Exit status: 0

This yields an average of 77.71ms per extension to load.  This is an increase of 0.29ms from the base tce-load which is about 0.4% decrease in performance.  This is pretty close, the difference in performance here may have nothing to do with the script itself.  I would have to run this scenario a few more times to see what kind of range it would give.

Scenario 5:

   User time (seconds): 21.91
   System time (seconds): 76.81
   Percent of CPU this job got: 63%
   Elapsed (wall clock) time (h:mm:ss or m:ss): 2m 36.12s
   Average shared text size (kbytes): 0
   Average unshared data size (kbytes): 0
   Average stack size (kbytes): 0
   Average total size (kbytes): 0
   Maximum resident set size (kbytes): 3520
   Average resident set size (kbytes): 0
   Major (requiring I/O) page faults: 0
   Minor (reclaiming a frame) page faults: 9911401
   Voluntary context switches: 118517
   Involuntary context switches: 40630
   Swaps: 0
   File system inputs: 0
   File system outputs: 0
   Socket messages sent: 0
   Socket messages received: 0
   Signals delivered: 0
   Page size (bytes): 4096
   Exit status: 0

This scenario is the same as Scenario 3 but with the sed calls changed to the shell replacement that maro used.  This decreased the load time per extension but was not as good of results as above. Average 78.06ms per extension loaded, which is 1.28ms longer per kernel module extension (about 1.6% increase)

I started looking into a way to combine Scenario 4 with the replace only when needed model of Scenarios 3 and 5 but ran into some errors before calling it a night.  I may be able to get to this more this weekend.