Understanding tc: Why is “mount on boot” not super fast?

[Stefann]

Hi,
Sorry if this common knowledge or if it is already described somewhere.
I’m trying to learn.

At this moment I have several apps installed as “mount at boot”. I DO understand that I could have chosen for “on demand” which would save mount time and maybe I will do that. At this moment I just want to understand.

What I see happening during startup is that the time spend “mounting all those packages” takes quite some time (I did not time it exactly but I guess 30..60 seconds).
In my (ignorant) assumption “mounting” is just setting some pointers and does not really actively “do” something. After all… the whole reason “not to choose copy mode” is to avoid taking time to load them in ram. If that is the case mounting of all applications together should take less than a second.

For that reason my question: “why is it taking considerable time”? And/or “any smart thoughts about the philosophy behind things” (because this choice between “on demand”/“on boot”/“copy” I consider a very core aspect of TC, i would not be surprised if there would be an untold “philosophy”/“deeper thought” on this).

Note. Practically speaking the boot time is no problem at all. I use tc as os behind a 24/7 home control computer. I boot it only in case of some maintenance. It’s just curiosity.

[curaga]

The book contains a lot of info on this, on why symlinks vs unionfs etc.

[Stefann]

The book contains a lot of info on this, on why symlinks vs unionfs etc.

Thanks, I needed that pointer.
I had read the book already once but “message did not compute at that time”.
The “unionfs” pointer helpt me locating the chapter.
Please be aware I’m not a Linux expert. I’m learning.

I’m highly intrigued and impressed by the tiny core approach though. It is quite brilliant.
- the frugal approach keeps the os clean.
- the .filetool.lst file contains a highly transparent and simple overview of user modifications to the os.
- all in ram makes it super fast and keeps the flashdrive alive by not writing to it (unless specifically asked by the user application)

[Stefann]

Ok… I studied the book “with red ears”.. wow 

I guess the answer to my question is:
- the tgz is a squashfs based compressed filesystem.
- on boot it creates several mount points in the main filesystem.
- squashfs needs “few blocks in ram” to do that.

So the time is consumed in:
- creating the several mount points in the main filesystem to begin with
- inserting the squashhfs overhead in ram needed to efficiently “make this work”

The method “maintains the original filestructure” of the application (which is good because the os than not generates unexpected omissions).
But that also means that complex applications (such as the gcc compiler) need some time to get mounted IF they need a lot of mountpoints.

>> makes perfect sense! Happy to have learned this.
(Including having broadened my general Linux knowledge)

[CentralWare]

What I see happening during startup is that the time spend “mounting all those packages” takes quite some time (I did not time it exactly but I guess 30..60 seconds).

@Stefann: Good morning!

It sounds to me as though you have a desktop environment and quite a few extensions loading during boot.
If boot time were a concern, you can delegate some extensions to load AFTER the desktop launches.

For example, let's say you had OpenSSH or Dropbear loading to give you remote shell access to the machine. This doesn't HAVE to be something that's done right this second while it's booting up as it'll delay getting to the desktop and it has nothing to DO with the desktop's operations. That said, you could take OpenSSH out of /etc/sysconfig/tcedir/onboot.lst and instead, load it later...  in the background... after your desktop is finished and operational.

The extensions you take out of onboot.lst...  move them into onlater.lst in the same location.  onlater.lst is just a name I came up with, it is not in the TCL text book.
Do not add drivers/firmware, Xorg support extensions or file systems to onlater.lst as it may have adverse results or even stop the desktop from loading.

Add a new line somewhere near the end of /opt/bootlocal.sh

Code: [Select]

/opt/bootlater.sh &
Then create /opt/bootlater.sh

Code: [Select]

#!/bin/sh
TCEDIR=/etc/sysconfig/tcedir
for EXT in $(cat $TCEDIR/onlater.lst)
do
     su -c "tce-load -i ${TCEDIR}/optional/${EXT}" tc
     sleep 1
done
Run backup if you are using backup to persist your changes so that /opt/bootlater.sh gets saved.
The extensions you load in the background, I would recommend limiting any extensions that add icons to the toolbar/wbar/etc. as too many of these being done at the same time can get odd results.
The sleep() call above slows down loading extensions in the background to help prevent some of these oddities.

Please be aware I’m not a Linux expert. I’m learning.

That is exactly why we do what we do here...

[Stefann]

Thanks a lot,
that makes a lot of sense.

You are very right,...  I AM loading a lot of stuff I do not need directly.
Especially the c compiler gcc. Also "make" not directly needed.
Also I have feeling I have too much php booting
php-8.3-mod.tcz
php-8.3-fpm.tcz
php-8.3-dev.tcz
php-8.3-cli.tcz
php-8.3-cgi.tcz
while I only need simple php scripts called from apache... no worry, I'll figure out which I can remove by trial and error.
On the other hand I have not included python yet and that is on the todo list.

Not sure I will follow your suggestion (hope you don't mind),
My main question was on "understanding", not necessarily "changing".
Booting the computer is not something I do very often (main reason would be after a crash that only very rarely happens) so the inconvenience is very bearable.
Although clear instructions, I have a big preference to keep things close to standard. I have been running this thing on DSL (damn small linux) since 2008 and expect to run it on TC for an other 15..20 years (lets say... until I die, I'm 62 now). This means I will experience OS upgrades. I want to avoid "OS upgrades requiring work that I totally forgot how to do".

I'm happy I at least "understand" where the "on demand time" is spend on, I can than avoid overloading it.
Also "as the booting took so much time", I was a bit afraid the stuff would also bring cpu-load during normal operation (avoiding that IS a big priority). I now better understand the impact.
If boottime becomes very long I will do as you say.

[CentralWare]

Not sure I will follow your suggestion (hope you don't mind),

Of course I don't mind!  In fact, this thread gives me an idea for a project we're working on.

Additionally, if you were to modify the list of boot codes you're using and add the code showapps, should you ever need to reboot, this code would display the progress of which extensions are loaded / being loaded and if there was one extension in particular that was taking a good deal of time loading, it would be visually apparent.

Good luck and take care!

[Stefann]

Of course I don't mind!  In fact, this thread gives me an idea for a project we're working on.

Additionally, if you were to modify the list of boot codes you're using and add the code showapps, should you ever need to reboot, this code would display the progress of which extensions are loaded / being loaded and if there was one extension in particular that was taking a good deal of time loading, it would be visually apparent.

Good luck and take care!

Thanks!
That is actually what I did, and how I noticed that the mounting takes significant time.
I was under the impression that mounting would be similarly simple as creating some symbolic links so that long time surprised me (although I already wondered how “linking into a compressed file” would work).
With all of the knowledge I gathered over past few days I much better understand the concept.

Monitoring the boot- actions on a regular basis will not be possible. In principle I run the thing headless. It’s located in a utility room where hooking up a console is quite cumbersome. I had to do it to solve bios boot-order issues which allowed me to see all this happening. In normal operation I will not be able to see this.
Which actually is a reason to “like to understand this at this moment in time”. It will prevent me from going in wrong directions later.

One thing I WILL do to improve performance is getting rid of desktop/vnc.
At this moment I run fwlm desktop with vnc to access.
I have shh and samba fully functional which means I can actually do all my normal work without desktop/vnc. At this moment I’m still regularly reverting to some of the tiny core gui-based tools so I like the desktop for now.
Once all setup is robustly working without need for tweaks over a week or so I will insert the “text” boot-option and prevent fwlm en vnc from starting. This will improve startup but especially save cpu load in normal operation.
I’ll basically bring 2 labels in de extlinux.conf: one with and one without text-mode. That will allow me to change default label to text or non-text using nano and force that mode by a reboot.

Again, thanks for explaining/guiding me in directions. All ideas as brought are highly appreciated. Even if I don’t do them directly they will feed into my furture plans.

[Stefann]

@Stefann: Good morning!

It sounds to me as though you have a desktop environment and quite a few extensions loading during boot.
If boot time were a concern, you can delegate some extensions to load AFTER the desktop launches.

For example, let's say you had OpenSSH or Dropbear loading to give you remote shell access to the machine. This doesn't HAVE to be something that's done right this second while it's booting up as it'll delay getting to the desktop and it has nothing to DO with the desktop's operations. That said, you could take OpenSSH out of /etc/sysconfig/tcedir/onboot.lst and instead, load it later...  in the background... after your desktop is finished and operational.

The extensions you take out of onboot.lst...  move them into onlater.lst in the same location.  onlater.lst is just a name I came up with, it is not in the TCL text book.
Do not add drivers/firmware, Xorg support extensions or file systems to onlater.lst as it may have adverse results or even stop the desktop from loading.

Add a new line somewhere near the end of /opt/bootlocal.sh

Code: [Select]

/opt/bootlater.sh &
Then create /opt/bootlater.sh

Code: [Select]

#!/bin/sh
TCEDIR=/etc/sysconfig/tcedir
for EXT in $(cat $TCEDIR/onlater.lst)
do
     su -c "tce-load -i ${TCEDIR}/optional/${EXT}" tc
     sleep 1
done
Run backup if you are using backup to persist your changes so that /opt/bootlater.sh gets saved.
The extensions you load in the background, I would recommend limiting any extensions that add icons to the toolbar/wbar/etc. as too many of these being done at the same time can get odd results.
The sleep() call above slows down loading extensions in the background to help prevent some of these oddities.

@Centralware
Just rethinking this…
I not yet have figured out how to start an application that was “not on boot mounted”. Will need to wait until next weekend.
But.. having said that..
Instead of adding this extra script,…
Could I not just mount the delayed applications by calling them?
My main delayed time is gcc. Ik probably could just add a “make alert” call in the bootlocal.sh
And have a rule for “alert” that basically just calls gcc for a status message.

This would have the advantage that I don’t need to add an extra .sh file. Reason I am bit hesitant to add such file:
- if I do a tinycore upgrade (say) 4 years from now I may have forgotten about that and fail the upgrade running into a lot of trouble shooting (realistic,… I just upgraded from latest damn small Linux version-2008 to tinycore 15).
- I’m not really a bash-script wizard. I’m struggling enormously with the syntax. I have done a few but all either extremely simple (without variables, without conditional statements), or succeeding only after a lot of trial and error. So…. Of course it would be very nice if your script “just works”, but if I need to modify it 2 years from now that will cost me significant time.

Havingd said that…….I may end up doing it anyway. I like to learn, if I really did not have wanted this I would just have bought something instead of doing it all myself. My homecontrol system is programmed in C from scratch. Started the project before home assistant was a thing.
But… for something as “un-annoying as this” I prefer to keep things simple.

[CentralWare]

@Stefann: Have you experimented with OnDemand?
I'm going through the steps right now while describing it to ensure I'm being as accurate as possible.

1. tce-load -wo curl
     This loads the extension curl as an OnDemand program; I just rebooted that machine.  Curl is NOT loaded.

2. curl http://www.google.com
     The curl extension is loaded/mounted and then launched. Google's heavily obfuscated javascript content is returned, as expected.

3. Until the machine is rebooted, the curl extension remains loaded.

I'd imagine this doesn't operate as expected on every extension, but for compiler tools and other things you may not use very often, this may be a way to speed up booting AND reduce memory usage AND in the process, reduce CPU during startup by reducing the number of extensions being dealt with at that time.  It should also be "upgrade" friendly.

Note: Upgrades to the operating system should never be affected by /opt scripts as your backups or persistent directories generally fall outside the core operation files and extensions.

REMEMBERING changes you've made, however, is another story.  It's always wise if you make changes and need to remember 5 years from now WHY you did something or what you did it for, in shell scripts like bootlocal.sh, bootsync.sh (and other .sh files) simply leave yourself detailed notes/comments preceded by a pound sign (#)

Code: [Select]

# put other system startup commands hereis a good example found at the start of /opt/bootlocal.sh  The "#" and everything that follows it on that line is ignored by the system.
This is a similar concept to the comments you're accustomed to in C.

[Rich]

Hi CentralWare

... I'd imagine this doesn't operate as expected on every extension, ...

It doesn't work for every extension. You don't want to use it for
libraries, kernel modules, or firmware.

It works for curl because a script that loads curl.tcz is created and that
script is called curl. It is found because $PATH includes the ondemand
directory:

Code: [Select]

tc@E310:~$ echo $PATH
/home/tc/.local/bin:/usr/local/sbin:/usr/local/bin:/apps/bin:/usr/sbin:/usr/bin:/sbin:/bin:/etc/sysconfig/tcedir/ondemand
tc@E310:~$
It will likely fail with tar.tcz because busybox /bin/tar will be found before
/etc/sysconfig/tcedir/ondemand/tar.

It will only work if the name of the extension exactly matches the name
of the program you want to run.

Anyone placing compiletc.tcz ondemand expecting gcc to magically
load and execute will be disappointed.

[Stefann]

@Stefann: Have you experimented with OnDemand?
I'm going through the steps right now while describing it to ensure I'm being as accurate as possible.

1. tce-load -wo curl
     This loads the extension curl as an OnDemand program; I just rebooted that machine.  Curl is NOT loaded.

2. curl http://www.google.com
     The curl extension is loaded/mounted and then launched. Google's heavily obfuscated javascript content is returned, as expected.

3. Until the machine is rebooted, the curl extension remains loaded.

I'd imagine this doesn't operate as expected on every extension, but for compiler tools and other things you may not use very often, this may be a way to speed up booting AND reduce memory usage AND in the process, reduce CPU during startup by reducing the number of extensions being dealt with at that time.  It should also be "upgrade" friendly.

Note: Upgrades to the operating system should never be affected by /opt scripts as your backups or persistent directories generally fall outside the core operation files and extensions.

REMEMBERING changes you've made, however, is another story.  It's always wise if you make changes and need to remember 5 years from now WHY you did something or what you did it for, in shell scripts like bootlocal.sh, bootsync.sh (and other .sh files) simply leave yourself detailed notes/comments preceded by a pound sign (#)

Code: [Select]

# put other system startup commands hereis a good example found at the start of /opt/bootlocal.sh  The "#" and everything that follows it on that line is ignored by the system.
This is a similar concept to the comments you're accustomed to in C.

Thanks,
That is indeed how I was hoping this would work.
No I did not try this yet. As I have a day job it will need to wait until Saturday.
And... even worse... today it became clear I will need to help my son with his business for some time. I'm afraid "tiny core time" will be very restricted in the weeks to come (but I will need some tiny core relax time now and than  ).
I'll keep you posted on progress.

Yes... indeed I make notes on how I configure things.
preferably as comments in the files I have as you say
I made a full text file that describes step-by-step the whole process that I went through to get tiny core working. Including sections and config info for each application I loaded.
For that reason I feel quite "fine" to add some lines to boot local.sh
But feel more reserved on adding an other script.sh file:
- extra lines in boot local.sh I can simply accompany by comment
- an extra script.sh file will need an extra section in my step-by-step documentation. Which makes it more cluttered. As said... I may still do it... but I have a basic reluctance against extra files, especially if the benefit is very marginal.

For info: I timed a reboot yesterday. The console is no longer there to monitor boot-steps but I can of course time between reboot-command & functional again. I am currently at 80 seconds (on a 500MHz single core 32bit ultra low power VIA EDEN processor). In principle "that is very acceptable", just enough to make a coffee.

It doesn't work for every extension. You don't want to use it for
libraries, kernel modules, or firmware.

Anyone placing compiletc.tcz ondemand expecting gcc to magically
load and execute will be disappointed.

Ah.. thanks for that update.
So... it will require testing. And it will need "retesting" for any application.
We'll see... as mentioned above, it looks like I will not have much time in coming weeks, I keep you guys posted.

And... main goal of the start of this tread is certainly achieved: I got guided into understanding things and I will built on that.

[CentralWare]

@Stefann: The extra script isn't a requirement; the same content can be added to bootlocal.sh directly; it's merely my own practice to separate things for organizational reasons.
(bootlocal.sh is already run as a background task.)

@Rich: compiletc...  yes, I hadn't considered that one but can imagine the reaction!  Until recently, I didn't utilize macro extensions at all - nor did OnDemand seem useful for what I do on a day-to-day basis, so putting the two together wasn't going to come from experience   I do, however, miss the "TC" macro concept!  (TCL 8.x I believe - "instant desktop")

[Rich]

Hi CentralWare

... Until recently, I didn't utilize macro extensions at all ...

I guess you are referring to  meta-extensions ?

- nor did OnDemand seem useful for what I do on a day-to-day basis, ...

I've never used OnDemand myself. I felt it was too easy to misuse. Though it's been
a while, there have been some occasions where a user saw OnDemand and assumed
it meant every command could be at your fingertips by placing extensions there
instead of OnBoot.

... It will only work if the name of the extension exactly matches the name
of the program you want to run.

Anyone placing compiletc.tcz ondemand expecting gcc to magically
load and execute will be disappointed.

Just to hammer the point home (or beat it to death), here are a handfull of
(non-meta extension) examples that are also not candidates for OnDemand:

Code: [Select]

ace-of-penguins.tcz
alsa.tcz
alsa-config.tcz
aoetools.tcz
apache2.4.tcz
bind.tcz
binutils.tcz
blueman.tcz
bluez.tcz
bluez5.tcz
cairo.tcz
cdrtools.tcz
connman.tcz
coreutils.tcz
cups.tcz
ffmpeg3.tcz
findutils.tcz
fox.tcz
freebasic.tcz
ijs.tcz
inetutils.tcz
inotify-tools.tcz
pci-utils.tcz
usbutils.tcz
util-linux.tcz
Also, most, if not all of the following:

Code: [Select]

*-dev.tcz
*-doc.tcz
*-icon*.tcz
lib*.tcz     # Plus library extensions by other names (cairomm.tcz, clucene.tcz, clutter.tcz, freeglut.tcz, glu.tcz, ... )
*-locale.tcz
*-theme*.tcz

[Stefann]

@Stefann: The extra script isn't a requirement; the same content can be added to bootlocal.sh directly; it's merely my own practice to separate things for organizational reasons.
(bootlocal.sh is already run as a background task.)

Than actually makes a lot of sense.
I will try it.
As said… time will be bit of a problem upcoming weeks

Just to hammer the point home (or beat it to death), here are a handfull of
(non-meta extension) examples that are also not candidates for OnDemand…..

Shortened the quote… but you fully made your point.

Amen..

Will not go that route