Understanding tc: Why is “mount on boot” not super fast?

[Leee]

I've been following this thread and I've got to say, I just learned a -lot- about loading extensions ondemend.  I like to keep my toolbox well stocked, so to speak, so knowing some of the gritty details about ondemand is kind of satisfying.
Having said that, I've never actually used ondemand, nor do I expect to... but you never know.

[CentralWare]

I guess you are referring to  meta-extensions ?

@Rich: LOL...  nah!
META="...to one's self" (like self-reflection) or showing or suggesting an explicit awareness of itself or oneself as a member of its category"
MACRO="a single instruction that expands automatically into a set of instructions to perform a particular task."

For example, the
META-TAG (in HTML) describes the page it lives within; sometimes more as an appendage as time progressed.
META-MUCIL...  doesn't get more "self aware" of what's "within" than that!
"eh...  whadda I know!?"

@Leee: I can imagine for those short on memory...  I mean RAM... loading things on-the-fly could really come in handy.
I'd imagine it being even more useful if those same items could be unloaded when they've completed their tasks...  but the argument would then be "...who is to say WHEN it's no longer needed?..."

[Rich]

Hi CentralWare

I guess you are referring to  meta-extensions ?

@Rich: LOL...  nah!
META="...to one's self" (like self-reflection) or showing or suggesting an explicit awareness of itself or oneself as a member of its category"
MACRO="a single instruction that expands automatically into a set of instructions to perform a particular task." ...

So you weren't referring to extensions like  compiletc.tcz ? 

Title:          compiletc.tcz
Description:    compiletc metapackage
Version:        0.1
Author:         n/a
Original-site:  tinycorelinux.com
Copying-policy: n/a
Size:           4KB
Extension_by:   N. Carigon
Tags:           compiletc metapackage
Comments:       This is a meta-extension for a compile environment
 ----- Snip ----- 

[CentralWare]

@Rich: Sorry, was making fun of the "meta" name (per Webster's)

[Rich]

Hi CentralWare
I know, I was just messing with you.

[yvs]

Hi CentralWare

... I'd imagine this doesn't operate as expected on every extension, ...

It doesn't work for every extension.

I suppose that tips suggesting extensions containing some program (or might be even with a prompt to download/install them) can be done by shell hooks on user side (most of shells have some not_found hook).
Those handy tips are often used in many other distributions.

Something like

Code: [Select]

tc@tc-x86_64:~$ cc
cc found in 'gcc'

tc@tc-x86_64:~$ as
as found in 'binutils'

tc@tc-x86_64:~$ yacc
yacc found in 'bison'

tc@tc-x86_64:~$ locate
locate found in 'findutils'

tc@tc-x86_64:~$ hmm   
hmm: not found

The only thing that makes it a bit annoying it's manual support and update a list of those binaries because lack of centralized db/lists of packaged files (it's several shell lines but anyway it can easy forget to update those lists locally by end-user) and possible difference in naming of the same extensions among TCL for different architectures

[Stefann]

Note… in response to my own original question:
Understanding tc: Why is “mount on boot” not super fast?
Where my initial objective was mostly “understanding”, not necessarily “solving”.
(For reason I intend to use this OS for 15+ year, I like to understand it)

I had a bit of a brainwave….
Mounting is probably reasonably fast,
But the full compressed archive will probably need a read

That would mean that the “on boot” startup time would be  proportional to the disk space of that all “on demand” .tcz files take.

And this is where I probably went wrong in very initial assumption:
- a normal mount just mounts from a location one can find in an instant
- this mounting of a compressed archive probably requires disk operation to read the full archive file.

In weekend I will see which package has a large footprint and see what happens if I exclude it from boot.
My thought is triggered because I notice the “disk activity” led showing lots of action during boot. That is not mounting…. That is reading.

[Rich]

Hi Stefann

... My thought is triggered because I notice the “disk activity” led showing lots of action during boot. That is not mounting…. That is reading.

The disk activity is likely from all of the links being created from
the filesystem to the files in each extension.

[curaga]

@yvs: Personally that really annoys me on Ubuntu and others. When I make a typo, that hook causes significant delay, with cold caches the first time is several seconds.

[yvs]

@yvs: Personally that really annoys me on Ubuntu and others. When I make a typo, that hook causes significant delay, with cold caches the first time is several seconds.

agree I usually disable it and use `apt-file search` when it's needed, it's slow there maybe because of enormous number of packages and files in that, or probably some compression to keep that info, I haven't dig into it.
In case of not so big number of TC extensions it doesn't look slow to become annoying, for example for the first time not installed gcc:

Code: [Select]

tc@tc-x86_64:~$ time gcc
gcc found in 'gcc'
                                                                               
real    0m0.009s
user    0m0.004s
sys     0m0.005s


it's in case of a simple bash hook (not tried busybox ash compiled with CONFIG_ASH_BASH_NOT_FOUND_HOOK yet)

Code: [Select]

# it can be boosted a bit using filelist on ramfs and pregrepping before sed

EXT_BIN_LIST="/etc/sysconfig/tcedir/not_found.aux"

function command_not_found_handle() {
  test -f "$EXT_BIN_LIST" && {
    bin="$1"; shift
    exts="$(sed -n "s,^\(.*\)[ ]\+.*/$bin/.*,\1,p" <"$EXT_BIN_LIST")"
    test -n "$exts" && echo "$bin found in '$exts'" || echo "$bin: not found"
  }
  return 127
}

and simple text format like

Code: [Select]

% grep '^gcc' /etc/sysconfig/tcedir/not_found.aux
gcc /c++/cc/cpp/g++/gcc/gcc-ar/gcc-nm/gcc-ranlib/gcov/gcov-dump/gcov-tool/lto-dump/x86_64-pc-linux-gnu-c++/x86_64-pc-linux-gnu-g++/x86_64-pc-linux-gnu-gcc/x86_64-pc-linux-gnu-gcc-13.2.0/x86_64-pc-linux-gnu-gcc-ar/x86_64-pc-linux-gnu-gcc-nm/x86_64-pc-linux-gnu-gcc-ranlib/

Anyway command_not_found_handle() can be set/unset by user, it's always good to have options

[yvs]

- this mounting of a compressed archive probably requires disk operation to read the full archive file.

Not sure if that's related: on Ubuntu with its squashfs use (for snaps) there were many discussions of use different compression formats (for example lzo zstd xz etc.) trading-of between size and performance

[mocore]

Ubuntu with its squashfs use (for snaps) there were many discussions of use different compression formats (for example lzo zstd xz etc.) trading-of between size and performance

 some perhaps (historically) relevant content on that subject linked in the below quote 

wrt squashfs and compression
iv just found this [linked below]  post lamenting the state of Ubuntu's Snap single-threaded decompression   
... "scripts to recreate these measurements are included" ! .. if any one with similar hw is interested in making comparison?
https://forum.snapcraft.io/t/squashfs-is-a-terrible-storage-format/9466

[Stefann]

head of the optional folder sorted on size.

Indeed gcc is super big
-> I definitely can do with a later load of that.
Would not be surprised if that is eating 50% of boottime.

To my (kind of) surprise samba3 is also big
-> One of the first things I use, cannot load later.
>> guess samba did not succeed in "having small footprint" like many other apps.

mmmh... I did. not install perl. guess this is a dependancy somewhere

ouch... I have 2 pythons, I can get rid of one.

After that it becomes peanuts

Will need to find time in weekend to explore
 

Code: [Select]

tc@huis:/mnt/sda1/tce/optional$ ls -Sl
total 215860
-rw-rw-r--    1 tc       staff     61710336 Aug 17 06:26 gcc.tcz
-rw-rw-r--    1 tc       staff     35917824 Aug 17 06:25 samba3.tcz
-rw-rw-r--    1 tc       staff     16601088 Aug 17 06:25 perl5.tcz
-rw-rw-r--    1 tc       staff     16556032 Aug 17 06:25 python3.6.tcz
-rw-rw-r--    1 tc       staff     15740928 Aug 17 06:25 python3.9.tcz
-rw-rw-r--    1 tc       staff      6410240 Aug 17 06:25 binutils.tcz
-rw-rw-r--    1 tc       staff      4218880 Aug 17 06:25 php-8.3-dev.tcz
-rw-rw-r--    1 tc       staff      3239936 Aug 17 06:25 php-8.3-ext.tcz
-rw-rw-r--    1 tc       staff      3125248 Aug 17 06:25 glibc_base-dev.tcz
-rw-rw-r--    1 tc       staff      3063808 Aug 17 06:25 glibc_gconv.tcz
-rw-rw-r--    1 tc       staff      2994176 Aug 17 06:25 gcc_libs-dev.tcz
-rw-rw-r--    1 tc       staff      2834432 Aug 17 06:25 openssl.tcz
-rw-rw-r--    1 tc       staff      2764800 Aug 17 06:26 realvnc.tcz

[Rich]

Hi Stefann
I don't think size is the main issue. I think it's the number
of symlinks that need to be created. That is a function of
how many files are in the extension.

[Stefann]

Hi @rich,
I think you are right,
I did do some reading on squashfs, it’s indeed intended to be a fast compressed filesystem (you of course knew that ), and what’s more… the meta data is kept in some special place. So.. not the full file needs to be read to identify all mount points.
Anyway… the practical implication is probably less.
Biggest file gcc is 61MB, 7th ranking big file is php with 4MB. I guess that the amount of mountpoints for php will be similarly smaller.
I will just do a test this weekend by removing gcc from the onboot list and see what happens.

Note: by now it’s just for fun and learning. There is hardly a real need to do this.
Having said that….  if it gives a significant gain.. I will probably remove gcc from the onboot list permanently and load it via bootlocal.sh as @centralware suggested. A single line in bootlocal.sh i probably consider worth the “clutter deviating it from standard”.