Understanding tc: Why is “mount on boot” not super fast?

[CentralWare]

@Rich: I was about to give a little advice regarding extension loading (ie: $platform/tcz/src build file examples) until it dawned on me, I don't have an answer on a piece or three of the puzzle.
tce, tce-load, etc. - I'm guessing all of these scripts assume $USER is a sudoer?

SO... to test the theory I created a test script to clone how tce might be doing its (-i)

Code: [Select]

#!/bin/sh
. /etc/init.d/tc-functions
clear; CUR=$(pwd)

### tcl-load -i theory:

# Gather our extension name from $1 (just for this test, loop from $@ normally)
EXT=$1
EXT=${EXT/.tcz/}
EXT=${EXT/KERNEL/$(uname -r)}
echo "${WHITE}Loading ${YELLOW}$EXT${NORMAL}"

# 1. Mount the extension $EXT in /tmp/tcloop2
if [ ! -d /tmp/tcloop2/$EXT ]; then
    echo "  ${WHITE}Mounting ${YELLOW}$EXT${NORMAL}"
    mkdir -p /tmp/tcloop2/$EXT
    sudo mount /etc/sysconfig/tcedir/optional/${EXT}.tcz /tmp/tcloop2/$EXT || $(echo "${RED}ERROR! ${WHITE}Failed to load ${YELLOW}${EXT}${NORMAL}" && exit 1) # <-- SUDO Unnecessary?
fi

# 2. Navigate to /tmp/tcloop2/$EXT
cd /tmp/tcloop2/$EXT

# 3. I'm seeing LINKS and FILES in tce.installed...  LINKS are from /tmp/tcloop/EXT, FILES are dummy/empty that didn't exist.
<skipped>

# 4. Use find -exec to mkdir all DIRECTORIES to /fake/*
sudo find * -type d -exec sudo mkdir -p /fake/{} \;                                                     # <-- SUDO

# 5. Use find to LINK all FILES into /fake/*
sudo find * -type f -exec sudo ln -s /tmp/tcloop2/${EXT}/{} /fake/{} \;                                 # <-- SUDO

# 6. I'm guessing HERE is a good place to launch /usr/local/tce.installed/$EXT
[ -e /usr/local/tce.installed/$EXT ] && sudo sh /usr/local/tce.installed/$EXT                           # <-- SUDO

# 7. I don't know if files are SCANNED (libraries) or if ldconfig is called "just in case"
sudo /sbin/ldconfig >/dev/null 2>&1                                                                     # <-- SUDO

# 8. I'm also guessing for hardware/firmware/modules we need to call udevadm - unsure of our trigger
# find * -type d -name modules?  Or -name modules* needed for x64/pi/etc?

UPDATE: Move #6 to the end

...it was quicker than dredging through TCE
Any insight from experience? (When you have a minute!)

Thanks!

[Stefann]

Welll…. I did the test.. removed gcc from the onboot.lst.
No luck. Boot time exactly the same (within about a 1 second margin as I was just timing it with my phone, viusually watching the status leds).

I was able to cut 10 seconds by deleting a bunch of old error logs that were part of the backup.
I’m now at 100 seconds From “press physical start button to application running”.
That’s actually not real bad.
(In earlier post I said 80secs which was timed less accurate by simply counting)
Of course already 5 seconds get lost by “waitusb” and I feel the bios also eats 5 seconds as it takes 10 seconds to see any disk activity.
I think time is limited by disk access, I see the disk-led flashing constantly. It’s an old slow 500MHz processor, the 10 seconds I gained by deleting approximately 30MB of error logs, so yeahhhh… it’s slow. And that’s ok as it is also very low power.
As said earlier, the system is currently headless without console. I can only check the status leds.

It’s ok.
I consider the topic closed.
I learned a lot.
Application is running and controlling my home.
And it’s fun.
Thanks to all who put their thoughts in this.

[Rich]

Hi Stefann

... I was able to cut 10 seconds by deleting a bunch of old error logs that were part of the backup. ...

Backup/restore functions can really eat up time if many and/or
large files are involved.

Try to be selective by backing up individual files.

Avoid backing up entire directories when possible.

If your home directory has a lot of files, consider
switching to a persistent home. See the Wiki for
persistence.

Or you can try something like this:

Code: [Select]

mkdir "$(readlink /etc/sysconfig/tcedir)/../MyData"
ln -s "$(readlink /etc/sysconfig/tcedir)/../MyData" MyData
The first command finds the device your tce directory is located
on and creates a directory called MyData on it.

The second command creates a  link  (shortcut) in your home
directory that points to the MyData directory you created.

You can use that link like a regular directory. Copy or move
files to it, create sub-directories in it, etc. This would be a
place to keep some files you might otherwise place in your
home directory. When you run a backup, only the link gets
backed up, not the contents it's pointing to. That is fine
because the contents it's pointing to are on persitent
storage.

[Stefann]

Thanks,
And ehh… I already had that 
My home directory is really empty, merely administrative  stuff and profiles. The backup is now under 5 seconds.
I created 2 partitions on my disk, 1st with tc, and the 2nd partition with all my user data. Indeed in the bootlocal I create  a directory /work and mount that to the 2nd partition.
Actually I “mount” it to /work, not “link”. I’m always a bit confused about the pros and cons but i normally approach the machine as a network drive by the samba server. That works better with a mounted location than a link.

I did consciously chose NOT to use persistent opt and home.
By doing this I can always copy the tc and backup file to the 2nd partition and than do a complete whipe and reinstall of tinycore on the 1st partition.
In other words… I keep tinycore pristine on partition 1. Without snippets like “persistent home and opt” on partition 2.
All deltas from stock are in the backup file.
While partition 2 is fully for user data.

I thought long about this choice. On dams small Linux where I come from I DID have persistent home and opt but after few years both got polluted. Once they are polluted it becomes very difficult to separate OS content from self inserted content.

Note….
The tricky thing is that one “might” think about the home directory as a directory with user data only but that is completely  true. It is loaded with hidden files that configure the desktop and the login behaviors. So…
I consider the home directory as a directory that sure carries the preferences of the user but belongs to the OS, not to the “user space”.
(And by taking that approach the backup remains really really small)

Note…
As I said earlier. I started this in 2008 on damn small Linux. That was also my 1st experience with Linux which means that by the time all was working the thing was pretty cluttered.
Now I’m making a fresh start with tiny core I make sure I really really really have the concepts right.
(Which was the reason for the original question… wanting to understand to avoid going in a direction I would later regret).

And… but… having all said that….
I sure appreciate the inputs.
Some I already did. Others hint me.
And (as you I hope recognize) I’m sure interested to understand the concepts.

[Rich]

Hi CentralWare

... I'm guessing all of these scripts assume $USER is a sudoer?

Yes.

...it was quicker than dredging through TCE
Any insight from experience? (When you have a minute!)

There is a lot going on there.

#1 is actually:

Code: [Select]

# If load local or install then also check if already installed.
if [ "$INSTALL" ] && [ ! "$BOOTING" ]; then
        if [ -f "$TCEINSTALLED/$APPNAME" ]; then
                echo "$APPNAME is already installed!"
                continue
        fi

Code: [Select]

# 3. I'm seeing LINKS and FILES in tce.installed...

The links are post install scripts. The rest are empty files.
All entries in this directory match extension names without .tcz.
These are used by other script(s) to test if an extension is already installed.

Code: [Select]

# 4. Use find -exec to mkdir all DIRECTORIES to /fake/*
sudo find * -type d -exec sudo mkdir -p /fake/{} \;                                                     # <-- SUDO

# 5. Use find to LINK all FILES into /fake/*
sudo find * -type f -exec sudo ln -s /tmp/tcloop2/${EXT}/{} /fake/{} \;                                 # <-- SUDO

find  is only used to determine if an extension is empty (meta extension)
or contains kernel modules.
Files get linked in like this:

Code: [Select]

yes "$FORCE" | sudo /bin/cp -ais /tmp/tcloop/"$APPNAME"/* / 2>/dev/null

Code: [Select]

# 7. I don't know if files are SCANNED (libraries) or if ldconfig is called "just in case"

If $BOOTING ldconfig is called after all of onboot.lst is installed.
The rest of the time, it is called after each extensions is installed.

Code: [Select]

# 8. I'm also guessing for hardware/firmware/modules we need to call udevadm - unsure of our trigger
# find * -type d -name modules?  Or -name modules* needed for x64/pi/etc?

Right after creating links from extension into file system:

Code: [Select]

[ -n "`find /tmp/tcloop/$APPNAME -type d -name modules`" ] && MODULES=TRUEThen update_system gets called which includes this:

Code: [Select]

                if [ "$MODULES" ]; then
                        sudo /sbin/depmod -a 2>/dev/null
                        sudo /sbin/udevadm trigger
copy2fs.list, copy2fs.flg, and $BOOTING will also affect what happens.

Code: [Select]

# 6. I'm guessing HERE is a good place to launch /usr/local/tce.installed/$EXT
[ -e /usr/local/tce.installed/$EXT ] && sudo sh /usr/local/tce.installed/$EXT                           # <-- SUDO

It depends:

Code: [Select]

        if [ -x "$TCEINSTALLED"/$2 ]; then
                if [ "$BOOTING" ] ; then
                        echo "$TCEINSTALLED"/$2 >> /tmp/setup.lst
                else
                        sudo "$TCEINSTALLED"/$2
                fi
        else
                touch "$TCEINSTALLED"/$2
        fi

[curaga]

If you add the label or uuid option to waitusb, it will proceed as soon as the device shows up, with the 5 secs becoming the max value.

[Stefann]

If you add the label or uuid option to waitusb, it will proceed as soon as the device shows up, with the 5 secs becoming the max value.

Ahh.. thanks...
No: I did not know that
Sadly: it will not help as the uuid is already there (I created the install by using the tiny-core "create install gui"

I guess in that case the bios takes more time. I start seeing disk-activity after about 10secs

Note, its not really bad, the 100sec total start time includes starting the desktop and VNC and samba and ssh.
Like everything until my application starts.
On a 500MHz low power 1watt cpu from 2008. that's not bad.

[Juanito]

Sadly: it will not help as the uuid is already there (I created the install by using the tiny-core "create install gui"

You use the uuid already created - search these forums or see the wiki to see how to do it.

[gadget42]

did a forum search for "waitusb=uuid" via the forum search page
(https://forum.tinycorelinux.net/index.php?action=search)
(change the search order to "most recent topics first")

and there are currently four results

[Stefann]

did a forum search for "waitusb=uuid" via the forum search page
(https://forum.tinycorelinux.net/index.php?action=search)
(change the search order to "most recent topics first")

and there are currently four results

Sadly: it will not help as the uuid is already there (I created the install by using the tiny-core "create install gui"

You use the uuid already created - search these forums or see the wiki to see how to do it.

That's what I mean, I'm already doing that:

Code: [Select]

DEFAULT core

LABEL core
KERNEL /tce/boot/vmlinuz
INITRD /tce/boot/core.gz
APPEND quiet  host=huis syslog cron tz=CET-1CEST,M3.5.0,M10.5.0/3 waitusb=5:UUID="a6f362cf-6c77-4fb7-bb05-76724d3b8113" tce=UUID="a6f362cf-6c77-4fb7-bb05-76724d3b8113"