copy2fs.lst ignored by tc-config on boot, only works with tce-load

[zharr]

I'm using piCore and trying to have all auto-loaded extensions copied to RAM (to unmount the SD card during operation) but normally mount optional extensions loaded later.
However this only seems to work when using tce-load directly, on boot it always seems to mount the extensions (though the copy2fs.flg works).
Seeing how the /etc/init.d/tc-config script has it's own implementation handling copy2fs.lst (different from tce-load), the issue might lie there.
That's why I'm posting here - I don't think it's necessarily a piCore packaging issue.
Sadly, I have not found a good way to edit that script to iterate on (or even run it safely).
Any help would be appreciated!

[Rich]

Hi zharr

... Seeing how the /etc/init.d/tc-config script has it's own implementation handling copy2fs.lst (different from tce-load), the issue might lie there. ...

I don't think so. The section of  /etc/init.d/tc-config  you are referring to looks like this:

Code: [Select]

if [ -n "$PRETCE" ]; then
        PKG=optional ; LOADER="tce-load -i" ; LIST=onboot.lst
        RAM=/tmp/tce
        CPTOFS=""
        if [ "$PRETCE" == "RAM" ] ; then
                SRC=$RAM
                [ -f $SRC/copy2fs.flg ] && CPTOFS=1
        else
                PPATH=${PRETCE%%/*}
                mkdir -p /mnt/$PPATH
                mount /dev/$PPATH /mnt/$PPATH
                SRC=/mnt/$PRETCE
                [ -x $SRC/onpre.sh ] && cp $SRC/onpre.sh /tmp
                [ -f $SRC/copy2fs.flg ] && CPTOFS=1
                if [ $CPTOFS ] ; then
                        cp $SRC/$PKG/* $RAM/$PKG/
                        cp $SRC/$LIST $RAM/
                        SRC=$RAM
                        umount /mnt/$PPATH 2>/dev/null
                fi
        fi
        MNT=""
        while read NAME OPT
        do
                [ $CPTOFS ] || grep -q $NAME $SRC/copy2fs.lst 2>/dev/null && FL="-c" || FL=''
                [ $FL ] || MNT=1
                su "$USER" -c "/usr/bin/$LOADER $SRC/$PKG/$NAME"
                [ "$SRC" == "$RAM" ] && [ "$FL" == "-c" ] && rm $RAM/$PKG/$NAME
        done < $SRC/$LIST
        [ "$SRC" != "$RAM" ] && [ $MNT ] || umount /mnt/$PPATH 2>/dev/null
        rm $RAM/copy2fs.flg $RAM/copy2fs.lst 2>/dev/null
        [ -x /tmp/onpre.sh ] && /tmp/onpre.sh
fi
Unless you are using the  pretce=  boot code, that section will not execute.

[zharr]

Right, figured it must execute since that's the only place I found where it reads onboot.lst in the boot process (using corebook as reference).
Color me surprised when I found the actual TCE loading is done by a python script! /usr/bin/tce-bootload to be exact, getting passed the onboot.lst contents.
And sure enough, it doesn't check copy2fs.lst at all:

Code: [Select]

for p in tczout:

        pid = os.fork()

        if pid == 0:
                #This is the child process.
                if showapps:
                        print(p[:-4] + ' ', end="")
                        print('\033[1;33m', end="")

                os.mkdir(root + '/tmp/tcloop/' + p[:-4])
                system_command('sudo /usr/bin/mount -t squashfs -o loop ' + sys.argv[1] + '/optional/' + p + ' ' + root + '/tmp/tcloop/' + p[:-4], p[:-4])
                #Child process exits
                sys.exit(0)

I assume this is something that can only be fixed by remastering, or using "base" bootcode and manually loading onboot TCEs (will try that first).

[Rich]

Hi zharr

... or using "base" bootcode and manually loading onboot TCEs (will try that first).

I think this might work:

Use the  base  bootcode.

Add this to /opt/bootsync.sh:

Code: [Select]

/usr/bin/tce-setup "booting" > /dev/null 2>&1

[zharr]

Executed that manually, "Skipping regular Extension loading as requested on the command line"
Still investigating, but it seems tc-restore.sh won't work at all with base bootcode, nor will tce-load (because /etc/sysconfig/tcedir just points to /tmp/tce).
So I removed base, and instead used an empty onboot.lst, loading them manually in bootsync with:

Code: [Select]

# Manually load intial tces with normal tce-load
TCEDIR=$(/usr/bin/readlink /etc/sysconfig/tcedir)
sudo -u tc /usr/bin/tce-load -cli $(cat $TCEDIR/onstartup.lst)(note I skipped copy2fs.lst entirely here with the -c flag, but it does work just fine with the copy2fs.lst instead)

Now I finally have an embedded system that's RAM-only for low SD card wear, and can mount more dev packages on-demand when remoting in.
Thanks for your help!

[Paul_123]

It’s only taken about 6 years for someone to mention this, from when the bootload script was introduced.  There are a few changes needed to the bootload script  I’ll look at them soon.

But for piCore16, what you did should work for you.

[Paul_123]

Note: tce-bootload only had copy2fs.flg implemented.   I'll look at copy2fs.lst, while, I'm changing the force overwrite of currently present files.  ref: https://forum.tinycorelinux.net/index.php?topic=28174.msg182872#msg182872

[zharr]

Thanks - thats a bit over my head. Don't even know the reasoning for tce-bootload to exist when it should do something very similar to tce-load, but haven't looked too closely at what it does, or if tce-load is also parallelised as tce-bootload is. But they could share a lot of the code to be consistent (if tce-bootload was a shell script just like tce-load). Just, one is executed by root on boot, while the other uses the fixed configured TCE path and is explicitly non-root.

[Rich]

Hi zharr

... Don't even know the reasoning for tce-bootload to exist when it should do something very similar to tce-load, ...

I think it was created to speed up extension loading when piCore boots, which
is when the most extension loading typically takes place.

Also, I think a lot of pi based computers use SD cards for storage, which are
possibly slower than modern spinning disks and SSD disks found in laptop
or desktop computers.

[Paul_123]

tce-load is not parallelized.  And I would not even want to try to go there.   There is a lot of awk in tce-load (To step through the dependency chain)

piCore had micropython since it's beginning days, and tce-bootload was created to take advantage of python.  When Raspi released multicore devices, parallelization was added to tce-bootload.  If you were loading 20-30 extensions, it cut the loading time in half.

Anyway, I've updated tce-bootload to handle copy2fs.lst.  I'll merge it into 17.0 when I release the next images.

[zharr]

tce-load is not parallelized.

Also, I think a lot of pi based computers use SD cards for storage, which are possibly slower than modern spinning disks and SSD disks found in laptop or desktop computers.

Right, since I'm copying from RAM anyway I don't think I'd see a slowdown from using tce-load instead (should test though). Especially since I AM copying from a slow SD card, so it will be the bottleneck.

Anyway, I've updated tce-bootload to handle copy2fs.lst.  I'll merge it into 17.0 when I release the next images.

Thanks! Likely won't be able to upgrade due to reasons outside of piCore/tinyCore, but glad it's back as a feature for those edge cases.

[Rich]

Hi zharr

...

Also, I think a lot of pi based computers use SD cards for storage, which are possibly slower than modern spinning disks and SSD disks found in laptop or desktop computers.

Right, since I'm copying from RAM anyway I don't think I'd see a slowdown from using tce-load ...

Using copy2fs.lst or copy2fs.flg both copy extensions to RAM, which means
they are already loaded, so there is nothing for tce-load to do.