Tiny Core Linux

General TC => Programming & Scripting - Unofficial => Topic started by: CNK on February 24, 2023, 06:30:19 PM

Title: Tinkering with rebuildfstab
Post by: CNK on February 24, 2023, 06:30:19 PM
Following the recent rejected rewrite (http://forum.tinycorelinux.net/index.php/topic,26114.0.html) of the device identification in Tiny Core's rebuildfstab script, intended for quicker execution time, I thought I'd play around with the suggestion in the GitHub discussion (https://github.com/tinycorelinux/Core-scripts/pull/39) that:
Quote
Perhaps a large part of the time taken is by the "find" command, where adding "-maxdepth 2" may give some speedup.

Testing in Tiny Core Pure64 13, I found that this indeed sped it up to a small degree. But this suggestion also meant that there are never any desired "dev" files deeper than one level below the /sys/block directory, in which case the find command can simply be replaced by shell file name globbing. Also the DEVNAME and DEVMAJOR variables could be set using shell string manipulations instead of calling external programs.

Comparison of old/new moved to attachment (rebuildfstab_changes.txt) to avoid forum error.

While not near the stats of the earlier proposed modification, the speed-up still looks significant, and the changes here are much more minor.

(running twice to reduce cache effects, which are less significant on this particular PC than they are on others)
Code: [Select]
# echo modified: ;time rebuildfstab-mod; echo original: ; time rebuildfstab; echo original: ; time rebuildfstab;echo modified: ;time rebuildfstab-mod;
modified:
real    0m 0.39s
user    0m 0.08s
sys     0m 0.18s
original:
real    0m 0.63s
user    0m 0.20s
sys     0m 0.40s
original:
real    0m 0.60s
user    0m 0.21s
sys     0m 0.38s
modified:
real    0m 0.37s
user    0m 0.07s
sys     0m 0.17s

The only potential behaviour change is that the order of the entries in /etc/fstab will be different if an unpartitioned block device (eg. /dev/sda) is added there. I'm guessing that the exact order of auto-generated fstab entries doesn't matter much to anyone anyway.

Attached is the full file, where I also added ext4 at the mount options bit. I don't really think the execution time of rebuildfstab is much of an issue, it was just a fun thing to play around with, so whether TC wants to adopt this change or not, I don't really care. But if so, then go right ahead.
Title: Re: Tinkering with rebuildfstab
Post by: curaga on February 25, 2023, 12:36:55 AM
Thanks for the changes. There are some issues:
- the changed DEVMAJOR line uses wrong var (i)
- it calls fstype potentially many more times
- it calls grep many times, which is what an earlier patch specifically changed to the combined awk
- if there is no glob match, the pattern is passed as-is (I guess the -b test was added for this)

It may not be simple to apply just the globbing; when I tried, it was slower.
Title: Re: Tinkering with rebuildfstab
Post by: curaga on February 25, 2023, 12:55:51 AM
It does seem the find line is the main source of slowness, but also one that is not easily rewritten in shell while keeping minimal changes (to minimize the chance of regressions). If anyone is interested in tackling this, replacing just the find command with a small C program could be a nice exercise (glob(), strstr()).
Title: Re: Tinkering with rebuildfstab
Post by: hiro on February 25, 2023, 01:33:01 AM
did you measure how much time blkid takes when you run it once vs. many times?
i think if find is slow then it might be bugged. it shouldn't be supposed to do very much! :O
Title: Re: Tinkering with rebuildfstab
Post by: hiro on February 25, 2023, 01:37:36 AM
on my machine, running blkid for the first time take 6x as much time as find.
and after everything is cached blkid takes the same amount of time as find. and it seems like there's no cache-related variation with find.
grep is at least a magnitude faster.

i'm happy to help obsess some more about this at some point, as it's such a small contained neat thing. ;)
Title: Re: Tinkering with rebuildfstab
Post by: CNK on February 25, 2023, 03:17:23 AM
Thanks for the changes. There are some issues:
- the changed DEVMAJOR line uses wrong var (i)
Oops, yes I messed that one up completely, it should be "DEVMAJOR=${DEVMAJOR%%:*}" there of course.

Quote
- it calls fstype potentially many more times

How do you come to this conclusion (besides due to the DEVMAJOR error)?

For me this script shows that the same number of file system device lines are run through the parts of the loop where the fstype and grep calls are located. If the DEVNAME and DEVMAJOR variables are also the same in each version, then I don't see where there can be a difference in the behaviour.

Code: [Select]
#!/bin/busybox ash
rm -f /tmp/blkdev.txt

for i in /sys/block/*/dev /sys/block/*/*/dev; do
  case "$i" in
    *loop*|*ram*)
      continue
      ;;
  esac

  DEVNAME=${i%*/dev}
  DEVNAME=${DEVNAME##*/}
  [ -b "/dev/$DEVNAME" ] || continue

  echo "$i" >> /tmp/blkdev.txt
done

echo rebuildfstab-mod:
wc -l /tmp/blkdev.txt
rm /tmp/blkdev.txt

for i in `find /sys/block/*/ -name dev`; do
  case "$i" in
    *loop*|*ram*)
      continue
      ;;
  esac

  echo "$i" >> /tmp/blkdev.txt
done

echo rebuildfstab:
wc -l /tmp/blkdev.txt
rm /tmp/blkdev.txt

Quote
- it calls grep many times, which is what an earlier patch specifically changed to the combined awk

Oh right, I forgot to check the current version (https://github.com/tinycorelinux/Core-scripts/blob/master/usr/sbin/rebuildfstab) on Github and just based this on the version in TC 13. Now that I've seen that, I see what you're talking about (it took a while) - you're using "read" to advance through the list of block devices now, and feeding it from find and awk.

Quote
- if there is no glob match, the pattern is passed as-is (I guess the -b test was added for this)

Yes the -b test catches that.

Quote
It may not be simple to apply just the globbing; when I tried, it was slower.

OK. Testing against the 'new' rebuildfstab on GitHub the execution time is much closer with my version Vs 'rebuildfstab-new', and that's on a test machine with only four mountable file systems so not many calls to Grep which affect the older design that I based mine on. I'll believe you that the new layout is no faster with globbing (fed to awk via "echo", presumably), I've spent enough time on this already now.

Code: [Select]
# echo original:; time rebuildfstab; echo github: ; time ./rebuildfstab-new;echo modified: ;time rebuildfstab-mod;echo modified: ;time rebuild
fstab-mod; echo github: ; time ./rebuildfstab-new;echo original:; time rebuildfstab;
original:
real    0m 0.82s
user    0m 0.22s
sys     0m 0.50s
github:
real    0m 0.37s
user    0m 0.09s
sys     0m 0.26s
modified:
real    0m 0.33s
user    0m 0.12s
sys     0m 0.16s
modified:
real    0m 0.36s
user    0m 0.12s
sys     0m 0.19s
github:
real    0m 0.37s
user    0m 0.09s
sys     0m 0.26s
original:
real    0m 0.81s
user    0m 0.21s
sys     0m 0.51s
EDIT: Oops again, posted test results before applying the DEVMAJOR fix to my version before.
Title: Re: Tinkering with rebuildfstab
Post by: curaga on February 25, 2023, 08:09:43 AM
did you measure how much time blkid takes when you run it once vs. many times?
i think if find is slow then it might be bugged. it shouldn't be supposed to do very much! :O
It's not that find is buggy, it's how it works (and bb find is slower than GNU find too). It runs a stat call for each file, and in the searched directories there are a lot of files. We're only interested in the file names in this invocation, file types/sizes/times don't matter. So this part is the lowest hanging fruit currently, I believe.

i'm happy to help obsess some more about this at some point, as it's such a small contained neat thing. ;)
Everyone is welcome to go over the TC scripts. rebuildfstab's speed is fast enough for me personally, but if you find it too slow, please do look into it.

Quote
- it calls fstype potentially many more times

How do you come to this conclusion (besides due to the DEVMAJOR error)?
The system version runs blkid once (it's in the END block in awk), and parses the output. Your version runs it once per device. The later calls should be cached and fast, but it's still extra calls.
Title: Re: Tinkering with rebuildfstab
Post by: hiro on February 25, 2023, 09:58:08 AM
The system version runs blkid once (it's in the END block in awk), and parses the output. Your version runs it once per device. The later calls should be cached and fast, but it's still extra calls.

thanks for clearing that up. i actually missed the END and thought it was run multiple times.

if the GNU find is faster than bb find, then maybe it's not worth optimizing for this. as in the long-term it would be more useful if the busybox find was made faster instead (not our problem).

any remaining bottlenecks are now much less obvious to my intuition and i agree we're probably hitting diminishable returns here.

but i'm gonna still offer to try out more solutions, even on a bigger SAS array, if that would be of any help to the remaining benchmark fanatics :P
Title: Re: Tinkering with rebuildfstab
Post by: CNK on February 25, 2023, 02:18:39 PM
Quote
- it calls fstype potentially many more times
How do you come to this conclusion (besides due to the DEVMAJOR error)?
The system version runs blkid once (it's in the END block in awk), and parses the output. Your version runs it once per device. The later calls should be cached and fast, but it's still extra calls.

Yeah I was still comparing with the TC 13 version there. If before posting I'd remembered to check on GitHub to see if rebuildfstab had changed since TC 13 then I would have seen the new layout and never talked about my experiments on the old version in first palce. Sorry to waste time.
Title: Re: Tinkering with rebuildfstab
Post by: hiro on February 25, 2023, 03:44:57 PM
nah, you didn't start this "time-waste".
it's at least entertaining and at best educational to the next reader.
Title: Re: Tinkering with rebuildfstab
Post by: Rich on February 25, 2023, 10:46:01 PM
Hi curaga
... Everyone is welcome to go over the TC scripts. rebuildfstab's speed is fast enough for me personally, but if you find it too slow, please do look into it. ...
I think I may have an approach that doesn't need the  find  command.
The  /dev/block/*  directory seems to contain the information being searched for.
I used a cut down version of  rebuildfstab  containing the  find  loop that isolates
the value of  DEVNAME  and  DEVMAJOR as a baseline. I included an  echo
statement to list the found devices.
Here are the timing results:
Code: [Select]
tc@E310:~/rebuildfstab$ time ./findblockdevs > /dev/null
real    0m 0.71s
user    0m 0.22s
sys     0m 0.53s
tc@E310:~/rebuildfstab$ time ./newfindblockdevs  > /dev/null
real    0m 0.09s
user    0m 0.04s
sys     0m 0.05s
tc@E310:~/rebuildfstab$

It finds the same devices (order does not matter):
Code: [Select]
tc@E310:~/rebuildfstab$ time ./findblockdevs
DEVNAME=sda4    DEVMAJOR=8
DEVNAME=sda2    DEVMAJOR=8
DEVNAME=sda     DEVMAJOR=8
DEVNAME=sda7    DEVMAJOR=8
DEVNAME=sda5    DEVMAJOR=8
DEVNAME=sda3    DEVMAJOR=8
DEVNAME=sda1    DEVMAJOR=8
DEVNAME=sda6    DEVMAJOR=8
DEVNAME=sdb     DEVMAJOR=8
DEVNAME=sdb1    DEVMAJOR=8
DEVNAME=sdc     DEVMAJOR=8
DEVNAME=sdd     DEVMAJOR=8
DEVNAME=sde     DEVMAJOR=8
DEVNAME=sdf     DEVMAJOR=8
DEVNAME=sr0     DEVMAJOR=11
DEVNAME=sr1     DEVMAJOR=11
real    0m 0.70s
user    0m 0.22s
sys     0m 0.52s
tc@E310:~/rebuildfstab$ time ./newfindblockdevs
DEVNAME=sr0     DEVMAJOR=11
DEVNAME=sr1     DEVMAJOR=11
DEVNAME=sda     DEVMAJOR=8
DEVNAME=sda1    DEVMAJOR=8
DEVNAME=sdb     DEVMAJOR=8
DEVNAME=sdb1    DEVMAJOR=8
DEVNAME=sda2    DEVMAJOR=8
DEVNAME=sda3    DEVMAJOR=8
DEVNAME=sdc     DEVMAJOR=8
DEVNAME=sda4    DEVMAJOR=8
DEVNAME=sdd     DEVMAJOR=8
DEVNAME=sda5    DEVMAJOR=8
DEVNAME=sda6    DEVMAJOR=8
DEVNAME=sde     DEVMAJOR=8
DEVNAME=sda7    DEVMAJOR=8
DEVNAME=sdf     DEVMAJOR=8
real    0m 0.09s
user    0m 0.03s
sys     0m 0.06s
tc@E310:~/rebuildfstab$

I've attached the baseline file (findblockdevs) and the version with my changes (newfindblockdevs).
Lines beginning with  ###  were commented out by me.
Title: Re: Tinkering with rebuildfstab
Post by: curaga on February 25, 2023, 11:07:01 PM
I worry about a potential race there - is it certain those devices (/the newest of them) exist when rebuildfstab is called. /sys gets populated early, but I don't remember when exactly udev creates devices wrt the called scripts.

No interest in the C approach :P
Title: Re: Tinkering with rebuildfstab
Post by: Rich on February 25, 2023, 11:23:19 PM
Hi curaga
How do you feel about using  /proc/partitions? I can cut the time
by another factor of 2 parsing that source.
Title: Re: Tinkering with rebuildfstab
Post by: hiro on February 26, 2023, 02:19:30 AM
7x faster, nice :D

maybe we should also replace blkid with a shellscript, then? /s
Title: Re: Tinkering with rebuildfstab
Post by: curaga on February 26, 2023, 07:01:51 AM
/proc should be fine.
Title: Re: Tinkering with rebuildfstab
Post by: Rich on February 26, 2023, 09:18:34 AM
Hi curaga
New timing results:
Code: [Select]
tc@E310:~/rebuildfstab$ time ./newerfindblockdevs > /dev/null
real    0m 0.03s
user    0m 0.01s
sys     0m 0.02s
tc@E310:~/rebuildfstab$

This version also filters out my card readers empty slots (sdc - sdf):
Code: [Select]
tc@E310:~/rebuildfstab$ time ./newerfindblockdevs
DEVNAME=sda     DEVMAJOR=8
DEVNAME=sda1    DEVMAJOR=8
DEVNAME=sda2    DEVMAJOR=8
DEVNAME=sda3    DEVMAJOR=8
DEVNAME=sda4    DEVMAJOR=8
DEVNAME=sda5    DEVMAJOR=8
DEVNAME=sda6    DEVMAJOR=8
DEVNAME=sda7    DEVMAJOR=8
DEVNAME=sdb     DEVMAJOR=8
DEVNAME=sdb1    DEVMAJOR=8
DEVNAME=sr0     DEVMAJOR=11
DEVNAME=sr1     DEVMAJOR=11
DEVNAME=sdg     DEVMAJOR=8
real    0m 0.03s
user    0m 0.01s
sys     0m 0.02s
tc@E310:~/rebuildfstab$

I've attached the script that contains the modified section of rebuildfstab
I'm working on speeding up.
Title: Re: Tinkering with rebuildfstab
Post by: CNK on February 26, 2023, 01:57:17 PM
Great spotting Rich! That's a much neater source for the info, as well as a quicker one.
Title: Re: Tinkering with rebuildfstab
Post by: Rich on February 26, 2023, 02:25:06 PM
Hi CNK
Thanks, accessing information that the kernel already exposes is much faster.
I'm currently evaluating a replacement for the  /usr/sbin/fstype  script.

Title: Re: Tinkering with rebuildfstab
Post by: Rich on February 26, 2023, 09:20:45 PM
Hi curaga
I just looked at the github version and realized a lot of changes have
been made since the TC10 version I'm working with. I copied that
version and when I tried to run it on my TC10 system I got this error:
Code: [Select]
tc@E310:~/rebuildfstab$ time sudo ./rebuildfstab.git
./rebuildfstab.git: line 71: syntax error: unexpected redirection
Command exited with non-zero status 2
real    0m 0.01s
user    0m 0.00s
sys     0m 0.01s
tc@E310:~/rebuildfstab$

This is the section in question:
Code: [Select]
     67   if [ "$MOUNTPOINT" != "none" ]; then
     68     mkdir -p "/mnt/$DEVNAME" 2>/dev/null >/dev/null
     69   fi
     70   printf "%-15s %-15s %-8s %-20s %-s\n" "$DEVROOT/$DEVNAME" "$MOUNTPOINT" "$FSTYPE" "$OPTIONS" "0 0 $ADDEDBY" >>"$TMP"
     71 done < <(find /sys/block/*/ -maxdepth 2 -name dev ! -regex '.*\(/loop\|/ram\|/zram\|\s\).*' | awk -v devroot="$DEVROOT" -v tmptab="$TMP" '
     72     BEGIN {
     73         while (getline < tmptab > 0) remain[$1]=1
     74         FS="/|!"
     75     }

Any idea why it's not backward compatible?
Title: Re: Tinkering with rebuildfstab
Post by: curaga on February 26, 2023, 10:55:04 PM
I think that's just missing functionality in the older ash.
Title: Re: Tinkering with rebuildfstab
Post by: Rich on February 27, 2023, 07:38:45 AM
Hi curaga
Thanks, I think you're right. I changed the shebang to  bash  and it runs:
Code: [Select]
tc@E310:~/rebuildfstab$ time sudo ./rebuildfstab.git
real    0m 0.92s
user    0m 0.14s
sys     0m 0.32s
tc@E310:~/rebuildfstab$ time sudo ./rebuildfstab.git
real    0m 0.69s
user    0m 0.11s
sys     0m 0.36s
tc@E310:~/rebuildfstab$
Title: Re: Tinkering with rebuildfstab
Post by: Rich on March 05, 2023, 09:41:22 AM
Hi CNK
I changed how information is gathered but kept what is done with
it basically the same.

For each test, I cleared the cache, then ran the script 3 times in succession.

Timing on Dell E310 desktop TC10 x86
Original rebuildfstab from TC10:
Code: [Select]
tc@E310:~/rebuildfstab$ time sudo rebuildfstab
real    0m 1.98s
user    0m 0.37s
sys     0m 0.83s
tc@E310:~/rebuildfstab$ time sudo rebuildfstab
real    0m 1.42s
user    0m 0.36s
sys     0m 0.83s
tc@E310:~/rebuildfstab$ time sudo rebuildfstab
real    0m 1.41s
user    0m 0.40s
sys     0m 0.76s

rebuildfstab from git, shebang changed to bash so it would run under TC10:
Code: [Select]
tc@E310:~/rebuildfstab$ time sudo ./rebuildfstab.git
real    0m 1.05s
user    0m 0.17s
sys     0m 0.54s
tc@E310:~/rebuildfstab$ time sudo ./rebuildfstab.git
real    0m 0.78s
user    0m 0.15s
sys     0m 0.39s
tc@E310:~/rebuildfstab$ time sudo ./rebuildfstab.git
real    0m 0.71s
user    0m 0.12s
sys     0m 0.39s

My version of rebuildfstab:
Code: [Select]
tc@E310:~/rebuildfstab$ time sudo ./rebuildfstab.rewrite
real    0m 0.72s
user    0m 0.06s
sys     0m 0.12s
tc@E310:~/rebuildfstab$ time sudo ./rebuildfstab.rewrite
real    0m 0.16s
user    0m 0.09s
sys     0m 0.06s
tc@E310:~/rebuildfstab$ time sudo ./rebuildfstab.rewrite
real    0m 0.23s
user    0m 0.07s
sys     0m 0.11s

I've done testing on 2 desktops (TC9 x86 and TC10 x86) and 1 laptop (TC14.0alpha1 x86_64)
and can no longer find any issues.

Timing on the laptop
rebuildfstab from TC14.0alpha1:
Code: [Select]
tc@box:~$ time sudo ./rebuildfstab
real    0m 0.36s
user    0m 0.06s
sys     0m 0.14s
tc@box:~$ time sudo ./rebuildfstab
real    0m 0.27s
user    0m 0.05s
sys     0m 0.11s
tc@box:~$ time sudo ./rebuildfstab
real    0m 0.31s
user    0m 0.05s
sys     0m 0.12s

My version of rebuildfstab:
Code: [Select]
tc@box:~$ time sudo ./rebuildfstab.rewrite
real    0m 0.14s
user    0m 0.04s
sys     0m 0.06s
tc@box:~$ time sudo ./rebuildfstab.rewrite
real    0m 0.13s
user    0m 0.04s
sys     0m 0.05s
tc@box:~$ time sudo ./rebuildfstab.rewrite
real    0m 0.09s
user    0m 0.04s
sys     0m 0.04s

I've attached a copy for you and any other interested parties to
test, examine the code, and provide feedback.

For those that feel code is self commenting, you can remove most of them like this:
Code: [Select]
tc@E310:~/rebuildfstab$ grep -v "### " rebuildfstab.rewrite > rebuildfstab.stripped
tc@E310:~/rebuildfstab$ ls -l rebuildfstab.git rebuildfstab.rewrite rebuildfstab.stripped
-rwxr-xr-x 1 tc staff 2595 Feb 27 08:56 rebuildfstab.git
-rwxr-xr-x 1 tc staff 4145 Mar  5 10:13 rebuildfstab.rewrite
-rw-r--r-- 1 tc staff 2735 Mar  5 12:33 rebuildfstab.stripped
tc@E310:~/rebuildfstab$
Title: Re: Tinkering with rebuildfstab
Post by: CNK on March 05, 2023, 10:28:26 PM
Nice work, it's the fastest yet on the first PC I've tested it on (TC 13, x86_64). It does query blkid multiple times, which as Curaga pointed out earlier is avoided in the current Git-repo code, although the "blkid -s TYPE" command that the current script uses actually takes longer to complete on its own than your whole script does in my tests. Maybe that's because that PC only has six hardware block devices present though.

Also I notice that you didn't add ext4 to the part where "relatime" gets added to the mount options for ext file systems (line 135). I see that the script in the Git repo doesn't do that at all though, and I see that the mount man page says "relatime" is now the default, so that line can probably be removed from your script too.

I'll test it out in TC 13 on another x86_64 PC later. Maybe sometime I'll try it on my 100MHz i486 laptop too and see how many hours it takes. :)
Title: Re: Tinkering with rebuildfstab
Post by: curaga on March 05, 2023, 11:31:22 PM
This version also misses custom mounts, and I feel the queue logic is worse than the current checks. If two more scripts happen to be spawned while one is running, the old version would only rerun once, this would rerun them all.

Regarding comments, some additions are definitely welcome, like the 98 block major etc. Though some new ones are overboard IMHO.

I'd prefer a minimal patch just replacing the find line and editing the awk script correspondingly (and a few comments). Otherwise adding custom mounts and running blkid just once to your version may need multiple grep calls, etc.
Title: Re: Tinkering with rebuildfstab
Post by: Rich on March 06, 2023, 08:34:47 AM
Hi curaga
This version also misses custom mounts, ...
Could you provide a detailed example of what gets missed please.

Quote
... and I feel the queue logic is worse than the current checks. If two more scripts happen to be spawned while one is running, the old version would only rerun once, this would rerun them all. ...
Exactly. A program calling rebuildfstab expects to find an updated fstab.
Consider 2 programs calling at the same time:
  | ProgramA | rebuildfstab scans devices, updates fstab, launches rebuildfstab again | ProgramA reads fstab |
  | ProgramB | rebuildfstab touches rescan file and exits | ProgramB reads fstab |

ProgramA is blocked until rebuildfstab updates fstab.
ProgramB is allowed to continue, unaware that rebuildfstab did not run for it.

While I don't expect this type of situation to happen often I felt it was a
better way of handling it.

Quote
... Otherwise adding custom mounts and running blkid just once ...
I'll take a closer look at that.
Title: Re: Tinkering with rebuildfstab
Post by: curaga on March 06, 2023, 10:28:02 AM
Custom mounts: say you have a line mounting /dev/sda1 to /files.

The primary caller of rebuildfstab is udev. The other known callers are at boot time and mnttool, and mnttool does not expect that behavior. So I don't think such a guarantee is needed.
Title: Re: Tinkering with rebuildfstab
Post by: Rich on March 06, 2023, 05:31:31 PM
Hi curaga
Custom mounts: say you have a line mounting /dev/sda1 to /files. ...
Do you mean something like this might already be in  /etc/fstab:
Code: [Select]
/dev/sda1       /files          ext4     noauto,users,exec    0 0and in this example I should skip adding:
Code: [Select]
/dev/sda1       /mnt/sda1       ext4     noauto,users,exec    0 0 # Added by TC
Title: Re: Tinkering with rebuildfstab
Post by: CNK on March 06, 2023, 05:55:39 PM
The old (TC13) script checked the "$TMP" list of manually-generated fstab entries for any entry already matching the device that it was about to add. It called grep once per round to do this.

One alternative approach besides the Awk solution in the new script (which I don't understand Awk well enough to touch), would be to use "grep -v" to exclude from the lines fed to the "while read" loop all those matching the first field in existing fstab entries stored in $TMP.

For example if you have "grep $FILTERS -v /proc/partitions | while read -r DEVMAJOR DEVMINOR BLOCKS DEVNAME". Start with FILTERS="-e 'loop[0-9]*$' -e 'ram[0-9]*$' -e '^$' -e ' name$'", then write a loop that adds an "-e" onto $FILTERS for each first field of "$TMP". EDIT: "grep $FILTERS -v /proc/partitions" doesn't work, I haven't got time to figure out exactly how wrong I am here...

I didn't really have time to write this post clearly, sorry if it's just confusing.
Title: Re: Tinkering with rebuildfstab
Post by: CNK on March 06, 2023, 06:34:14 PM
For example if you have "grep $FILTERS -v /proc/partitions | while read -r DEVMAJOR DEVMINOR BLOCKS DEVNAME". Start with FILTERS="-e 'loop[0-9]*$' -e 'ram[0-9]*$' -e '^$' -e ' name$'", then write a loop that adds an "-e" onto $FILTERS for each first field of "$TMP". EDIT: "grep $FILTERS -v /proc/partitions" doesn't work, I haven't got time to figure out exactly how wrong I am here...

Use backslashes (\) to escape the spaces between arguments, instead of double quotes ("), when setting the FILTERS variable and it works.
Title: Re: Tinkering with rebuildfstab
Post by: curaga on March 07, 2023, 12:12:12 AM
Rich, precisely.
Title: Re: Tinkering with rebuildfstab
Post by: Rich on March 10, 2023, 10:37:03 PM
Hi curaga
I've made the following changes:
1. The arbitration logic still queues up multiple scripts, but after the first script
   exits, any scripts waiting in the queue will exit instead of processing fstab again.
2. I removed the test for  98) # User-mode virtual block devices.  If someone creates
   one of these devices and it contains a file system, the section labeled
    ### Loop through block devices  will pick it up.
3. Added  ext4  to  ext2|ext3) OPTIONS="${OPTIONS},relatime" ;;  test.
4. blkid now only gets called once with a filtered device list to process.

For consistency, I cleared the cache and ran the script 3 times with a fixed delay
for each benchmark.

Timing on Dell E310 desktop TC10 x86
rebuildfstab from git, shebang changed to bash so it would run under TC10:
Code: [Select]
tc@E310:~/rebuildfstab$ time sudo ./rebuildfstab.git; sleep 1; echo; time sudo ./rebuildfstab.git; sleep 1; echo; time sudo ./rebuildfstab.git
real    0m 1.02s
user    0m 0.18s
sys     0m 0.53s

real    0m 0.77s
user    0m 0.16s
sys     0m 0.37s

real    0m 0.83s
user    0m 0.15s
sys     0m 0.39s

My version of rebuildfstab:
Code: [Select]
tc@E310:~/rebuildfstab$ time sudo ./rebuildfstab.devlist; sleep 1; echo; time sudo ./rebuildfstab.devlist; sleep 1; echo; time sudo ./rebuildfstab.devlist
real    0m 0.48s
user    0m 0.04s
sys     0m 0.05s

real    0m 0.08s
user    0m 0.03s
sys     0m 0.04s

real    0m 0.13s
user    0m 0.03s
sys     0m 0.05s

The execution times have improved over my previous version:
... My version of rebuildfstab:
Code: [Select]
tc@E310:~/rebuildfstab$ time sudo ./rebuildfstab.rewrite
real    0m 0.72s
user    0m 0.06s
sys     0m 0.12s
tc@E310:~/rebuildfstab$ time sudo ./rebuildfstab.rewrite
real    0m 0.16s
user    0m 0.09s
sys     0m 0.06s
tc@E310:~/rebuildfstab$ time sudo ./rebuildfstab.rewrite
real    0m 0.23s
user    0m 0.07s
sys     0m 0.11s
...

A copy of the new version of the script is attached if anyone
wants to do any testing.
Title: Re: Tinkering with rebuildfstab
Post by: curaga on March 10, 2023, 11:26:14 PM
Mostly looks good, but it was a bit buggy when I tested. zram0 and sr0 got added twice for me when testing on tc 13.x, and it added loops. This happened when the VM had no drives. I also still dislike the queue logic.

I edited it a bit, and made some other changes:
- comments should not be aligned like devices
- auto filesystems should still get a dir created
- the relatime line is unneeded nowadays

Please try the git version and see if it looks ok.
Title: Re: Tinkering with rebuildfstab
Post by: CNK on March 10, 2023, 11:45:41 PM
One thing that I'd add is that if the rebuildfstab process crashes, or gets killed due to running out of RAM, the $TMP file will probably be left over. Then sometime later rebuildfstab could run again with the same process ID, and any fstab entries added before it died the previous time would be duplicated or incorrect in the final /etc/fstab.

So I suggest adding a line to delete $TMP at the start if it already exists. EDIT: Or use mktemp.

Otherwise it seems to run fine for me in some brief tests.
Title: Re: Tinkering with rebuildfstab
Post by: Rich on March 11, 2023, 09:06:47 AM
Hi curaga
...  This happened when the VM had no drives. ...
Yes, I forgot if you pass  blkid  an empty  $DevList  it spits out every device it can find.
Thanks for piping  blkid  output into  read.  Don't know how I missed that. :-[

Quote
... comments should not be aligned like devices ...
I saw that the comment was present, but my brain didn't register that it was indented.
Now I understand why you added the  "#"*) # Comments  clause.

Quote
... auto filesystems should still get a dir created ...
Yup. casualty of refactoring the script so  blkid  would only need to be called once.

Quote
... Please try the git version and see if it looks ok.
Runs well. Looks good ...

Quote
... I also still dislike the queue logic. ...
It's atomic and it works, but OK.

However, I would like to point out I think this is kind of pointless:
Code: [Select]
# If another copy tried to run while we were running, rescan.
if [ -e /var/run/rebuildfstab.rescan ]; then
  rm -f /var/run/rebuildfstab.rescan
  exec $0 "$@"
fi
As far as I can tell, the only program that knows to check for  /var/run/rebuildfstab.rescan
is  rebuildfstab  when it calls itself.
Title: Re: Tinkering with rebuildfstab
Post by: curaga on March 11, 2023, 09:25:32 AM
Thanks CNK, added rm.
Title: Re: Tinkering with rebuildfstab
Post by: Rich on March 12, 2023, 10:10:30 AM
Hi curaga
Found a bug.
Changed:
Code: [Select]
mkdir -p "/mnt/$DEVNAME" 2>/dev/null >/dev/nullTo:
Code: [Select]
mkdir -p "$MOUNTPOINT" 2>/dev/null >/dev/nullIn 2 places. Only the second instance caused an error, but both should
be using  "$MOUNTPOINT"  just like the  printf  statements do.

Pull request created.

Quick question. I logged into github and clicked on  Sync fork  so my
copy matches the Tinycore copy.

Locally, I have this:
Code: [Select]
c@E310:~$ cd TC-Git/
tc@E310:~/TC-Git$ ls -l
total 12
-rw-r--r-- 1 tc staff  170 Feb 28 13:13 CloneCommands.txt
drwxr-sr-x 9 tc staff 4096 Feb 28 13:39 Core-scripts/
-rwxrwxrwx 1 tc staff 1927 Mar  1 20:31 depends-on.sh
tc@E310:~/TC-Git$ git clone https://github.com/rarost/Core-scripts.git
fatal: destination path 'Core-scripts' already exists and is not an empty directory.
tc@E310:~/TC-Git$
I deleted  Core-scripts/  and ran  git clone https://github.com/rarost/Core-scripts.git
again and made my changes.

What command should I have run to make my local copy match my copy on github?
Title: Re: Tinkering with rebuildfstab
Post by: curaga on March 12, 2023, 10:29:46 AM
"git pull" should be enough. Merged.