Perhaps a large part of the time taken is by the "find" command, where adding "-maxdepth 2" may give some speedup.
# echo modified: ;time rebuildfstab-mod; echo original: ; time rebuildfstab; echo original: ; time rebuildfstab;echo modified: ;time rebuildfstab-mod;
modified:
real 0m 0.39s
user 0m 0.08s
sys 0m 0.18s
original:
real 0m 0.63s
user 0m 0.20s
sys 0m 0.40s
original:
real 0m 0.60s
user 0m 0.21s
sys 0m 0.38s
modified:
real 0m 0.37s
user 0m 0.07s
sys 0m 0.17s
Thanks for the changes. There are some issues:Oops, yes I messed that one up completely, it should be "DEVMAJOR=${DEVMAJOR%%:*}" there of course.
- the changed DEVMAJOR line uses wrong var (i)
- it calls fstype potentially many more times
#!/bin/busybox ash
rm -f /tmp/blkdev.txt
for i in /sys/block/*/dev /sys/block/*/*/dev; do
case "$i" in
*loop*|*ram*)
continue
;;
esac
DEVNAME=${i%*/dev}
DEVNAME=${DEVNAME##*/}
[ -b "/dev/$DEVNAME" ] || continue
echo "$i" >> /tmp/blkdev.txt
done
echo rebuildfstab-mod:
wc -l /tmp/blkdev.txt
rm /tmp/blkdev.txt
for i in `find /sys/block/*/ -name dev`; do
case "$i" in
*loop*|*ram*)
continue
;;
esac
echo "$i" >> /tmp/blkdev.txt
done
echo rebuildfstab:
wc -l /tmp/blkdev.txt
rm /tmp/blkdev.txt
- it calls grep many times, which is what an earlier patch specifically changed to the combined awk
- if there is no glob match, the pattern is passed as-is (I guess the -b test was added for this)
It may not be simple to apply just the globbing; when I tried, it was slower.
# echo original:; time rebuildfstab; echo github: ; time ./rebuildfstab-new;echo modified: ;time rebuildfstab-mod;echo modified: ;time rebuild
fstab-mod; echo github: ; time ./rebuildfstab-new;echo original:; time rebuildfstab;
original:
real 0m 0.82s
user 0m 0.22s
sys 0m 0.50s
github:
real 0m 0.37s
user 0m 0.09s
sys 0m 0.26s
modified:
real 0m 0.33s
user 0m 0.12s
sys 0m 0.16s
modified:
real 0m 0.36s
user 0m 0.12s
sys 0m 0.19s
github:
real 0m 0.37s
user 0m 0.09s
sys 0m 0.26s
original:
real 0m 0.81s
user 0m 0.21s
sys 0m 0.51s
EDIT: Oops again, posted test results before applying the DEVMAJOR fix to my version before.
did you measure how much time blkid takes when you run it once vs. many times?It's not that find is buggy, it's how it works (and bb find is slower than GNU find too). It runs a stat call for each file, and in the searched directories there are a lot of files. We're only interested in the file names in this invocation, file types/sizes/times don't matter. So this part is the lowest hanging fruit currently, I believe.
i think if find is slow then it might be bugged. it shouldn't be supposed to do very much! :O
i'm happy to help obsess some more about this at some point, as it's such a small contained neat thing. ;)Everyone is welcome to go over the TC scripts. rebuildfstab's speed is fast enough for me personally, but if you find it too slow, please do look into it.
The system version runs blkid once (it's in the END block in awk), and parses the output. Your version runs it once per device. The later calls should be cached and fast, but it's still extra calls.Quote- it calls fstype potentially many more times
How do you come to this conclusion (besides due to the DEVMAJOR error)?
The system version runs blkid once (it's in the END block in awk), and parses the output. Your version runs it once per device. The later calls should be cached and fast, but it's still extra calls.
The system version runs blkid once (it's in the END block in awk), and parses the output. Your version runs it once per device. The later calls should be cached and fast, but it's still extra calls.Quote- it calls fstype potentially many more timesHow do you come to this conclusion (besides due to the DEVMAJOR error)?
... Everyone is welcome to go over the TC scripts. rebuildfstab's speed is fast enough for me personally, but if you find it too slow, please do look into it. ...I think I may have an approach that doesn't need the find command.
tc@E310:~/rebuildfstab$ time ./findblockdevs > /dev/null
real 0m 0.71s
user 0m 0.22s
sys 0m 0.53s
tc@E310:~/rebuildfstab$ time ./newfindblockdevs > /dev/null
real 0m 0.09s
user 0m 0.04s
sys 0m 0.05s
tc@E310:~/rebuildfstab$
tc@E310:~/rebuildfstab$ time ./findblockdevs
DEVNAME=sda4 DEVMAJOR=8
DEVNAME=sda2 DEVMAJOR=8
DEVNAME=sda DEVMAJOR=8
DEVNAME=sda7 DEVMAJOR=8
DEVNAME=sda5 DEVMAJOR=8
DEVNAME=sda3 DEVMAJOR=8
DEVNAME=sda1 DEVMAJOR=8
DEVNAME=sda6 DEVMAJOR=8
DEVNAME=sdb DEVMAJOR=8
DEVNAME=sdb1 DEVMAJOR=8
DEVNAME=sdc DEVMAJOR=8
DEVNAME=sdd DEVMAJOR=8
DEVNAME=sde DEVMAJOR=8
DEVNAME=sdf DEVMAJOR=8
DEVNAME=sr0 DEVMAJOR=11
DEVNAME=sr1 DEVMAJOR=11
real 0m 0.70s
user 0m 0.22s
sys 0m 0.52s
tc@E310:~/rebuildfstab$ time ./newfindblockdevs
DEVNAME=sr0 DEVMAJOR=11
DEVNAME=sr1 DEVMAJOR=11
DEVNAME=sda DEVMAJOR=8
DEVNAME=sda1 DEVMAJOR=8
DEVNAME=sdb DEVMAJOR=8
DEVNAME=sdb1 DEVMAJOR=8
DEVNAME=sda2 DEVMAJOR=8
DEVNAME=sda3 DEVMAJOR=8
DEVNAME=sdc DEVMAJOR=8
DEVNAME=sda4 DEVMAJOR=8
DEVNAME=sdd DEVMAJOR=8
DEVNAME=sda5 DEVMAJOR=8
DEVNAME=sda6 DEVMAJOR=8
DEVNAME=sde DEVMAJOR=8
DEVNAME=sda7 DEVMAJOR=8
DEVNAME=sdf DEVMAJOR=8
real 0m 0.09s
user 0m 0.03s
sys 0m 0.06s
tc@E310:~/rebuildfstab$
tc@E310:~/rebuildfstab$ time ./newerfindblockdevs > /dev/null
real 0m 0.03s
user 0m 0.01s
sys 0m 0.02s
tc@E310:~/rebuildfstab$
tc@E310:~/rebuildfstab$ time ./newerfindblockdevs
DEVNAME=sda DEVMAJOR=8
DEVNAME=sda1 DEVMAJOR=8
DEVNAME=sda2 DEVMAJOR=8
DEVNAME=sda3 DEVMAJOR=8
DEVNAME=sda4 DEVMAJOR=8
DEVNAME=sda5 DEVMAJOR=8
DEVNAME=sda6 DEVMAJOR=8
DEVNAME=sda7 DEVMAJOR=8
DEVNAME=sdb DEVMAJOR=8
DEVNAME=sdb1 DEVMAJOR=8
DEVNAME=sr0 DEVMAJOR=11
DEVNAME=sr1 DEVMAJOR=11
DEVNAME=sdg DEVMAJOR=8
real 0m 0.03s
user 0m 0.01s
sys 0m 0.02s
tc@E310:~/rebuildfstab$
tc@E310:~/rebuildfstab$ time sudo ./rebuildfstab.git
./rebuildfstab.git: line 71: syntax error: unexpected redirection
Command exited with non-zero status 2
real 0m 0.01s
user 0m 0.00s
sys 0m 0.01s
tc@E310:~/rebuildfstab$
67 if [ "$MOUNTPOINT" != "none" ]; then
68 mkdir -p "/mnt/$DEVNAME" 2>/dev/null >/dev/null
69 fi
70 printf "%-15s %-15s %-8s %-20s %-s\n" "$DEVROOT/$DEVNAME" "$MOUNTPOINT" "$FSTYPE" "$OPTIONS" "0 0 $ADDEDBY" >>"$TMP"
71 done < <(find /sys/block/*/ -maxdepth 2 -name dev ! -regex '.*\(/loop\|/ram\|/zram\|\s\).*' | awk -v devroot="$DEVROOT" -v tmptab="$TMP" '
72 BEGIN {
73 while (getline < tmptab > 0) remain[$1]=1
74 FS="/|!"
75 }
tc@E310:~/rebuildfstab$ time sudo ./rebuildfstab.git
real 0m 0.92s
user 0m 0.14s
sys 0m 0.32s
tc@E310:~/rebuildfstab$ time sudo ./rebuildfstab.git
real 0m 0.69s
user 0m 0.11s
sys 0m 0.36s
tc@E310:~/rebuildfstab$
tc@E310:~/rebuildfstab$ time sudo rebuildfstab
real 0m 1.98s
user 0m 0.37s
sys 0m 0.83s
tc@E310:~/rebuildfstab$ time sudo rebuildfstab
real 0m 1.42s
user 0m 0.36s
sys 0m 0.83s
tc@E310:~/rebuildfstab$ time sudo rebuildfstab
real 0m 1.41s
user 0m 0.40s
sys 0m 0.76s
tc@E310:~/rebuildfstab$ time sudo ./rebuildfstab.git
real 0m 1.05s
user 0m 0.17s
sys 0m 0.54s
tc@E310:~/rebuildfstab$ time sudo ./rebuildfstab.git
real 0m 0.78s
user 0m 0.15s
sys 0m 0.39s
tc@E310:~/rebuildfstab$ time sudo ./rebuildfstab.git
real 0m 0.71s
user 0m 0.12s
sys 0m 0.39s
tc@E310:~/rebuildfstab$ time sudo ./rebuildfstab.rewrite
real 0m 0.72s
user 0m 0.06s
sys 0m 0.12s
tc@E310:~/rebuildfstab$ time sudo ./rebuildfstab.rewrite
real 0m 0.16s
user 0m 0.09s
sys 0m 0.06s
tc@E310:~/rebuildfstab$ time sudo ./rebuildfstab.rewrite
real 0m 0.23s
user 0m 0.07s
sys 0m 0.11s
tc@box:~$ time sudo ./rebuildfstab
real 0m 0.36s
user 0m 0.06s
sys 0m 0.14s
tc@box:~$ time sudo ./rebuildfstab
real 0m 0.27s
user 0m 0.05s
sys 0m 0.11s
tc@box:~$ time sudo ./rebuildfstab
real 0m 0.31s
user 0m 0.05s
sys 0m 0.12s
tc@box:~$ time sudo ./rebuildfstab.rewrite
real 0m 0.14s
user 0m 0.04s
sys 0m 0.06s
tc@box:~$ time sudo ./rebuildfstab.rewrite
real 0m 0.13s
user 0m 0.04s
sys 0m 0.05s
tc@box:~$ time sudo ./rebuildfstab.rewrite
real 0m 0.09s
user 0m 0.04s
sys 0m 0.04s
tc@E310:~/rebuildfstab$ grep -v "### " rebuildfstab.rewrite > rebuildfstab.stripped
tc@E310:~/rebuildfstab$ ls -l rebuildfstab.git rebuildfstab.rewrite rebuildfstab.stripped
-rwxr-xr-x 1 tc staff 2595 Feb 27 08:56 rebuildfstab.git
-rwxr-xr-x 1 tc staff 4145 Mar 5 10:13 rebuildfstab.rewrite
-rw-r--r-- 1 tc staff 2735 Mar 5 12:33 rebuildfstab.stripped
tc@E310:~/rebuildfstab$
This version also misses custom mounts, ...Could you provide a detailed example of what gets missed please.
... and I feel the queue logic is worse than the current checks. If two more scripts happen to be spawned while one is running, the old version would only rerun once, this would rerun them all. ...Exactly. A program calling rebuildfstab expects to find an updated fstab.
| ProgramA | rebuildfstab scans devices, updates fstab, launches rebuildfstab again | ProgramA reads fstab |
| ProgramB | rebuildfstab touches rescan file and exits | ProgramB reads fstab |
... Otherwise adding custom mounts and running blkid just once ...I'll take a closer look at that.
Custom mounts: say you have a line mounting /dev/sda1 to /files. ...Do you mean something like this might already be in /etc/fstab:
/dev/sda1 /files ext4 noauto,users,exec 0 0
and in this example I should skip adding:/dev/sda1 /mnt/sda1 ext4 noauto,users,exec 0 0 # Added by TC
For example if you have "grep $FILTERS -v /proc/partitions | while read -r DEVMAJOR DEVMINOR BLOCKS DEVNAME". Start with FILTERS="-e 'loop[0-9]*$' -e 'ram[0-9]*$' -e '^$' -e ' name$'", then write a loop that adds an "-e" onto $FILTERS for each first field of "$TMP". EDIT: "grep $FILTERS -v /proc/partitions" doesn't work, I haven't got time to figure out exactly how wrong I am here...
tc@E310:~/rebuildfstab$ time sudo ./rebuildfstab.git; sleep 1; echo; time sudo ./rebuildfstab.git; sleep 1; echo; time sudo ./rebuildfstab.git
real 0m 1.02s
user 0m 0.18s
sys 0m 0.53s
real 0m 0.77s
user 0m 0.16s
sys 0m 0.37s
real 0m 0.83s
user 0m 0.15s
sys 0m 0.39s
tc@E310:~/rebuildfstab$ time sudo ./rebuildfstab.devlist; sleep 1; echo; time sudo ./rebuildfstab.devlist; sleep 1; echo; time sudo ./rebuildfstab.devlist
real 0m 0.48s
user 0m 0.04s
sys 0m 0.05s
real 0m 0.08s
user 0m 0.03s
sys 0m 0.04s
real 0m 0.13s
user 0m 0.03s
sys 0m 0.05s
... My version of rebuildfstab:Code: [Select]tc@E310:~/rebuildfstab$ time sudo ./rebuildfstab.rewrite
...
real 0m 0.72s
user 0m 0.06s
sys 0m 0.12s
tc@E310:~/rebuildfstab$ time sudo ./rebuildfstab.rewrite
real 0m 0.16s
user 0m 0.09s
sys 0m 0.06s
tc@E310:~/rebuildfstab$ time sudo ./rebuildfstab.rewrite
real 0m 0.23s
user 0m 0.07s
sys 0m 0.11s
... This happened when the VM had no drives. ...Yes, I forgot if you pass blkid an empty $DevList it spits out every device it can find.
... comments should not be aligned like devices ...I saw that the comment was present, but my brain didn't register that it was indented.
... auto filesystems should still get a dir created ...Yup. casualty of refactoring the script so blkid would only need to be called once.
... Please try the git version and see if it looks ok.Runs well. Looks good ...
... I also still dislike the queue logic. ...It's atomic and it works, but OK.
# If another copy tried to run while we were running, rescan.
if [ -e /var/run/rebuildfstab.rescan ]; then
rm -f /var/run/rebuildfstab.rescan
exec $0 "$@"
fi
As far as I can tell, the only program that knows to check for /var/run/rebuildfstab.rescanmkdir -p "/mnt/$DEVNAME" 2>/dev/null >/dev/null
To:mkdir -p "$MOUNTPOINT" 2>/dev/null >/dev/null
In 2 places. Only the second instance caused an error, but both shouldc@E310:~$ cd TC-Git/
tc@E310:~/TC-Git$ ls -l
total 12
-rw-r--r-- 1 tc staff 170 Feb 28 13:13 CloneCommands.txt
drwxr-sr-x 9 tc staff 4096 Feb 28 13:39 Core-scripts/
-rwxrwxrwx 1 tc staff 1927 Mar 1 20:31 depends-on.sh
tc@E310:~/TC-Git$ git clone https://github.com/rarost/Core-scripts.git
fatal: destination path 'Core-scripts' already exists and is not an empty directory.
tc@E310:~/TC-Git$
I deleted Core-scripts/ and ran git clone https://github.com/rarost/Core-scripts.git