General TC > Programming & Scripting - Unofficial
Tinkering with rebuildfstab
CNK:
--- Quote from: curaga on February 25, 2023, 03:36:55 AM ---Thanks for the changes. There are some issues:
- the changed DEVMAJOR line uses wrong var (i)
--- End quote ---
Oops, yes I messed that one up completely, it should be "DEVMAJOR=${DEVMAJOR%%:*}" there of course.
--- Quote ---- it calls fstype potentially many more times
--- End quote ---
How do you come to this conclusion (besides due to the DEVMAJOR error)?
For me this script shows that the same number of file system device lines are run through the parts of the loop where the fstype and grep calls are located. If the DEVNAME and DEVMAJOR variables are also the same in each version, then I don't see where there can be a difference in the behaviour.
--- Code: ---#!/bin/busybox ash
rm -f /tmp/blkdev.txt
for i in /sys/block/*/dev /sys/block/*/*/dev; do
case "$i" in
*loop*|*ram*)
continue
;;
esac
DEVNAME=${i%*/dev}
DEVNAME=${DEVNAME##*/}
[ -b "/dev/$DEVNAME" ] || continue
echo "$i" >> /tmp/blkdev.txt
done
echo rebuildfstab-mod:
wc -l /tmp/blkdev.txt
rm /tmp/blkdev.txt
for i in `find /sys/block/*/ -name dev`; do
case "$i" in
*loop*|*ram*)
continue
;;
esac
echo "$i" >> /tmp/blkdev.txt
done
echo rebuildfstab:
wc -l /tmp/blkdev.txt
rm /tmp/blkdev.txt
--- End code ---
--- Quote ---- it calls grep many times, which is what an earlier patch specifically changed to the combined awk
--- End quote ---
Oh right, I forgot to check the current version on Github and just based this on the version in TC 13. Now that I've seen that, I see what you're talking about (it took a while) - you're using "read" to advance through the list of block devices now, and feeding it from find and awk.
--- Quote ---- if there is no glob match, the pattern is passed as-is (I guess the -b test was added for this)
--- End quote ---
Yes the -b test catches that.
--- Quote ---It may not be simple to apply just the globbing; when I tried, it was slower.
--- End quote ---
OK. Testing against the 'new' rebuildfstab on GitHub the execution time is much closer with my version Vs 'rebuildfstab-new', and that's on a test machine with only four mountable file systems so not many calls to Grep which affect the older design that I based mine on. I'll believe you that the new layout is no faster with globbing (fed to awk via "echo", presumably), I've spent enough time on this already now.
--- Code: ---# echo original:; time rebuildfstab; echo github: ; time ./rebuildfstab-new;echo modified: ;time rebuildfstab-mod;echo modified: ;time rebuild
fstab-mod; echo github: ; time ./rebuildfstab-new;echo original:; time rebuildfstab;
original:
real 0m 0.82s
user 0m 0.22s
sys 0m 0.50s
github:
real 0m 0.37s
user 0m 0.09s
sys 0m 0.26s
modified:
real 0m 0.33s
user 0m 0.12s
sys 0m 0.16s
modified:
real 0m 0.36s
user 0m 0.12s
sys 0m 0.19s
github:
real 0m 0.37s
user 0m 0.09s
sys 0m 0.26s
original:
real 0m 0.81s
user 0m 0.21s
sys 0m 0.51s
--- End code ---
EDIT: Oops again, posted test results before applying the DEVMAJOR fix to my version before.
curaga:
--- Quote from: hiro on February 25, 2023, 04:33:01 AM ---did you measure how much time blkid takes when you run it once vs. many times?
i think if find is slow then it might be bugged. it shouldn't be supposed to do very much! :O
--- End quote ---
It's not that find is buggy, it's how it works (and bb find is slower than GNU find too). It runs a stat call for each file, and in the searched directories there are a lot of files. We're only interested in the file names in this invocation, file types/sizes/times don't matter. So this part is the lowest hanging fruit currently, I believe.
--- Quote from: hiro on February 25, 2023, 04:37:36 AM ---i'm happy to help obsess some more about this at some point, as it's such a small contained neat thing. ;)
--- End quote ---
Everyone is welcome to go over the TC scripts. rebuildfstab's speed is fast enough for me personally, but if you find it too slow, please do look into it.
--- Quote from: CNK on February 25, 2023, 06:17:23 AM ---
--- Quote ---- it calls fstype potentially many more times
--- End quote ---
How do you come to this conclusion (besides due to the DEVMAJOR error)?
--- End quote ---
The system version runs blkid once (it's in the END block in awk), and parses the output. Your version runs it once per device. The later calls should be cached and fast, but it's still extra calls.
hiro:
--- Quote from: curaga on February 25, 2023, 11:09:43 AM ---The system version runs blkid once (it's in the END block in awk), and parses the output. Your version runs it once per device. The later calls should be cached and fast, but it's still extra calls.
--- End quote ---
thanks for clearing that up. i actually missed the END and thought it was run multiple times.
if the GNU find is faster than bb find, then maybe it's not worth optimizing for this. as in the long-term it would be more useful if the busybox find was made faster instead (not our problem).
any remaining bottlenecks are now much less obvious to my intuition and i agree we're probably hitting diminishable returns here.
but i'm gonna still offer to try out more solutions, even on a bigger SAS array, if that would be of any help to the remaining benchmark fanatics :P
CNK:
--- Quote from: curaga on February 25, 2023, 11:09:43 AM ---
--- Quote from: CNK on February 25, 2023, 06:17:23 AM ---
--- Quote ---- it calls fstype potentially many more times
--- End quote ---
How do you come to this conclusion (besides due to the DEVMAJOR error)?
--- End quote ---
The system version runs blkid once (it's in the END block in awk), and parses the output. Your version runs it once per device. The later calls should be cached and fast, but it's still extra calls.
--- End quote ---
Yeah I was still comparing with the TC 13 version there. If before posting I'd remembered to check on GitHub to see if rebuildfstab had changed since TC 13 then I would have seen the new layout and never talked about my experiments on the old version in first palce. Sorry to waste time.
hiro:
nah, you didn't start this "time-waste".
it's at least entertaining and at best educational to the next reader.
Navigation
[0] Message Index
[#] Next page
[*] Previous page
Go to full version