How to find which extension provides a file

[patrikg]

Sorry for not getting the performance working with sqlite3 ---
Tried to get it working so the command line should report the performance.

I can't getting it working with the ".timer on" with the command line version of sqlite3.
The ".timer on" doesn't report the speed sadly when executing from the command line.

But if you get into the interactive mode of sqlite3 it just works so you get a report how many seconds the question took.

It says in "man" you could supply the -cmd ".timer on" but it doesn't work for me, but some another commands like -cmd ".help" works out of the box.

And if "you" choose sqlite3 you could also store some more metadata like version, date and so on.

Maybe also store ldd output of the executable, so you can easily see the dependency's.

[Rich]

Hi nick65go

... OR, if the speed is not the main goal (but nice for multi-CPU core) then maybe the user confort for seach parameters (GUI options will be nice also). But of course, the pro[fessionals] already know the use of "" and back-quotes, etc.

I think backwards compatibility and user friendly search parameters
should be the primary goals.

I did a rewrite of provides.sh as follows:
1. The default search mode is unchanged.
2. No backslashes to enter. The script adds them if needed.
3. No quoting of search term.
4. End of search term can be marked with an anchor ($).
5. Added a brief usage message.
6. This works for both the command line and the GUI.

Examples:
Search for plain file name:

Code: [Select]

tc@E310:~/Scripting/Provides$ ./provides.sh grep | wc -l
51
tc@E310:~/Scripting/Provides$
Search for file name with leading slash:

Code: [Select]

tc@E310:~/Scripting/Provides$ ./provides.sh /grep
boost-1.65-dev.tcz
boost-dev.tcz
emacs.cedet.tcz
emacs.doc.tcz
emacs.minimal.tcz
emacs.prog.tcz
fpc-src.tcz
fpc.tcz
grep.tcz
kompozer.tcz
libreoffice.tcz
mariadb-test.tcz
python3.6.tcz
tc@E310:~/Scripting/Provides$
Search for file name with partial path:

Code: [Select]

tc@E310:~/Scripting/Provides$ ./provides.sh bin/grep
emacs.minimal.tcz
fpc.tcz
grep.tcz
tc@E310:~/Scripting/Provides$
Search for file name with partial path and trailing anchor:

Code: [Select]

tc@E310:~/Scripting/Provides$ ./provides.sh bin/grep$
grep.tcz
tc@E310:~/Scripting/Provides$
Added help message:

Code: [Select]

tc@E310:~/Scripting/Provides$ ./provides.sh --help

Finds extension(s) that provide a filename.
Filenames in list being searched include full paths, for example:
        usr/local/bin/grep

Usage:
        provides.sh FileName

Examples:
        This also finds all extensions that include local in their pathes.
        provides.sh cal

        This finds *bin/cal*. Including sbin/cal, bin/calibrate, etc.
        provides.sh bin/cal

        This finds *bin/cal. Including sbin/cal, usr/bin/cal, etc.
        provides.sh bin/cal$

The more specific the search, the fewer the results returned.


tc@E310:~/Scripting/Provides$
And the script itself:

Code: [Select]

#!/bin/busybox ash
. /etc/init.d/tc-functions
useBusybox

# --------------------------------------------------------------------------- #
Usage()
{
        echo "
Finds extension(s) that provide a filename.
Filenames in list being searched include full paths, for example:
usr/local/bin/grep

Usage:
        ${0##*/} FileName

Examples:
This also finds all extensions that include local in their pathes.
        ${0##*/} cal

This finds *bin/cal*. Including sbin/cal, bin/calibrate, etc.
        ${0##*/} bin/cal

This finds *bin/cal. Including sbin/cal, usr/bin/cal, etc.
        ${0##*/} bin/cal$

The more specific the search, the fewer the results returned.

"
        exit
}
# --------------------------------------------------------------------------- #



# See if user needs help.
case "$1" in
        -h) Usage;;
        -help) Usage;;
        --help) Usage;;
        "") exit 1;;
esac

# Flag to indicate whether we want an exact match. 0=No  1=Yes.
Exact=0

TARGET="$1"

TCEDIR="/etc/sysconfig/tcedir"
DB="provides.db"

getMirror
cd "$TCEDIR"
if zsync -i "$TCEDIR"/"$DB" -q "$MIRROR"/"$DB".zsync
then
rm -f "$DB".zs-old
else
if [ ! -f "$TCEDIR"/"$DB" ]
then
  wget -O "$TCEDIR"/"$DB".gz "$MIRROR"/"$DB".gz
  gunzip "$TCEDIR"/"$DB".gz
fi
fi
chmod g+rw "$TCEDIR"/"$DB"
cd - > /dev/null

# Save string length of TARGET.
Length=${#TARGET}

# Remove trailing $ (exact match request) if present.
TARGET="${TARGET%$}"

# If TARGET is now shorter, exact match was requested.
[ ${#TARGET} -lt $Length ] && Exact=1

# Replace all instances of / with \/.
TARGET="${TARGET//\//\\\/}"

if [ $Exact -eq 0 ]
then
awk 'BEGIN {FS="\n";RS=""} /'${TARGET}'/{print $1}' "$TCEDIR"/"$DB"
else
awk 'BEGIN {FS="\n";RS=""} /'${TARGET}'\n/||/'${TARGET}'$/{print $1}' "$TCEDIR"/"$DB"
fi

[polikuo]

Hi, Rich.

Do you think its possible to implement some checker so that Provides.sh only sync with the repo if the db file is older than ... say a day ?
Syncing takes a lot of time.
I don't know the exact syntax, but "find -mtime" should work ?
Perhaps you could wrap up the syncing code into a shell function ?

[Rich]

Hi polikuo
A quick read of the man page suggests that only the provides.db.zsync
file should get downloaded and only if it changed since the last download.

This a time based version I used to use:

Code: [Select]

# Maximum age of  provides.db  in seconds before it is considered to be stale and
# needs to be redownloaded.
MaxAge=3600

PROVIDES="provides.db"


RefreshProvidesDB()
{
# This downloads a fresh copy of provides.db if any of the following are true:
# 1. The file is not in the current directory.
# 2. The file is older than 1 hour (3600 seconds).

cd /etc/sysconfig/tcedir

# Make sure the proper  provides.db  exists.
if [ -f "$PROVIDES" ]
then
        # Compute number of seconds since provides.db modified (downloaded).
        Age=$(( $(date +%s) - $(date -r "$PROVIDES"  +%s) ))
        if [ $Age -lt $MaxAge ]
        then
                # File is recent enough to use.
                return
        fi
        # File is too old, delete it.
        rm "$PROVIDES"
fi

# Fetch a fresh copy of the file.
wget -q -O "$PROVIDES" "$URL$PROVIDES"
if [ $? -ne 0 ]
then
        echo "Download failed for: $URL$PROVIDES"
        exit 1
fi

# Make sure it has a current timestamp.
touch "$PROVIDES"

cd - > /dev/null

}


[Rich]

Hi polikuo

... Perhaps you could wrap up the syncing code into a shell function ?

Done.

Code: [Select]

#!/bin/busybox ash
. /etc/init.d/tc-functions
useBusybox

# --------------------------------------------------------------------------- #
UpdateProvidesDB()
{
getMirror
cd "$TCEDIR"
if zsync -i "$TCEDIR"/"$DB" -q "$MIRROR"/"$DB".zsync
then
rm -f "$DB".zs-old
else
if [ ! -f "$TCEDIR"/"$DB" ]
then
  wget -O "$TCEDIR"/"$DB".gz "$MIRROR"/"$DB".gz
  gunzip "$TCEDIR"/"$DB".gz
fi
fi
chmod g+rw "$DB"
cd - > /dev/null
}
# --------------------------------------------------------------------------- #

# --------------------------------------------------------------------------- #
Usage()
{
        echo "
Finds extension(s) that provide a filename.
Filenames in list being searched include full paths, for example:
usr/local/bin/grep

Usage:
        ${0##*/} FileName

Examples:
This also finds all extensions that include local in their pathes.
        ${0##*/} cal

This finds *bin/cal*. Including sbin/cal, bin/calibrate, etc.
        ${0##*/} bin/cal

This finds *bin/cal. Including sbin/cal, usr/bin/cal, etc.
        ${0##*/} bin/cal$

The more specific the search, the fewer the results returned.

"
        exit
}
# --------------------------------------------------------------------------- #



# See if user needs help.
case "$1" in
        -h) Usage;;
        -help) Usage;;
        --help) Usage;;
        "") exit 1;;
esac

# Flag to indicate whether we want an exact match. 0=No  1=Yes.
Exact=0

TARGET="$1"

TCEDIR="/etc/sysconfig/tcedir"
DB="provides.db"

# Run zsync on the provides.db file.
UpdateProvidesDB

# Save string length of TARGET.
Length=${#TARGET}

# Remove trailing $ (exact match request) if present.
TARGET="${TARGET%$}"

# If TARGET is now shorter, exact match was requested.
[ ${#TARGET} -lt $Length ] && Exact=1

# Replace all instances of / with \/.
TARGET="${TARGET//\//\\\/}"

if [ $Exact -eq 0 ]
then
awk 'BEGIN {FS="\n";RS=""} /'${TARGET}'/{print $1}' "$TCEDIR"/"$DB"
else
awk 'BEGIN {FS="\n";RS=""} /'${TARGET}'\n/||/'${TARGET}'$/{print $1}' "$TCEDIR"/"$DB"
fi

[patrikg]

I think this script can also detect some fault tcz's, that the squashfs is making so you don't clobbing your own old files and directorys.

Here is the old thread talking about that.

https://forum.tinycorelinux.net/index.php/topic,25229.msg161137.html#msg161137

[Rich]

Hi patrikg
The version that is currently part of Tinycore is also capable
of that. The syntax for that version is a bit more cryptic:

Code: [Select]

provides.sh "\/usr_1\/"
Using the version attached to my last post would look like this:

Code: [Select]

provides.sh /usr_1/

[patrikg]

Hi @Rich

Ok, thanks for that. Didn't know.
This is because my main dist is not TC.
I am just here to help out, because i want that TC survives.

I like it a lot with embedded systems like Raspberry PI (PICore).

And if somebody want to learn how Linux works i suggest to use TC, it's
not bloated with so many things.

It's not just the kernel(Core) that's are Tiny.





 

[mocore]

ftr
a vaguely  related tangent ... expanding *generally* on some themes in this topic
can be found hear https://forum.tinycorelinux.net/index.php/topic,27364.0.html

[Rich]

Hi polikuo
Focusing on ease of use and some speed improvements, I did
some more work on provides.sh:
1. The default search mode is unchanged. Any programs/scripts calling provides.sh should see no breakage.
2. No backslashes to enter. The script adds them if needed.
3. No quoting of search term, even if it includes embedded spaces.
4. End of search term can be marked with an anchor ($).
5. Added a brief help message (-h, -help, or --help).
6. Added alias to /usr/local/bin/awk if it exists.
7. Added -nz switch to skip zsync for faster responses.
8. These features all work for both the command line and the GUI.

Regular search:

Code: [Select]

tc@E310:~/Scripting/Provides$ time ./provides.sh bin/grep$
grep.tcz
real    0m 1.36s
user    0m 0.49s
sys     0m 0.23s
tc@E310:~/Scripting/Provides$
Same search, no zsync:

Code: [Select]

tc@E310:~/Scripting/Provides$ time ./provides.sh -nz bin/grep$
grep.tcz
real    0m 0.22s
user    0m 0.19s
sys     0m 0.03s
tc@E310:~/Scripting/Provides$-nz switch can be placed before or after the search term.
-nz switch also works in the Apps GUI Provides field.

Search, no zsync, embedded space:

Code: [Select]

tc@E310:~/Scripting/Provides$ time ./provides.sh Black Gnome -nz
volumeicon.tcz
real    0m 0.13s
user    0m 0.11s
sys     0m 0.02s
tc@E310:~/Scripting/Provides$
tc@E310:~/Scripting/Provides$ time ./provides.sh  Make Cocoa Headers.txt -nz
fpc-src.tcz
real    0m 0.13s
user    0m 0.10s
sys     0m 0.02s
tc@E310:~/Scripting/Provides$No quotes required.

And the help message:

Code: [Select]

tc@E310:~/Scripting/Provides$ ./provides.sh --help

Find extension(s) that provide a filename.
Filenames in list being searched include full paths, for example:
        usr/local/bin/grep

Usage:
   provides.sh [ -nz ] FileName

   -nz    Skip updating (zsync) the provides.db file. This speeds up
          the search, but might miss items if provides.db is outdated.

Examples:
   provides.sh cal           Finds cal anywhere in FileName

   provides.sh bin/cal       Finds bin/cal anywhere in FileName

   provides.sh bin/cal$      Finds FileName that ends in bin/cal

   provides.sh Black Gnome   Finds FileName with embedded spaces

The more specific the search, the fewer the results returned.

tc@E310:~/Scripting/Provides$
I've attached a copy of this version. I would appreciate it if some forum
members download, test, and provide comments.

After downloading, run:

Code: [Select]

sudo cp provides.sh /usr/binThe new provides.sh will now be functional for both command line and Apps GUI.

I will be removing the attachments from previous posts.

[nick65go]

@Rich: From my point of view, you did a really good job! I mean you kept compatibility for old i486 CPU and improved the search comfort, BOTH in GUI and cmd line. Using just awk and shell, the search time (for you) is less than 2 seconds!

The number of extensions (in the future) will NOT dramatically increase (because missing MORE significant number of developers).
My assumptions are that the SIZE of database will NOT increase too much (+20% maxim as GZIP-ed) and the database is LOCATED in RAM memory, plus the UPDATE of the database is not so often performed. So, it seams that by this path you will soon arrive at diminishing return (Pareto principle). All is just an intellectual challenge, which I like.

Even if you can achieve, lets say search in 1/1000 seconds, which is 2,000.00 time faster than 2 seconds, the practical implication could be exploited only in loops or pipes programs, but not in GUI interface.

I wonder what (old?) computer specification you used (CPU, HDD, RAM), if you want to share, for your test.

[polikuo]

Hi, Rich.
Nice work !!
You don't mind if I tweak a little right ?
Cause I really like to skip the zsync routine.

Code: [Select]

--- provides.sh
+++ /usr/local/bin/provides.sh
@@ -32,6 +32,7 @@
 fi
 chmod g+rw "$DB"
 cd - > /dev/null
+touch $LIST
 }
 # --------------------------------------------------------------- #

@@ -85,7 +86,7 @@
 [ -z "$TARGET" ] && exit 1

 # Run zsync on the provides.db file if NZ equals 1.
-[ $NZ -eq 1 ] && UpdateProvidesDB
+[ $NZ -eq 1 ] || [ -n "$(find $LIST -mmin +15)" ] && UpdateProvidesDB

 # Save string length of TARGET.
 Length=${#TARGET}

The idea here is similar to the sudo session.
On Ubuntu, for example, after typing the password in a sudo command, the system will not ask for password in the following 15 minutes.
The newer version of busybox supports -mmin option.

Code: [Select]

tc@pi4:/tmp$ uname -a
Linux pi4 6.6.47-piCore-v8 #21 SMP PREEMPT Sat Aug 31 15:08:05 EDT 2024 aarch64 GNU/Linux
tc@pi4:/tmp$ busybox find --help
BusyBox v1.36.1 (2024-04-07 15:54:04 UTC) multi-call binary.
...
...
        -mtime DAYS     mtime is greater than (+N), less than (-N),
                        or exactly N days in the past
        -mmin MINS      mtime is greater than (+N), less than (-N),
                        or exactly N minutes in the past


[Rich]

Hi nick65go

... Using just awk and shell, the search time (for you) is less than 2 seconds! ...

Most of that time is consumed by zsync checking if the servers version of provides.db
is newer than the one on your computer.

If you use the -nz switch it skips the zsync check and the time drops to under 0.2 seconds.

... I wonder what (old?) computer specification you used (CPU, HDD, RAM), if you want to share, for your test.

I retired my old 800Mhz  i686 (coppermine), 512 Meg RAM, PATA drives system
a year or two ago. It was a good machine for testing speed impact of optimizations.

Now I mostly use this machine:

Code: [Select]

tc@E310:~$ sudo inxi -CfmDGnSM
System:
  Host: E310 Kernel: 4.19.10-tinycore i686 bits: 32 Desktop: FLWM
  Distro: TinyCore 10.1
Machine:
  Type: Desktop System: Dell product: Dell DV051 v: N/A serial: 9Q5X2B1
  Mobo: Dell model: 0JC474 serial: ..CN481115B2077P. BIOS: Dell v: A04
  date: 04/04/2006
Memory:
  RAM: total: 2.96 GiB used: 1.30 GiB (44.1%)
  Array-1: capacity: 4 GiB note: est. slots: 2 EC: Single-bit ECC
  Device-1: DIMM_1 size: 2 GiB speed: 533 MHz
  Device-2: DIMM_2 size: 2 GiB speed: 533 MHz
CPU:
  Topology: Single Core model: Intel Pentium 4 bits: 64 type: MT
  L2 cache: 1024 KiB
  Speed: 2793 MHz min/max: N/A Core speeds (MHz): 1: 2793 2: 2793
  Flags: acpi apic bts cid clflush cmov constant_tsc cpuid cx16 cx8 de
  ds_cpl dtes64 dts fpu fxsr ht lahf_lm lm mca mce mmx monitor msr mtrr pae
  pat pbe pebs pge pni pse pse36 sep ss sse sse2 tm tsc vme xtpr
Graphics:
  Device-1: Intel 82915G/GV/910GL Integrated Graphics driver: i915 v: kernel
  Display: server: X.Org 1.20.4 driver: intel resolution: 1920x1080~60Hz
  OpenGL: renderer: Mesa DRI Intel 915G x86/MMX/SSE2 v: 1.4 Mesa 19.2.3
Network:
  Device-1: Intel 82562ET/EZ/GT/GZ - PRO/100 VE Ethernet driver: e100
  IF: eth0 state: up speed: 100 Mbps duplex: full mac: 00:13:20:c4:4a:20
  IF-ID-1: dummy0 state: down mac: 6e:93:1c:04:f2:dc
  IF-ID-2: ip_vti0 state: down mac: 00:00:00:00
  IF-ID-3: tunl0 state: down mac: 00:00:00:00
Drives:
  Local Storage: total: 931.52 GiB used: 144.75 GiB (15.5%)
  ID-1: /dev/sda vendor: HP model: MB0500EBNCR size: 465.76 GiB
  ID-2: /dev/sdb vendor: HP model: MB0500EBNCR size: 465.76 GiB
tc@E310:~$

[Rich]

Hi polikuo

... Cause I really like to skip the zsync routine. ...

I get the feeling you don't like waiting for results from your computer.

This is the timing for busybox awk:

Code: [Select]

tc@E310:~/Scripting/Provides$ time ./provides.sh -nz glxinfo$
mesa-demos.tcz
real    0m 0.45s
user    0m 0.41s
sys     0m 0.04s
tc@E310:~/Scripting/Provides$ time ./provides.sh -nz glxinfo$
mesa-demos.tcz
real    0m 0.47s
user    0m 0.42s
sys     0m 0.04s
tc@E310:~/Scripting/Provides$
This is the timing when gawk.tcz is installed:

Code: [Select]

tc@E310:~/Scripting/Provides$ time ./provides.sh -nz glxinfo$
mesa-demos.tcz
real    0m 0.21s
user    0m 0.16s
sys     0m 0.04s
tc@E310:~/Scripting/Provides$ time ./provides.sh -nz glxinfo$
mesa-demos.tcz
real    0m 0.22s
user    0m 0.18s
sys     0m 0.03s
tc@E310:~/Scripting/Provides$I override the busybox default and use GNU awk if I detect it's present.

... The newer version of busybox supports -mmin option. ...

The current version of busybox on TC10 x86 does too.

Code: [Select]

+[ $NZ -eq 1 ] || [ -n "$(find $LIST -mmin +15)" ] && UpdateProvidesDB

That's pretty clever, but was the intent to override the -nz command
line switch every 15 minutes?

Maybe it makes more sense to remove the -nz command line switch
and just let it run a check when the file is at least 15 minutes old.

I would like to hear some opinions on this please.

[CNK]

Works fine here (TC15 x86_64).

Maybe it makes more sense to remove the -nz command line switch
and just let it run a check when the file is at least 15 minutes old.

I would like to hear some opinions on this please.

I like the switch - it would avoid trouble when internet access isn't working and connections are timing out. If someone really runs it on a 486 then there would be a performance advantage too.

I'd also suggest showing the help info if provides.sh is run with no arguments.