How to find which extension provides a file

[Rich]

Hi CNK

... Maybe only output the help info when called with no arguments if "[ -t 1 ]" is true, ...

Done.

[GNUser]

Hi Rich. FWIW I like the idea of automatically updating provides.db and would be happy with whatever maximum age is default.

Splitting the update into its own command or special flag (à la apt-file) would mean that using provides.sh would be a 2-step process for every user, every time (first checking age +/- updating the db, then searching the db). No autoupdate would also be a gotcha for less experienced users.

P.S. Folks that want maximum speed or more granular control can use the environmental variable you proposed to stop db updates (and/or you could add a "no db update" flag).

[GNUser]

Hi Rich. Can you please post/attach your current provides.sh? I'd love to start using it right away. Thanks
I hope it becomes part of the base system for TCL16.

[Rich]

Hi GNUser

... No autoupdate would also be a gotcha for less experienced users. ...

There will be an auto update (zsync) by default for that very reason. I think
15 minutes is long enough for most users to not get in the way when doing
multiple searches.

Also, unless you're the unluckiest person in the world, and the item you are
searching for gets added to provides.db in between auto updates, is short
enough not to cause problems.

... P.S. Folks that want maximum speed or more granular control can use the environmental variable you proposed to stop db updates (and/or you could add a "no db update" flag).

That's already in place. Including  -nz  (no zsync) in the command tells
provides.sh not to call  UpdateProvidesDB().

Latest version attached. Give it a shake and let me know if any bugs fall out.
I'll start looking into adding an environmental variable for advanced users.

[GNUser]

Thanks, Rich. I'll give it a good shake 

[CNK]

... The 1s ping might be so short it gives false negatives sometimes ...

If I change it to this:

Code: [Select]

ping -A -W 1 -c 2 8.8.8.8 2>&1 > /dev/null
It timed out in 2 seconds with no connectivity and still
returned instantly with connectivity.

The -A flag made the instant response possible:

Code: [Select]

-A              Ping as soon as reply is recevied

-A isn't supported by GNU Inetutils Ping. With inetutils.tcz loaded I get:

Code: [Select]

ping: invalid option -- 'A'

Ping (surprisingly) isn't covered by useBusybox(), so the script needs to use /bin/ping if using -A.

[Rich]

Hi CNK
Thanks for the heads up. ping changed to /bin/ping.

[Rich]

Hi GNUser

... Latest version attached. Give it a shake and let me know if any bugs fall out. ...

I did some more testing and found it ignored + signs and choked on the [ character.
I did some work on sanitizing the input and it seems to handle almost everything:

Code: [Select]

tc@E310:~/Scripting/Provides$ ./provides.sh be@latin.strings
abiword.tcz
tc@E310:~/Scripting/Provides$ ./provides.sh corrupt_t1#P#p1.MYI
mariadb-10.3-test.tcz
mariadb-10.4-test.tcz
mariadb-test.tcz
tc@E310:~/Scripting/Provides$ ./provides.sh 100%_Opaque.myb
mypaint-brushes.tcz
mypaint.tcz
tc@E310:~/Scripting/Provides$ ./provides.sh 9vx-iso
9vx.tcz
tc@E310:~/Scripting/Provides$ ./provides.sh [
coreutils.tcz
tc@E310:~/Scripting/Provides$ ./provides.sh {972ce4c6-7e08-4474-a285-3208198ce6fd}
firefox-ESR.tcz
icecat.tcz
kompozer.tcz
minefield21.tcz
palemoon.tcz
seamonkey-noSSE2.tcz
seamonkey.tcz
thunderbird.tcz
tc@E310:~/Scripting/Provides$ ./provides.sh NetSNMP::ASN.3
net-snmp-dev.tcz
net-snmp-doc.tcz
tc@E310:~/Scripting/Provides$ ./provides.sh ISO_8859-1,GL.gz
glibc_i18n_locale.tcz
tc@E310:~/Scripting/Provides$ ./provides.sh avahi-ui-gtk3~1.desktop
avahi-ui-gtk3.tcz
tc@E310:~/Scripting/Provides$ ./provides.sh c++filt
binutils.tcz
tc@E310:~/Scripting/Provides$ ./provides.sh AAC_\(Nero_FAAC\).txt
deadbeef.tcz
tc@E310:~/Scripting/Provides$As you can see, it handles  @ # % _ - [ { } : , ~  and  +.
It also handles  !  and should work with  ^ , the provides.db had no ^ characters.
I couldn't get it to accept ( or ), so those need to be escaped. i.e.  \(  and  \).
The $ sign can only be used at the end as an anchor, no where else.
The following should be avoided because the shell grabs them & | " ' ` < >.
You can try escaping them, but I'd hope none of the file names use them.

Edit: It also handles  ! ^ =  and  ].

... I'll start looking into adding an environmental variable for advanced users.

Accommodations for a  PROVIDESUPDATE  environmental variable have been added.
It's documented right in the beginning of the script if you want to use it.

[polikuo]

Hi, Rich

How about escaping everything that's not alphanumeric into hex for pattern matching ?
awk script

Code: [Select]

tc@pi4:/mnt/mmcblk0p2/notes/Shell-Tricks$ cat hex.awk
#!/usr/bin/awk -f
function hex(h,_,r,cmd) {
  cmd = "printf \x27" h "\x27 | hexdump -e \x272/1 \x22%X\x22\x27"
  cmd | getline r
  close(cmd)
  return r
}
function qry2ptn(str,_,i,c,s) {
  for (i=1;i<=length(str);i++){
    c = substr(str,i,1)
    if(c ~ /[[:alnum:]]/) s = s c
    else {
      if(c ~ /\x27/) s = s "\x5Cx27" # single quote
      else if(c ~ /\x25/) s = s "\x5Cx25" # %
      else s = s "\x5Cx" hex(c)
    }
  }
  return s
}
{
  print qry2ptn($0)
}

Example run

Code: [Select]

$ printf '\x21\x22\x23\x24\x25\x26\x27\x28\x29\x2A\x2B\x2C\x2D\x2E\x2F\x3A\x3B\x3C\x3D\x3E\x3F\x40\x5B\x5C\x5D\x5E\x5F\x60\n' > ptn
$ cat ptn
!"#$%&'()*+,-./:;<=>?@[\]^_`
$ ./hex.awk ptn
\x21\x22\x23\x24\x25\x26\x27\x28\x29\x2A\x2B\x2C\x2D\x2E\x2F\x3A\x3B\x3C\x3D\x3E\x3F\x40\x5B\x5C\x5D\x5E\x5F\x60

The only thing that needs adjusting is the leading "^" and the trailing "$" that are defined by your script for special purpose.

[Rich]

Hi polikuo
Thanks, but most non-alphanumeric don't even need to be escaped.

As I said in reply #36:

That was the plan:
  Use a / to mark the beginning of the search term if desired.
  Type in your search term.
  Use a $ sign to mark the end of the search term if desired.
  Hit enter.

I am:
1. Trying to simplify the search so you don't need to enter or learn
   escape or quoting techniques to find what you're looking for.
2. Make the search return a tiny list instead of hundreds of results.

This version of provides.sh can accept the following characters in the command line:

Code: [Select]

! # % + , - . / : = ? @ [ ] ^ _ { } ~  and blank spacesMost appear in various provides.db files.
All have been tested.

I have not found the following in any provides.db files:

Code: [Select]

< > & * ; ' " `The shell will try to act on them and may do bad things.

The shell will try to act on these but I found them in TC10 x86 provides.db file:

Code: [Select]

( )
It is a short list and you don't need ( or ) to find any of them:

Code: [Select]

tc@E310:~/Scripting/Provides$ grep '(' /etc/sysconfig/tcedir/provides.db
usr/local/share/gtk-doc/html/cairo/cairo-Quartz-(CGFont)-Fonts.html
/usr/local/lib/deadbeef/convpresets/AAC_(Nero_FAAC).txt
/usr/local/lib/deadbeef/convpresets/FLAC_(compression_level_5).txt
/usr/local/lib/deadbeef/convpresets/MP3_CBR_320_Kbps_(Lame).txt
/usr/local/lib/deadbeef/convpresets/MP3_VBR_192Kbps_(Lame).txt
/usr/local/lib/deadbeef/convpresets/Ogg_Vorbis_(-q_5).txt
/usr/local/share/vim/vim82/lang/menu_chinese(gb)_gb.936.vim
/usr/local/share/vim/vim82/lang/menu_chinese(taiwan)_taiwan.950.vim
/usr/local/share/mypaint-data/1.0/brushes/kaerhon_v1/smudge_ink(0.7)_sm.myb
/usr/local/share/mypaint-data/1.0/brushes/kaerhon_v1/smudge_ink(0.7)_sm_prev.png
/usr/local/lib/python3.6/site-packages/setuptools/script (dev).tmpl
/usr/local/lib/python2.7/site-packages/setuptools-39.0.1-py2.7.egg/setuptools/script (dev).tmpl
usr/local/share/vim/vim72/lang/menu_chinese(gb)_gb.936.vim
usr/local/share/vim/vim72/lang/menu_chinese(taiwan)_taiwan.950.vim
usr/local/share/gtk-doc/html/libxfce4panel-1.0/libxfce4panel-Panel-Plugin-Register-Macros-(4.6-Style).html
tc@E310:~/Scripting/Provides$

... The only thing that needs adjusting is the leading "^" ...

There is no leading ^ in this type of search. All filenames include a path, so
to anchor the beginning, just add a leading slash.

Let's say you're looking for program called apps:

Code: [Select]

tc@E310:~/Scripting/Provides$ ./provides.sh apps | wc -l
95That's way too many. To narrow the search, you add an anchor:

Code: [Select]

tc@E310:~/Scripting/Provides$ ./provides.sh /apps | wc -l
78That narrowed it some, but you know it's a program, so you extend the anchor:

Code: [Select]

tc@E310:~/Scripting/Provides$ ./provides.sh bin/apps
Xprogs.tcz

[GNUser]

Hi Rich. Thanks for sharing this improved version. I have put it on all my TCL boxes

It would be helpful if the Usage() function also outputted provides.sh's version number and/or release date.

[Rich]

Hi GNUser
I've been meaning to add that. Now that it's pretty much
stabilized, I'll do that.

[Rich]

Hi GNUser
After further testing I added the period to the list of
punctuation I'm escaping.

Version information has also been added:

Code: [Select]

tc@E310:~/Scripting/Provides$ ./provides.sh

Version 0.3 Nov 25, 2024

Find extension(s) that provide a filename.
Filenames in list being searched include full paths, for example:
        usr/local/bin/grep

Usage:
   provides.sh [ -nz ] FileName

   -nz    Skip updating (zsync) the provides.db file. This speeds up
          the search, but might miss items if provides.db is outdated.

Examples:
   provides.sh cal           Finds cal anywhere in FileName
   provides.sh bin/cal       Finds bin/cal anywhere in FileName
   provides.sh bin/cal$      Finds FileName that ends in bin/cal
   provides.sh Black Gnome   Finds FileName with embedded spaces

Searches are case sensitive.
The $ sign can only be used at the end of your search term.

tc@E310:~/Scripting/Provides$
The updated version is attached. The previous version will be removed.

[GNUser]

Hi Rich. It looks great! Thanks again for adding all this polish to this script.

[patrikg]

Sorry for not getting the performance working with sqlite3 ---
Tried to get it working so the command line should report the performance.

I can't getting it working with the ".timer on" with the command line version of sqlite3.
The ".timer on" doesn't report the speed sadly when executing from the command line.

But if you get into the interactive mode of sqlite3 it just works so you get a report how many seconds the question took.

It says in "man" you could supply the -cmd ".timer on" but it doesn't work for me, but some another commands like -cmd ".help" works out of the box.

And if "you" choose sqlite3 you could also store some more metadata like version, date and so on.

Maybe also store ldd output of the executable, so you can easily see the dependency's.

Now when i looked at it again, i see that sqlite need the select command to stdin, so
if i make the command like this i get the performance with sqlite3.
I think i can be useful for another to see it can be done.

So the search.sh file should be like this:

Code: [Select]

echo "SELECT * FROM tcz_files WHERE file LIKE '%$1%';" | sqlite3 -cmd ".timer on" -box files.db
make.sh

Code: [Select]

cat << 'EOF' > make.sh
#!/bin/bash
sqlite3 files.db "CREATE TABLE tcz_files (name TEXT NOT NULL,file TEXT NOT NULL);"
for filename in $(ls *.tcz)
do
   tczfile=$filename;for file in $(unsquashfs -lc $tczfile); do sqlite3 files.db "INSERT INTO tcz_files ( NAME , FILE ) VALUES ( '$tczfile' , '$(basename $file)' );" ; done
done
EOF

chmod u+x make.sh
search.sh

Code: [Select]

cat << 'EOF' > search.sh
#!/bin/bash
echo "SELECT * FROM tcz_files WHERE file LIKE '%$1%';" | sqlite3 -cmd ".timer on" -box files.db
EOF

chmod u+x search.sh
PS, so I take back what I said before it's sqlite bug, I have to read RTFM, to get it

Happy coding !!!