tgrex.pl - tcz/scm full text info search and download tool

[webb]

i made this script to streamline finding and installing extensions.

i have it bash-aliased as tgre (pronounced "tea-gruh") because it's more fun to type than ~/scripts/tgrex.pl

tgre update - downloads all the .info files
tgre x (or tgre search x) - full text search of the info files
tgre info x (or show x) - show's the info for extension x
tgre install x - downloads and loads extension x (and adds filename to tce/afterboot.lst (not terribly useful unless you're loading extensions from afterboot (e.g., in bootlocal.sh)))

todo:
tgre disable x - mv optional/x.* disabled/.
tgre update - also retrieve and save /tmp/tcloop/...bin/* filenames to the .tgrex cache
tgre x - if x would be provided by a currently unloaded extension, download the extension, load it, and run x
tgre why vmlinuz wuz compiled with 4gb ram limit? - jeez i don't know, i thought that was a weird decision too

anyways, the script is admittedly a bit slapdash (you'll see what i mean if you have the misfortune to look at the code), but i use it all the time now, and perhaps some of you may find something useful in it as well.

best,
webb

[tinypoodle]

It may be preferable to store executables in ~/.local/bin which is in $PATH

[bmarkus]

It may be preferable to store executables in ~/.local/bin which is in $PATH

/usr/local/bin would be better

[curaga]

tgre update - downloads all the .info files

Yikes, that's hitting the mirror hard

tgre why vmlinuz wuz compiled with 4gb ram limit? - jeez i don't know, i thought that was a weird decision too

If you have 64-bit hw, you should run a 64-bit kernel. PAE is a hack.

[althalus]

Unless you really REALLY want to be able to do real full text searching, rather than download all the info files and search them yourself, why not just make use of the hard work already put in for tools like tce-ab? They're all shell scripts that you can open up and poke around inside of.

/usr/bin/tce-ab
/usr/bin/provides.sh
/usr/bin/search.sh

[tinypoodle]

... and that without creating a dependency on perl

[mocore]

tgre update - downloads all the .info files

Yikes, that's hitting the mirror hard

all .info files  compressed
  gz or even xz
seem to me at least use full to have locally
i often have found
 trying out this and that extension
 that i have the tcz file locally but not info (to read and get things setup correctly )
 usually when i have no connection

+ it could make app browser and ab faster to browse .info for each app
   if tcz-info.xz was downloaded once
   and the files viewed from the archive localy
   rather than downloaded individual while browsing through the apps
   
 tho that may require  adding dependency of what ever compression was used

 imho , it would be nice to have the option if possible to use this method

 .. tho i geus that having a local mirror of the repo is the alternative option

[webb]

> Unless you really REALLY want to be able to do real full text searching, -althalus
yes, i do! searching by the initial string of a package name won't help me unless i already know the name of the package. how does this help me discover feh? or zsync? how can i see all tinycore font packages? with tgre, use tgre ttf, or if you're compiling something and you get "error: missing x" then usually tgre x, tgre info x', tgre install x' solves it. this has saved me hours.

> provides.sh search.sh -althalus
that is super-cool, i didn't know about those. provides.sh appears to retrieve a list of packages with matching info text via cgi, which will likely be sufficient for most users' needs. still, now that tgrex exists, i think i'll keep it around for awhile i sometimes want to manually regex search my local info files, or read an info file, as dubcore suggests.

> Yikes, that's hitting the mirror hard :p -curaga
um...yes...i meant to ask about that ...

so, it doesn't re-download .info files that it already has. with a persistent home, it takes about two minutes of server time (real 2m0.309s), with an average download speed of 60K/s during that time. updating the first time was more like downloading a coreplus iso. i hope that's not excessive. for non-persistent home, though, this tool is inadvisable for the user or the server.

> and that without creating a dependency on perl -tinypoodle
yes, dependencies are regrettable. i've gotten better since i wrote this last year and am planning to rewrite it in bash and make some improvements. such as tgre disable x, searching by which executables an extension provides, and of course deciding whether ~/.local/bin or /usr/local/bin is better

-webb

ps:

> If you have 64-bit hw, you should run a 64-bit kernel. -curaga
agreed! 3.0.21-tinycore64 won't boot on my 2010 thinkpad. i tried compiling my own 64 bit kernel by piecing together scraps of information from the wiki overview, forums, other sources. i got it to boot, but not to load core.gz (something about a missing zcache or zram kernel module). lucky for me, i then found that 3.0.3-tinycore64 does boot on my hardware.
http://wiki.tinycorelinux.net/wiki:64_bit_compatibility

[althalus]

> Unless you really REALLY want to be able to do real full text searching, -althalus
yes, i do! searching by the initial string of a package name won't help me unless i already know the name of the package. how does this help me discover feh? or zsync? how can i see all tinycore font packages? with tgre, use tgre ttf, or if you're compiling something and you get "error: missing x" then usually tgre x, tgre info x', tgre install x' solves it. this has saved me hours.

Actually, the tags.db that tce-ab (and presumably the gui app browser) search also includes the short description available in the info files, so it's slightly more useful than that.

Code: [Select]

tc@box:~$ search.sh -t image viewer
eog-dev.tcz
eog-doc.tcz
eog-locale.tcz
eog-plugins-locale.tcz
eog-plugins.tcz
eog.tcz
feh-1.3.4.tcz
gqview1-locale.tcz
gqview1.tcz
gqview2.tcz
gthumb-dev.tcz
gthumb-locale.tcz
gthumb.tcz
qiv.tcz
viewnior-doc.tcz
viewnior-locale.tcz
viewnior.tcz
xfi.tcz
xzgv.tcz

(I do get where you're coming from though, even that short description often isn't enough if you're not sure what to search for)

[webb]

# a few comparisons

thinki time search.sh eurofurence
[no results]
real   0m2.315s <-- ouch

thinki time tgre eurofurence
ttf-monofur.tcz <-- sweet monospaced font
real   0m0.234s <-- 10x faster than tc's builtin!

thinki time cscope -L -0 eurofurence # (i just thought of this particular piece of overkill...)
ttf-monofur.tcz.info <global> 14 sans serif font based on the eurofurence
real   0m0.038s <-- 6x faster than mine!!


# more tests:

thinki time provides.sh bin\/chattr$
[no results]
real   0m1.603s

thinki time provides.sh chattr
e2fsprogs.tcz
kdebindings.tcz <-- false positive
kdepimlibs-dev.tcz <-- false positive
real   0m1.487s

thinki time provides chattr
e2fsprogs.tcz: /usr/local/bin/chattr
kdebindings.tcz: usr/local/share/sip/PyKDE4/akonadi/persistentsearchattribute.sip <-- more obviously false positives
kdepimlibs-dev.tcz: usr/local/include/akonadi/persistentsearchattribute.h <---------------
real   0m0.078s <-- 20x faster

thinki time provides bin\/chattr$ <-- regex allowed
e2fsprogs.tcz: /usr/local/bin/chattr <-- shows path
real   0m0.070s

# ps here's my "provides":
grep  -H $1 ~/.tgrex/cache/*.list | sed 's/.*\/\(.*\)\.list:\(.*\)/\1: \2/'
# "depends" is similar

time grep ^chattr$ provides.db <-- dl'd from tc ibiblio server
chattr
real   0m0.010s <-- 7x faster still!, and the fastest result yet! win tc!!


i would be happy to offer up my rather underutilized web server as a tinycore mirror btw to support the project and make up for the added strain my fulltext .info, .list, and .dep syncing causes.

-webb