Proposal: new file dep.db.gz on tc server, having all *.tcz.dep inside

[nick65go]

On any tc server, or its mirrors, we have the "core-index" files: tags.db.gz + provides.db.gz + sizelist.gz
These provide functionality to tce-ab and its app-GUI, for tcz management.

I propose to have also a file, named like dep.db.gz on tc server, consisting of concatenation of all *.tcz.dep

It is a simple task, very fast to do it; this task will not very often run on server, because very seldom a new tcz version will add a new dep. Also the size of this dep.db.gz file will be insignificant, so is not a burden for the server maintenance.

A new proposal could be also to enhance the app-GUI to do the same (which tcz ask for a given Y.tcz ?), similar to TAG selection), IF the dep.db.gz proposal is approved/ implemented.


Motivation:
In most linux distributions (Debian, Archlinux, Alpinelinux, just to name some of them) the user could browse the repository for any package/sub-package (these are like tcz extensions) and find out who is using a specific package.

sample: poppler.tcz is needed by whom, what tcz ask for it (will not run without it)?

according to this link
http://forum.tinycorelinux.net/index.php/topic,26121.msg167760.html#msg167760
this task can NOT be done by a common user, because the user needs to download all *.tcz.dep from the server/mirrors, and this action is rude/impolite to server resources.

[nick65go]

Just an example why will not be a burden on the server, using the venerable/stable TC13:

Code: [Select]

Index of /tinycorelinux/13.x/x86/tcz/:  15,668 files
Search ".tcz.tree" (1773 hits) size=22,411,412 Bytes
Search ".tcz.dep"  (1773 hits) size=   668,732 Bytes
gzip compression for already existing provides.db (14,611,683 bytes) as provides.db.gz (1,487,225 bytes) means ratio 10.18%. the bigger the text files sum(*.tcz.tree) the better the compression, but let be conservative, same 11%. these means that 
- a new tcz.tree.db.gz will be =~ 2,281,107 bytes [aka 2,227 KB, or 2,17 MB]
- a new tcz.dep.db.gz will be =~ 65,306 bytes [aka 63.77 KB, or 0.062 MB]
and so on for *.tcz.info.db.gz

[curaga]

OK. 14.x x86* now have this file. It uses the same format as provides.db, with the filename first and then deps if any.

For a "depends-on.sh" and corresponding Apps/tce-ab support, patches welcome.

[nick65go]

@curaga: Thank you!
Good to see the inception of tcz.###.db.gz as accepted files on the server, for meta data query.
I will start to think/build/patch some scripts, with my rusty skills. All other/ experts script-writes are welcome to compete  ; I am not afraid to get the bottom fame position.

[Rich]

Hi curaga

... For a "depends-on.sh" and corresponding Apps/tce-ab support, patches welcome.

I've attached a first cut of  depends-on.sh.

[curaga]

Rich, that looks good. Please commit it to core-scripts.

[Rich]

Hi curaga
OK, I'm done fighting with github. The procedure I used in July of 2020 no
longer worked. When i went to push, I received a message that passwords
were no longer being accepted. So I created a "personal access token" and
tried again:

Code: [Select]

tc@E310:~/TC-Git$ git clone https://github.com/rarost/Core-scripts.git
Cloning into 'Core-scripts'...
remote: Enumerating objects: 1033, done.
remote: Counting objects: 100% (169/169), done.
remote: Compressing objects: 100% (93/93), done.
remote: Total 1033 (delta 81), reused 135 (delta 74), pack-reused 864
Receiving objects: 100% (1033/1033), 175.10 KiB | 0 bytes/s, done.
Resolving deltas: 100% (517/517), done.
Checking connectivity... done.
tc@E310:~/TC-Git$ cp depends-on.sh Core-scripts/usr/bin/
tc@E310:~/TC-Git$ cd Core-scripts/
tc@E310:~/TC-Git/Core-scripts$ git add -A
tc@E310:~/TC-Git/Core-scripts$ git commit -a
[master 02a01c0] Added depends-on.sh
 1 file changed, 54 insertions(+)
 create mode 100755 usr/bin/depends-on.sh
tc@E310:~/TC-Git/Core-scripts$ git push
Username for 'https://github.com': rarost
Password for 'https://rarost@github.com':
Counting objects: 5, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (5/5), done.
Writing objects: 100% (5/5), 1.08 KiB | 0 bytes/s, done.
Total 5 (delta 2), reused 0 (delta 0)
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To https://github.com/rarost/Core-scripts.git
   3384bfc..02a01c0  master -> master
tc@E310:~/TC-Git/Core-scripts$
I then went to  https://github.com/rarost/Core-scripts
Clicked on  Contribute -> Open pull request
Filled in a comment.
Clicked on  Create pull request

[GNUser]

Hi Rich. That's really nice, thank you for putting it together. I look forward to seeing the script in the base system!

I played around with the script and think I found three small bugs:

1. Running depends-on.sh gtk3
- Expected matches: Only extensions that contain gtk3.tcz in their .dep file
- Actual matches: Some unexpected hits such as blueman.tcz which contain gtk3-gir.tcz (but not gtk3.tcz) in their .dep file

2. Running depends-on.sh iwd or depends-on.sh iwd.tcz
- Expected matches: None, because no packages depend on iwd.tcz
- Actual matches: Extensions that depend on eiwd.tcz

3. Running any depends-on.sh foo or depends-on.sh foo.tcz
- Expected matches: Should not include foo.tcz because it seems silly to say that an extension depends on itself
- Actual matches always include foo.tcz

I attached a little patch that fixes these issues.

[GNUser]

P.S. The patch assumes that dependencies listed in .dep files always include the trailing .tcz. Hopefully that's a safe assumption. If not, then the logic in the patch needs to be reworked.

[Greg Erskine]

Will this do what you are trying to do?

Code: [Select]

EXTN="${EXTN%.tcz}.tcz"

[curaga]

Rich, your pull request succeeded and I merged it.

About the fuzzy match: I kind of expect it to be fuzzy, if I can't be bothered to type the full name. This could be handled with an option, so that both a fuzzy and exact match are available. (which is the default and which is the option doesn't matter to me)

You can check for an option and then use "shift" if it was there, so the search param is $1 in both cases.

edit: dep files always include the .tcz.

[GNUser]

Yes it does, Greg. Thank you. I like your more succinct way.

Hi, curaga. Good to know the dep files always include the .tcz. It simplifies the logic for an exact search.

I'm submitting a pull request now. Fuzzy search is default, -e turns will turn on exact search.

Edit: Rich, if you prefer exact search to be default, let me know and I can use -f for fuzzy and easily flip the logic.

[Rich]

Hi GNUser

Code: [Select]

EXTN="${EXTN%.tcz}.tcz"

That's exactly how I handled it in my  FetchExt.sh  script. 

I'm guessing if someone is searching for which extensions depend on something, they
would probably want an exact match (noise free result) most of the time.

Once again, I'm guessing a search like this would usually be done to assess the impact
of doing something to/with an extension will have on other extensions or how an
installation behaves.

I can see a fuzzy search if you want to include legacy extensions like libpng12 vs libpng, or
poppler vs poppler07 for example. A fuzzy search would also include -bin, -dev, -gir, etc.
versions of these extensions.

For a fuzzy search you should probably strip the  .tcz  from the search term:

Code: [Select]

EXTN="${EXTN%.tcz}"

... Edit: Rich, if you prefer exact search to be default, let me know and I can use -f for fuzzy and easily flip the logic.

OK, short answer, I prefer exact for a default.

[GNUser]

That makes sense, Rich. I'm on it.

[GNUser]

Hi Rich. I created a pull request on GitHub to change  depends-on.sh  as you suggested.

I'm glad it's your preference to default to an exact search. I imagine that I'll be doing an exact search most of the time.