tce/app-browser , sparing of storage or network

[mocore]

thinking about repository and related tools 

using app browser or tce to browse the extention info , request's separatly each .info/ect file viewed

and also afair separate request for each dependent tcz

i have in the past wandered ,..

0) could the time/bandwidth be reduced ?
if repo metadata (as mentioned in the below quote ) and  individual extention metadata files could be
downloaded once then used/viewed localy?.. as apposed to repeatedly requested

i guess this makes less diffrence to those with a local mirror or who know which packages are required

for the case of browseing the apps and searching/exploring posilities before testing ( if the build works at all / has required dep's ect )
waiting for many request to compleate takes time , at this point storage and bandwidth are abubdent for the most part , time is however constrained constantly

1) when downloading extentions & dependencies  could the initial conection be reused ??!!


which rased the question *exactly* what version of http is suported by tce/app-browser / tce-fetch.sh /wget and or the by the server
as this is sort of an implicit dependency of tc operation

one imho woth some consideration
and which seem a fair question(s) considering tc philosophies
 http://tinycorelinux.net/concepts.html - 'Easy, fast, and simple renew-ability and stability is a principle goal of Tiny Core.'
this appears to fall under speed consideration

i wander if anyone else has given this any thaught ( dont remenber finding any post with similar jist in the past )

all though i except ( statisticly) nothing has explicitly changed since 1.x ( apart from perhaps the server to nginx https://forum.tinycorelinux.net/index.php/topic,20499.0.html )
so it is statistically unlikely to do so in the future 

@mocore: I'm not certain the purpose you're after, but we already have a number of text based files found in the repo itself which may suit your purpose.
Bare in mind they're a part of the repository as opposed to the forum.

tcz/info.lst is basically a file listing of TCZs
tcz/sizelist is the above plus file lengths
tcz/tags.db is a g-zipped version of info plus tag words
tcz/provides.db is similar to an INI file with file listings for each extension
tcz/md5.db is, as you guessed it, the signatures for each file
^{* most of the items above are available in g-zip form, some are gz only.}

[mocore]

some similar/relevant observations
wrt network connections ect ... apparently inspired an alternative backed  for the core package manager
( at least that's the gist i get from scanning the thread )

http://forum.tinycorelinux.net/index.php/topic,12458.0.html - "corepkg - a new core package and updates manager"

to match all current md5 files against the ones in the repository, which generates a LOT of small web server requests to the repository and can take a long time, depending on how many extensions you have,

[nick65go]

@mocore: I'm inclined to think like you.

To change a little the main scripts, like tce-fetch.sh for (tags.db.gz + provides.db.gz + sizelist.gz + md5.db.gz), such as it will first check (for example) and if the files are ALREADY in /tmp then it will not download them again 
 (eventually check also they date, hm.. not sure why these files could not-authorized arrived in  /tmp, so maybe not-necessary to touch/check they date).

[rant] Time is vital in our life! because is finit on this planet. Storage space not so. So speed (time involved) is the focus, for me. Even [hated] M$soft at one moment break-it with DOS compatibility, from win 9.x to advance in winNT.

I am not suggesting to break tcz compatibility regarding running locally on host machine/PC/laptop; however some tcz do not fit with core 486 CPU compatibility (see firefox which needs CPU with SSE instructions, etc).
But TC started in 2009, we are now in 2023, so servers (or their MIRRORS) storage and protocols advanced in the mean time. The assumptions today are not like they were 14 years ago. [/rant]

[nick65go]

looking in /usr/bin/tce-ab, it download specific *.tcz.tree or *.tcz.dep for each on-demand; But on the server there is no file to contain all / concatenated  *.tcz.tree or *.tcz.dep;

So, when the used look-around in tce or it app-GUI, lets say for 100 individual programs (because the user just evaluate the TC software repository potential), the user will download (connect + disconnect) to server for 100 times (with wget).

I re-discovered (LOL, some will say: the cold water, ) a small program in TC14 also:

Code: [Select]

Title:          tdb.tcz
Description:    trivial database
Version:        1.2.12
Size: 48KB
Extension_by:   juanito
                ----------
Change-log:     first version
Current:        2013/11/21
I wonder (play with it) if is faster /more usefully instead of classic tce (awk, sed, grep), to process a local database, created from downloaded server files (tags.db.gz + provides.db.gz + sizelist.gz + md5.db.gz). Anybody can share their experience of tcz alternative-management?

[GNUser]

Anybody can share their experience of tcz alternative-management?

My tcz alternative for large applications that I only use rarely (e.g., libreoffice, gimp, qbittorrent, electrum) is AppImage. AppImages work perfectly, load quickly, and add nothing to boot time.

Two caveats:
1. Most AppImages these days are 64-bit only, so 64-bit TCL is needed
2. A few minor tweaks are needed to allow AppImages to work on TCL (see here)

[nick65go]

@GNUser: Thanks, good tip. But now you must trust TC and AppImage.
But.. you do not use TC applications. I was asking for something like scripts, to search TCZ, for example:
- show me all apps maintained by GNUuser, when I run from aterm
- list all tcz which depend on fltk-1.3.tcz, when nothing is installed (just kernel + busy-box).
PS:
- for big apps, like ex: ffmpeg (used by firefox), I have a big ffmpeg-all.tcz (containing all its dependencies, and it will populate also /usr/local/tc-local/ with null dep loaded, etc)

- for new versions of (lets say) VLC, I chroot and un-squash an Alpinelinux-VLC.sqfs, but usually I have also a partition with full Alpinelinux installed in its root.

[GNUser]

@GNUser: Thanks, good tip. But now you must trust TC and AppImage.
But.. you do not use TC applications. I was asking for something like scripts, to search TCZ, for example...

Hi nick65go. A few thoughts about this.

If you download AppImage directly from the developer, there is actually less trust needed on your part. If you were to use electrum.tcz (it is not available in the repo, but let's pretend) you would have to trust the electrum developers and the TCL contributor who put together the extension. This is a sensitive application because it manages one's bitcoin wallet. Downloading AppImage directly from developer eliminates the need to trust a middle man creating the TCL extension.

I actually do use TC extensions for most applications. I'm comfortable trusting the extension contributors in vast majority of cases.

I'm not sure how to search for extensions by maintainer (short of having your own TCL repo and grepping through the .info files). If you discover a way to do that, please share.

Your approach for big apps is ingenious but sounds labor-intensive. I'm too lazy to do something like create ffmpeg-all.tcz (BTW, there is a statically-linked--i.e., no dependencies--version of ffmpeg. It's available here.)

[nick65go]

@GNUser: Each with their paranoia security level mine is big I think.

1. G: "grepping through the .info files)"
-  so that I propose to the server admins small / insignificant sized files like tcz.info.db.gz, tcz.dep.db.gz, tcz.tree.db.gz etc, to allow users to build/query their own local database with META data for tcz.

2. G: "Your approach for big apps is ingenious but sounds labor-intensive."
- you only do it one time in life (OK, maybe few times), and only for few (let's say 10) tczs, like Xorg-3D, gtk3, ffmpeg. ex:

Code: [Select]

for i in $your_list; do unshashfs -F $i /tmp/x; done. So all tcz into same /tmp/x folder. Your list is seen from cpanel /stat, to see all loop AFTER all mandatory Xorg loops already loaded. Do not forget to populate /usr/local/tclocal with fake null dep and have a start script named like your extension (ex: gnumeric needs this script for gkt-icons/menu/schema init).

3. G:"there is a statically-linked--i.e., no dependencies--version of ffmpeg"
- No, all built-in libs will/ are already loaded by Xorg or other apps later, so waste of RAM.

[GNUser]

n: "I propose to the server admins small / insignificant sized files like tcz.info.db.gz..."
That's an excellent idea. It sounds like curaga is receptive to these files existing on the server. If you have the time and inclination, please build the search functionality into tce (CLI extension explorer). I think it would be a significant contribution to the distro.

n: "foo-all.tcz"
It is a very interesting idea. I do care about boot time, but not that much. I may give your idea a shot sometime.

n: "waste of RAM"
Good point. That's one of the drawbacks of all portable packaging strategies, whether AppImage or statically-linked binary or whatever else. Convenience always comes at a price.

[curaga]

Info and tree combined files would have less of use, IMHO.

[nick65go]

Info and tree combined files would have less of use, IMHO.

Well, in the absence of a good html page (as was with tc9), the contactenation of *.tcz,info into sothing like tcz.info.db.gz will allow to search for few criteria/fields, like Version:, Current: Extension_by: Comments:;
Right, it is not for functionality in TC, but skimming on version, meta data.
I think the size will be small, but could you give a size for a tc13_x86 sum(*.tcz.info.) and its gzip size? just to see its impact for server. Thanks.

[Rich]

Hi nick65go

... I think the size will be small, but could you give a size for a tc13_x86 sum(*.tcz.info.) ...

The sum of the apparent sizes is 3801251 bytes.

[nick65go]

Rich: "The sum of the apparent sizes is 3801251 bytes." sum(*.tcz.info.) ...
IF not already gzip-ed these 3.8MB, then will be 390 KB gzip-ed, as statistically after compression remain at 10 %. 
And... the app-GUi, or tce-ab needs an improvement to use it to search by field criteria.
So we can happy spy the [GN ]user nuggets contributions, which we/some uses. Frankly, I first looked for/ at curaga (and others TC founders) for small / nice gold nuggets before I jump to fat-equivalents.

[curaga]

That's the thing, I expect searching by submitter to be a very rare use case.

[mocore]

tree combined files would have less of use,

this makes me wander ..
...how are the .tree files created ? ( is/colud the script be published) ?

i thik iv read they are created by combining the .dep files ? *somehow*

given that now dep.db.gz exists
i guess with a/the mk_tree script it would be possible to (dynamicly)create .tree from dep.db ??...


i think tcz.info.db.gz or repo-info.tcz (all .info) or even! repo-meta.tcz collecting  all the .gz files ( could be easily scripted )
 would be nice/convenient  to have  all the repo meta data (&  perhaps meta data creation scripts!! ) accessible to tc system / user .