New package manager

[CNK]

How does it acquire the data to search?  I would presume it downloads all the .info files for all the packages...  Without a server-side search script of some type, it's eating up bandwidth.

For files contained in extensions there's a provides.db.gz file in the repo that's downloaded initially to the tce directory then checked for updates using zsync (which only downloads changed parts) on each run. Check it yourself by running "less /usr/bin/provides.sh". There's also a more sophisticated version of the script available with the provides.tcz extension.

As I examined here, the provides.db.gz file is downloaded initially when that's present on the mirror using Zsync, even though it looks in the script like the uncompressed provides.db file is used.

For keywords there's a tags.db.gz file with all the tags for each extension in the repo, which is downloaded by the search.sh script.

The package manager I just uploaded to the repo can list optional dependencies in the package .dep files (using the format [optional.tcz|installed.tcz]).

OK, but changes to the .dep file format should really be compatible with the tools in the base system, or else this is really still belongs in a fork of Tiny Core Linux.

I see you've already submitted a differently formatted info file for "pax", missing lots of the usual fields and omitting the upper-case letters at the start of fields. I expect this is incompatible with the programs generating these "db" files, hence "pax.tcz" is not in the tags.db.gz file, and therefore doesn't get found when searching with tce-ab. I don't know whether there's an issue with adding fields to the "info" files, but it seems obvious that you should be still including the existing ones in the same way as others. "Title:" should be the extension file name, but you're using "title:" for what should be in the "Description:" field, for example.

"Size:" isn't there in any form, and it's a field I often look for in the extension browser.

There's a switch to install them as well.  I know this won't help you, but it was something I thought about and incorporated.  There's also the ability to install certain files out of a package instead of the entire contents.

It's good to see these new ideas, but for my use I'd want things to still work with existing tools such as tce-ab and tce-update. Especially when it comes to things like the formatting of info files for extensions in the official repo.

[wysiwyg]

There are some files with information:

Code: [Select]

dep.db.gz Lists each extension followed by its direct dependencies.
info.lst Lists which extensions are available in the repo.
md5.db.gz Lists md5sum for each extension.
provides.db Lists which files each extension provides including their paths.
sizelist Lists how big each extension is.
tags.db.gz Lists which tags show up in each extensions .info file.

Hey Rich, thanks for the reply!

So then it basically does download the entire repo's worth of non-software package files.  This does seem very odd to me for several reasons...

First, there is a pinned post at the top of this board saying not to abuse the bandwidth that is generously being donated to the project.  Yet the way these default tools work is by doing just that.  Even with using zsync, that would still fall under that category - at least in my eyes.  Without putting these files somewhere that they get saved, they would have to be re-downloaded every time a device reboots.  Again, I would say that a server-side script is much more efficient in this regard.  One could be crafted using simple grep and sed calls, removing even the need for a database server.  Then a small script could be created to simply call the URL with the proper parameter.  This would involve the addition of something like cURL, but it would be far more efficient (in both storage space and bandwidth) than the current solution.

The second point that stands out to me is the fact that TC takes great strides at being as small as possible, yet to perform this task with the existing package manager, it has to bloat the system with the entire software library.  The fact that TC attempts to achieve this slim profile is actually one of the reasons why I incorporated the ability to install specified files from packages instead of the whole package.

It just seems odd to me that this was the direction that was taken based on the goals of the project.  I'm not complaining, I'm just pointing out these oddities.

[wysiwyg]

The package manager I just uploaded to the repo can list optional dependencies in the package .dep files (using the format [optional.tcz|installed.tcz]).

OK, but changes to the .dep file format should really be compatible with the tools in the base system, or else this is really still belongs in a fork of Tiny Core Linux.

I see you've already submitted a differently formatted info file for "pax", missing lots of the usual fields and omitting the upper-case letters at the start of fields. I expect this is incompatible with the programs generating these "db" files, hence "pax.tcz" is not in the tags.db.gz file, and therefore doesn't get found when searching with tce-ab. I don't know whether there's an issue with adding fields to the "info" files, but it seems obvious that you should be still including the existing ones in the same way as others. "Title:" should be the extension file name, but you're using "title:" for what should be in the "Description:" field, for example.

"Size:" isn't there in any form, and it's a field I often look for in the extension browser.

Hey CNK, thanks for the convo!

Honestly, I didn't even know there was any consistency with those files! lol  I worked on trying to standardize the repo here years ago, but didn't have much help.  As I processed those files I immediately noticed that there were so many inconsistencies that I just made my own.  For instance, not all of them capitalized the words.  Some had missing fields.  Some had extras.  I get it, it's a volunteer project.  I tried to fix this, but it was too large of a task without significant help.

I can go back and update the .info file to match what TC requires.  Is there a template somewhere to use by chance?

In regards to the compatibility with the existing tools and optional dependencies, I can understand your request.  But that functionality isn't present, so there's nothing to be compatible with.  If the tce-* scripts would like to incorporate that form of processing from my package manager, they are more than welcomed to do so - I do not mind.  But I will say that it may involve a little bit of work to incorporate as my package manager does not share the same code as the original tce-* scripts.

There's a switch to install them as well.  I know this won't help you, but it was something I thought about and incorporated.  There's also the ability to install certain files out of a package instead of the entire contents.

It's good to see these new ideas, but for my use I'd want things to still work with existing tools such as tce-ab and tce-update. Especially when it comes to things like the formatting of info files for extensions in the official repo.

Thanks, I agree!  New ideas always propel tech forward.  Currently pax has no ability to search the repo so the tce-ab script would still be required.  I assume that tce-update is used to update existing packages?  If so, this tool has been replaced within pax, although it would presently require 2 commands instead of 1.  A future update will resolve this, as well as add the ability to have this whole process automated if desired.

One of the other desired features was the ability to unload/uninstall packages.  TC currently has no ability to do so, but you can with pax.

[wysiwyg]

To quickly go back to the start of this additional conversation... Part of the reason pax can not search packages is because it looked like it just pulled down the files ondemand.  Based on our conversation, it appears that I was right.  Before trying to add in that feature, I wanted to see about having something server-side - again for efficiency, mostly.  Plus having this would allow for consistency across various Linux distros making pax for universal than trying to incorporate the way it's done in TC (which may be unique to just this distro and non-portable to others).

I do appreciate the feedback from you guys!

[patrikg]

This topic has being discussed before, how our package manager deal with how to determine what files is in what package ?

I suggested a SQLite solution, so you could also add more fields like description size and so on.
But i see the point of what TC is all about, simple and small. And not lot of dependencies, in this SQLite.
 
Let me see on my computer if i have the POC saved ?

Code: [Select]

$ cat make.sh
#!/bin/sh
sqlite3 files.db "CREATE TABLE IF NOT EXISTS tcz_files (name TEXT NOT NULL,file TEXT NOT NULL);"
for filename in $(ls *.tcz)
do
   tczfile=$filename;for file in $(unsquashfs -lc $tczfile); do sqlite3 files.db "INSERT INTO tcz_files ( NAME , FILE ) VALUES ( '$tczfile' , '$(basename $file)' );" ; done
done

Code: [Select]

$ cat search.sh
#!/bin/sh
sqlite3 -cmd ".timer on" -box files.db <<< "SELECT name as Package, file as File FROM tcz_files WHERE file LIKE '%$1%';"
Maybe you could steal my POC and make something about it.

Happy hacking

[CNK]

I can go back and update the .info file to match what TC requires. Is there a template somewhere to use by chance?

This is the example I follow. Also the submitqc script in submitqc.tcz checks the info file for wrong or missing fields, among other things.

In regards to the compatibility with the existing tools and optional dependencies, I can understand your request. But that functionality isn't present, so there's nothing to be compatible with.

I was more concerned that like with the info file you'd submit extensions with dep files in a format that breaks the regular tce* tools. If you have something like separate "extension.tcz.dep.opt" files then that might work better, but otherwise I'd hope the tce* tools get support added for any new formatting features before they're used in the repo.

To quickly go back to the start of this additional conversation... Part of the reason pax can not search packages is because it looked like it just pulled down the files ondemand. Based on our conversation, it appears that I was right.

Yes, but I'm not sure why you said Debian's package system is more "server-side" than on TC. Debian also downloads a full package list to search locally, and it downloads the description and dependency info equivalent to the info and dep files for all packages too. Debian do have a web interface that allows searching in a web browser, but apt-get and other package managers I've used there all locally search the downloaded package lists.

[Rich]

Hi wysiwyg

... So then it basically does download the entire repo's worth of non-software package files. ...

I'm working from memory here, and referring to the Apps GUI:

Code: [Select]

These get loaded for browsing/installing extensions:
info.lst Used to populate the left window of Apps
provides.db Used to populate the  Files  tab of Apps,
sizelist Used when populating the  Size  tab of Apps.

These get loaded for functions under Apps->Maintenance:
dep.db.gz Dependencies And Deletions, Check Onboot Unneeded, Check for Orphans
md5.db.gz Md5 checking, Check for Updates

Used in Search options:
provides.db Search for extensions by provided file
tags.db.gz Search for extensions by listed tags

... First, there is a pinned post at the top of this board saying not to abuse the bandwidth that is generously being donated to the project. ...

I'm not sure which post you are referring to, but that was likely in response
to individuals downloading entire repos.

... Even with using zsync, that would still fall under that category - at least in my eyes.  Without putting these files somewhere that they get saved, they would have to be re-downloaded every time a device reboots. ...

A reboot does not automatically result in a re-download.
sizelist  and  tags.db  are in  /tmp , so they would get re-downloaded if they're needed.
dep.db  and  provides.db  are in the  tce  directory, so they are persistent.

... TC takes great strides at being as small as possible, yet to perform this task with the existing package manager, it has to bloat the system with the entire software library.

I don't understand what you are trying to say there.
How are we bloating the system with the entire software library?

[wysiwyg]

This topic has being discussed before, how our package manager deal with how to determine what files is in what package ?

I suggested a SQLite solution, so you could also add more fields like description size and so on.
But i see the point of what TC is all about, simple and small. And not lot of dependencies, in this SQLite.
 
Let me see on my computer if i have the POC saved ?

Code: [Select]

$ cat make.sh
#!/bin/sh
sqlite3 files.db "CREATE TABLE IF NOT EXISTS tcz_files (name TEXT NOT NULL,file TEXT NOT NULL);"
for filename in $(ls *.tcz)
do
   tczfile=$filename;for file in $(unsquashfs -lc $tczfile); do sqlite3 files.db "INSERT INTO tcz_files ( NAME , FILE ) VALUES ( '$tczfile' , '$(basename $file)' );" ; done
done

Code: [Select]

$ cat search.sh
#!/bin/sh
sqlite3 -cmd ".timer on" -box files.db <<< "SELECT name as Package, file as File FROM tcz_files WHERE file LIKE '%$1%';"
Maybe you could steal my POC and make something about it.

Happy hacking

Hey Patrikg!

Thanks for the code snippets!  It looks like you are trying to use sqlite for a local repository on each client.  I'm actually trying to get an online solution for searching non-local files.  With the files that are already downloaded, they can have their corresponding .info files parsed - at least with pax.  The tce-* scripts require the downloading of the entire non-software package files.  This could be useful for offline devices, but then they couldn't download anything anyways making this a non-point.  The vast majority of devices are connected to the internet making this wasteful.  An online search takes zero space on your local storage while providing you the results you seek, for either local or non-local packages.

[wysiwyg]

I can go back and update the .info file to match what TC requires. Is there a template somewhere to use by chance?

This is the example I follow. Also the submitqc script in submitqc.tcz checks the info file for wrong or missing fields, among other things.

Awesome!  I'll grab that and adjust my .info files to match.

In regards to the compatibility with the existing tools and optional dependencies, I can understand your request. But that functionality isn't present, so there's nothing to be compatible with.

I was more concerned that like with the info file you'd submit extensions with dep files in a format that breaks the regular tce* tools. If you have something like separate "extension.tcz.dep.opt" files then that might work better, but otherwise I'd hope the tce* tools get support added for any new formatting features before they're used in the repo.

Gotcha!  I don't think you'd have to worry about this because it would take modifying the .dep files for various packages in the repo, of which I have no control over.  Nor would I want to do that without approval anyways.  But, if that functionality was added into the tce-* scripts from pax, I'd suggest to adopt the other forms of entries to those files as well (versioning, copying specific files from the package, etc).

To quickly go back to the start of this additional conversation... Part of the reason pax can not search packages is because it looked like it just pulled down the files ondemand. Based on our conversation, it appears that I was right.

Yes, but I'm not sure why you said Debian's package system is more "server-side" than on TC. Debian also downloads a full package list to search locally, and it downloads the description and dependency info equivalent to the info and dep files for all packages too. Debian do have a web interface that allows searching in a web browser, but apt-get and other package managers I've used there all locally search the downloaded package lists.

Hmmm can you point to where this list is?  I checked /var/lib/apt and the files in there only contain hashes to (past?) files, or brief info for installed packages.  I couldn't find anything where the entire repo's package listing is stored locally.  And yes, this is the equivalent to .info and .dep files for already downloaded packages being on the device.  This is not what I'm referring to.  pax can already parse those files for local searches.  For example, if a user wants to know what database servers are available in the online repo, those files are no good.  The current setup that TC has is to download the entire repo's worth of information files and search locally.  An online search would be much more efficient in terms of bandwidth and storage, right?

[wysiwyg]

Hey Rich, thanks for the reply!

... First, there is a pinned post at the top of this board saying not to abuse the bandwidth that is generously being donated to the project. ...

I'm not sure which post you are referring to, but that was likely in response
to individuals downloading entire repos.

Yes, that was the post that I was referring to. But wouldn't the partial download of the entire repo also fall under that category?  If the same results can be achieved without downloading anything, isn't that the most efficient use of bandwidth that is being donated?

... Even with using zsync, that would still fall under that category - at least in my eyes.  Without putting these files somewhere that they get saved, they would have to be re-downloaded every time a device reboots. ...

A reboot does not automatically result in a re-download.
sizelist  and  tags.db  are in  /tmp , so they would get re-downloaded if they're needed.
dep.db  and  provides.db  are in the  tce  directory, so they are persistent.

I understand.  I wasn't sure where the files were being placed.  It was just an observation based on the non-permanent nature of parts of the OS.

... TC takes great strides at being as small as possible, yet to perform this task with the existing package manager, it has to bloat the system with the entire software library.

I don't understand what you are trying to say there.
How are we bloating the system with the entire software library?

So if TC optimizes their packages during compile to be smaller in size (which they do) to shave off bytes per binary, I'd say that's going through effort to have the smallest distro possible.  If a normal user that uses the GUI for the tce-* scripts does any type of search, they are automatically downloading entire parts of the repo.  If it doesn't matter in that instance, why go through the efforts elsewhere to make this as small as possible?

If the search abilities for packages, whether for local or remote, can be accomplished with the same results, and one doesn't add anything to your storage (also saving donated bandwidth) vs one that does, wouldn't that fall in line with other aspects?  Let alone just being a more efficient system overall?

[CNK]

Yes, but I'm not sure why you said Debian's package system is more "server-side" than on TC. Debian also downloads a full package list to search locally, and it downloads the description and dependency info equivalent to the info and dep files for all packages too. Debian do have a web interface that allows searching in a web browser, but apt-get and other package managers I've used there all locally search the downloaded package lists.

Hmmm can you point to where this list is?  I checked /var/lib/apt and the files in there only contain hashes to (past?) files, or brief info for installed packages.  I couldn't find anything where the entire repo's package listing is stored locally.

Checking on a Debian system, they are in /var/lib/apt/lists. Clearly all the packages are in the lists, including those not installed. 30MB total for Debian Bullseye.

Code: [Select]

$ head /var/lib/apt/lists/deb.debian.org_debian_dists_bullseye-updates_main_binary-arm64_Packages
Package: ca-certificates-java
Version: 20190909+deb11u1
Installed-Size: 43
Maintainer: Debian Java Maintainers <pkg-java-maintainers@lists.alioth.debian.org>
Architecture: all
Depends: ca-certificates (>= 20121114), default-jre-headless | java8-runtime-headless, libnss3 (>= 3.12.10-2~)
Description: Common CA certificates (JKS keystore)
Multi-Arch: foreign
Description-md5: 304cd3554728e5d076f8ecbb3b5057d8
Tag: protocol::ssl, role::app-data, security::authentication,

$ aptitude show ca-certificates-java | grep ^State:
State: not installed

The lists are downloaded from URLs like http://ftp.debian.org/debian/dists/bullseye/main/binary-arm64/Packages.gz

An online search would be much more efficient in terms of bandwidth and storage, right?

Yes but Linux mirror sites only host static content, so instead of having the load spread over various mirrors it would all be on the site hosting the search, and that would be a single point of failure (although I guess a package manager could fall back to downloading the lists the old way from a static mirror if it went down).

Also, keep the sizes in perspective. That tags.db.gz file tce-ab downloads in order to search extension names and tags/keywords is about 30KB. One thread page on these forums is bigger than that. The provides.db.gz file is bigger at 2MB, but it gets cached. The extra server CPU load from getting hit by lots of search requests (these days probably including lots of bots just trying to hack or crash it, like I've experienced on my own website) would probably be more of an issue than transferring that little extra data.

[wysiwyg]

Hey CNK, thanks for the reply!

Yes, but I'm not sure why you said Debian's package system is more "server-side" than on TC. Debian also downloads a full package list to search locally, and it downloads the description and dependency info equivalent to the info and dep files for all packages too. Debian do have a web interface that allows searching in a web browser, but apt-get and other package managers I've used there all locally search the downloaded package lists.

Hmmm can you point to where this list is?  I checked /var/lib/apt and the files in there only contain hashes to (past?) files, or brief info for installed packages.  I couldn't find anything where the entire repo's package listing is stored locally.

Checking on a Debian system, they are in /var/lib/apt/lists. Clearly all the packages are in the lists, including those not installed. 30MB total for Debian Bullseye.

Code: [Select]

$ head /var/lib/apt/lists/deb.debian.org_debian_dists_bullseye-updates_main_binary-arm64_Packages
Package: ca-certificates-java
Version: 20190909+deb11u1
Installed-Size: 43
Maintainer: Debian Java Maintainers <pkg-java-maintainers@lists.alioth.debian.org>
Architecture: all
Depends: ca-certificates (>= 20121114), default-jre-headless | java8-runtime-headless, libnss3 (>= 3.12.10-2~)
Description: Common CA certificates (JKS keystore)
Multi-Arch: foreign
Description-md5: 304cd3554728e5d076f8ecbb3b5057d8
Tag: protocol::ssl, role::app-data, security::authentication,

$ aptitude show ca-certificates-java | grep ^State:
State: not installed

The lists are downloaded from URLs like http://ftp.debian.org/debian/dists/bullseye/main/binary-arm64/Packages.gz

Well, I stand corrected!  Oddly enough, I renamed the /var/lib/apt directory to have a .org extension, then ran apt-cache search and it still populated with results.  I disconnected from the Internet, and it still gave results!  I'm not sure how that was possible if the local database was unreachable, as well as the Internet...

An online search would be much more efficient in terms of bandwidth and storage, right?

Yes but Linux mirror sites only host static content, so instead of having the load spread over various mirrors it would all be on the site hosting the search, and that would be a single point of failure (although I guess a package manager could fall back to downloading the lists the old way from a static mirror if it went down).

Also, keep the sizes in perspective. That tags.db.gz file tce-ab downloads in order to search extension names and tags/keywords is about 30KB. One thread page on these forums is bigger than that. The provides.db.gz file is bigger at 2MB, but it gets cached. The extra server CPU load from getting hit by lots of search requests (these days probably including lots of bots just trying to hack or crash it, like I've experienced on my own website) would probably be more of an issue than transferring that little extra data.

If they only host static content, then yes you are setting up a single point of failure.  However, depending on the hosting company, they will have numerous mirrors themselves to prevent just such an issue.  Now that does not preclude an issue with the server itself having a problem...

I agree that tags.db.gz file is probably not too big of an issue.  But multiply the provides.db.gz by just 100 users and that's already at 200mb.  That's a lot of searching that can be done to meet or exceed that using an online script.  I'm quite frankly surprised that Debian does this.  There's no telling what their bandwidth bill is at the end of every month lol

I guess I'll throw in the ability to process searches this way too.  It seems like a giant waste of resources (more so for someone like Debian), but I'm not footing the bill so...

[nick65go]

I followed this discussion with interest, tce* scripts are good, no perfect! For me, all is just an intellectual challenge, but not a BIG win in any direction-solution developers will chose to pursuit.
For "modern" hardware (multi-core CPU, SDD or Nvram storage, fast 100MB..1GB network links, etc) is .. not worthy. So the "new" pax package could have a meaningful gain only for outdated surviving hardware like 486 CPU, with small HDD storage, small 128MB RAM, 10MB NIC card etc.
Because, as Rich said 0.2 .. 2 seconds is enough to load all "non-software" repo metadata. Looking into https://mirrors.dotsrc.org/tinycorelinux/17.x/x86_64/tcz/ I see provides.db.gz =3.1 MB, but provides.db.zsync = 175KB!!

1. Using TC on USB, users will not premature destroy their pen/stick with too many updates.
2. Using TC on HDD, will have permanent /tce/optional folder (aka static on HDD, sometime even cache in RAM for a session), and seldom using zsync for 175KB extra. 

From my past experience, many TCZ extension are not often updated, because new versions have bigger size. But tiny is about just core (not very useful by itself without apps), not the full system as used by most users. It seams that TC developers think that users initially search for apps, then users settle with some apps and updates seldom.

In summary the idea to use a new package management revolves in replacing wget (from busybox)  with SQLite, or maybe CURL+dependencies, by increasing the TINY base/core.

[Leee]

It seams that TC developers think that users initially search for apps, then users settle with some apps and updates seldom.

That's pretty much what I do.  I have recently started doing occasional extension updates but I haven't exactly made a habit of it.

[wysiwyg]

Hey Nick!  Thanks for adding to the discussion!

I followed this discussion with interest, tce* scripts are good, no perfect! For me, all is just an intellectual challenge, but not a BIG win in any direction-solution developers will chose to pursuit.
For "modern" hardware (multi-core CPU, SDD or Nvram storage, fast 100MB..1GB network links, etc) is .. not worthy. So the "new" pax package could have a meaningful gain only for outdated surviving hardware like 486 CPU, with small HDD storage, small 128MB RAM, 10MB NIC card etc.

I agree that the TC developers are most likely going to leave the tce-* scripts as is, which is fine. They are a barebones way to accomplish their task.  But to say that pax is only meaningful for older hardware, would be incorrect.  pax has a much larger feature set than the default tce-* scripts, including one that has been requested for many years - the ability to unload software.  I would invite you to read the .info file associated with the project to see the abilities it has over the default manager(s).

http://tinycorelinux.net/16.x/x86/tcz/pax.tcz.info

Which is actually the original version.  I sent the latest version at the end of last week that hasn't been processed yet.  There isn't much change by way of code, but it does provide an in-depth man page to further explain some of the features.  Of course, feedback is always welcome to make it better if something is unclear.

Because, as Rich said 0.2 .. 2 seconds is enough to load all "non-software" repo metadata. Looking into https://mirrors.dotsrc.org/tinycorelinux/17.x/x86_64/tcz/ I see provides.db.gz =3.1 MB, but provides.db.zsync = 175KB!!

1. Using TC on USB, users will not premature destroy their pen/stick with too many updates.
2. Using TC on HDD, will have permanent /tce/optional folder (aka static on HDD, sometime even cache in RAM for a session), and seldom using zsync for 175KB extra. 

From my past experience, many TCZ extension are not often updated, because new versions have bigger size. But tiny is about just core (not very useful by itself without apps), not the full system as used by most users. It seams that TC developers think that users initially search for apps, then users settle with some apps and updates seldom.

My original gripe was more so about being efficient with things.  I did also think that the way TC was handling it was an exception, not the rule.  And yes, 3.1MB is not very much in size.  But when 10 people use it, that becomes 31MB, when 100 people use it is becomes 310MB, when 1000 people use it is becomes 3.1GB, etc.  And remember that this is donated bandwidth.  I'll still add in the ability to search using an online URL, but I'll also have to expand that to include entire repo db's to work across various distros.

In summary the idea to use a new package management revolves in replacing wget (from busybox)  with SQLite, or maybe CURL+dependencies, by increasing the TINY base/core.

I'm not sure what advantage that would give you.  You would still have to download the entire database to your local device, defeating the efficiency I was aiming for.  And the reason wget was chosen is because with basic package management, you only need to download packages, which wget works very well at - plus its part of busybox so no additional dependencies.  cURL adds another dependency to the mix, adding extra size to the base - of which TC tries to keep as small as possible.  I will attempt to use wget to handle the searching side, but I may have to revert to cURL.  But my design goals are different than TC.  I do want small and efficient, but not at the expense of traditional features.