Tiny Core Extensions > TCE Talk
New package manager
CNK:
--- Quote from: wysiwyg on January 28, 2026, 07:26:22 PM ---How does it acquire the data to search? I would presume it downloads all the .info files for all the packages... Without a server-side search script of some type, it's eating up bandwidth.
--- End quote ---
For files contained in extensions there's a provides.db.gz file in the repo that's downloaded initially to the tce directory then checked for updates using zsync (which only downloads changed parts) on each run. Check it yourself by running "less /usr/bin/provides.sh". There's also a more sophisticated version of the script available with the provides.tcz extension.
As I examined here, the provides.db.gz file is downloaded initially when that's present on the mirror using Zsync, even though it looks in the script like the uncompressed provides.db file is used.
For keywords there's a tags.db.gz file with all the tags for each extension in the repo, which is downloaded by the search.sh script.
--- Quote from: wysiwyg on January 28, 2026, 07:26:22 PM ---The package manager I just uploaded to the repo can list optional dependencies in the package .dep files (using the format [optional.tcz|installed.tcz]).
--- End quote ---
OK, but changes to the .dep file format should really be compatible with the tools in the base system, or else this is really still belongs in a fork of Tiny Core Linux.
I see you've already submitted a differently formatted info file for "pax", missing lots of the usual fields and omitting the upper-case letters at the start of fields. I expect this is incompatible with the programs generating these "db" files, hence "pax.tcz" is not in the tags.db.gz file, and therefore doesn't get found when searching with tce-ab. I don't know whether there's an issue with adding fields to the "info" files, but it seems obvious that you should be still including the existing ones in the same way as others. "Title:" should be the extension file name, but you're using "title:" for what should be in the "Description:" field, for example.
"Size:" isn't there in any form, and it's a field I often look for in the extension browser.
--- Quote from: wysiwyg on January 28, 2026, 07:26:22 PM ---There's a switch to install them as well. I know this won't help you, but it was something I thought about and incorporated. There's also the ability to install certain files out of a package instead of the entire contents.
--- End quote ---
It's good to see these new ideas, but for my use I'd want things to still work with existing tools such as tce-ab and tce-update. Especially when it comes to things like the formatting of info files for extensions in the official repo.
wysiwyg:
--- Quote from: Rich on January 28, 2026, 08:55:56 PM ---There are some files with information:
--- Code: ---dep.db.gz Lists each extension followed by its direct dependencies.
info.lst Lists which extensions are available in the repo.
md5.db.gz Lists md5sum for each extension.
provides.db Lists which files each extension provides including their paths.
sizelist Lists how big each extension is.
tags.db.gz Lists which tags show up in each extensions .info file.
--- End code ---
--- End quote ---
Hey Rich, thanks for the reply!
So then it basically does download the entire repo's worth of non-software package files. This does seem very odd to me for several reasons...
First, there is a pinned post at the top of this board saying not to abuse the bandwidth that is generously being donated to the project. Yet the way these default tools work is by doing just that. Even with using zsync, that would still fall under that category - at least in my eyes. Without putting these files somewhere that they get saved, they would have to be re-downloaded every time a device reboots. Again, I would say that a server-side script is much more efficient in this regard. One could be crafted using simple grep and sed calls, removing even the need for a database server. Then a small script could be created to simply call the URL with the proper parameter. This would involve the addition of something like cURL, but it would be far more efficient (in both storage space and bandwidth) than the current solution.
The second point that stands out to me is the fact that TC takes great strides at being as small as possible, yet to perform this task with the existing package manager, it has to bloat the system with the entire software library. The fact that TC attempts to achieve this slim profile is actually one of the reasons why I incorporated the ability to install specified files from packages instead of the whole package.
It just seems odd to me that this was the direction that was taken based on the goals of the project. I'm not complaining, I'm just pointing out these oddities.
wysiwyg:
--- Quote from: CNK on January 28, 2026, 08:58:09 PM ---
--- Quote from: wysiwyg on January 28, 2026, 07:26:22 PM ---The package manager I just uploaded to the repo can list optional dependencies in the package .dep files (using the format [optional.tcz|installed.tcz]).
--- End quote ---
OK, but changes to the .dep file format should really be compatible with the tools in the base system, or else this is really still belongs in a fork of Tiny Core Linux.
I see you've already submitted a differently formatted info file for "pax", missing lots of the usual fields and omitting the upper-case letters at the start of fields. I expect this is incompatible with the programs generating these "db" files, hence "pax.tcz" is not in the tags.db.gz file, and therefore doesn't get found when searching with tce-ab. I don't know whether there's an issue with adding fields to the "info" files, but it seems obvious that you should be still including the existing ones in the same way as others. "Title:" should be the extension file name, but you're using "title:" for what should be in the "Description:" field, for example.
"Size:" isn't there in any form, and it's a field I often look for in the extension browser.
--- End quote ---
Hey CNK, thanks for the convo!
Honestly, I didn't even know there was any consistency with those files! lol I worked on trying to standardize the repo here years ago, but didn't have much help. As I processed those files I immediately noticed that there were so many inconsistencies that I just made my own. For instance, not all of them capitalized the words. Some had missing fields. Some had extras. I get it, it's a volunteer project. I tried to fix this, but it was too large of a task without significant help.
I can go back and update the .info file to match what TC requires. Is there a template somewhere to use by chance?
In regards to the compatibility with the existing tools and optional dependencies, I can understand your request. But that functionality isn't present, so there's nothing to be compatible with. If the tce-* scripts would like to incorporate that form of processing from my package manager, they are more than welcomed to do so - I do not mind. But I will say that it may involve a little bit of work to incorporate as my package manager does not share the same code as the original tce-* scripts.
--- Quote from: CNK on January 28, 2026, 08:58:09 PM ---
--- Quote from: wysiwyg on January 28, 2026, 07:26:22 PM ---There's a switch to install them as well. I know this won't help you, but it was something I thought about and incorporated. There's also the ability to install certain files out of a package instead of the entire contents.
--- End quote ---
It's good to see these new ideas, but for my use I'd want things to still work with existing tools such as tce-ab and tce-update. Especially when it comes to things like the formatting of info files for extensions in the official repo.
--- End quote ---
Thanks, I agree! New ideas always propel tech forward. Currently pax has no ability to search the repo so the tce-ab script would still be required. I assume that tce-update is used to update existing packages? If so, this tool has been replaced within pax, although it would presently require 2 commands instead of 1. A future update will resolve this, as well as add the ability to have this whole process automated if desired.
One of the other desired features was the ability to unload/uninstall packages. TC currently has no ability to do so, but you can with pax.
wysiwyg:
To quickly go back to the start of this additional conversation... Part of the reason pax can not search packages is because it looked like it just pulled down the files ondemand. Based on our conversation, it appears that I was right. Before trying to add in that feature, I wanted to see about having something server-side - again for efficiency, mostly. Plus having this would allow for consistency across various Linux distros making pax for universal than trying to incorporate the way it's done in TC (which may be unique to just this distro and non-portable to others).
I do appreciate the feedback from you guys!
patrikg:
This topic has being discussed before, how our package manager deal with how to determine what files is in what package ?
I suggested a SQLite solution, so you could also add more fields like description size and so on.
But i see the point of what TC is all about, simple and small. And not lot of dependencies, in this SQLite.
Let me see on my computer if i have the POC saved ?
--- Code: ---$ cat make.sh
#!/bin/sh
sqlite3 files.db "CREATE TABLE IF NOT EXISTS tcz_files (name TEXT NOT NULL,file TEXT NOT NULL);"
for filename in $(ls *.tcz)
do
tczfile=$filename;for file in $(unsquashfs -lc $tczfile); do sqlite3 files.db "INSERT INTO tcz_files ( NAME , FILE ) VALUES ( '$tczfile' , '$(basename $file)' );" ; done
done
--- End code ---
--- Code: ---$ cat search.sh
#!/bin/sh
sqlite3 -cmd ".timer on" -box files.db <<< "SELECT name as Package, file as File FROM tcz_files WHERE file LIKE '%$1%';"
--- End code ---
Maybe you could steal my POC and make something about it.
Happy hacking :)
Navigation
[0] Message Index
[#] Next page
[*] Previous page
Go to full version