Tiny Core Linux
Tiny Core Extensions => TCE Talk => Topic started by: wysiwyg on January 22, 2026, 02:40:58 PM
-
I have recently updated the package manager used in a forked distro of TC and wanted to add it to the repo for TC since it has many more features than the tce-* equivalents. Reading through the instructions here:
https://forum.tinycorelinux.net/index.php/topic,2072.msg11002.html#msg11002
says that one of the final steps in the use the bcrypt binary to encode it so that the attachments go through to the gmail address listed in that document. Unfortunately it does not appear that bcrypt is in the Debian repo any longer. Is there some other encryption to use? Or should this step just be ignored?
Thanks!
-
Hi wysiwyg
Why can't you bcrypt the tar file using Tinycore?
-
Hey Rich, thanks for the reply. I don't even see bcrypt in the repo there either.
-
Hi wysiwyg
It's part of the base system:
tc@E310:~$ ls -l /usr/bin/bcrypt
-rwxr-xr-x 1 root root 14836 Jun 9 2019 /usr/bin/bcrypt
-
Oh so it is!!! Thanks for the tip. I couldn't find it anywhere lol
-
For anyone interested, the package manager is called 'pax'. It should be in the repo as soon as it's processed.
-
Which is the forked distro? I'm just curious =_)
tnx
-
Hey s1ckn3s5!
The distro is called XiniX. It was originally forked back in 2017:
https://forum.tinycorelinux.net/index.php/topic,19366.0.html
It is very dated at this point and would be considered a security nightmare due to the outdated software. I'm in the process of slowly reviving it to the current version of TC, but being a one man show with various projects makes it a slow process. I'm going to be porting several of the projects used with it over to TC as they are being updated over the coming months.
If you or anyone else has any questions about it, it would be best to DM me. This is a forum for TinyCore after all :)
-
I've been notified that the package is now live for the x86 and x86_64 variants of TC. Still waiting on inclusion for the Pi's.
Please provide any feedback for the project - especially any bugs. None are currently known, but if any are found, I can fix them.
-
One final note. Currently there is no GUI for this package, so any usage will have to be done from the command line only. It is discouraged to use the default tce-* scripts and this package manager interchangeably as certain cache files will not get placed when using the tce-* scripts. This can be resolved by calling 'pax -z' beforehand, but it is an extra step that most will forget. If you want to make pax your default manager, run 'pax --install' and it will walk you through the process.
I'm working on the man page for this project now and will include it in the next release. Running 'pax --help' should provide you enough information to successfully use the software though. DM me with any questions you may have.
-
Just an FYI to the TinyCore web team, it appears that the Repo Browser is offline. I get a 404 error when attempting to access it.
Would the team be interested in a server-side that is similar to Debian's? This way package managers could actually search through the packages. I know there is the ability in the native tce GUI, but it is rudimentary and not a search based on keywords or words in descriptions (nor for files).
-
Hi wysiwyg
I just checked. Repo appears to be up.
-
Would the team be interested in a server-side that is similar to Debian's? This way package managers could actually search through the packages. I know there is the ability in the native tce GUI, but it is rudimentary and not a search based on keywords or words in descriptions (nor for files).
The "Provides" search in the terminal tce-ab package manager allows searching for keywords and files. Or run "provides.sh" to search for files.
I find tce-ab already works as well as Debian package managers work for me. The only feature I miss is the provision for Debian packages to have optional dependencies. On TC I have to manually edit .dep files (which isn't well accomodated by TC's extension updater scripts, but at least that's easier than it is with Debian).
-
How does it acquire the data to search? I would presume it downloads all the .info files for all the packages... Without a server-side search script of some type, it's eating up bandwidth.
The package manager I just uploaded to the repo can list optional dependencies in the package .dep files (using the format [optional.tcz|installed.tcz]). There's a switch to install them as well. I know this won't help you, but it was something I thought about and incorporated. There's also the ability to install certain files out of a package instead of the entire contents.
-
Hi wysiwyg
How does it acquire the data to search? ...
There are some files with information:
dep.db.gz Lists each extension followed by its direct dependencies.
info.lst Lists which extensions are available in the repo.
md5.db.gz Lists md5sum for each extension.
provides.db Lists which files each extension provides including their paths.
sizelist Lists how big each extension is.
tags.db.gz Lists which tags show up in each extensions .info file.
Some of these files are also available in a compressed format.
Here's a list:
36592 Jan 28 10:21 dep.db.gz
56002 Jan 28 10:21 info.lst
14221 Jan 28 10:21 info.lst.gz
89232 Jan 28 10:21 md5.db.gz
33378319 Jan 28 10:21 provides.db
3249743 Jan 28 10:21 provides.db.gz
179537 Jan 28 10:21 provides.db.zsync
79429 Jan 28 10:21 sizelist
23267 Jan 28 10:21 sizelist.gz
36370 Jan 28 10:21 tags.db.gz
-
How does it acquire the data to search? I would presume it downloads all the .info files for all the packages... Without a server-side search script of some type, it's eating up bandwidth.
For files contained in extensions there's a provides.db.gz (http://tinycorelinux.net/16.x/x86_64/tcz/provides.db.gz) file in the repo that's downloaded initially to the tce directory then checked for updates using zsync (which only downloads changed parts) on each run. Check it yourself by running "less /usr/bin/provides.sh". There's also a more sophisticated version of the script (https://forum.tinycorelinux.net/index.php/topic,27363.msg176490.html#msg176490) available with the provides.tcz extension.
As I examined here (https://forum.tinycorelinux.net/index.php/topic,26130.msg167812.html#msg167812), the provides.db.gz file is downloaded initially when that's present on the mirror using Zsync, even though it looks in the script like the uncompressed provides.db file is used.
For keywords there's a tags.db.gz (http://tinycorelinux.net/16.x/x86_64/tcz/tags.db.gz) file with all the tags for each extension in the repo, which is downloaded by the search.sh script.
The package manager I just uploaded to the repo can list optional dependencies in the package .dep files (using the format [optional.tcz|installed.tcz]).
OK, but changes to the .dep file format should really be compatible with the tools in the base system, or else this is really still belongs in a fork of Tiny Core Linux.
I see you've already submitted a differently formatted info file (http://tinycorelinux.net/16.x/x86_64/tcz/pax.tcz.info) for "pax", missing lots of the usual fields and omitting the upper-case letters at the start of fields. I expect this is incompatible with the programs generating these "db" files, hence "pax.tcz" is not in the tags.db.gz file, and therefore doesn't get found when searching with tce-ab. I don't know whether there's an issue with adding fields to the "info" files, but it seems obvious that you should be still including the existing ones in the same way as others. "Title:" should be the extension file name, but you're using "title:" for what should be in the "Description:" field, for example.
"Size:" isn't there in any form, and it's a field I often look for in the extension browser.
There's a switch to install them as well. I know this won't help you, but it was something I thought about and incorporated. There's also the ability to install certain files out of a package instead of the entire contents.
It's good to see these new ideas, but for my use I'd want things to still work with existing tools such as tce-ab and tce-update. Especially when it comes to things like the formatting of info files for extensions in the official repo.
-
There are some files with information:
dep.db.gz Lists each extension followed by its direct dependencies.
info.lst Lists which extensions are available in the repo.
md5.db.gz Lists md5sum for each extension.
provides.db Lists which files each extension provides including their paths.
sizelist Lists how big each extension is.
tags.db.gz Lists which tags show up in each extensions .info file.
Hey Rich, thanks for the reply!
So then it basically does download the entire repo's worth of non-software package files. This does seem very odd to me for several reasons...
First, there is a pinned post at the top of this board saying not to abuse the bandwidth that is generously being donated to the project. Yet the way these default tools work is by doing just that. Even with using zsync, that would still fall under that category - at least in my eyes. Without putting these files somewhere that they get saved, they would have to be re-downloaded every time a device reboots. Again, I would say that a server-side script is much more efficient in this regard. One could be crafted using simple grep and sed calls, removing even the need for a database server. Then a small script could be created to simply call the URL with the proper parameter. This would involve the addition of something like cURL, but it would be far more efficient (in both storage space and bandwidth) than the current solution.
The second point that stands out to me is the fact that TC takes great strides at being as small as possible, yet to perform this task with the existing package manager, it has to bloat the system with the entire software library. The fact that TC attempts to achieve this slim profile is actually one of the reasons why I incorporated the ability to install specified files from packages instead of the whole package.
It just seems odd to me that this was the direction that was taken based on the goals of the project. I'm not complaining, I'm just pointing out these oddities.
-
The package manager I just uploaded to the repo can list optional dependencies in the package .dep files (using the format [optional.tcz|installed.tcz]).
OK, but changes to the .dep file format should really be compatible with the tools in the base system, or else this is really still belongs in a fork of Tiny Core Linux.
I see you've already submitted a differently formatted info file (http://tinycorelinux.net/16.x/x86_64/tcz/pax.tcz.info) for "pax", missing lots of the usual fields and omitting the upper-case letters at the start of fields. I expect this is incompatible with the programs generating these "db" files, hence "pax.tcz" is not in the tags.db.gz file, and therefore doesn't get found when searching with tce-ab. I don't know whether there's an issue with adding fields to the "info" files, but it seems obvious that you should be still including the existing ones in the same way as others. "Title:" should be the extension file name, but you're using "title:" for what should be in the "Description:" field, for example.
"Size:" isn't there in any form, and it's a field I often look for in the extension browser.
Hey CNK, thanks for the convo!
Honestly, I didn't even know there was any consistency with those files! lol I worked on trying to standardize the repo here years ago, but didn't have much help. As I processed those files I immediately noticed that there were so many inconsistencies that I just made my own. For instance, not all of them capitalized the words. Some had missing fields. Some had extras. I get it, it's a volunteer project. I tried to fix this, but it was too large of a task without significant help.
I can go back and update the .info file to match what TC requires. Is there a template somewhere to use by chance?
In regards to the compatibility with the existing tools and optional dependencies, I can understand your request. But that functionality isn't present, so there's nothing to be compatible with. If the tce-* scripts would like to incorporate that form of processing from my package manager, they are more than welcomed to do so - I do not mind. But I will say that it may involve a little bit of work to incorporate as my package manager does not share the same code as the original tce-* scripts.
There's a switch to install them as well. I know this won't help you, but it was something I thought about and incorporated. There's also the ability to install certain files out of a package instead of the entire contents.
It's good to see these new ideas, but for my use I'd want things to still work with existing tools such as tce-ab and tce-update. Especially when it comes to things like the formatting of info files for extensions in the official repo.
Thanks, I agree! New ideas always propel tech forward. Currently pax has no ability to search the repo so the tce-ab script would still be required. I assume that tce-update is used to update existing packages? If so, this tool has been replaced within pax, although it would presently require 2 commands instead of 1. A future update will resolve this, as well as add the ability to have this whole process automated if desired.
One of the other desired features was the ability to unload/uninstall packages. TC currently has no ability to do so, but you can with pax.
-
To quickly go back to the start of this additional conversation... Part of the reason pax can not search packages is because it looked like it just pulled down the files ondemand. Based on our conversation, it appears that I was right. Before trying to add in that feature, I wanted to see about having something server-side - again for efficiency, mostly. Plus having this would allow for consistency across various Linux distros making pax for universal than trying to incorporate the way it's done in TC (which may be unique to just this distro and non-portable to others).
I do appreciate the feedback from you guys!
-
This topic has being discussed before, how our package manager deal with how to determine what files is in what package ?
I suggested a SQLite solution, so you could also add more fields like description size and so on.
But i see the point of what TC is all about, simple and small. And not lot of dependencies, in this SQLite.
Let me see on my computer if i have the POC saved ?
$ cat make.sh
#!/bin/sh
sqlite3 files.db "CREATE TABLE IF NOT EXISTS tcz_files (name TEXT NOT NULL,file TEXT NOT NULL);"
for filename in $(ls *.tcz)
do
tczfile=$filename;for file in $(unsquashfs -lc $tczfile); do sqlite3 files.db "INSERT INTO tcz_files ( NAME , FILE ) VALUES ( '$tczfile' , '$(basename $file)' );" ; done
done$ cat search.sh
#!/bin/sh
sqlite3 -cmd ".timer on" -box files.db <<< "SELECT name as Package, file as File FROM tcz_files WHERE file LIKE '%$1%';"
Maybe you could steal my POC and make something about it.
Happy hacking :)
-
I can go back and update the .info file to match what TC requires. Is there a template somewhere to use by chance?
This is the example I follow (https://wiki.tinycorelinux.net/doku.php?id=wiki:creating_extensions#tczinfo_example). Also the submitqc script in submitqc.tcz checks the info file for wrong or missing fields, among other things.
In regards to the compatibility with the existing tools and optional dependencies, I can understand your request. But that functionality isn't present, so there's nothing to be compatible with.
I was more concerned that like with the info file you'd submit extensions with dep files in a format that breaks the regular tce* tools. If you have something like separate "extension.tcz.dep.opt" files then that might work better, but otherwise I'd hope the tce* tools get support added for any new formatting features before they're used in the repo.
To quickly go back to the start of this additional conversation... Part of the reason pax can not search packages is because it looked like it just pulled down the files ondemand. Based on our conversation, it appears that I was right.
Yes, but I'm not sure why you said Debian's package system is more "server-side" than on TC. Debian also downloads a full package list to search locally, and it downloads the description and dependency info equivalent to the info and dep files for all packages too. Debian do have a web interface that allows searching in a web browser, but apt-get and other package managers I've used there all locally search the downloaded package lists.
-
Hi wysiwyg
... So then it basically does download the entire repo's worth of non-software package files. ...
I'm working from memory here, and referring to the Apps GUI:
These get loaded for browsing/installing extensions:
info.lst Used to populate the left window of Apps
provides.db Used to populate the Files tab of Apps,
sizelist Used when populating the Size tab of Apps.
These get loaded for functions under Apps->Maintenance:
dep.db.gz Dependencies And Deletions, Check Onboot Unneeded, Check for Orphans
md5.db.gz Md5 checking, Check for Updates
Used in Search options:
provides.db Search for extensions by provided file
tags.db.gz Search for extensions by listed tags
... First, there is a pinned post at the top of this board saying not to abuse the bandwidth that is generously being donated to the project. ...
I'm not sure which post you are referring to, but that was likely in response
to individuals downloading entire repos.
... Even with using zsync, that would still fall under that category - at least in my eyes. Without putting these files somewhere that they get saved, they would have to be re-downloaded every time a device reboots. ...
A reboot does not automatically result in a re-download.
sizelist and tags.db are in /tmp , so they would get re-downloaded if they're needed.
dep.db and provides.db are in the tce directory, so they are persistent.
... TC takes great strides at being as small as possible, yet to perform this task with the existing package manager, it has to bloat the system with the entire software library.
I don't understand what you are trying to say there.
How are we bloating the system with the entire software library?
-
This topic has being discussed before, how our package manager deal with how to determine what files is in what package ?
I suggested a SQLite solution, so you could also add more fields like description size and so on.
But i see the point of what TC is all about, simple and small. And not lot of dependencies, in this SQLite.
Let me see on my computer if i have the POC saved ?
$ cat make.sh
#!/bin/sh
sqlite3 files.db "CREATE TABLE IF NOT EXISTS tcz_files (name TEXT NOT NULL,file TEXT NOT NULL);"
for filename in $(ls *.tcz)
do
tczfile=$filename;for file in $(unsquashfs -lc $tczfile); do sqlite3 files.db "INSERT INTO tcz_files ( NAME , FILE ) VALUES ( '$tczfile' , '$(basename $file)' );" ; done
done$ cat search.sh
#!/bin/sh
sqlite3 -cmd ".timer on" -box files.db <<< "SELECT name as Package, file as File FROM tcz_files WHERE file LIKE '%$1%';"
Maybe you could steal my POC and make something about it.
Happy hacking :)
Hey Patrikg!
Thanks for the code snippets! :) It looks like you are trying to use sqlite for a local repository on each client. I'm actually trying to get an online solution for searching non-local files. With the files that are already downloaded, they can have their corresponding .info files parsed - at least with pax. The tce-* scripts require the downloading of the entire non-software package files. This could be useful for offline devices, but then they couldn't download anything anyways making this a non-point. The vast majority of devices are connected to the internet making this wasteful. An online search takes zero space on your local storage while providing you the results you seek, for either local or non-local packages.
-
I can go back and update the .info file to match what TC requires. Is there a template somewhere to use by chance?
This is the example I follow (https://wiki.tinycorelinux.net/doku.php?id=wiki:creating_extensions#tczinfo_example). Also the submitqc script in submitqc.tcz checks the info file for wrong or missing fields, among other things.
Awesome! I'll grab that and adjust my .info files to match.
In regards to the compatibility with the existing tools and optional dependencies, I can understand your request. But that functionality isn't present, so there's nothing to be compatible with.
I was more concerned that like with the info file you'd submit extensions with dep files in a format that breaks the regular tce* tools. If you have something like separate "extension.tcz.dep.opt" files then that might work better, but otherwise I'd hope the tce* tools get support added for any new formatting features before they're used in the repo.
Gotcha! I don't think you'd have to worry about this because it would take modifying the .dep files for various packages in the repo, of which I have no control over. Nor would I want to do that without approval anyways. But, if that functionality was added into the tce-* scripts from pax, I'd suggest to adopt the other forms of entries to those files as well (versioning, copying specific files from the package, etc).
To quickly go back to the start of this additional conversation... Part of the reason pax can not search packages is because it looked like it just pulled down the files ondemand. Based on our conversation, it appears that I was right.
Yes, but I'm not sure why you said Debian's package system is more "server-side" than on TC. Debian also downloads a full package list to search locally, and it downloads the description and dependency info equivalent to the info and dep files for all packages too. Debian do have a web interface that allows searching in a web browser, but apt-get and other package managers I've used there all locally search the downloaded package lists.
Hmmm can you point to where this list is? I checked /var/lib/apt and the files in there only contain hashes to (past?) files, or brief info for installed packages. I couldn't find anything where the entire repo's package listing is stored locally. And yes, this is the equivalent to .info and .dep files for already downloaded packages being on the device. This is not what I'm referring to. pax can already parse those files for local searches. For example, if a user wants to know what database servers are available in the online repo, those files are no good. The current setup that TC has is to download the entire repo's worth of information files and search locally. An online search would be much more efficient in terms of bandwidth and storage, right?
-
Hey Rich, thanks for the reply!
... First, there is a pinned post at the top of this board saying not to abuse the bandwidth that is generously being donated to the project. ...
I'm not sure which post you are referring to, but that was likely in response
to individuals downloading entire repos.
Yes, that was the post that I was referring to. But wouldn't the partial download of the entire repo also fall under that category? If the same results can be achieved without downloading anything, isn't that the most efficient use of bandwidth that is being donated?
... Even with using zsync, that would still fall under that category - at least in my eyes. Without putting these files somewhere that they get saved, they would have to be re-downloaded every time a device reboots. ...
A reboot does not automatically result in a re-download.
sizelist and tags.db are in /tmp , so they would get re-downloaded if they're needed.
dep.db and provides.db are in the tce directory, so they are persistent.
I understand. I wasn't sure where the files were being placed. It was just an observation based on the non-permanent nature of parts of the OS.
... TC takes great strides at being as small as possible, yet to perform this task with the existing package manager, it has to bloat the system with the entire software library.
I don't understand what you are trying to say there.
How are we bloating the system with the entire software library?
So if TC optimizes their packages during compile to be smaller in size (which they do) to shave off bytes per binary, I'd say that's going through effort to have the smallest distro possible. If a normal user that uses the GUI for the tce-* scripts does any type of search, they are automatically downloading entire parts of the repo. If it doesn't matter in that instance, why go through the efforts elsewhere to make this as small as possible?
If the search abilities for packages, whether for local or remote, can be accomplished with the same results, and one doesn't add anything to your storage (also saving donated bandwidth) vs one that does, wouldn't that fall in line with other aspects? Let alone just being a more efficient system overall?
-
Yes, but I'm not sure why you said Debian's package system is more "server-side" than on TC. Debian also downloads a full package list to search locally, and it downloads the description and dependency info equivalent to the info and dep files for all packages too. Debian do have a web interface that allows searching in a web browser, but apt-get and other package managers I've used there all locally search the downloaded package lists.
Hmmm can you point to where this list is? I checked /var/lib/apt and the files in there only contain hashes to (past?) files, or brief info for installed packages. I couldn't find anything where the entire repo's package listing is stored locally.
Checking on a Debian system, they are in /var/lib/apt/lists. Clearly all the packages are in the lists, including those not installed. 30MB total for Debian Bullseye.
$ head /var/lib/apt/lists/deb.debian.org_debian_dists_bullseye-updates_main_binary-arm64_Packages
Package: ca-certificates-java
Version: 20190909+deb11u1
Installed-Size: 43
Maintainer: Debian Java Maintainers <pkg-java-maintainers@lists.alioth.debian.org>
Architecture: all
Depends: ca-certificates (>= 20121114), default-jre-headless | java8-runtime-headless, libnss3 (>= 3.12.10-2~)
Description: Common CA certificates (JKS keystore)
Multi-Arch: foreign
Description-md5: 304cd3554728e5d076f8ecbb3b5057d8
Tag: protocol::ssl, role::app-data, security::authentication,
$ aptitude show ca-certificates-java | grep ^State:
State: not installed
The lists are downloaded from URLs like http://ftp.debian.org/debian/dists/bullseye/main/binary-arm64/Packages.gz
An online search would be much more efficient in terms of bandwidth and storage, right?
Yes but Linux mirror sites only host static content, so instead of having the load spread over various mirrors it would all be on the site hosting the search, and that would be a single point of failure (although I guess a package manager could fall back to downloading the lists the old way from a static mirror if it went down).
Also, keep the sizes in perspective. That tags.db.gz file tce-ab downloads in order to search extension names and tags/keywords is about 30KB. One thread page on these forums is bigger than that. The provides.db.gz file is bigger at 2MB, but it gets cached. The extra server CPU load from getting hit by lots of search requests (these days probably including lots of bots just trying to hack or crash it, like I've experienced on my own website) would probably be more of an issue than transferring that little extra data.
-
Hey CNK, thanks for the reply!
Yes, but I'm not sure why you said Debian's package system is more "server-side" than on TC. Debian also downloads a full package list to search locally, and it downloads the description and dependency info equivalent to the info and dep files for all packages too. Debian do have a web interface that allows searching in a web browser, but apt-get and other package managers I've used there all locally search the downloaded package lists.
Hmmm can you point to where this list is? I checked /var/lib/apt and the files in there only contain hashes to (past?) files, or brief info for installed packages. I couldn't find anything where the entire repo's package listing is stored locally.
Checking on a Debian system, they are in /var/lib/apt/lists. Clearly all the packages are in the lists, including those not installed. 30MB total for Debian Bullseye.
$ head /var/lib/apt/lists/deb.debian.org_debian_dists_bullseye-updates_main_binary-arm64_Packages
Package: ca-certificates-java
Version: 20190909+deb11u1
Installed-Size: 43
Maintainer: Debian Java Maintainers <pkg-java-maintainers@lists.alioth.debian.org>
Architecture: all
Depends: ca-certificates (>= 20121114), default-jre-headless | java8-runtime-headless, libnss3 (>= 3.12.10-2~)
Description: Common CA certificates (JKS keystore)
Multi-Arch: foreign
Description-md5: 304cd3554728e5d076f8ecbb3b5057d8
Tag: protocol::ssl, role::app-data, security::authentication,
$ aptitude show ca-certificates-java | grep ^State:
State: not installed
The lists are downloaded from URLs like http://ftp.debian.org/debian/dists/bullseye/main/binary-arm64/Packages.gz
Well, I stand corrected! Oddly enough, I renamed the /var/lib/apt directory to have a .org extension, then ran apt-cache search and it still populated with results. I disconnected from the Internet, and it still gave results! I'm not sure how that was possible if the local database was unreachable, as well as the Internet...
An online search would be much more efficient in terms of bandwidth and storage, right?
Yes but Linux mirror sites only host static content, so instead of having the load spread over various mirrors it would all be on the site hosting the search, and that would be a single point of failure (although I guess a package manager could fall back to downloading the lists the old way from a static mirror if it went down).
Also, keep the sizes in perspective. That tags.db.gz file tce-ab downloads in order to search extension names and tags/keywords is about 30KB. One thread page on these forums is bigger than that. The provides.db.gz file is bigger at 2MB, but it gets cached. The extra server CPU load from getting hit by lots of search requests (these days probably including lots of bots just trying to hack or crash it, like I've experienced on my own website) would probably be more of an issue than transferring that little extra data.
If they only host static content, then yes you are setting up a single point of failure. However, depending on the hosting company, they will have numerous mirrors themselves to prevent just such an issue. Now that does not preclude an issue with the server itself having a problem...
I agree that tags.db.gz file is probably not too big of an issue. But multiply the provides.db.gz by just 100 users and that's already at 200mb. That's a lot of searching that can be done to meet or exceed that using an online script. I'm quite frankly surprised that Debian does this. There's no telling what their bandwidth bill is at the end of every month lol
I guess I'll throw in the ability to process searches this way too. It seems like a giant waste of resources (more so for someone like Debian), but I'm not footing the bill so...
-
I followed this discussion with interest, tce* scripts are good, no perfect! For me, all is just an intellectual challenge, but not a BIG win in any direction-solution developers will chose to pursuit.
For "modern" hardware (multi-core CPU, SDD or Nvram storage, fast 100MB..1GB network links, etc) is .. not worthy. So the "new" pax package could have a meaningful gain only for outdated surviving hardware like 486 CPU, with small HDD storage, small 128MB RAM, 10MB NIC card etc.
Because, as Rich said 0.2 .. 2 seconds is enough to load all "non-software" repo metadata. Looking into https://mirrors.dotsrc.org/tinycorelinux/17.x/x86_64/tcz/ (https://mirrors.dotsrc.org/tinycorelinux/17.x/x86_64/tcz/) I see provides.db.gz =3.1 MB, but provides.db.zsync = 175KB!!
1. Using TC on USB, users will not premature destroy their pen/stick with too many updates.
2. Using TC on HDD, will have permanent /tce/optional folder (aka static on HDD, sometime even cache in RAM for a session), and seldom using zsync for 175KB extra.
From my past experience, many TCZ extension are not often updated, because new versions have bigger size. But tiny is about just core (not very useful by itself without apps), not the full system as used by most users. It seams that TC developers think that users initially search for apps, then users settle with some apps and updates seldom.
In summary the idea to use a new package management revolves in replacing wget (from busybox) with SQLite, or maybe CURL+dependencies, by increasing the TINY base/core.
-
It seams that TC developers think that users initially search for apps, then users settle with some apps and updates seldom.
That's pretty much what I do. I have recently started doing occasional extension updates but I haven't exactly made a habit of it.
-
Hey Nick! Thanks for adding to the discussion!
I followed this discussion with interest, tce* scripts are good, no perfect! For me, all is just an intellectual challenge, but not a BIG win in any direction-solution developers will chose to pursuit.
For "modern" hardware (multi-core CPU, SDD or Nvram storage, fast 100MB..1GB network links, etc) is .. not worthy. So the "new" pax package could have a meaningful gain only for outdated surviving hardware like 486 CPU, with small HDD storage, small 128MB RAM, 10MB NIC card etc.
I agree that the TC developers are most likely going to leave the tce-* scripts as is, which is fine. They are a barebones way to accomplish their task. But to say that pax is only meaningful for older hardware, would be incorrect. pax has a much larger feature set than the default tce-* scripts, including one that has been requested for many years - the ability to unload software. I would invite you to read the .info file associated with the project to see the abilities it has over the default manager(s).
http://tinycorelinux.net/16.x/x86/tcz/pax.tcz.info
Which is actually the original version. I sent the latest version at the end of last week that hasn't been processed yet. There isn't much change by way of code, but it does provide an in-depth man page to further explain some of the features. Of course, feedback is always welcome to make it better if something is unclear.
Because, as Rich said 0.2 .. 2 seconds is enough to load all "non-software" repo metadata. Looking into https://mirrors.dotsrc.org/tinycorelinux/17.x/x86_64/tcz/ (https://mirrors.dotsrc.org/tinycorelinux/17.x/x86_64/tcz/) I see provides.db.gz =3.1 MB, but provides.db.zsync = 175KB!!
1. Using TC on USB, users will not premature destroy their pen/stick with too many updates.
2. Using TC on HDD, will have permanent /tce/optional folder (aka static on HDD, sometime even cache in RAM for a session), and seldom using zsync for 175KB extra.
From my past experience, many TCZ extension are not often updated, because new versions have bigger size. But tiny is about just core (not very useful by itself without apps), not the full system as used by most users. It seams that TC developers think that users initially search for apps, then users settle with some apps and updates seldom.
My original gripe was more so about being efficient with things. I did also think that the way TC was handling it was an exception, not the rule. And yes, 3.1MB is not very much in size. But when 10 people use it, that becomes 31MB, when 100 people use it is becomes 310MB, when 1000 people use it is becomes 3.1GB, etc. And remember that this is donated bandwidth. I'll still add in the ability to search using an online URL, but I'll also have to expand that to include entire repo db's to work across various distros.
In summary the idea to use a new package management revolves in replacing wget (from busybox) with SQLite, or maybe CURL+dependencies, by increasing the TINY base/core.
I'm not sure what advantage that would give you. You would still have to download the entire database to your local device, defeating the efficiency I was aiming for. And the reason wget was chosen is because with basic package management, you only need to download packages, which wget works very well at - plus its part of busybox so no additional dependencies. cURL adds another dependency to the mix, adding extra size to the base - of which TC tries to keep as small as possible. I will attempt to use wget to handle the searching side, but I may have to revert to cURL. But my design goals are different than TC. I do want small and efficient, but not at the expense of traditional features.
-
It seams that TC developers think that users initially search for apps, then users settle with some apps and updates seldom.
That's pretty much what I do. I have recently started doing occasional extension updates but I haven't exactly made a habit of it.
In a future version pax will have the ability to alert and/or automatically perform this task. I just needed to iron out what needed to be done beforehand. Most people don't do updates if they don't know they are available. Actually most people don't install updates even when they're told they have them. This is obviously unsafe and leaves you subject to exploits...
-
I follow the forums here pretty closely, so if there's an update to an extension I use, I'll usually know about it that way and update as needed. Not a big fan of automatic updates in general.
That's not say I;m not interested in pax - but at this point it's really just a curiosity to me.
-
I completely understand. I'm also not a big fan of automatic updates (which is why I didn't mandate them in pax). To me freedom is better than anything. Although freedom comes with responsibility. If the user chooses not to update their system, then the consequences fall on them.
I do appreciate your curiosity for the project, and look forward to hearing any feedback you can provide. I want the software to be as useful to people as possible, of course.
-
looking into https://github.com/cliquesoft/pax/blob/main/bin/pax it seams it is a /bin/sh script, which is OK.
- An 'html' output option: it needs a "small browser" to see this output (you can be inspired by how fluff.tcz shows its help)
- A graphical interface: FLTK is in the base. [not GTK1/2/3 neither QT4/5/6. nor Xdialog when Wayland is the future].
As https://mirrors.dotsrc.org/tinycorelinux/17.x/x86_64/tcz/pax.tcz.tree shows, pax depends on "many" other tczpax.tcz
coreutils.tcz
gmp.tcz
squashfs-tools.tcz
liblzma.tcz
lzo.tcz
libzstd.tcz
liblz4.tczThe package-manager belongs in the base, the base must / should be small, but pax+deps is not small size for now by "TC developers standards". Maybe I am wrong, but I know how hard / long was the discussion on TC boot / config parameters: a 4 byte file (containing just a boot option as text) have 4KB (a page) in RAM etc.
I wish you success, because I want TC to evolve. From my point of view the "libs dependency hell" was solved by an AWK engine or can be a C-compiled software (like in Arch-linux or Alpine-linux ). Maybe your focus is in download speed, or parallel sources; where is the bulk CPU consumption, in server or client, etc. I mean do you want to "protect" the web-server energy/bandwide by transferring the energy bill to the client-user, etc. I just ask...
If I may: It will be nice to define first what you improve (in Tinycore!), to measure what we have already (for a common user) and then you state by how much you WANT (not necessarily to achieve for now) to improve (in KB size, KB/second, or % or whatever).
-
I am a novice in terms of software licensing. But I saw that "pax" has a "Cliquesoft Public License [CPLv2]".
I try to do not agree / obey (shame on me) with IP (intellectual property, aka new feudalism), the only acceptable license is something like this https://landley.net/toybox/license.html (https://landley.net/toybox/license.html) ; YMMV
-
Hey nick65go, thanks for adding to the conversation!
looking into https://github.com/cliquesoft/pax/blob/main/bin/pax it seams it is a /bin/sh script, which is OK.
- An 'html' output option: it needs a "small browser" to see this output (you can be inspired by how fluff.tcz shows its help)
- A graphical interface: FLTK is in the base. [not GTK1/2/3 neither QT4/5/6. nor Xdialog when Wayland is the future].
As https://mirrors.dotsrc.org/tinycorelinux/17.x/x86_64/tcz/pax.tcz.tree shows, pax depends on "many" other tczpax.tcz
coreutils.tcz
gmp.tcz
squashfs-tools.tcz
liblzma.tcz
lzo.tcz
libzstd.tcz
liblz4.tczThe package-manager belongs in the base, the base must / should be small, but pax+deps is not small size for now by "TC developers standards". Maybe I am wrong, but I know how hard / long was the discussion on TC boot / config parameters: a 4 byte file (containing just a boot option as text) have 4KB (a page) in RAM etc.
I wish you success, because I want TC to evolve. From my point of view the "libs dependency hell" was solved by an AWK engine or can be a C-compiled software (like in Arch-linux or Alpine-linux ). Maybe your focus is in download speed, or parallel sources; where is the bulk CPU consumption, in server or client, etc. I mean do you want to "protect" the web-server energy/bandwide by transferring the energy bill to the client-user, etc. I just ask...
I completely agree with everything you said. I'll tackle them one at a time...
1. Yes the project is a shell script to make is portable to all CPU architectures without having to compile anything. It's not as efficient as a compiled binary, but there is no gain this way since the tce-* package manager is also scripts.
2. The various output options are so that pax can be integrated into various software and not just for command line usage. Currently there is the default text, and two XML options that can be used with any web-based GUI. There is also a fifo option so that pax can run in a "client control mode". This will allow pax to basically work in a network where an administrator can perform software admin tasks on the clients without having to visit them (e.g. remotely installing, uninstalling, updating, etc). I still have to build the server-side, but one can be put in place by a savvy admin. I plan on expanding to include two more options: html and json.
3. The graphical interface for pax will be a web-based interface. I just have to port several other applications into TC before I can get that going. However, if any other developers on here are interested in building a different type of interface (e.g. GTK, QT, FLTK, etc), I will be at their disposal to assist.
4. The coreutils dependency is required because pax needs the 'stat' binary to perform some tasks. Unfortunately the busybox applet is not part of the TC ramdisk, so it requires that dependency. If pax were the default package manager in TC, it could just install a single file out of the coreutils package and leave the rest, but the default tce-* scripts don't have this ability, so I'm stuck installing the entire package. The squashfs-tools dependency I'm actually going to remove in the next version of pax. It's only used when building from source code or packaging pre-compiled software, of which the vast majority of users do not do.
5. Yes, everything should be as small as possible. If the TC team wants to add in the 'stat' busybox applet, that can reduce one of the two dependencies. I can remove the other one in a future version.
If I may: It will be nice to define first what you improve (in Tinycore!), to measure what we have already (for a common user) and then you state by how much you WANT (not necessarily to achieve for now) to improve (in KB size, KB/second, or % or whatever).
This is a great point. So there are several items:
1. Efficiency. Unless the code has been updated, the tce-* scripts install each package (and their dependencies), then call ldconfig, udevadm, and then the package installation script. While this is not a big deal when installing a package at a time in a running system, it's inefficient during the boot process when everything is getting loaded. pax doesn't do this. It installs all the requested package (and dependency) files, then executes those calls only once.
There is also an install option to load packages in a single or dual-stage process when booting the system. This can allow the device to boot faster while loading supplemental software in the background. This is also not possible with the tce-* package manager.
2. Uniform execution. I'm not sure about the tce-* scripts, but the syntax with pax is uniform across commands. For example, to copy, extract, or install packages from a different source (to an optional target directory) all has the same syntax with different actions:
pax -S /path/to/packages -{cei} PACKAGE(S) [TARGET DIRECTORY]
3. Single file. With pax there is one script to call, not many to perform different tasks. This has an advantage and a disadvantage. The advantage is that there's one program to learn, not multiple. It also enables synergy among it's switches (as defined above). The disadvantage is that the entire script must be loaded to perform a task, vs smaller individual script that do specific things.
4. Versatility. For example, pax can take a space separated list, list in a text file, or a directory as the parameter to use when performing an action:
pax -c package1 package2 package3
pax -c textfile.txt
pax -c /path/to/packages
This is not possible with the tce-* scripts.
5. Features. The feature set in pax is larger than that of the tce-* equivalents. There are only one or two features not yet implemented (one of which is searching) because I needed to see how TC performed this task beforehand so that it could be designed properly. It takes a long time to code, test/debug, and document all of this (4 months for the latest version of pax).
I think I've addressed the improvements in this reply - reduction of dependencies (which will require the TC team for coreutils) and the addition of a few outstanding features.
If you have anything else nick65go, feel free to reply. I enjoy the conversation!
-
I am a novice in terms of software licensing. But I saw that "pax" has a "Cliquesoft Public License [CPLv2]".
I try to do not agree / obey (shame on me) with IP (intellectual property, aka new feudalism), the only acceptable license is something like this https://landley.net/toybox/license.html (https://landley.net/toybox/license.html) ; YMMV
I'm a big fan of Rob Landley! I do agree that the 0BSD is one of the best licenses for freedom. The CPLv2 is basically the GPLv2 without some of its clauses making it less restrictive. So I'm actually making the software more free than by using the GPLv2 (which is a very widely used license for freedom). See https://www.cliquesoft.org/#Licenses for more information.
BUT as I stated in the prior response, it takes a long time to build software. That means I have to donate my time to do so instead of doing something else to make money. It's expensive to build free software! Again, it took me 4 months to re-design and re-build pax without earning a penny of revenue for my time spent. As a way to potentially earn some money, I offer the ability to re-license the software using a version of the BSD for a fee (under the CPLv1). This will probably never materialize, but it is an option to earn back some money.
-
@wysiwyg: please let me clarify few things. I am a user not a developer. Tinycore is one of my hobbies, but not the only one. I freely and shameless express my controversial / personal opinions on this forum, taken the risk to be banned by admins.
For now I use CachyOS (KDE plasma) and I have fun with Archlinux or Alpinelinux. I keep an eye on Tinycore development, but I do not currently use it as my main OS, because my (pseudo) obsession with security (TC does not sign tcz packages, it used md5 instead of SHA*, default X* server as root, slow patch vulnerability, etc.).
I would like to recommend you to spend a little more time to analyze tce* scripts. Mostly tc-config and tce-load. Because if I am not wrong, about efficiency:
1. udevadm is used at boot-time (and stays in RAM) and one/two times when you load a device (sound card, wifi, etc). but not for usual tcz apps.
2. I had the impression that ldconfig is loaded one time (at boot-time for all onboot.lst), and then one-time after each user-chose selected/nominated tcz LIST.
I see you very passionate/ determined, so I do not wish to diminish your achievements. Especially, if you read the forum, you will see that many (good in my opinion) proposals were rejected just because TC started and is committed for low resources aka 486 CPU, 64-128MB RAM etc. In this restricted environment every bytes count and improvements are hard to design (for core/base). In base is kernel + libc + busybox, so playground is just sh scripts...and ideas recycled from 2009-2025.
-
Hey nick65go, thanks for the continued conversation!
What's your draw to Arch and Alpine?
I have analyzed the tce-load script, which is what pax was originally forked from. It is no longer a fork and is its own project. In regards to your points, they are both wrong. udevadm is what is used to trigger udev to basically re-examine the attached devices and bring any new ones onboard, or discard removed ones from the device. ldconfig is used when libraries are added or removed. This is why those commands have to be executed (sometimes) depending on particular files that get installed in packages. As I stated, pax only (potentially) runs these commands once per execution regardless of how many packages are being installed, whereas the tce-load script calls them repeatedly (if applicable) per package being requested to install. Thus, pax is more efficient in this manner.
I do agree that there are parts that get dismissed due to their goals of being small. This is why I was "shocked" about how package searching was handled.
And no offense was taken. I am passionate about my projects. I have spent a considerable amount of my life working on all of them lol
-
Hi wysiwyg
... As I stated, pax only (potentially) runs these commands once per execution regardless of how many packages are being installed, whereas the tce-load script calls them repeatedly (if applicable) per package being requested to install. ...
Except when using:
tce-load -bwhich tells it you are booting. Then depmod, ldconfig, and tce.installed get run in the end.
I mention some of that here:
https://forum.tinycorelinux.net/index.php/topic,27231.msg175177.html#msg175177
Run these two commands to see where some of this takes place:
grep -n BOOTING /usr/bin/tce*
grep -n "/etc/sysconfig/newmodules" /usr/bin/tce*
-
Hey Rich!
Was the code updated at some point to include that? The original fork from tce-load was back somewhere around 2017 (almost 10 years ago) and I don't think that was the case. But then again, it was very long ago...
Just out of curiosity, why not have that functionality at all times?
-
Hi wysiwyg
I don't know when it was enacted. This is from TC10 x86:
tc@E310:~$ grep -n BOOTING /usr/bin/tce*
/usr/bin/tce-load:10:unset WGET INSTALL COPYINSTALL BOOTING ONDEMAND DOWNLOAD_ONLY LOAD_ONLY SUPPRESS
/usr/bin/tce-load:41: if [ "$BOOTING" ]; then
/usr/bin/tce-load:56: b) BOOTING=TRUE ;;
/usr/bin/tce-load:92: [ "$BOOTING" ] || rmdir /mnt/test
/usr/bin/tce-load:96: if [ "$BOOTING" ]; then
/usr/bin/tce-load:107: if [ "$BOOTING" ] ; then
/usr/bin/tce-load:132: if [ ! "$BOOTING" ]; then
/usr/bin/tce-load:146: if [ ! "$BOOTING" ]; then
/usr/bin/tce-load:154: [ "$BOOTING" ] && [ "$SHOWAPPS" ] && echo -n "${YELLOW}$APPNAME ${NORMAL}"
/usr/bin/tce-load:241:if [ "$INSTALL" ] && [ ! "$BOOTING" ]; then
/usr/bin/tce-load:292:[ "$BOOTING" ] && exit 0I think that's from around 2019.
-
Hey Rich!
Ok, if it was implemented around that time, that would probably explain why I didn't see it in the version I originally forked from (which was 7.x in 2017). When browsing through the source, I didn't understand why it was being called the way it was. Obviously I adjusted that in my own project. If that update was made in 2019, that would most likely have been TC 9.x...
Thanks for the tidbit!
-
Hi wysiwyg
I don't know when it was implemented. I took a look at
rootfs.gz for TC5 (2013/2014) and found this:
tc@E310:~/TinycoreISOs/RootfsTC5$ grep -n BOOTING usr/bin/tce*
usr/bin/tce-load:10:unset WGET INSTALL COPYINSTALL BOOTING ONDEMAND DOWNLOAD_ONLY LOAD_ONLY SUPPRESS
usr/bin/tce-load:50: b) BOOTING=TRUE ;;
usr/bin/tce-load:108: if [ "$BOOTING" ]; then
usr/bin/tce-load:119: if [ "$BOOTING" ] ; then
usr/bin/tce-load:144: if [ ! "$BOOTING" ]; then
usr/bin/tce-load:159: if [ ! "$BOOTING" ]; then
usr/bin/tce-load:167: [ "$BOOTING" ] && [ "$SHOWAPPS" ] && echo -n "${YELLOW}$APPNAME ${NORMAL}"
usr/bin/tce-load:223:if [ "$INSTALL" ] && [ ! "$BOOTING" ]; then
usr/bin/tce-load:304:[ "$BOOTING" ] && exit 0
At least some of this was already in place back then.
-
What's your draw to Arch and Alpine?
I will not buy/repair OLD devices anymore, I have some of them already FULL functional (if you forget their heat/noise, dead battery etc). Most with 1- 4+GB RAM, 100-250+ GB HDD, i686+ CPU, starting from year 2006 (20 years ago!).
Today I can only buy x86-64 NEW devices. So, compatibility with 486 CPU is not useful for me for the future. A USB stick is 17-40 EUR for a 150-400 MB/s speed and 64-90GB, with a live span of 2-5 years (aka one beer/year). Compare this with all tcz packages at 4GB total. Even a full Archlinux is 20GB storage.
Capitalist markets force me to buy, every 5-8 years (because planed obsolesce), a device with more RAM/HDD than I really need, therefore my focus shifted to security. This means distro with more developers, faster patching, better secured channels for updating, etc. Yes, this means undesirable bloat/fat, but I already paid for not-need hardware. It depends what someone use the software for. I spend money to gain time, for things that matter to me. (Tinycore is still on my list).
-
Security should always be a top focus with projects today. We live in a different time when scammers and malicious people are everywhere and can do a lot of harm due to the amount of "things" being moved into the digital sphere. It used to just be a countries government that they had to worry about, but that changed decades ago.
I wish there were more effort with security in TC, but it is a volunteer OS and as a result there are some things that fall short as a result. I do have another project that would help keep packages more up-to-date, but I doubt it will get implemented. I'll try to get it added to the repo anyways in the coming weeks...
Speaking of the repo, I've been trying to get pax into the RPi repo's, but the powers that be are preventing it due to "more feedback needed." I've tried to get clarification for days, but get nothing. I don't understand why this package is coming under this scrutiny. It's also non-nonsensical because they are limited the amount of people that could potentially provide feedback. Plus this is script so it works across all platforms and CPU architectures so it can be obtained from one and applied to any. For anyone that is using the RPi versions of TC and would like to try pax, simply execute these commands at a prompt:
cd
wget http://tinycorelinux.net/16.x/x86/tcz/pax.tcz
wget http://tinycorelinux.net/16.x/x86/tcz/pax.tcz.dep
wget http://tinycorelinux.net/16.x/x86/tcz/pax.tcz.md5.txt
tce-load -i ./pax.tcz
A quick point about capitalism. It is simply a method of operation for economies, just like communism is another one. Capitalism is not responsible for planned obsolesce in products. That falls on the company that manufactures the products. That same company could operate identically under any economic method. This is like saying that because I take my body temperature with my armpit vs under my tongue, I got sick. One has nothing to do with the other.
-
in the past, economist Milton Friedman was arguing that the interests of shareholders should be placed above those of consumers and other stakeholders. Now ephemeral Trump politics versus his "EU friends"+ neighborhoods, etc. So please, let agree to disagree about how this late stage capitalism is influencing hardware / software development & cooperation world-wide. If communism is/was bad it does not mean capitalism is the best, it is just that humans had limited imagination to organize the society -- because poor choose of their leaders.
I will stop here, to not deviate from the main topic - new package-manager for tinycore.
-
I wasn't arguing one was better than the other. I was simply saying that capitalism is not responsible for a company making planned obsolesce in their products, and that it could do that under any type of economy - capitalism, communism, socialism, etc.
But I do agree, that a discussion among those things is definitely off topic. :)
-
After 30+ days, pax is still not in all the repo. @gutmensch can you explain exactly what it is you're looking for and why? Maybe by bringing this to light whatever it is that you're looking for can be found.
-
pax posted to piCore/piCore64 16.x repos
-
Thanks @juanito I do appreciate it!