corepkg - a new core package and updates manager

[martin]

Sorry, one more response.

Another idea I had was, I provide a tcz that contains corepkg and coresome (in it's various stages). CorePkg will get the package database from my server where it is generated and hosted.

On my server, I only grab the tcz file names, the md5 files and the info files from the repository. I use those to create the package database. This way, it can still track updated files, TinyCore Linux users can still benefit from CorePkg, all downloads are still made from the TinyCore Linux repository network and the TinyCore Linux admins don't need to run anything on the server. I do all that, and if users want to, they download corepkg through appbrowser and use it in place of tce-update if they want to. All under the TinyCore Linux banner.

Does that suit better?

edit: plus this gets me out of having to host all that source!

[vinnie]

+1 to latest martin post

sorry if I intervene again, but I wanted to try to understand something

Martin, you wanted that they were integrating your package manager enlarging the distribution of 400 kb?
I understand why they have said no, if that was the request.
I saw the size of tinycore shaping slowly, externalizing parts no longer needed or considered accessory (ex. tool for install merged into core plus) and also programs of good quality but left to the choice of users (like dirwin).
In this case I can understand the reaction of developers, a similar request would clash with the coherence maintained so far.

However, if enough hosting corepkg in the repository and generate a db on the server for use, I think it would be feasible test with hypothetical benefical feedback.

[martin]

My apologies, I sometimes think of the most complex way to solve a situation or problem before the simplest occurs to me.

If there is no problem with my last proposal, I will re-change the original website I worked on to explain corepkg and what it does, and my Tiny Core Linux derivative, Nucleus Desktop, will be no more.

[Rich]

Hi Martin C
I don't see a problem if you wish to offer an extension with an alternative package management
system so long as:
1. It does not create excessive traffic on the Tinycore repository, which it doesn't sound like it will.
2. It conforms with how AppBrowser and AppsAudit installs, removes, and deals with packages.

[martin]

1. It does not create excessive traffic on the Tinycore repository, which it doesn't sound like it will.

Absolutely not. That was one of the core (pun not intended) purposes and motivations for its design.

2. It conforms with how AppBrowser and AppsAudit installs, removes, and deals with packages.

When it does install a package, it wraps around tce-load, and absolutely does not get in the way of AppBrowser nor AppsAudit. CoreSoMe is meant to be complementary to AppBrowser and provide similar functionality but uses the CorePkg system instead for working, and part AppsAudit functionality is re-made in CorePkg, so users can user either/or at any time and not be in conflict.

edit: Hey, I want to call a truce to prevent any (further) unnecessary hostilities here. I mean it's all about continuing to make the great TinyCore Linux project even greater, right?

[Rich]

Hi Martin C
While I did not think there would be a problem, I felt those two points were worth highlighting in the
interest of avoiding any conflicts.

[martin]

As a point of closure, I want to show that [Link removed due to forum policy violation] is no longer as it was, but I will leave [Link removed due to forum policy violation] as it is temporarily so that Robert can see for himself that I clearly credited and attributed the project to TinyCore Linux from the very beginning. The USB download is also no longer available.

The main information page (Well, I bought the domain, might as well do something with it) is at [Link removed due to forum policy violation] but it will mostly be to just explain what CorePkg is. I will put the current manual on the front page there, as well as a secondary download link for the tcz.

The .info file will contain a condensed version of the manual.

This is all non-commercial and all GPL'ed, of course.

[martin]

After processing 3,520 packages, the package database is done (comes to 3.5MB). I am now making the corepkg.tcz for download and then everyone will be able to try it out.

[roberts]

You are persisting on ignoring our forum rules.
Please read Forum Rules and Guidelines and specifically rule 2.
Wouldn't you expect users of your forums to be respectful of your posted rules?

[Rich]

Hi Martin C
I removed the links from your post #36. While I feel you started this project with good intentions,
there was some friction created due to some misunderstandings. Hopefully this is now just water
under the bridge and we can all move on.

[martin]

You are persisting on ignoring our forum rules.
Please read Forum Rules and Guidelines and specifically rule 2.

Forum Rules and Guidelines (http://forum.tinycorelinux.net/index.php/topic,7738.0.html)
"2. No attachments/links of binary extensions".

I'm not posting anything.

there was some friction created due to some misunderstandings. Hopefully this is now just water
under the bridge and we can all move on.

I'm trying to, really hard, but it seems roberts doesn't want to let me. I really don't know what I've done to offend so much, really I don't,

and yes, there are one or two forums I help over see in a semi-official capacity, and they don't care about what links are posted, as long as they aren't blatantly commercial. There are implicitly commercial links on these forums in several places, but it seems only I'm being picked on.

edit 2: I had a brief look over some history here before, and roberts is nearly approaching this matter (similar hostility level) along the lines of that guy who tried to create a project called tinycore on sourceforge. Look, that aint me, I just want to create something awesome for TinyCore Linux. I became a fan over 2 years ago, and have been very heavily using it since early December last year, hoping to go to it full-time very soon, but this is all leaving a very sour taste in my mouth.

edit 3: Sent last PM to robert to try and find out, privately, what I have done to annoy him so much. I'm going to make an honorable and civil gesture to find out what is wrong so I can make amends. Is that not a good approach to this? If he doesn't respond, I'll keep working on this, and submitting stuff, but I do not know why he is unresponsive to me.

[Guy]

Martin

What are the advantages and disadvantages of your method compared with the existing.

For example, an advantage may be quicker checking for updates. How much quicker?

What else?

- - - - - - - - - - - - -

Robert

Why don't you give a direct answer expressing what you really think. When you avoid the real issue, people get the impression that you have bad feelings towards them, but they don't know why. Martin was trying to make a positive contribution. He does not know why you are reacting this way.

For example, you may say something like:

"Martin, I appreciate that you are trying to make a contribution, but after the time and effort I put into making the existing apps, I don't want to replace it with another one which has only minor improvements."

That way Martin will know what you are really thinking.

If you show appreciation for him trying to make a contribution, he may not have bad feelings.

Robert, one more thing: Does Martins app offer any genuine advantages? If it does, you may want to include them. That way Tinycore improves, Martin made a positive contribution, and everyone is happy.

[martin]

What are the advantages and disadvantages of your method compared with the existing.

Advantages:
* Scenario: You have two computers, one with an internet connection, one without. The one without, you are trying to install TinyCore Linux on. You know what extension you want, but you don't want to have manually figure out what dependencies it has, plus their dependencies, to copy over. CorePkg can do this. This helps also when downloading larger segments on a restricted on-peak/off-peak internet connection. Get the remainder parts when more convenient and the larger parts in the cheaper data time. One command will fetch them all, after detecting what you already have.
* You grabbed the TCZs, but you didn't get the MD5's. CorePkg can re-generate the md5's for you.
* Primary: tce-update checks all current md5s against those it remotely fetches and compares them. This means network traffic being used, internet speed and repository speed possibly limiting, etc just to check if there is anything newer to retrieve. The time it takes depends on how many extensions you have (and of course, their dependencies). CorePkg does all this without using any network connection at all, except for whenever you fetch the latest database. Think of Debian "apt-get update" (uses network) and "apt-get upgrade" (doesn't use network unless you want it to).
* Instant keyword search. This is just a consequence of the design. In appbrowser, you search for an item, you select Provides, Keyword, etc and it takes a second or so to find what you want, but you may have to select one of the three options to find what you are after. CorePkg does a keyword search in less time than I can measure (I might time it). This is nice, but it's not even a part worth boasting about, it's just a good side benefit.

Disadvantage:
* The program embeds SQLite3 and therefore is approximately 425KB in size. This would make Core Linux that much bigger if included in core.gz (which I'm not asking for at all).
* Until CoreSoMe is mature, the only way to engage the tool is via the command line.

I simply wanted to link to an online manual I wrote for it here and the link was taken down. There was nothing commercial on the page whatsoever, in fact, there were no less than 5 links on one page to tinycorelinux.com.

For example, an advantage may be quicker checking for updates. How much quicker?

Depending on how many extensions you have, a tce-update could take a minute or two. I have a 19Mbps connection and I had to wait about 2 minutes for the check to be done, which depended on the speed of the repository. Using CorePkg, it can verify everything in 1-2 seconds, but I will have to time it first to be more accurate.

edit: timings
corepkg search Xorg
(results excluded)
Found 128 results.

real    0m0.036s
user    0m0.028s
sys     0m0.004s

corepkg -u (detect upgradeable packages) - out of a record of 3,520 packages in the known repository:
456 files processed. (this is how many I had in my tce/optional directory)
Your files are all up to date!

real    0m1.953s
user    0m1.012s
sys     0m0.940s

[Guy]

Martin, I don't make any decisions, Robert and the team may disagree with everything I say.

The program embeds SQLite3 and therefore is approximately 425KB in size. This would make Core Linux that much bigger if included in core.gz (which I'm not asking for at all).

It is unlikely that they would want to include it in core.gz. But then you are not asking them to.


As more and more extensions are added to the Tinycore repository, searching, and checking for updates, may take longer and longer. Robert and the team may consider using a database to make this quicker. A script could be written to update the database every time an extension is updated.

I don't know how this is done in distros with large repositories. Does anyone know?


Martin: If Robert and the team would allow you to submit this as an extension in the Tinycore repository, would you want to. You would need to be responsible for improving it, keeping it up to date, fixing any bugs, etc. You would need to develop it into a professional app. If you did not follow up with these, it is probably better that you do not start.

There would also need to be some agreement on having the database and how to keep it updated.

Robert: Would you consider allowing Martin to submit this as an extension (if he wants to)? Would you consider having a database, and some method of keeping it updated? All options could be considered, but one option is for the database to be another extension.

[martin]

but one option is for the database to be another extension.

That is all I want, and the extension is being built. I'm cleaning up the current new version of a few things and will make the tcz very soon.

It is unlikely that they would want to include it in core.gz. But then you are not asking them to.

That's right.

A script could be written to update the database every time an extension is updated.

I offered to provide this for them. But if not, I have already found a way to keep it pretty up to date with the latest extensions.

You would need to be responsible for improving it, keeping it up to date, fixing any bugs, etc. You would need to develop it into a professional app. If you did not follow up with these, it is probably better that you do not start.

Just affirming, if I wasn't prepared for this, I wouldn't have offered. I definitely would like to be the overseer and maintainer for it.

There would also need to be some agreement on having the database and how to keep it updated.

How I made the current 3.5MB pkg.db:
On my VPS:
* Used lynx to get the directory listing from ibiblio.
* Convert the lynx dump to text.
* Used a bash script to extract only the tcz name.
* Use a bash script to create a fake .tcz file name of the real one (so CorePkg knows to process an extension of that name), download the .info and .md5.txt files from ibiblio.
* Process the folder containing all this with corepkg.

Voila. That generated the package database of all 3.5 thousand current known extensions from the repository. I have automated this to need little work and little oversight for myself, there is no problem doing this 1-2 even 3 times a week. The package database generation took me just a few minutes to do once, then it is available for the client to fetch and use.