User mirroring of the repo.

[Jason W]

Ibiblio has been kind enough to grant us server space and bandwidth to host Tinycore.  I have from the start kept a mirror of our stuff on ibiblio since I am the one responsible for maintaining extensions and their sources on the server.  And as the keeper of our repo, I feel the need to mention my concern for the trend of users to download the entire ibiblio repo.  I know there are one or some of you that maintain a copy of the tcz repo to run checks on for discrepencies and report back to us.  As always we appreciate help in finding errors and those kind of things.  But for the average user unless there is a good reason, such as setting up a company intranet wide extension server, planning for an extended period with no web access,  or whatever, I am asking users not to engage in downloading the entire repo.  It is being a much better patron of ibiblio to download what you need to your tce directory.  The appsaudit and tce-update tools will keep your extensions up to date without rsyncing or wgetting the entire repo.

Our full tinycore/ directory contains 11GB of data.  There really is no reason that a normal user needs to sync this to their PC.  You may notice that at times ibiblio seems slow.  Users syncing that 11GB of data (or the 1.9GB /tcz repo directory) can only contribute to that slowness.  

Gentoo has rules set up on their servers that limit the amount of "emerge --sync" that can be done in a days time based on IP address.  And they ask their community to respect the resources of their servers.  I am simply asking the TC community to show ibiblio the same respect.  I would rather us police ourselves than to force ibiblio to place limits on Tinycore.

[AlabamaPaul]

I fear I may be guilty of this, as I maintain an internal mirror for use on multiple TC machines.
However, I only perform this sync once a day around 01:11 CST (07:11 GMT).
I do not download the entire repo, my script uses the following basic algorithm:
* Download info.lst file
* For each entry XXX in info.lst:
** Download XXX.md5.txt file
** If local XXX.md5.txt is missing, or does not match:
**** Download extension files (XXX.tcz, XXX.dep, XXX.list, XXX.info)

The script then builds the local info.lst and an index.html file similar to one found on Ibiblio

Is this acceptable ? If not, then what would you suggest I change ?

I realize there are some mirrors I could use, but I believe there is nothing "official".
Perhaps the project could maintain a mirrors.lst file in the repository directory that contains a list of "official" mirrors. This way, I (and others like me) could download this list and use a mirror.
If this works, then eventually, appbrowser could be modified to take advantage of the mirror list as well.

[roberts]

I am working on a mirrors.lst.  But it use will be via Select Mirror GUI from the Control Panel.
appbrowser, ab, and appbrowser-cli get the site via /opt/.tcrc. Select Mirror is the GUI for maintenace of that file.

[AlabamaPaul]

Great.
Is there any reason why you should not place the mirrors.lst file in the repository directory ?
By placing the mirrors.lst file in the repository, the Select Mirrors, Control Panel GUI could have an option to allow the user to download the most recent mirrors list.

[gerald_clark]

Why not mirrors.tcz ?

[AlabamaPaul]

Because I'm not suggesting an extension (which would use the .tcz naming convention). I was suggesting a simple text file that contained a list of all the "official" mirrors, each on a new line.
This is similar to the info.lst file that appbroswer (and all it's variants) uses. The info.lst file contains a list of all extensions available in the repository.

[gerald_clark]

This list can be in a tcz.
Then there is no requirement for any special "Control Panel GUI" code to update it when it changes.
Tce-update will update it automatically.

[roberts]

The GUI is to select which one for the list!
You have to be kidding to waste a mount point or require a special flag for a tiny text file!

[gerald_clark]

Robert, I was responding to AlabamaPaul's suggestion to have a Control Panel GUI to download the Mirror List.
My comment was that the mirror list could be mirror-list.tcz.
Then Appbrowser could be used to download the mirror list, and tce-update would update it automatically.
This requires no special Control Panel GUI.
This requires no special flag.
Is it a waste of a mount point? If so, the file could be copied, and the tcz unmounted.

[AlabamaPaul]

Gerald,
I believe you misunderstood my post. I am not proposing a new application, only a simple list. There is already a Control Panel GUI for Select Mirror, I was merely suggesting that the existing GUI could have an option to download the list.

Paul

[roberts]

The mirror.lst is posted in tcz repository.
Yes. The plan is that the GUI mirror selection program populates from this list.

[gerald_clark]

Understood now.

[oleg]

It is not possible to prevent people from downloading the whole repo. Because some try to install TC many times before finding the best configuration. Waiting every time to download extensions is not handy. Additional caching proxy is necessary or manual manipulating of the files. And then dependencies. Pain.

My proposition: Select most popular stable extensions which are not in testing mode. Create iso image and ask people to use torrent.

[roberts]

If, as we have politely asked, users would only download their "most popular" extensions, we would not be experiencing slow downs which affect everybody. Nobody requires eveything in the repository. It is a mater of being a good net citizen and using our free hosted bandwidth respectfully.

Since the idea of creating a custom prebuilt iso keeps coming up, let me try to restate my vision and reason for starting this project...

It used to be that we went to the music store and purchased music cds. Many times these music cds had songs that we did not care for. I know I have many times used the remote to skip certain songs. On some cd players you can program tracks. Then along came the ipod like devices. They do not come with preselected music. You download and pay for just the titles that you like. Music cds have begun to disappear as giving the users choice is obviously better.

The same concept applies to Tiny/Micro Core. Think of Core as an ipod like device for playing your favorite Linux tunes (apps). Our repository is like itunes, you get to select that which you wish to use.

Now just as you would not try to download every title from itunes... Well you get the picture on both downloading everything and prebuilt isos.

[helander]

I imagine that one of the reasons why people download the full repo is because there is no support for installing, including resolving dependencies, from multiple repos.

In case you have your own extensions not available in the official repo and they depend on extensions in the official repo (which most of them likely) you are caught in error prone manual operations. I was caught in this about 8 months ago so I modified tce-load etc to support loading from multiple repos,(one local, one at a friend and the official) in that order. It worked very fine but with the pace that tce-load et al changed it became too much work to keep that solution up-to-date and reverted to having a large portion of the official repo available locally. I do an rsync (only tranfser differences) about twice a week in order not to be a too bad citizen.

If you would be interested to include this kind of multirepo support, provided by many other distros, I would be happy to provide a protoytype implementation.

It will not likely solve all the bandwidth problems, but it could possibly reduce some of them.


Kind Regards

Lars