Proposed improvement of repo and related tools

[helander]

The idea described here came from a tool I have made. This tools helps to automate the process of creating .dep files for the extensions I am building. The tool needs to have access to the *.tcz.list files from the repo. So one request I have is to have all these files packed into a tar.gz file (tcz.list.tar.gz) and have that file available to download from the repo (e.g. using wget). If the proposal described below will not find its way into the repo/tools, I hope that at least tcz.list.tar.gz could be made available

I think appbrowser today loads info, list and dep files on demand. Sometimes when the repo server is loaded or the network (local or broadband) is loaded, it could take time to navigate in appbroswer, due to the loading of these files. What if, these files were available locally, then appbrowser would only need to contact the repo when actually downloading a .tcz.

The sizes of these files are approx:

   tcz.info.tar.gz 200kB
   tcz.dep.tar.gz   25lB
   tcz.list.tar.gz       1MB

These fles could be downloaded when you connect to the repo and having a checksum file associated with each of them that could be checked to see if an up to date file is already available locally. On an 8 Mbps connection the loading of these files (overhead at connect if up to date files not cached locally) should take just a second or two. The real benefit for the repo server will be that most sessions will only hit the server with requests when .tcz files are downloaded and not when the user is navigating around among the extensions.


Kind Regards

Lars

[curaga]

Our cloud users would be hitting it for 1.225mb for every boot, instead of the few kilobytes of their own browsing.

[helander]

Not on every boot, but the first time after boot using appbrowser (or related tool). For users with low bandwidth connections, they could save these files as part of their backup, however the rate by which the repo changes may still require a download due to non-matching checksum.

Another way to deal with the problem of users with low bandwidth connections is to have a menu entry in appbrowser that allow you to download these files and once that have been done appbrowser work on the local cache instead of fetching each extension specific file from the repo.

[roberts]

I think we need to stay on track to use the existing infrastructure provided.
Starting with v2.8 we now have easy access to additional mirrors.

Much effort has been made to make the the system more simple to use rather than to add optional methods which would imply even larger downloads, which would only get even larger, or otherwise trying to stay in sync with a regularly updated repository.

[maro]

Hmmm, a bit a tricky question. On the one hand I fully support Robert's view to keep things simple and minimise the downloads for cloud users. OTOH, I can see the benefit of the proposal from a development POV. I guess one of the main reasons for creating my own private TCZ directory mirror was that I wanted to know the content of all the extensions (and not only via the appbrowser on a per session basis).

Well, I now wonder to which degree the SSH access of the core team on to ibiblio.org would allow to run scripts (or even cron jobs). I guess that way it would be possible to create updates of the proposed archives (if changes have occured) without wasting too much precious time of the core team members. Since I've done these kind of scripts at work I can offer my services here to help out.

Regarding the appbrowser, I'd agree with the view expressed by Robert. But there might be room for a community created extension of a "appbrowser DeLuxe". Surely not as part of the base system, and probably not as something that the core team is expected to maintain. I can see a potential conflict here, but since AFAIK the source code is avaiable it should be relatively easy to stay compatible. In the end (at least as far as I understand it) the appbrowser is "just" a GUI on top of 'tce-load'.

So in summary: It would be really nice if the proposal could be implemented on a purely optional basis.
Just my 2c ...