ibiblio out of space issues

[bmarkus]

rsync would really help with mirror updates, though.

Sure. But luck of rsync is not a blocking factor. I'm just mirroring the armv6 repo from ibiblio with lftp. lftp can do what you need:

- recursive transfer
- keeps original dates
- copies symlinks (on LINUX)
- check date for new file
- removes files from target if deleted in source
- multiple parallel connections
- etc.

Really nice.

http://www.cyberciti.biz/faq/lftp-mirror-example/

[curaga]

It still does full file downloads, even if only X bytes changed, right?

[bmarkus]

It still does full file downloads, even if only X bytes changed, right?

Yes. I'm not saying it is the best solution, just it works if no better solution is available.

[roberts]

Hi Rich,

Given the current situation I don't see any reason to object.

[roberts]

Without quotas on ibiblio, and with the constantly "out of disk space" something needs to be done.

1. Private server, at home, at least in the US, would have very slow uplink. Our internet access choices are very limited, very slow (compared to many), and very expensive.
2. Most of our mirrors are not "ours" they are mirroring ibiblio. We do not have direct contact. There have been very few that have asked us to become a mirror. As such we do not have upload access for extensions.

I am most interested in any and all proposals.

I work for an Australia based hosting company, but our services are sort of boutique, I guess,  and we don't really do much work that requires massive amounts of drive space. On top of that I do a bit of web dev and hosting on the side, too - If you can get me some stats to work with, I can see if I can pitch something to my boss, or make some arrangement with my own services to help, but I can't guarantee anything. At the very least, I could probably offer to mirror a couple of the smaller repositories (corepure64 and maybe the ARM repos?), but that would be on resources that have a 200GB transfer limit per month. (I think I saw in another thread 47G storage to hold everything, but no other stats really available? What would we be looking at if older versions are kept on ibiblio and frozen, and willing providers were to host individual repositories?).

Current 4.x is 20GB. That is with all older versions 1.x, 2.x, and 3.x to stay on ibiblio.  Sorry, no bandwidth stats are available. Armv6 is already being hosted elsewhere, but I am sure that a mirror would be welcomed. Armv7 is currently using import with much progress and is functional on Google Code. The bulk is x86 and growing x86_64. It is going to be difficult with package management without a single master as ibiblio has provided. Each hosted platform would require a single site to have master status and our scripts would need to be run to generate the support files, indexes and trees, etc.

BTW I have sent one more email to ibiblio inquiring of the possibility of quotas. If that could have happen we could manage our own disk space.

[bmarkus]

Armv6 is already being hosted elsewhere, but I am sure that a mirror would be welcomed.

True. Master repo is in Hungary. I already set up a mirror on my GoDaddy account hosted in the Netherlands. If someone is ready to offer mirror just send me a PM. We can use different mirrors.tcz for each platform, so this can be managed.

[Rich]

Hi roberts
Thanks. I will start a thread for posting links for extensions for testing only with a note that the thread may
be removed once our storage problems are resolved, since this meant as a temporary stopgap measure.
I'll also add a note that all bug reports be handled under  TCE Bugs  so that information remains available
once there is space available in the repository and the extensions are moved there.

[Rich]

Hi roberts
Does this look reasonable?
http://forum.tinycorelinux.net/index.php/topic,14484.msg81969.html#msg81969

[roberts]

OK.

[bmarkus]

Any news on ibiblio space issue? I have updates waiting in the queue for x86 repository. Just asking after reding Juanito's post about new extensions in his Coreoure64 repo

[curaga]

Still out of space, unfortunately.

[Rich]

I took the liberty of adding an index to the thread linked in reply #22 in case that thread starts to get a bit long.

[genec]

Thanks again for a great distro toolkit.  I'm glad to hear a resolution to this issue is in the works and await the final resolution.

For future-resistance, has anyone considered using DNS to help with this in the future?

Code: [Select]

http://repo.tinycorelinux.net/pub/tinycorelinux
or the like?

[bmarkus]

Thanks again for a great distro toolkit.  I'm glad to hear a resolution to this issue is in the works and await the final resolution.

For future-resistance, has anyone considered using DNS to help with this in the future?

Code: [Select]

http://repo.tinycorelinux.net/pub/tinycorelinux
or the like?

ARM v6 (Raspberry Pi) repo is already using http://repo.picorelinux.org

But the core issue is lack of storage space. DNS will not give you a realible and  free storage capacity with rsync.

[genec]

But the core issue is lack of storage space. DNS will not give you a realible and  free storage capacity with rsync.

Yes.  I realize this.  I'm more thinking that if by some chance things need to move again in the future.