ibiblio out of space issues

[bmarkus]

We see frequent ibiblio out of space issues, they do not have enough storage to host growing repositories including Tiny Core Linux. Fortunately, there are more than 10 mirrors replicating repo all around the World. My proposal is to drop ibiblio as a repository and use only mirrors with a private master. In such case master is not listed as mirror, so can be anywhere even at home with 7/24 broadband connection. Except initial setup it is not bandwith critical. Storage is cheap, an old PC would be OK with a RAID1 HD. Even you can use Core as OP system If  master is lost, a new master can be resynced from a mirror. Using a subdomain what you can online maintain you can move server around without reconfiguring morrors!

This is not a unique solution, there are vendors with no own repo offered.

Also you can segment repo on platforms with different mirror list and mirrors (and different master) for x86, armv6, ...

I can offer a master for armv6 with a dedicated domain sitting on a broadband link if there are mirrors.

It would work and would offer better realibility cutting dependency on ibiblio.

[aus9]

Hi

Would you recommend users delay submitting new packages, until a new repository system is in place?

cheers

[Lee]

I have a hosting account that is very under-utilized.  The hosting service has been very reliable over the years - though I haven't ever exactly stressed it.

What are the requirements for hosting a repo - in terms of storage and bandwidth?

[bmarkus]

I have a hosting account that is very under-utilized.  The hosting service has been very reliable over the years - though I haven't ever exactly stressed it.

What are the requirements for hosting a repo - in terms of storage and bandwidth?

Is it a shared WEB hosting, virtual server or dedicated machine? Check you contract. If it is a WEB account, most suppliers do not allow different use, like file sharing even if you are within bandwidth or storage limit specially if it is 'unlimited'.

[Lee]

Yes, this is shared web hosting and yes, "unlimited" means -as long as it doesn't impact server performance or capacity- rather than what you or I or Webster Dictionary mean by "unlimited".

[bmarkus]

For mirror, you must run rsync periodically, for master repo must provide rsync daemon running.

Usual cause that content out of WEB pages are prohibited

[Lee]

I could run whatever I want on the server in my cellar - but my bandwidth would be a limiting factor, especially outbound.

[roberts]

As for a shared web hosting, recall, Tiny Core was kicked off Bluehost for excessive bandwidth and server load, And that was just our forum and wiki! And I was paying for the "unlimited" service.

[aus9]

http://sourceforge.net/ has certain rules for allowing projects to have downloads....summary  open source

Each project has all their files under one name eg http://sourceforge.net/projects/rkhunter/
But that project removes older versions and offers not much else as its not a distro

an E17 Linux distro beginning with B uses it to host isos and image files but uses a different site for their individual files as they use an APT package system

I assume you have dismissed this site because of its non-repository type structure
but had you considered it for the just isos and images, kernels and core.gzs?

cheers

EDIT

They allow forums and wikis there as well but you won't like the wiki structure too much, the one I am involved with is a "closed" wiki meaning needs a login and password approved by some admin type

http://sourceforge.net/apps/trac/rkhunter/wiki/MPRKH#Contents

[roberts]

We see frequent ibiblio out of space issues, they do not have enough storage to host growing repositories including Tiny Core Linux. Fortunately, there are more than 10 mirrors replicating repo all around the World. My proposal is to drop ibiblio as a repository and use only mirrors with a private master. In such case master is not listed as mirror, so can be anywhere even at home with 7/24 broadband connection. Except initial setup it is not bandwith critical. Storage is cheap, an old PC would be OK with a RAID1 HD. Even you can use Core as OP system If  master is lost, a new master can be resynced from a mirror. Using a subdomain what you can online maintain you can move server around without reconfiguring morrors!

This is not a unique solution, there are vendors with no own repo offered.

Also you can segment repo on platforms with different mirror list and mirrors (and different master) for x86, armv6, ...

I can offer a master for armv6 with a dedicated domain sitting on a broadband link if there are mirrors.

It would work and would offer better realibility cutting dependency on ibiblio.

Without quotas on ibiblio, and with the constantly "out of disk space" something needs to be done.

1. Private server, at home, at least in the US, would have very slow uplink. Our internet access choices are very limited, very slow (compared to many), and very expensive.
2. Most of our mirrors are not "ours" they are mirroring ibiblio. We do not have direct contact. There have been very few that have asked us to become a mirror. As such we do not have upload access for extensions.

I am most interested in any and all proposals.

If there are any providers that benefit from Tiny Core and are able to host, please post here or PM me.
If there are any other Universities that are able to offer file distribution services, please post here or PM me.

We only need file distribtion, limited upload only by dedicated team member, with unlimited downloads by the community.

We do not need web page, forum, or wiki hosting. We have that covered. Only file repository.

Even if the repository is broken up by platform with several provides hosting only a single platform would be great.

We are only a small team of volunteers. There has never been a request for donations. Money has never been a need, goal ir desire for this project. The team, including myself, do not wish to be involved with finances. So we are hoping that some or several providers will step forward to offer free file hosting.

We can freeze older repositories and hopefully leave them here on ibiblio. Only move 4.x which includes x86, x86_64, armv6, and armv7.

[aus9]

fair enough so I won't link to the forum post but some soul with the same intent posted

FYI - I finally solved my problem by going to DistroWatch, looking at the mirrors for the top distros, and e-mailing the administrators of each one I found asking if they would host the new distro. I stopped after finding three. While mirror lists like the one that Aron googled up were helpful, most of the mirrors contained on those lists didn't respond. University LUGs were the most helpful.

looking at the secondary mirrors for debian as example I can see strings like
something.bjtu.edu.cn
ftp.monash.edu.au
mirror.aarnet.edu.au

so I hope that helps

what I am suggesting is you delegate one or more of your dev team to email the unis direct and ask for hosting assistance

If there are any other Universities that are able to offer file distribution services, please post here or PM me.

I am assuming you have no objections if members take initiative and email their home country unis?

If so,  I can email aussie unis but will wait for confirmatiion that is what you prefer

2) If so, what exact links to you what them to see

would these suffice?
http://distro.ibiblio.org/tinycorelinux/downloads.html

and

http://distro.ibiblio.org/tinycorelinux/arm.html

and

this actual thread



thanks for reading

[althalus]

Without quotas on ibiblio, and with the constantly "out of disk space" something needs to be done.

1. Private server, at home, at least in the US, would have very slow uplink. Our internet access choices are very limited, very slow (compared to many), and very expensive.
2. Most of our mirrors are not "ours" they are mirroring ibiblio. We do not have direct contact. There have been very few that have asked us to become a mirror. As such we do not have upload access for extensions.

I am most interested in any and all proposals.

I work for an Australia based hosting company, but our services are sort of boutique, I guess,  and we don't really do much work that requires massive amounts of drive space. On top of that I do a bit of web dev and hosting on the side, too - If you can get me some stats to work with, I can see if I can pitch something to my boss, or make some arrangement with my own services to help, but I can't guarantee anything. At the very least, I could probably offer to mirror a couple of the smaller repositories (corepure64 and maybe the ARM repos?), but that would be on resources that have a 200GB transfer limit per month. (I think I saw in another thread 47G storage to hold everything, but no other stats really available? What would we be looking at if older versions are kept on ibiblio and frozen, and willing providers were to host individual repositories?).

[Rich]

Hi roberts
I received a PM from a forum member requesting guidance on the possibility of posting extensions for testing
since ibiblio is out of space. Maybe it would be OK to post a link to code.google.com under
Programming & Scripting - Unofficial  in the interim? Your thoughts please.

[bmarkus]

Just a correction, to have rsync is not a must have. You can do mirroring with other tools, like lftp.

[curaga]

rsync would really help with mirror updates, though.