Tiny Core Linux
General TC => General TC Talk => Topic started by: bmarkus on November 20, 2012, 09:43:15 AM
-
We see frequent ibiblio out of space issues, they do not have enough storage to host growing repositories including Tiny Core Linux. Fortunately, there are more than 10 mirrors replicating repo all around the World. My proposal is to drop ibiblio as a repository and use only mirrors with a private master. In such case master is not listed as mirror, so can be anywhere even at home with 7/24 broadband connection. Except initial setup it is not bandwith critical. Storage is cheap, an old PC would be OK with a RAID1 HD. Even you can use Core as OP system :) If master is lost, a new master can be resynced from a mirror. Using a subdomain what you can online maintain you can move server around without reconfiguring morrors!
This is not a unique solution, there are vendors with no own repo offered.
Also you can segment repo on platforms with different mirror list and mirrors (and different master) for x86, armv6, ...
I can offer a master for armv6 with a dedicated domain sitting on a broadband link if there are mirrors.
It would work and would offer better realibility cutting dependency on ibiblio.
-
Hi
Would you recommend users delay submitting new packages, until a new repository system is in place?
cheers
-
I have a hosting account that is very under-utilized. The hosting service has been very reliable over the years - though I haven't ever exactly stressed it.
What are the requirements for hosting a repo - in terms of storage and bandwidth?
-
I have a hosting account that is very under-utilized. The hosting service has been very reliable over the years - though I haven't ever exactly stressed it.
What are the requirements for hosting a repo - in terms of storage and bandwidth?
Is it a shared WEB hosting, virtual server or dedicated machine? Check you contract. If it is a WEB account, most suppliers do not allow different use, like file sharing even if you are within bandwidth or storage limit specially if it is 'unlimited'.
-
Yes, this is shared web hosting and yes, "unlimited" means -as long as it doesn't impact server performance or capacity- rather than what you or I or Webster Dictionary mean by "unlimited".
-
For mirror, you must run rsync periodically, for master repo must provide rsync daemon running.
Usual cause that content out of WEB pages are prohibited
-
I could run whatever I want on the server in my cellar - but my bandwidth would be a limiting factor, especially outbound.
-
As for a shared web hosting, recall, Tiny Core was kicked off Bluehost for excessive bandwidth and server load, And that was just our forum and wiki! And I was paying for the "unlimited" service.
-
http://sourceforge.net/ has certain rules for allowing projects to have downloads....summary open source
Each project has all their files under one name eg http://sourceforge.net/projects/rkhunter/
But that project removes older versions and offers not much else as its not a distro
an E17 Linux distro beginning with B uses it to host isos and image files but uses a different site for their individual files as they use an APT package system
I assume you have dismissed this site because of its non-repository type structure
but had you considered it for the just isos and images, kernels and core.gzs?
cheers
EDIT
They allow forums and wikis there as well but you won't like the wiki structure too much, the one I am involved with is a "closed" wiki meaning needs a login and password approved by some admin type
http://sourceforge.net/apps/trac/rkhunter/wiki/MPRKH#Contents
-
We see frequent ibiblio out of space issues, they do not have enough storage to host growing repositories including Tiny Core Linux. Fortunately, there are more than 10 mirrors replicating repo all around the World. My proposal is to drop ibiblio as a repository and use only mirrors with a private master. In such case master is not listed as mirror, so can be anywhere even at home with 7/24 broadband connection. Except initial setup it is not bandwith critical. Storage is cheap, an old PC would be OK with a RAID1 HD. Even you can use Core as OP system :) If master is lost, a new master can be resynced from a mirror. Using a subdomain what you can online maintain you can move server around without reconfiguring morrors!
This is not a unique solution, there are vendors with no own repo offered.
Also you can segment repo on platforms with different mirror list and mirrors (and different master) for x86, armv6, ...
I can offer a master for armv6 with a dedicated domain sitting on a broadband link if there are mirrors.
It would work and would offer better realibility cutting dependency on ibiblio.
Without quotas on ibiblio, and with the constantly "out of disk space" something needs to be done.
1. Private server, at home, at least in the US, would have very slow uplink. Our internet access choices are very limited, very slow (compared to many), and very expensive.
2. Most of our mirrors are not "ours" they are mirroring ibiblio. We do not have direct contact. There have been very few that have asked us to become a mirror. As such we do not have upload access for extensions.
I am most interested in any and all proposals.
If there are any providers that benefit from Tiny Core and are able to host, please post here or PM me.
If there are any other Universities that are able to offer file distribution services, please post here or PM me.
We only need file distribtion, limited upload only by dedicated team member, with unlimited downloads by the community.
We do not need web page, forum, or wiki hosting. We have that covered. Only file repository.
Even if the repository is broken up by platform with several provides hosting only a single platform would be great.
We are only a small team of volunteers. There has never been a request for donations. Money has never been a need, goal ir desire for this project. The team, including myself, do not wish to be involved with finances. So we are hoping that some or several providers will step forward to offer free file hosting.
We can freeze older repositories and hopefully leave them here on ibiblio. Only move 4.x which includes x86, x86_64, armv6, and armv7.
-
fair enough so I won't link to the forum post but some soul with the same intent posted
FYI - I finally solved my problem by going to DistroWatch, looking at the mirrors for the top distros, and e-mailing the administrators of each one I found asking if they would host the new distro. I stopped after finding three. While mirror lists like the one that Aron googled up were helpful, most of the mirrors contained on those lists didn't respond. University LUGs were the most helpful.
looking at the secondary mirrors for debian as example I can see strings like
something.bjtu.edu.cn
ftp.monash.edu.au
mirror.aarnet.edu.au
so I hope that helps
what I am suggesting is you delegate one or more of your dev team to email the unis direct and ask for hosting assistance
If there are any other Universities that are able to offer file distribution services, please post here or PM me.
I am assuming you have no objections if members take initiative and email their home country unis?
If so, I can email aussie unis but will wait for confirmatiion that is what you prefer
2) If so, what exact links to you what them to see
would these suffice?
http://distro.ibiblio.org/tinycorelinux/downloads.html
and
http://distro.ibiblio.org/tinycorelinux/arm.html
and
this actual thread
thanks for reading
-
Without quotas on ibiblio, and with the constantly "out of disk space" something needs to be done.
1. Private server, at home, at least in the US, would have very slow uplink. Our internet access choices are very limited, very slow (compared to many), and very expensive.
2. Most of our mirrors are not "ours" they are mirroring ibiblio. We do not have direct contact. There have been very few that have asked us to become a mirror. As such we do not have upload access for extensions.
I am most interested in any and all proposals.
I work for an Australia based hosting company, but our services are sort of boutique, I guess, and we don't really do much work that requires massive amounts of drive space. On top of that I do a bit of web dev and hosting on the side, too - If you can get me some stats to work with, I can see if I can pitch something to my boss, or make some arrangement with my own services to help, but I can't guarantee anything. At the very least, I could probably offer to mirror a couple of the smaller repositories (corepure64 and maybe the ARM repos?), but that would be on resources that have a 200GB transfer limit per month. (I think I saw in another thread 47G storage to hold everything, but no other stats really available? What would we be looking at if older versions are kept on ibiblio and frozen, and willing providers were to host individual repositories?).
-
Hi roberts
I received a PM from a forum member requesting guidance on the possibility of posting extensions for testing
since ibiblio is out of space. Maybe it would be OK to post a link to code.google.com under
Programming & Scripting - Unofficial in the interim? Your thoughts please.
-
Just a correction, to have rsync is not a must have. You can do mirroring with other tools, like lftp.
-
rsync would really help with mirror updates, though.
-
rsync would really help with mirror updates, though.
Sure. But luck of rsync is not a blocking factor. I'm just mirroring the armv6 repo from ibiblio with lftp. lftp can do what you need:
- recursive transfer
- keeps original dates
- copies symlinks (on LINUX)
- check date for new file
- removes files from target if deleted in source
- multiple parallel connections
- etc.
Really nice.
http://www.cyberciti.biz/faq/lftp-mirror-example/
-
It still does full file downloads, even if only X bytes changed, right?
-
It still does full file downloads, even if only X bytes changed, right?
Yes. I'm not saying it is the best solution, just it works if no better solution is available.
-
Hi Rich,
Given the current situation I don't see any reason to object.
-
Without quotas on ibiblio, and with the constantly "out of disk space" something needs to be done.
1. Private server, at home, at least in the US, would have very slow uplink. Our internet access choices are very limited, very slow (compared to many), and very expensive.
2. Most of our mirrors are not "ours" they are mirroring ibiblio. We do not have direct contact. There have been very few that have asked us to become a mirror. As such we do not have upload access for extensions.
I am most interested in any and all proposals.
I work for an Australia based hosting company, but our services are sort of boutique, I guess, and we don't really do much work that requires massive amounts of drive space. On top of that I do a bit of web dev and hosting on the side, too - If you can get me some stats to work with, I can see if I can pitch something to my boss, or make some arrangement with my own services to help, but I can't guarantee anything. At the very least, I could probably offer to mirror a couple of the smaller repositories (corepure64 and maybe the ARM repos?), but that would be on resources that have a 200GB transfer limit per month. (I think I saw in another thread 47G storage to hold everything, but no other stats really available? What would we be looking at if older versions are kept on ibiblio and frozen, and willing providers were to host individual repositories?).
Current 4.x is 20GB. That is with all older versions 1.x, 2.x, and 3.x to stay on ibiblio. Sorry, no bandwidth stats are available. Armv6 is already being hosted elsewhere, but I am sure that a mirror would be welcomed. Armv7 is currently using import with much progress and is functional on Google Code. The bulk is x86 and growing x86_64. It is going to be difficult with package management without a single master as ibiblio has provided. Each hosted platform would require a single site to have master status and our scripts would need to be run to generate the support files, indexes and trees, etc.
BTW I have sent one more email to ibiblio inquiring of the possibility of quotas. If that could have happen we could manage our own disk space.
-
Armv6 is already being hosted elsewhere, but I am sure that a mirror would be welcomed.
True. Master repo is in Hungary. I already set up a mirror on my GoDaddy account hosted in the Netherlands. If someone is ready to offer mirror just send me a PM. We can use different mirrors.tcz for each platform, so this can be managed.
-
Hi roberts
Thanks. I will start a thread for posting links for extensions for testing only with a note that the thread may
be removed once our storage problems are resolved, since this meant as a temporary stopgap measure.
I'll also add a note that all bug reports be handled under TCE Bugs so that information remains available
once there is space available in the repository and the extensions are moved there.
-
Hi roberts
Does this look reasonable?
http://forum.tinycorelinux.net/index.php/topic,14484.msg81969.html#msg81969
-
OK.
-
Any news on ibiblio space issue? I have updates waiting in the queue for x86 repository. Just asking after reding Juanito's post about new extensions in his Coreoure64 repo :)
-
Still out of space, unfortunately.
-
I took the liberty of adding an index to the thread linked in reply #22 in case that thread starts to get a bit long.
-
Thanks again for a great distro toolkit. I'm glad to hear a resolution to this issue is in the works and await the final resolution.
For future-resistance, has anyone considered using DNS to help with this in the future?
http://repo.tinycorelinux.net/pub/tinycorelinux
or the like?
-
Thanks again for a great distro toolkit. I'm glad to hear a resolution to this issue is in the works and await the final resolution.
For future-resistance, has anyone considered using DNS to help with this in the future?
http://repo.tinycorelinux.net/pub/tinycorelinux
or the like?
ARM v6 (Raspberry Pi) repo is already using http://repo.picorelinux.org
But the core issue is lack of storage space. DNS will not give you a realible and free storage capacity with rsync.
-
But the core issue is lack of storage space. DNS will not give you a realible and free storage capacity with rsync.
Yes. I realize this. I'm more thinking that if by some chance things need to move again in the future.
-
If we ask to publish an article of request on information sites like distrowatch?
-
We're hopefully over the issue now: http://forum.tinycorelinux.net/index.php/topic,14546.0.html
Please report any issues in this thread.
bmarkus, would you like to keep the armv6 master still?
-
bmarkus, would you like to keep the armv6 master still?
Yes, it makes development more efficient in this phase, no wait for new extensions, changes appear in repo.
Of course we can rsync it to the official repo if there is a space and add to armv6 mirror list.
-
Hi curaga
what am I supposed to see when the server moves please?
my time is 8 hours ahead of UTC and your other post is time December 14, 2012, 12:36:36 PM
trivia below
when I click at TC main site to Downloads I see
http://distro.ibiblio.org/tinycorelinux/downloads.html
mirrors.tcz is showing a Nov date
cheers
we are a bit slow in Australia so take your time replying if at all ;D
-
Hm, yes, the top bar here at the forum still points to the old location. The wiki doesn't seem to have such links outside the wiki.
Mirrors.tcz and a new release defaulting to the new location will be out later, now it's mainly the website. When you put "tinycorelinux.net", you should see "tinycorelinux.net", not a redirect to ibiblio. It may take up to a day if your provider's DNS is slow.
edit: The top bar should now have correct links.
-
Hi curaga
The Tinycore website feels a lot snappier now. Page loads are almost instantaneous now.
Are you still in the process of moving files? When I navigate to http://tinycorelinux.net/4.x/x86/tcz/ all I see is
a page of text listing extensions, no way to download them. On http://tinycorelinux.net/4.x/x86/scm/ I get: mostly
dates and file sizes with the last 9 entries having partial file names. Clicking on them returns Not Found.
-
Hi curaga
Also the / entry in http://tinycorelinux.net/4.x/x86/scm/ takes me to the Tinycore website instead of moving up one
directory level.
-
Once you are seeing the new site and no longer being redirected to ibiblio then please edit /opt/tcemirror to use the new site: http://repo.tinycorelinux.net then apps GUI and scmaps GUI should work with the new site. I will be releasing a 7.2 update with /opt/tcemirror defaulting to repo.tinycorelinux.net in a couple of days.
-
Please report any issues in this thread.
Navigating to http://www.tinycorelinux.net/4.x/x86_64/tcz/ with a browser, I see a three column table - name/last time modified/size - name is blank, but the other two columns have data.
-
Repo access is really fast from Budapest. Well it is hosted in Germany, not in the States. Performace is enhanced, not just the space issue solved. From US it may look different.
-
The text list was intentional, it is a lot faster to just push info.lst than to generate a file listing of a huge directory. Also helps in gently nudging to avoid direct extension downloads ;)
The listing generation is a big cpu user if allowed in such a dir. Jason and Juanito, please adjust your scripts so that scm and x86_64/tcz get the unpacked info.lst generated for them for this reason.
-
Hi curaga
That's unfortunate. Every once in a while I download a package so I can look inside it without installing it and
all its dependencies.
-
The files are there of course, so copy-pasting the name will work. But we also tried a pre-generated html extension page before, and that was still huge (a couple mb IIRC).
-
Hi curaga
Thanks, that works. Sounds like a good trade off to help keep the repo running smoothly.
-
Rich, Apps GUI browser tabs, or tce select options, displays each of the files on the server.
If that is not sufficient there is always tce-fetch.sh to actually download an individual file.
-
Thankyou team for the change of main server, my links now ok
I rebooted with new tcemirror
running APPS -> check updates still finds the old Xprogs v 4.7
In case you had this on your todo list, forgive me
otherwise feel free to shoot the messenger
8)
-
Should be the correct Xprogs, however the info file had not been updated. I have now updated the info file.
Easy to verify, 4.7.1 Xprogs has add and delete buttons on Selective Copy Install to File System.
-
Thanks to whoever helped resolve this menacing issue. When will the new site be available for use?
-
Pages http://tinycorelinux.net/... are not available.
-
There appears to be a hardware fault.
-
I am glad you guys have found the cause. I don't run any servers so know zero but you may be interested in some checks?
http://www.htmlhelp.com/links/validators.htm
http://tool.motoricerca.info/robots-checker.phtml
and maybe for later, to cut down what bots are indexing
http://techpatterns.com/forums/about228.html
-
Does this image help anyone?
(http://oi48.tinypic.com/2lbjm8p.jpg)
based on that error maybe this helps?
http://www.faqforge.com/linux/controlpanels/how-to-solve-the-php-xcache-error-devzero-no-space-left-on-device/
http://stackoverflow.com/questions/5412435/what-does-write-failed-no-space-left-on-device-mean
sorry if I am misleading and wasting time
-
Hi aus9
I saw the same when I checked a few minutes ago, now the Wiki looks fine again.
-
As of now, I can connect to the forum page, but the home page still gives "unable to connect" error.
-
Hi thane and all forum members
No doubt moving the repo and the website is a massive task, so some hiccups and issues should be expected
along the way. The last report was that there was a hardware fault, which they may still be working on. Once
the new server is back on line, I'm sure it will be announced in this thread. Thank you for your patience.
-
The server experienced a hardware failure, resulting in that a new server is now being setup.
Obviously hardware failure is beyond our control. Now much work to do to start again.
We appreciate your patience during this time.
-
We should up be up now.
Note that appbrowser search/provides will be down until further notice, see http://forum.tinycorelinux.net/index.php/topic,13996.msg82480.html#msg82480
-
We should up be up now.
Note that appbrowser search/provides will be down until further notice, see http://forum.tinycorelinux.net/index.php/topic,13996.msg82480.html#msg82480
The wait for local provides searching won't be long. I have finished 4.7.2 for x86 and is currently in Team testing.
-
Hi roberts
Since the new repository is now up, would you like me to remove the Extensions for testing thread?
-
With Curaga having locked it and with a closing statement explaining the reason, that should be sufficient.
-
ibilio is still in the updated mirror list. Is there space now to keep repository content<
-
Yes, so far we've been able to sync to ibiblio, so it is current too (as are all mirrors fetching from ibiblio).
So if some closer mirror is faster than the primary, you can still use that, it'll just have an additional delay of one day compared to when ibiblio was the master.
-
What are plans now about repos of versions 1.x, 2.x, 3.x ?