Creating High Bandwidth mirror of the repo?

[tinydancer]

I noticed the pinned topic at the top of the forum discouraging the downloading of the entire mirror at distro.ibiblio.org, but I'm interested in creating a mirror myself. I have a helluva lot of bandwidth and I'd like to put it to good use. At the moment I barely get 100kB/s from the main mirror but from testing my connection speeds from a friends house I can upload at ~20M/S. What's the best way to get a copy of the mirror & publicise it?

Also, I just signed up and this forum seems to have a lot of anti-spam stuff? The whole click the similar image, then rotate the picture, then type in the letters you can barely read. Madness.

[curaga]

Ibiblio offers rsync. But for the initial fetch you might want to use a mirror closer to you, and then keep it up to date with rsync+ibiblio.

Publicizing it would consist of editing the wiki page "Mirrors"

The anti-spam measures will go away after a couple posts. We used to have a lot of spam.

[tinydancer]

Cheers for that... I'm starting an rsync grab right now and I'll post on how that goes

[tinydancer]

OK so the rsync worked, and my mirror is up & available at

http://213.138.101.164/tinycorelinux

so if you want to use that instead of the ibiblio mirror you can change /opt/tcemirror to reflect this.

NOTE - If you want to browse this in a web browser, you must visit the full URL of

http://213.138.101.164/tinycorelinux/4.x/x86/tcz

This is so I could tighten up my apache config and make that directory the only public facing one on my server.

Another point - I have the whole mirror on my machine and I can't guarantee how long it's going to be there for. (I can comfortably host it for 2-3 months if not longer, but I don't want to commit to it being on the wiki in case I have to drop out.) For that reason I'm perfectly happy for people to download my copy of the mirror. It's up to date as of 29/6/2012.

I can make this available by rsync or as a tgz over HTTP but I would much prefer to torrent this and decrease the load on my server.

If anyone can recommend a torrent tracker I would appreciate it.

Please note that in accordance with the GPL, all sources present on the original mirror *are* contained in this download. It is approx 12GB.

[curaga]

Why not just put it up on the pirate bay? We occasionally get requests for a torrent of the whole repo, so that'd be useful, even if only valid for a few months.

[xk2600]

What about just adding a CNAME (Alias) DNS entries for repo.tinycorelinux.net pointing to:

[countrycode].repo.tinycorelinux.net

Then have multiple A/CNAME records for each mirror in that country.

For example:

Code: [Select]

$ORIGIN tinycorelinux.net.
; ...
;
$ORIGIN repo.tinycorelinux.net
; catchall wildcard
*       IN  CNAME      mirror.repo.tinycorelinux.org.  ; ALL MIRRORS FOR ROUND ROBIN SELECTION
;
; primary rsync server
rsync   IN  CNAME      distro.ibiblio.org               ; Primary rsync repository on ibiblio.org
;
; mirrors cname list.
mirror IN  CNAME      us.repo.tinycorelinux.net.       ; United States Mirrors
mirror IN  CNAME      cn.repo.tinycorelinux.net.       ; China Mirrors
mirror IN  CNAME      id.repo.tinycorelinux.net.       ; Indonesia Mirrors
mirror IN  CNAME      nl.repo.timycorelinux.net.       ; Netherlands Mirrors
mirror IN  CNAME      ru.repo.tinycorelinux.net.       ; Russian Mirrors
mirror IN  CNAME      au.repo.tinycorelinux.net.       ; Austrailian Mirrors
;
; country specific mirrors
us      IN  CNAME      www.gtlib.gatech.edu.            ; United States
cn      IN  CNAME      mirrors.163.com.                 ; China
id      IN  CNAME      kambing.ui.ac.id.                ; Indonesia
nl      IN  CNAME      ftp.nluug.nl.                    ; Netherlands
nl      IN  CNAME      ftp.vim.org.                     ; Netherlands
ru      IN  CNAME      l4u-00.jinr.ru.                  ; Russia
au      IN  CNAME      tinycore.mirror.uber.com.au      ; Australia

Require each repo participant to accept HTTP-GET Requests destined for:

repo.tinycorelinux.net/ and [countrycode].repo.tinycorelinux.net/

For those with local repos, let them provide a CNAME to tclrepo in thier DNS root zone.

This could be further expanded for logical connectivity by modifying the tce-load Logic binary. Upon attempt to connect to the repo to perform an install or whatnot, simply pull the DNS bindings for 'tclrepo' and 'repo.tinycorelinux.net' send 2 pings out to each host and utilize which ever has the least average latency utilize for the rest of that session. If a local repo exists, it will win. If a file download fails on one, roll to the next DNS Record, but when moving to a different file, always start back at the least latent repo.

Then just let the systems acting as external facing mirrors run rsync pointed to rsync.repo.tinycorelinux.com to maintain sync of thier local repository.

This accomplishes two things:

• enforces those wanting to run local repos to utilize the public mirrors instead of the ibiblio repo.
• encourages people to run external facing mirrors.
• allows a local repo to provide extremely quick boot time when running term servers.

It might also be beneficial to provide logging data when a repo host/file is failing, by logging to a file locally and submitting that file via POST after completing the install. In priority to the primary repo.

The local repo owner could then decide if that should be sent on to the mirror upstream via some conf file.

Anyways.. just an idea... If this is something people would utilize and it doesn't conflict with the current concept as far as implementation goes, I'd be happy to put something together.

[curaga]

That approach seems a bit complicated, as we already have a text list and a speed check (downloading a small file, which should be a better indicator than ping).

If running your own terminal server with a local mirror, you likely have set the local mirror as the default for all clients.

Besides round-robin, I don't see much benefit for the added complexity?