WelcomeWelcome | FAQFAQ | DownloadsDownloads | WikiWiki

Author Topic: [Solved] fast way of syncing local mirror to official TCL repository?  (Read 1720 times)

Offline GNUser

  • Hero Member
  • *****
  • Posts: 1345
I sync my local TCL mirror to the official repo using rsync (the remote URL is repo.tinycorelinux.net::tc/$version/$arch/tcz/). Lately the download speed for the transfers has been in the 20-30 kB/s range, which causes the mirror operations to sometimes last several hours. My actual internet connection speed (not the theoretical speed advertised by ISP) is 35 Mbps so my internet dealings are otherwise very fast.

Is the server that hosts the official repository not doing well? Either way, do you know of a faster way of syncing local mirror with the official TCL repository?
« Last Edit: February 14, 2022, 12:48:32 PM by Rich »

Offline Rich

  • Administrator
  • Hero Member
  • *****
  • Posts: 11213

Offline GNUser

  • Hero Member
  • *****
  • Posts: 1345
Re: fast way of syncing local mirror to official TCL repository?
« Reply #2 on: February 14, 2022, 11:18:45 AM »
Hi, Rich. Thanks for that, but none of these support rsync. Also, I'd rather sync directly to the official repo because I don't know how up-to-date a given third-party mirror is.
« Last Edit: February 14, 2022, 11:20:20 AM by GNUser »

Offline patrikg

  • Wiki Author
  • Hero Member
  • *****
  • Posts: 673
Re: fast way of syncing local mirror to official TCL repository?
« Reply #3 on: February 14, 2022, 11:23:28 AM »
I Don't know if this repo list is working but you have to test.

http://forum.tinycorelinux.net/index.php/topic,21478.msg134397.html#msg134397
« Last Edit: February 14, 2022, 11:25:08 AM by patrikg »

Offline GNUser

  • Hero Member
  • *****
  • Posts: 1345
Re: fast way of syncing local mirror to official TCL repository?
« Reply #4 on: February 14, 2022, 11:27:08 AM »
Thanks but again, as long as the TCL developers continue to allow local/private mirrors to be synced directly to the official repo, that would be my preference (as stated in the title of the thread).
« Last Edit: February 14, 2022, 11:32:00 AM by GNUser »

Offline GNUser

  • Hero Member
  • *****
  • Posts: 1345
Re: fast way of syncing local mirror to official TCL repository?
« Reply #5 on: February 14, 2022, 11:48:59 AM »
I tried rsync'ing to official repo from a different machine with a different internet connection. Speeds shot up to several MB/s.

The machine that serves as my local mirror accesses the internet through a VPN tunnel. When I configure firewall and route so that official TCL repo access bypasses the tunnel, problem goes away.

So it doesn't seem like there's anything wrong with the official repo, after all :) Sorry for the noise.
« Last Edit: February 14, 2022, 11:51:21 AM by GNUser »

Offline Rich

  • Administrator
  • Hero Member
  • *****
  • Posts: 11213
Re: [Solved] fast way of syncing local mirror to official TCL repository?
« Reply #6 on: February 14, 2022, 12:52:10 PM »
Hi GNUser
It's not noise if the source of the issue can be identified and a solution can be found.

Offline GNUser

  • Hero Member
  • *****
  • Posts: 1345
Re: [Solved] fast way of syncing local mirror to official TCL repository?
« Reply #7 on: February 14, 2022, 03:22:54 PM »
Hi GNUser
It's not noise if the source of the issue can be identified and a solution can be found.
Well, I'm glad you see it that way :D

The solution in this case may help other users with a similar kind of problem. The solution involved bypassing the wireguard VPN tunnel for the mirror operation. (Why my VPN tunnel slows down the mirror operation is not clear to me--99% of the time the VPN tunnel does not cause any noticeable slowdown. It seems to be something specific to the TCL repo server.)

Here are the relevant parts of the shell script that I use to sync my mirror:
[Attached due to forum error]

Happy hacking!

Offline patrikg

  • Wiki Author
  • Hero Member
  • *****
  • Posts: 673
Re: [Solved] fast way of syncing local mirror to official TCL repository?
« Reply #8 on: February 14, 2022, 11:05:19 PM »
You have to traceroute outside to the vpn tunnel entry point to determine and from inside to the repo to figure out where the bottleneck is.

 
« Last Edit: February 14, 2022, 11:08:08 PM by patrikg »