User mirroring of the repo.

[beerstein]

Do I understand that correctly?

We could the personal sub-repo automatically synchronize with the ibiblio repo?

I have another question: If I run that sub-part-repo on my home server (lighttpd) are the
dependencies installed also? Is there extra coding work necessary for that?

thanks
beerstein

[hiro]

Yeah, I think you understand this correctly. No extra work necessary.
On your normal pc you could use tce-cache instaed of tce-load, so that your cache/mirror will serve that same version to all other clients afterwards.

You could of course just ssh into your mirror and then use tce-load just as usual, but I often look at e.g. .info files, so I'd want to change tce-load to also download them.

[khamara]

it shouldn't be hard to write a perl script for an Apache Web Server. I wrote a caching proxy server in perl a millenia ago, it can be reused here.

The idea:
1) have a directory on the web server containing the tcz files
2) 'offer' what's available from the main TC site
3) if the local server does not have the file, download from the main server, stream to the client and store for future downloads
4)it should also check for updates before offering any files

I am tempted to dig up that old code but I usually just copy files across my thumb drives. Unfortunately usb copies are not reliable for large files especially on windows and I have to do a 'sum' to make sure.

[SamK]

...
I am tempted to dig up that old code...

Sounds really useful, how about turning it in to an extension for the repository?
 

[beerstein]

Hi: I am glad you came up with this new post.
I am trying to do this for month. Could you please post your script and some more detailed infp?
An extension would be even better.

Thank you

[DoRight]

... having a large portion of the official repo available locally. I do an rsync (only tranfser differences) about twice a week in order not to be a too bad citizen.

I would like to rsync the Tiny Core repository to/for my terminal server.  I have very 'iffy' internet access except for between 1 - 4 AM (7 - 10 AM GMT).  I live in rural eastern Honduras where I am responsible for 2 school labs and assist with a 3rd.
I have not been able to figure out the source directory for the rsync.  Please assist.  Here is what I have tried:

Code: [Select]

rsync -av --stats  --progress --delete-after --exclude-from=/srv/tc.exclude --include-from=/srv/tc.include distro.ibiblio.org/pub/linux/distributions/tinycorelinux/3.x/tcz/* .Any suggestions for the exclusion file?  Especially what files should not be excluded?  I hope to start doing development on Tiny Core, especially Spanish language updates and some missing educational apps.  I will just rsync needed TCE's for now.  But my user base is growing rapidly in their computational sophistication.
Thanks, Do

[mbainrot]

Would it be possible to have just a gzipped copy of the repository?

Then atleast it would speed things up a little bit (by preventing multiple download threads and hung downloads)

edit:

... having a large portion of the official repo available locally. I do an rsync (only tranfser differences) about twice a week in order not to be a too bad citizen.

I would like to rsync the Tiny Core repository to/for my terminal server.  I have very 'iffy' internet access except for between 1 - 4 AM (7 - 10 AM GMT).  I live in rural eastern Honduras where I am responsible for 2 school labs and assist with a 3rd.
I have not been able to figure out the source directory for the rsync.  Please assist.  Here is what I have tried:

Code: [Select]

rsync -av --stats  --progress --delete-after --exclude-from=/srv/tc.exclude --include-from=/srv/tc.include distro.ibiblio.org/pub/linux/distributions/tinycorelinux/3.x/tcz/* .Any suggestions for the exclusion file?  Especially what files should not be excluded?  I hope to start doing development on Tiny Core, especially Spanish language updates and some missing educational apps.  I will just rsync needed TCE's for now.  But my user base is growing rapidly in their computational sophistication.
Thanks, Do

Ideally if I was you to get you started, I would get all of the binaries, thats all of the files in the tcz/ folder, then leave the sources for another day. I attempted a sync earlier today off ftp.vim.org and failed quite badly because I attempted to sync EVERYTHING. Now I am syncing just the contents of the tcz folder (excluding "source" and "src") to get my base copy.

[maro]

NOPE, using gzip on any of the already highly compressed tcz files (being 'squashFS' possibly using LZMA) is unlikely to provide any significant savings. I've sometimes even noted that if I want to compress a file that was already very well compressed that the size of the "compressed" output is a few bytes larger than the input as it just adds a few bytes of "wrapper" without being able to "squeeze out" any more.

Furthermore I personally would consider it "nuts" to sync more than the latest (i.e. 3.x) TCZ repository. Who needs all the sources and the older repositories (i.e. 1.x and 2.x) now?

[mbainrot]

My main reference was to reduce the CPU load caused by FTP clients handling large numbers of files.

Didn't know sources weren't needed, fairly new here

I guess my question is why are the sources put in the tcz dir when everyone knows most people will drag and drop the whole folder across.

I am only doing the contents (excluding sources) and its insane that I have to deal with 20,000+ files, Noting that I am doing my initial sync on my main workstation not on the server.

If I was to some how find some time in my insane study schedule and was to write up improved documentation, am I able to create a log in to the wiki and then just edit it myself or do I pm someone with my docs?

[Rich]

Hi mbainrot
According to a couple of posts I've read you can log into the wiki using your Tinycore name and
password, and add documentation.

[mbainrot]

Ok awesome, Once I finish my assignments (gotta love end of semester) I might write up some better documentation as it appears that the wiki formatting has munched the docs and its kinda unclear how to do it.

[hiro]

Yes, it's insane, why do you need all these extensions anyway?

[mbainrot]

I am setting up a complete local mirror as the other mirrors are really painful to get to from the land of oz.

Also I plan on using tiny core linux for pxe booting and have worked out that a local mirror and a slightly modified initrd (adding some scripts into /opt/bootlocal.sh to make it load stuff) should do the trick WAY better than slax.

Reason I need all packages is because I dont know exactly what my pxe booting setup is actually going to do yet.

In time when money permits I will prolly setup a high speed Australian mirror to make life a little more pleasant for us aussies.

My experience with mirroring is. Basing this on a linux web host
Assuming your www root resides in /var/www create the following directory structure.
/var/www/tc
/var/www/tc/3.x
/var/www/tc/3.x/tcz

use the tree command to verify the structure (noting my present working dir is /var/www)

Code: [Select]

[root@jupiter www]# tree -d tc
tc
`-- 3.x
    `-- tcz

2 directories
[root@jupiter www]#

1) Pick a mirror other than distro.ibbilio.org (i used ftp.vim.org) to get most of your stuff
2) Navigate to your version (e.g. 3.x)
4) Navigate to the tcz folder (note, this could take some time as generating a directory listing is pretty labourious)
5) For a full dump, copy all files inside of the tcz dir but avoid downloading the sources/src directory as they are quite large arn't needed for a local mirror.
6) When completed you'll probabally find that there are files missing, to rectify this connect to the main ftp server and then tell your client to only overwrite existing files if the source is newer. Whilst in theory it causes a load spike its short lived and its enough to get you going.
7) Check your folder permissions and then remaster your tiny core linux build so it uses your local repo.
Enjoy

Thats pretty much a detailed tl;dr of how to do it in easy to digest form.

[Pentium44]

Mind if I make a mirror of distro.ibiblio.org/tinycorelinux at vps.cddo.us/pub/tinycorelinux/ ?

[CentralWare]

@Jason / roberts / curaga / etc.: Would it be too much of an imposition to have the scripts currently run on the 3.x and newer TCZ directories to also be launched on the 2.x/tcz directory for sizelist and md5.db (and their gz counterparts?)  I know 2.x is likely a rare (if ever) recipient of updates due to its age, but considering there's otherwise no way I know of (without downloading every .info file for every .tcz) to determine if/when a file changes thus the above two items (md5 bare minimum) would be a terrific addition since 2x is still on the repo servers.

I'm currently writing up a system intended for downloading and maintaining a repository structure (I've recently noticed some odd things with WGET's mirroring capabilities and as such figured it's easier to draw it up from scratch as opposed to bug reporting and waiting) whereas the entire repo is going to be broken down between VERSION and PLATFORM where one or more combination of each can be selected and only those items are monitored for updates.  (ie: 9.x/armv6 and armv7 as well as 8.x/armv6 and 7 for someone who was solely focused on something like a RasPi, skipping over all of the unnecessary items - especially other platforms they'd otherwise have no use for.)  The 2.x, though, has no tracking (again, that I'm aware of) and I've read posts here and there with peeps still using 2x...)  The logic for these two files is simple: If MD5 and/or SIZE are different than what's on the user-based repo, update something.tcz and its associated files.

TCZ submission will be sent once she's completed.