Tiny Core Linux
Tiny Core Extensions => TCE Talk => Topic started by: Jason W on January 17, 2010, 08:52:06 AM
-
Ibiblio has been kind enough to grant us server space and bandwidth to host Tinycore. I have from the start kept a mirror of our stuff on ibiblio since I am the one responsible for maintaining extensions and their sources on the server. And as the keeper of our repo, I feel the need to mention my concern for the trend of users to download the entire ibiblio repo. I know there are one or some of you that maintain a copy of the tcz repo to run checks on for discrepencies and report back to us. As always we appreciate help in finding errors and those kind of things. But for the average user unless there is a good reason, such as setting up a company intranet wide extension server, planning for an extended period with no web access, or whatever, I am asking users not to engage in downloading the entire repo. It is being a much better patron of ibiblio to download what you need to your tce directory. The appsaudit and tce-update tools will keep your extensions up to date without rsyncing or wgetting the entire repo.
Our full tinycore/ directory contains 11GB of data. There really is no reason that a normal user needs to sync this to their PC. You may notice that at times ibiblio seems slow. Users syncing that 11GB of data (or the 1.9GB /tcz repo directory) can only contribute to that slowness.
Gentoo has rules set up on their servers that limit the amount of "emerge --sync" that can be done in a days time based on IP address. And they ask their community to respect the resources of their servers. I am simply asking the TC community to show ibiblio the same respect. I would rather us police ourselves than to force ibiblio to place limits on Tinycore.
-
I fear I may be guilty of this, as I maintain an internal mirror for use on multiple TC machines.
However, I only perform this sync once a day around 01:11 CST (07:11 GMT).
I do not download the entire repo, my script uses the following basic algorithm:
* Download info.lst file
* For each entry XXX in info.lst:
** Download XXX.md5.txt file
** If local XXX.md5.txt is missing, or does not match:
**** Download extension files (XXX.tcz, XXX.dep, XXX.list, XXX.info)
The script then builds the local info.lst and an index.html file similar to one found on Ibiblio
Is this acceptable ? If not, then what would you suggest I change ?
I realize there are some mirrors I could use, but I believe there is nothing "official".
Perhaps the project could maintain a mirrors.lst file in the repository directory that contains a list of "official" mirrors. This way, I (and others like me) could download this list and use a mirror.
If this works, then eventually, appbrowser could be modified to take advantage of the mirror list as well.
-
I am working on a mirrors.lst. But it use will be via Select Mirror GUI from the Control Panel.
appbrowser, ab, and appbrowser-cli get the site via /opt/.tcrc. Select Mirror is the GUI for maintenace of that file.
-
Great.
Is there any reason why you should not place the mirrors.lst file in the repository directory ?
By placing the mirrors.lst file in the repository, the Select Mirrors, Control Panel GUI could have an option to allow the user to download the most recent mirrors list.
-
Why not mirrors.tcz ?
-
Because I'm not suggesting an extension (which would use the .tcz naming convention). I was suggesting a simple text file that contained a list of all the "official" mirrors, each on a new line.
This is similar to the info.lst file that appbroswer (and all it's variants) uses. The info.lst file contains a list of all extensions available in the repository.
-
This list can be in a tcz.
Then there is no requirement for any special "Control Panel GUI" code to update it when it changes.
Tce-update will update it automatically.
-
The GUI is to select which one for the list!
You have to be kidding to waste a mount point or require a special flag for a tiny text file!
-
Robert, I was responding to AlabamaPaul's suggestion to have a Control Panel GUI to download the Mirror List.
My comment was that the mirror list could be mirror-list.tcz.
Then Appbrowser could be used to download the mirror list, and tce-update would update it automatically.
This requires no special Control Panel GUI.
This requires no special flag.
Is it a waste of a mount point? If so, the file could be copied, and the tcz unmounted.
-
Gerald,
I believe you misunderstood my post. I am not proposing a new application, only a simple list. There is already a Control Panel GUI for Select Mirror, I was merely suggesting that the existing GUI could have an option to download the list.
Paul
-
The mirror.lst is posted in tcz repository.
Yes. The plan is that the GUI mirror selection program populates from this list.
-
Understood now.
-
It is not possible to prevent people from downloading the whole repo. Because some try to install TC many times before finding the best configuration. Waiting every time to download extensions is not handy. Additional caching proxy is necessary or manual manipulating of the files. And then dependencies. Pain.
My proposition: Select most popular stable extensions which are not in testing mode. Create iso image and ask people to use torrent.
-
If, as we have politely asked, users would only download their "most popular" extensions, we would not be experiencing slow downs which affect everybody. Nobody requires eveything in the repository. It is a mater of being a good net citizen and using our free hosted bandwidth respectfully.
Since the idea of creating a custom prebuilt iso keeps coming up, let me try to restate my vision and reason for starting this project...
It used to be that we went to the music store and purchased music cds. Many times these music cds had songs that we did not care for. I know I have many times used the remote to skip certain songs. On some cd players you can program tracks. Then along came the ipod like devices. They do not come with preselected music. You download and pay for just the titles that you like. Music cds have begun to disappear as giving the users choice is obviously better.
The same concept applies to Tiny/Micro Core. Think of Core as an ipod like device for playing your favorite Linux tunes (apps). Our repository is like itunes, you get to select that which you wish to use.
Now just as you would not try to download every title from itunes... Well you get the picture on both downloading everything and prebuilt isos.
-
I imagine that one of the reasons why people download the full repo is because there is no support for installing, including resolving dependencies, from multiple repos.
In case you have your own extensions not available in the official repo and they depend on extensions in the official repo (which most of them likely) you are caught in error prone manual operations. I was caught in this about 8 months ago so I modified tce-load etc to support loading from multiple repos,(one local, one at a friend and the official) in that order. It worked very fine but with the pace that tce-load et al changed it became too much work to keep that solution up-to-date and reverted to having a large portion of the official repo available locally. I do an rsync (only tranfser differences) about twice a week in order not to be a too bad citizen.
If you would be interested to include this kind of multirepo support, provided by many other distros, I would be happy to provide a protoytype implementation.
It will not likely solve all the bandwidth problems, but it could possibly reduce some of them.
Kind Regards
Lars
-
why don't tc forum provide DVD of whole repo?
1: reduction of bandwidth
2: one kind donation way
3: easy to use
-
Hi: it takes about 6-10 hours to download the whole TC 3 repo. But when you have it on DVD or in
a folder I needed to install each demendency seperately.
Could you please give me a hint how to make that easier?
Do I need to run a http server on my machine?
Is there a way to integrate that repo into the app browser or the app auditor?
Thank you so much
-
Sounds like you did not download the dep files?
-
Hi: it takes about 6-10 hours to download the whole TC 3 repo.
Did you read the first post of this thread?
-
Heh, this thread is only encouraging it :D
I don't understand why people update so often. The extensions I'm using very rarely change at all.
-
I do think offering repo DVD is not suitable.
Here are the reasons in my opinion:
- The extensions updates very often
- There are a lot of extensions that are useless for most of the users (do you think it is impossible that most of you install extensions like 'aircrack-ng'?)
- It is a waste to store so many useless extensions into DVDs
My suggestion is to offer SOME useful extensions into CDs or provide an ISO for us.
-
My proposition: Select most popular stable extensions which are not in testing mode. Create iso image and ask people to use torrent.
but this idea has opened a way not to discard.
for the future have a repository on a torrent protocol could be revolutionary.
enough one button on the interface of type "keep active sharing" with a green/red icon (on/off).
if I remember well the tracker can be updated, md5's are not necessary with the support of hashing.
ok, I'm crazy :P
-
HI:
I suggest that users should make their own sub-repo at home or in the office:
In this sub-repo they should put the extensions they need and want the most. (20-50)
is what I always need. Then, depending on further developments and requirements the users
can add new extensions to their sub-repo at home. That home repo can be built up over time,
updated and stored on HD, CD, stick, card, DVD or whatever.
At the end it should work like a "hybrid - repo" using the one from ibiblio and the home repo.
It could save substantial bandwith.
This is just a suugestioen and I would be glad if you will comment on it.
-
Torrent is not revolutionary and a single tracker represents a high risk as a potential point of failure. What would be really revolutionary a distrubuted P2P DHT network. There are different opensource working models which can be used as a basis.
Just an idea, but can be a good University project for someone :)
-
Beerstein, I like your idea.
Currently I use tinycore on 2 pcs and 2 laptops. The tce is "cached" on multiple sd cards and usb flash drives. But there are problems in my current workflow when I change/update stuff:
1. The different tce dirs are not synced automatically
2. I don't want to keep all tces up to date, but compatible so that they work on all pcs.
3. If I update the base or extensions and everything is fine on one pc it might not work on the others.
4. If I then sync the dirs and the non-working version overwrites the old functional one it takes a lot of time to clean up the mess again. Especially if the old, working packages aren't available on the repo any more.
5. If I setup a new pc and download the extensions from ibiblio I might get newer versions than on the other pcs. The tce dirs will then potentially be incompatible even without an active upgrade.
So I haven't found a nice way to manage this yet, but a private partial repo mirror might be a good starting point. But I haven't tried it because it seems a bit more difficult then just starting a web browser in your tce dir, e.g. what about .info and .list files.?
-
Torrent is not revolutionary and a single tracker represents a high risk as a potential point of failure. What would be really revolutionary a distrubuted P2P DHT network. There are different opensource working models which can be used as a basis.
Just an idea, but can be a good University project for someone :)
The idea of app browser based on p2p is new for me, i know torrent system, but I trust your word, if you say it's better in that way
-
Hi hiro:
It seems to me that you do almost the same, what I am doing.
I have four (older Pcs - max 1.6 Ghz and 500 MB RAM) and three notebooks.
Once in a while I add a new extension. Then, after a while I just copy the /tce over to the other
ones. I was not able to synchronize all my /tce 's - that would be too much work.
Do you think - just copying the /tce is good enough in terms of errors and/or missing files?
Should I rather tar the /tce , copy that tar thing over and un-tar at the other machine?
Until I red you message (thank you) I thougt a lightpd or apache with all my files (the ones I already have)
should do the trick.
I think there is a way to update the files you mentioned automatically. But how?
If we could find a few more members interested in this special thing, may be we can find a solution.
Have a good day
beerstein
PS: To all members - please let me know if you are interested in this subject
-
I think there is a way to update the files you mentioned automatically. But how?
In the past I have found this a capable app with various Ubuntu setups - Apt-Cacher NG.
http://www.unix-ag.uni-kl.de/~bloch/acng
With it running on the LAN, a software request from any other machine on the LAN is directed to ACNG. If the package or update is found in the ACNG cache it is made available to the requesting machine. If the software is not present in the ACNG cache it is obtained from the WAN repository, made available to the requesting machine, and added to the ACNG cache for future use. If my memory is accurate it also concurrently handles multiple versions of the distribution.
It is primarily (but not exclusively) aimed at Debian based distributions, however it might provide ideas or a starting point for someone with coding skills.
-
Thank you SamK for this lead. I am afraid I am not that good to do a project like this, but
may be I can find a member here.
-
Just copying is fine.
Perhaps syncing tce dirs is not that important.
To load and update stuff on the mirror first we could write a tce-cache script for the clients like this:
#!/bin/sh
ssh mirror tce-load -w $2
tce-load $1 $2Then edit the tce-load on the mirror to include something like this:
wget `cat /opt/tcemirror`/3.x/tcz/$2.info
wget `cat /opt/tcemirror`/3.x/tcz/$2.list
and create a cache-update script, but I don't know the content yet ;)
And then we would start the httpd and echo our.mirror > /opt/tcemirror on all our clients.
-
Do I understand that correctly?
We could the personal sub-repo automatically synchronize with the ibiblio repo?
I have another question: If I run that sub-part-repo on my home server (lighttpd) are the
dependencies installed also? Is there extra coding work necessary for that?
thanks
beerstein
-
Yeah, I think you understand this correctly. No extra work necessary.
On your normal pc you could use tce-cache instaed of tce-load, so that your cache/mirror will serve that same version to all other clients afterwards.
You could of course just ssh into your mirror and then use tce-load just as usual, but I often look at e.g. .info files, so I'd want to change tce-load to also download them.
-
it shouldn't be hard to write a perl script for an Apache Web Server. I wrote a caching proxy server in perl a millenia ago, it can be reused here.
The idea:
1) have a directory on the web server containing the tcz files
2) 'offer' what's available from the main TC site
3) if the local server does not have the file, download from the main server, stream to the client and store for future downloads
4)it should also check for updates before offering any files
I am tempted to dig up that old code but I usually just copy files across my thumb drives. Unfortunately usb copies are not reliable for large files especially on windows and I have to do a 'sum' to make sure.
-
...
I am tempted to dig up that old code...
Sounds really useful, how about turning it in to an extension for the repository?
-
Hi: I am glad you came up with this new post.
I am trying to do this for month. Could you please post your script and some more detailed infp?
An extension would be even better.
Thank you
-
... having a large portion of the official repo available locally. I do an rsync (only tranfser differences) about twice a week in order not to be a too bad citizen.
I would like to rsync the Tiny Core repository to/for my terminal server. I have very 'iffy' internet access except for between 1 - 4 AM (7 - 10 AM GMT). I live in rural eastern Honduras where I am responsible for 2 school labs and assist with a 3rd.
I have not been able to figure out the source directory for the rsync. Please assist. Here is what I have tried:
rsync -av --stats --progress --delete-after --exclude-from=/srv/tc.exclude --include-from=/srv/tc.include distro.ibiblio.org/pub/linux/distributions/tinycorelinux/3.x/tcz/* .Any suggestions for the exclusion file? Especially what files should not be excluded? I hope to start doing development on Tiny Core, especially Spanish language updates and some missing educational apps. I will just rsync needed TCE's for now. But my user base is growing rapidly in their computational sophistication.
Thanks, Do
-
Would it be possible to have just a gzipped copy of the repository?
Then atleast it would speed things up a little bit (by preventing multiple download threads and hung downloads)
edit:
... having a large portion of the official repo available locally. I do an rsync (only tranfser differences) about twice a week in order not to be a too bad citizen.
I would like to rsync the Tiny Core repository to/for my terminal server. I have very 'iffy' internet access except for between 1 - 4 AM (7 - 10 AM GMT). I live in rural eastern Honduras where I am responsible for 2 school labs and assist with a 3rd.
I have not been able to figure out the source directory for the rsync. Please assist. Here is what I have tried:
rsync -av --stats --progress --delete-after --exclude-from=/srv/tc.exclude --include-from=/srv/tc.include distro.ibiblio.org/pub/linux/distributions/tinycorelinux/3.x/tcz/* .Any suggestions for the exclusion file? Especially what files should not be excluded? I hope to start doing development on Tiny Core, especially Spanish language updates and some missing educational apps. I will just rsync needed TCE's for now. But my user base is growing rapidly in their computational sophistication.
Thanks, Do
Ideally if I was you to get you started, I would get all of the binaries, thats all of the files in the tcz/ folder, then leave the sources for another day. I attempted a sync earlier today off ftp.vim.org and failed quite badly because I attempted to sync EVERYTHING. Now I am syncing just the contents of the tcz folder (excluding "source" and "src") to get my base copy.
-
NOPE, using gzip on any of the already highly compressed tcz files (being 'squashFS' possibly using LZMA) is unlikely to provide any significant savings. I've sometimes even noted that if I want to compress a file that was already very well compressed that the size of the "compressed" output is a few bytes larger than the input as it just adds a few bytes of "wrapper" without being able to "squeeze out" any more.
Furthermore I personally would consider it "nuts" to sync more than the latest (i.e. 3.x) TCZ repository. Who needs all the sources and the older repositories (i.e. 1.x and 2.x) now?
-
My main reference was to reduce the CPU load caused by FTP clients handling large numbers of files.
Didn't know sources weren't needed, fairly new here :)
I guess my question is why are the sources put in the tcz dir when everyone knows most people will drag and drop the whole folder across.
I am only doing the contents (excluding sources) and its insane that I have to deal with 20,000+ files, Noting that I am doing my initial sync on my main workstation not on the server.
If I was to some how find some time in my insane study schedule and was to write up improved documentation, am I able to create a log in to the wiki and then just edit it myself or do I pm someone with my docs?
-
Hi mbainrot
According to a couple of posts I've read you can log into the wiki using your Tinycore name and
password, and add documentation.
-
Ok awesome, Once I finish my assignments (gotta love end of semester) I might write up some better documentation as it appears that the wiki formatting has munched the docs and its kinda unclear how to do it.
-
Yes, it's insane, why do you need all these extensions anyway?
-
I am setting up a complete local mirror as the other mirrors are really painful to get to from the land of oz.
Also I plan on using tiny core linux for pxe booting and have worked out that a local mirror and a slightly modified initrd (adding some scripts into /opt/bootlocal.sh to make it load stuff) should do the trick WAY better than slax.
Reason I need all packages is because I dont know exactly what my pxe booting setup is actually going to do yet.
In time when money permits I will prolly setup a high speed Australian mirror to make life a little more pleasant for us aussies.
My experience with mirroring is. Basing this on a linux web host
Assuming your www root resides in /var/www create the following directory structure.
/var/www/tc
/var/www/tc/3.x
/var/www/tc/3.x/tcz
use the tree command to verify the structure (noting my present working dir is /var/www)
[root@jupiter www]# tree -d tc
tc
`-- 3.x
`-- tcz
2 directories
[root@jupiter www]#
1) Pick a mirror other than distro.ibbilio.org (i used ftp.vim.org) to get most of your stuff
2) Navigate to your version (e.g. 3.x)
4) Navigate to the tcz folder (note, this could take some time as generating a directory listing is pretty labourious)
5) For a full dump, copy all files inside of the tcz dir but avoid downloading the sources/src directory as they are quite large arn't needed for a local mirror.
6) When completed you'll probabally find that there are files missing, to rectify this connect to the main ftp server and then tell your client to only overwrite existing files if the source is newer. Whilst in theory it causes a load spike its short lived and its enough to get you going.
7) Check your folder permissions and then remaster your tiny core linux build so it uses your local repo.
8) Enjoy :)
Thats pretty much a detailed tl;dr of how to do it in easy to digest form.
-
Mind if I make a mirror of distro.ibiblio.org/tinycorelinux at vps.cddo.us/pub/tinycorelinux/ ?
-
@Jason / roberts / curaga / etc.: Would it be too much of an imposition to have the scripts currently run on the 3.x and newer TCZ directories to also be launched on the 2.x/tcz directory for sizelist and md5.db (and their gz counterparts?) I know 2.x is likely a rare (if ever) recipient of updates due to its age, but considering there's otherwise no way I know of (without downloading every .info file for every .tcz) to determine if/when a file changes thus the above two items (md5 bare minimum) would be a terrific addition since 2x is still on the repo servers.
I'm currently writing up a system intended for downloading and maintaining a repository structure (I've recently noticed some odd things with WGET's mirroring capabilities and as such figured it's easier to draw it up from scratch as opposed to bug reporting and waiting) whereas the entire repo is going to be broken down between VERSION and PLATFORM where one or more combination of each can be selected and only those items are monitored for updates. (ie: 9.x/armv6 and armv7 as well as 8.x/armv6 and 7 for someone who was solely focused on something like a RasPi, skipping over all of the unnecessary items - especially other platforms they'd otherwise have no use for.) The 2.x, though, has no tracking (again, that I'm aware of) and I've read posts here and there with peeps still using 2x...) The logic for these two files is simple: If MD5 and/or SIZE are different than what's on the user-based repo, update something.tcz and its associated files.
TCZ submission will be sent once she's completed.
-
The scripts might not run, as there's been changes over the years in info format, etc. However, there's been no changes to 2.x since 2012, and we don't take any for that old versions - please consider them as historical archives.
-
@curaga: okay, 2.x flagged as unsupported. Thanks!
-
@Jason / roberts / curaga / etc.
Here's a potential fix for bandwidth (TCL, Ibiblio and others) where people would not "need" a full mirror assuming there's an internet connection.
Caching Server Scenario:
Assume busybox-httpd or similar (small) web server
Add a custom CGI which catches 404's, puts the caller "on hold", fetches the requested file from a mirror, updates the local repo, then sends the file through to the caller.
This way, people can run a local repo on an as-needed basis. Figuring out which files/dependencies/etc. someone's going to need is a chore when you're building/rebuilding/testing/repeating.
Client machine:
/opt/tcemirror contains
http://192.168.111.222/tinycorelinux/
Repo Server:
/usr/local/bin/busybox-httpd runs our show
/usr/local/repowww/cgi-bin/index.cgi contains 404 downloader content
/usr/local/repowww/tinycorelinux is a soft-link to where ever the TCL repo is located