Tiny Core Linux

Tiny Core Extensions => TCE Q&A Forum => Topic started by: Ulysses_ on November 26, 2010, 02:08:15 PM

Title: Local mirror of all extensions
Post by: Ulysses_ on November 26, 2010, 02:08:15 PM
Rather than download extensions over and over in each attempt to get a combination that works, wasting server bandwidth and my time, I would like to save a copy of all currently existing extensions in a folder, and do my installations of selected extensions from there, instantly.

What's a simple way to do this?
Title: Re: Local mirror of all extensions
Post by: gerald_clark on November 26, 2010, 02:27:22 PM
This is covered in the wiki.
http://wiki.tinycorelinux.com/Local_appbrowser_server
Title: Re: Local mirror of all extensions
Post by: tinypoodle on November 26, 2010, 07:11:54 PM
If as you say saving bandwidth and time are your motivation, I'd doubt that mirroring could save any, in comparison to downloading every extension on first need and updating PPR occasionally.

Keep in mind that if downloading the whole repo of extensions initially, you will then have to keep in sync.
Title: Re: Local mirror of all extensions
Post by: Ulysses_ on November 26, 2010, 07:40:39 PM
What does "updating PPR" mean?

I have noticed danielibarnes installs his extensions using "tce-load -i" here (http://forum.tinycorelinux.net/index.php?topic=7652.msg41221#msg41221), rather than download and then install them using "tce-load -w -i".  I think I should do the same.  

Shouldn't I duplicate them all just once from my local mirror that I will set up, to a DVD or .iso file or local folder, so that "tce-load -i" can be used and is instant?
Title: Re: Local mirror of all extensions
Post by: tinypoodle on November 26, 2010, 07:49:21 PM
What does "updating PPR" mean?

Personal Persistent Repositories of extensions
Title: Re: Local mirror of all extensions
Post by: Ulysses_ on November 26, 2010, 08:10:31 PM
What about a caching proxy that uses hard disk space for its cache, is that simpler?  There must be a preconfigured vmware VM doing just that.
Title: Re: Local mirror of all extensions
Post by: tinypoodle on November 26, 2010, 08:27:59 PM
What about a caching proxy that uses hard disk space for its cache, is that simpler?  There must be a preconfigured vmware VM doing just that.

Simpler than what?
Title: Re: Local mirror of all extensions
Post by: Ulysses_ on November 26, 2010, 09:15:01 PM
A preconfigured persistent caching proxy running on another pc or vm would make the second time you tce-load -wi an extension much faster, which might be simpler than achieving the same speed increase by doing what is suggested above (getting required extensions from a local appbrowser server that needs setup, or downloading all existing extensions to a folder/DVD/.iso and duplicating them to the folder that tce-load -i installs from).
Title: Re: Local mirror of all extensions
Post by: tinypoodle on November 26, 2010, 10:23:48 PM
A preconfigured persistent caching proxy running on another pc or vm would make the second time you tce-load -wi an extension much faster, which might be simpler than achieving the same speed increase by doing what is suggested above (getting required extensions from a local appbrowser server that needs setup, or downloading all existing extensions to a folder/DVD/.iso and duplicating them to the folder that tce-load -i installs from).


http://forum.tinycorelinux.net/index.php?topic=7495.0

Quote
Before you post your question please note that it is very important to read the documentation provided.
Title: Re: Local mirror of all extensions
Post by: Ulysses_ on November 27, 2010, 01:57:40 PM
Look how much there is in the documentation:

http://wiki.tinycorelinux.com/tiki-index.php

Are you suggesting that I read everything remotely related to extensions in all these pages of documentation? 

And if I do that, then I will know that caching proxies are a bad idea, or that a live DVD holding all extensions (from which you choose what to install) is a bad idea?
Title: Re: Local mirror of all extensions
Post by: gerald_clark on November 27, 2010, 02:34:07 PM
Yes, I suggest you read the wiki.
People have envested a lot of time documenting TCL.
To ask them to re-write it here in the fora is selfish.

As to including all the extensions on the CD, that is impossible.
The repo contains over 8G and would take 2 DVDs.
Title: Re: Local mirror of all extensions
Post by: Jason W on November 27, 2010, 03:47:53 PM
Actually, the tcz repo minus the source directory is about 4.3GB, but nevertheless still very large.  And still much more efficient and recommended to only download what you need onto persistent storage, updating what you have occasionally as previously mentioned.
Title: Re: Local mirror of all extensions
Post by: tinypoodle on November 27, 2010, 05:46:48 PM
Look how much there is in the documentation:

http://wiki.tinycorelinux.com/tiki-index.php

Are you suggesting that I read everything remotely related to extensions in all these pages of documentation? 

And if I do that, then I will know that caching proxies are a bad idea, or that a live DVD holding all extensions (from which you choose what to install) is a bad idea?


Beats me how my post of a single link plus a single quote from the linked content - in order to stress that part - could be so blatantly misinterpreted.

The link points to a sticky and locked thread consisting of a single post, where the founder and lead developer of this project states very clearly what the preconditions in order to ask any question in this forum are:

Before posting your question ...

You are supposed to have read and respected this before asking questions.

All I did was giving a simple pointer to that by linking.



Title: Re: Local mirror of all extensions
Post by: Guy on November 27, 2010, 08:13:00 PM
Quote
Rather than download extensions over and over in each attempt to get a combination that works, wasting server bandwidth and my time, I would like to save a copy of all currently existing extensions in a folder, and do my installations of selected extensions from there, instantly.

What's a simple way to do this?

You can install extensions, and you can remove extensions. Rather than download extensions over and over, why not just remove those you don't want, and keep the others.

Here is a post on removing extensions.

http://forum.tinycorelinux.net/index.php?topic=7943.0


Extensions are updated often. If you do download all extensions, keeping up to date with the updates, will require a lot of downloading.
Title: Re: Local mirror of all extensions
Post by: Ulysses_ on November 28, 2010, 02:26:34 PM
You can install extensions, and you can remove extensions. Rather than download extensions over and over, why not just remove those you don't want, and keep the others.

Here is a post on removing extensions.

http://forum.tinycorelinux.net/index.php?topic=7943.0

Thanks.  Because of the backup, it seems possible to any beginner that adding 20 extensions and removing the 3 after some use and configuration may not the same as adding the 17 from scratch.  Removing extensions has been troubling people it seems, that must be why is has been debated to death.  I'd rather avoid it if possible.  It still seems possible, see below.

Quote
Extensions are updated often. If you do download all extensions, keeping up to date with the updates, will require a lot of downloading.

Of course, the idea was never to download all extensions from the repos over and over, but to use a mirror of my own or the equivalent that downloads them all just once and automatically syncs thereafter, and copy the lot from the local mirror at local gigabit rates, to a DVD or .iso or folder.  And do that at arbitrary intervals, just like any distro makes CD's/DVD's of itself at arbitrary intervals.

Installations would then be super fast, as they probably are already for TC developers when you guys have lots of extensions locally and can type "tce-load -i", rather than the "tce-load -w -i" that I have been typing.

Tinypoodle wait a moment, typing a reply to you.
Title: Re: Local mirror of all extensions
Post by: Ulysses_ on November 28, 2010, 02:55:29 PM
Tinypoodle, this is a sensitive subject, manuals versus forums.  When one has put so much effort in their part of documentation, it must be unpleasant to be expected to retype it in forums when some people do not bother to read their work.  It also takes some discipline to maintain manuals up to date, and someone who first comes across a new distro cannot possibly know whether this discipline exists.  Now when the discipline exists, is it really so difficult to say something like:

"Check out the xxx chapter in the documentation, it's explained there"

That's all we're entitled to really.  

Now you say I misintepreted your post, that you only provided a single link and some bold.  Sorry but you did not just provide a single link, you provided a gentle reminder of official policy on RTFM, and you did that in response to a question about proxies and DVD's.  

Does it not logically follow that you considered that question preventable, if only the manual had been read?  Preventable because either:

a. it is explicitly answered in the manual, or
b. the suggested idea (proxy, DVD, whatever) would violate the philosophy of TC explained in the manual.
Title: Re: Local mirror of all extensions
Post by: Ulysses_ on November 28, 2010, 03:02:41 PM
Actually, the tcz repo minus the source directory is about 4.3GB, but nevertheless still very large.  And still much more efficient and recommended to only download what you need onto persistent storage, updating what you have occasionally as previously mentioned.

What's to stop us from taking a snapshot of the tcz repo every few months, calling it a version of TC, and distributing it with bittorrent?  We'd be downloading the o/s from each other, and then getting updates from the repos. 

The repos server would hardly notice a new version of TC is out, no noticeable increase in bandwidth use.
Title: Re: Local mirror of all extensions
Post by: Guy on November 28, 2010, 10:58:18 PM
Quote
Because of the backup, it seems possible to any beginner that adding 20 extensions and removing the 3 after some use and configuration may not the same as adding the 17 from scratch.

Adding 20 extensions and removing the 3 after some use and configuration is the same as adding the 17 from scratch. Extensions are not modified when you use Tiny Core. This is one of the big advantages of Tiny Core. The only thing which may be changed is your settings and personal files.

Quote
Removing extensions has been troubling people it seems, that must be why is has been debated to death.

Some people have not known that extensions and dependencies can be removed with Apps Audit.

Removing extensions has not been debated. I will try to explain what has been debated. The way Tiny Core is designed, after updating or removing extensions, you need to shut the computer down and reboot, for the changes to come into effect. Some people have suggested that it should be designed so changes come into effect without rebooting. This is what has been debated.

Nobody has suggested that the existing system does not work well.

I highly recommend, you install extensions, and remove what you don't want. It is much easier to update the extensions you your computer, than to update the entire repository.
Title: Re: Local mirror of all extensions
Post by: tinypoodle on November 29, 2010, 12:20:53 AM
Tinypoodle, this is a sensitive subject, manuals versus forums.

IMHO, there is no such dilemma.

How To Ask Questions The Smart Way: www.catb.org/~esr/faqs/smart-questions.html
Title: Re: Local mirror of all extensions
Post by: Ulysses_ on November 29, 2010, 05:14:13 AM
Quote
Because of the backup, it seems possible to any beginner that adding 20 extensions and removing the 3 after some use and configuration may not the same as adding the 17 from scratch.

Adding 20 extensions and removing the 3 after some use and configuration is the same as adding the 17 from scratch. Extensions are not modified when you use Tiny Core. This is one of the big advantages of Tiny Core. The only thing which may be changed is your settings and personal files.

Alright, here's an example then.  You install Xorg-7.5-3d, create a good xorg.conf for it, put its path in /opt/.filetool.lst plus whatever else you have configured for X, observe a bug that seems to be the fault of Xorg, remove Xorg-7.5-3d, install Xorg-7.5-vmware, reboot.  

Then Xorg-7.5-vmware boots up with a xorg.conf, and may fail if the error is in there.  This is not the same as installing Xorg-7.5-vmware from scratch without ever messing with Xorg-7.5-3d.  

There may be more configuration files you have listed in /opt/.filetool.lst that are brought back to life and may be what causes the malfunction.

Quote
Nobody has suggested that the existing system does not work well.

I have observed firefox crash while playing an HD flash video (not TC's fault) and come back to life after a reboot with the home page and addons reset.  But in other distros if firefox crashes, the configuration survives.  Isn't the firefox configuration in the home directory, and doesn't /opt/.filetool.lst include the home directory by default?  If yes, there's a bug somewhere causing the the profile to be reset.  

If not, how can the user know what other directories the latest version of firefox or whatever uses for its configuration?  Or the extension creator is supposed to reliably reverse-engineer each version of firefox or whatever to find out where ALL its configuration is stored, and list his reverse-engineering results in the extension's .txt?  

This is inviting troubles, like reverse-engineered ntfs-3g it will take ages to get 100% reliable.

Firefox has changed the location and format of the bookmarks and other config several times from version to version, I know because I was struggling to keep track with it under sandboxie, which has some analogies with TC.  So its a never-ending chase.

Don't get me wrong, I like TC too much, I'd rather put up with occasional bugs if this is what it takes for outstanding performance.  The winning feature for me in TC is speed and low memory, not download size or hard disk space.
Title: Re: Local mirror of all extensions
Post by: Guy on November 29, 2010, 05:37:57 AM
With an extension like Xorg, you could delete the files which are being backed up, or remove its path from /opt/.filetool.lst, then reboot. There is no need to download the other extensions again.

In my early days of using Tiny Core, when doing an update, I would delete the /home and /opt directories.

Quote
Nobody has suggested that the existing system does not work well.

That is referring to the existing method of removing extensions.

I have found that Tiny Core as an operating system works well. However, people do come across bugs from time to time. When these are brought to the attention of the team, they are addressed.

I have never lost settings with Firefox like you have.
Title: Re: Local mirror of all extensions
Post by: Ulysses_ on November 29, 2010, 05:50:18 AM
Quote
Nobody has suggested that the existing system does not work well.

That is referring to the existing method of removing extensions.

Surely when you remove an extension with the existing method, keeping track of configuration files (whose location has been reverse-engineered) is not trivial.  It may be asking for trouble.
Title: Re: Local mirror of all extensions
Post by: tinypoodle on November 29, 2010, 06:59:22 AM
If not, how can the user know what other directories the latest version of firefox or whatever uses for its configuration?  Or the extension creator is supposed to reliably reverse-engineer each version of firefox or whatever to find out where ALL its configuration is stored, and list his reverse-engineering results in the extension's .txt

firefox is open source, it would make absolutely no sense to reverse-engineer it.
You can get the sources freely.
Title: Re: Local mirror of all extensions
Post by: Ulysses_ on November 29, 2010, 11:27:15 AM
Firefox or whatever it says.  Eg opera.  And while it can be done in theory, it doesn't seem likely that extrension creators will ever read the sources of everything they create extensions for, in order to find out where all config files are placed.
Title: Re: Local mirror of all extensions
Post by: Jason W on November 29, 2010, 12:25:16 PM
Firefoxes place thier config files in ~/.mozilla, pretty much been that way since the beginning of the Mozilla browser.  Icecat uses ~/.icecat to be different, it is a GNU version of Firefox that does things it's own way..

Opera will place it's config files in, guess where?     ~/.opera

No reading of source or reverse engineering needed.   Just simply place these directories in you backup, or use persistent home, and settings will be kept across sessions.  Or for the more adventurous, you can find which particular files in those directories relate the the specific settings you wont to keep, and just back up those files to be space efficient. 

But this is drifting away from the original topic of mirroring the repo.
Title: Re: Local mirror of all extensions
Post by: Ulysses_ on November 29, 2010, 03:43:53 PM
Sure it is. But if I may, do you really know for sure that all linux software follows the convention and never stores anything important elsewhere? 

Seen one that doesn't (xorg).  Probably vmware workstation and vmware tools too, if an extension could legally be created for them.
Title: Re: Local mirror of all extensions
Post by: Jason W on November 29, 2010, 03:52:16 PM
The browsers keep it pretty simple. 

But most other apps follow a similar form with keeping their system-wide config files in /etc, or in our case /usr/local/etc.  The files may be in a subdirectory usually named after the app.  Then, user config files that override the system wide ones are in the home directory.  That is the general gist of it,.  Usually the documentation of the app describes where configs are, and normally extension makers don't vary that location except for the /usr/llocal prefix.  Which would place the /etc files in /usr/local/etc, and so forth.
Title: Re: Local mirror of all extensions
Post by: Ulysses_ on November 29, 2010, 04:41:30 PM
Recognizing patterns seen before sounds like Sherlock Holmes investigating a crime scene, knowing how to question suspects, what to expect, etc.  A distant relative of reverse engineering really.

There must be tools similar to sandboxie in linux, where it's obvious what files a chosen executable has written to.  Encouraging the user to use such a tool might make configuration backups a little more explicit and perhaps reliable.
Title: Re: Local mirror of all extensions
Post by: Jason W on November 29, 2010, 05:09:49 PM
One way to do what you are wanting is to open a program, use it, and close it.  Then soon after enter this in a terminal:

sudo find /etc /usr/local/etc /home -mmin 1 -type f


This will tell you of all the files changed within the last minute in /etc, /usr/local/etc and /home.  Of course, just add directories to that command to expand the directories searched for changes.

That should help with the backup.
Title: Re: Local mirror of all extensions
Post by: Guy on November 29, 2010, 05:43:37 PM
Ulysses

When I first started using Tiny Core, I wasn't so familiar with what is saved where.

When I did an update, I would save a copy of my personal files, then delete the /home and /opt directories. This will delete anything you may be concerned about. It is like having a new installation.

As you spend more time using Tiny Core, you become more familiar with where things are saved. So you may just delete particular files or directories.
Title: Re: Local mirror of all extensions
Post by: tinypoodle on November 29, 2010, 07:08:55 PM
Sure it is. But if I may, do you really know for sure that all linux software follows the convention and never stores anything important elsewhere? 

Seen one that doesn't (xorg).

That is a very specific exception based on precedence:
Quote
An exception is made for the X Window System because of considerable precedent and widely-accepted practice.