Tiny Core Linux
Tiny Core Extensions => TCE Talk => Topic started by: bmarkus on August 04, 2009, 11:26:49 AM
-
Do we really need both .tce and .tcz extensions? You can
- mount .tcz
- copy content to RAM
- umount .tcz
It gives the same result as copying it from .tce, just requires a different load option either extension by extension or system wise and would significantly reduce extension creation, maintenance, etc.
-
Good thinking :)
If this is done I think we should have a boot option that tells how to install (tce or tcz way).
Different install models cloud, hard disk, etc could have different defaults but you should be
able to override the default with the mentioned boot option.
To defer the decision of the install mode to the actual install instead of package creation sounds like a good thing, even if the process to create both types could be automated (in the build system I use this is automated). However just having to care about one package type when loading the packages onto the system is good. Having the packages in the tce directory and then being able to restart
with some different boot codes to change from tce->tcz install (or vice versa) would be nice (without having to repopulate the tce directory)
Kind Regards
Lars
-
Use two directories, tce and tcz for example. Direcory will define loading mode. You can move extension from one to the other. Just one idea.
EDIT: In general I'm against the bootcode jungle. This is more simple. However you can use two bootcodes to override dirs, copy all to ram and mount all.
-
While true, you have to realize hindisght is typically better than foresight. TC in very short time has evolved very quickly. Some complain that it moves to quick.
We started with tce, tcz came later. tcz has gone through several iterations, ziofs, cramfs, and now squashfs.
tces being a simple tar archive is much easier for the new user to understand and thus create their first extension. tces are great for quick small personal extensions.
tczs may have embedded user.tar.gz which at first is typically another obstacle for the first time extension maker.
In the beginning we had limited loop devices to mount. Do you want every little exteions to occupy a loop? How would you automatically determine which tczs to mount/copy/unmount and which to mount and link? Introduce yet another directory or subdirectores to process at boot time? At the present you can easily have both tce and tcz in the same single PPR (tce directory).
I am certainly not against this. Lessening the load for Jason and having a single extension type is nice.
Perhaps the subdirectory approach would allow for an easy transitition and if we standardize on squashfs we would not even need to mount to perform the copy to ram.
-
Robert
thanks for the comments. I think it is worth to go further with this discussion. To have extensions duplicated makes the life complicated for all perties especially for the newcomers and for ordinary users when TC will be matured to a certain level, or distribution based on TC/MC will appear and also makes hard to understand the system.
KISS - Keep It Simple and Smile :)
Yes, TC's evolution is fast, and it is a beauty for developers (partly), pain for the users (?) :P
Regards... Béla
-
Robert
thanks for the comments. I think it is worth to go further with this discussion. To have extensions duplicated makes the life complicated for all perties especially for the newcomers and for ordinary users when TC will be matured to a certain level, or distribution based on TC/MC will appear and also makes hard to understand the system.
KISS - Keep It Simple and Smile :)
Yes, TC's evolution is fast, and it is a beauty for developers (partly), pain for the users (?) :P
Regards... Béla
I agree this deserves discussion. Speaking as a user, I agree migrating to a single type of extension makes a lot of sense. For me tces are in RAM and tczs are in usb flash memory so the tces run faster. I have not measured the exact difference in performance so I can't say if there is enough difference for me to care one way or the other. I wonder if the mounting of the tcz in ram makes it run at an equivalent speed as a tce in ram.
I do think keeping track of versions for the two classes of extensions is harder than it should be for all concerned.
I also think if the tcz creation procedure was turned into a user friendly utility program perhaps with a gui interface the complexity issue would go away.
-
This idea does seem worth pursuing. And I like bigpcman's suggestion that whatever of the extension creation process can be automated should be. However I generally find a non-gui process for this kind of thing to be more flexible, and in the end quite a bit user-friendlier than a GUI tool, which is always limited by it's author's imagination. Scripts + a good tutorial are worth their weight in gold. (Have a look at the Archlinux or Gentoo Wikis for good examples of this kind of thing.)
And since we're discussiong automation, an elegant build tool (like the one Arch linux has) would let package developers upgrade packages much more easily, since all the work is in setting up the automation initially, after that it runs by itself. Of course this can be done with scripting, but it seems that arch has taken it one generalization level further, resulting in a build process that is customized to the particularities of each extension, but also does all the appropirate safety and sanity checks. Just an idle thought.
-
Please, have a look tcbuild a very nice and simple build tool for extensions made by SvOlli. He announced it here on the form a few months ago and I have used it and tailored it to my own needs since that day. It is a tool in the same spirit as PKGBLD (Arch) and SLACKBUILD (slackware).
/Lars
-
I can agree on that a single extension type would reduce extension creation and maintenance time. It´s hard enough to keep track of all the additional files for a single extension type. At least when one is creating libraries and splitting the package into one run and one dev package - it quickly becomes a mess of all the files.
-
I too think we should reduce numbers of extension versions as tcz can be used also to load in RAM.
TC would handle both types of extension, however there should be only one online repository instead of two. The repository would contain only the .tcz* extension (normal case) OR the .tce* version (only in case the tcz version is not available)
-
Will tread lightly at first. Perhaps next RC.
-
Personally, I like the separation since it offers more choice. Sometimes a tcz won't work, whereas a tce will (eg, perl_xml.tcz if you've installed wine.gl.tczm; qt-4.5-base.tcz ---some conflict not present with qt-4.5-base.tce). I like to use tce's when they're small, even if a tcz is available. Choice might include how fast your computer is, RAM, etc.
I'll choose a tcz for temporary usage for frequent install/uninstall events. I'm not sure which is better regarding disk corruption, which is not infrequent. The tce folder (and backup) require frequent repair or you'll loose data.
-
Personally, I like the separation since it offers more choice.
Personally, I'm not sure what choice two parallel repositories actually offer. This is because as bmarkus noted in the first post of this thread, you should be able to do everything when a tcz is available: either load it (in RAM) or mount it.
Sometimes a tcz won't work, whereas a tce will
In my humble opinion this is precisely why we should have only one repository instead of two. If we have one extension that wouldn't work, the faulty extension would get fixed quicker.
There are at least two other issues I see with our two repositories:
1) Handling tce and tcz in parallel, and two dependencies files, and two md5sum files, and two descriptions is very cumbersome. I think this is preventing more people to contribute extensions.
2) We have two universes that are mostly parallel... but not entirely. And when they meet, it creates problems. As an example, the other day I was looking for the midnight commander application. Unfortuntely I didn't find it in the universe I usually use (tcz repo). It could have been the end of story, but I actually know that TC users are expected to look at TWO directories in order to find one app (!!). And indeed mc was in the other repo. So I downloaded it. Unfortunately not only it didn't work, but it started to make some other programs stop functioning to. Very frustrating. The reason was that mc uses glib1, so it downloaded it as a dependency. But I had glib1 installed already as a tcz because of some other programs. Luckily I quite quickly realized what was happening and found out the glib1.tce, removed it, and rebooted, but really it's a lot of frustration and fix for using a simple app.
With one single repository, even if mc had not been available as tcz, this problem would not have existed.
-
In my humble opinion this is precisely why we should have only one repository instead of two. If we have one extension that wouldn't work, the faulty extension would get fixed quicker.
I wouldn't want to eliminate tcz extensions altogether because there's a conflict in my personal setup. There's lots of variation regarding conflicts for a variety of setups.
1) Handling tce and tcz in parallel, and two dependencies files, and two md5sum files, and two descriptions is very cumbersome. I think this is preventing more people to contribute extensions.
I won't argue, but think that's unlikely. It take a few seconds to create the second md5sum file, & the info file is almost the same.
We have two universes that are mostly parallel... but not entirely. And when they meet, it creates problems. As an example, the other day I was looking for the midnight commander application. Unfortuntely I didn't find it in the universe I usually use (tcz repo). It could have been the end of story, but I actually know that TC users are expected to look at TWO directories in order to find one app (!!). And indeed mc was in the other repo. So I downloaded it. Unfortunately not only it didn't work, but it started to make some other programs stop functioning to. Very frustrating. The reason was that mc uses glib1, so it downloaded it as a dependency. But I had glib1 installed already as a tcz because of some other programs. Luckily I quite quickly realized what was happening and found out the glib1.tce, removed it, and rebooted, but really it's a lot of frustration and fix for using a simple app.
With one single repository, even if mc had not been available as tcz, this problem would not have existed
Appbrowser doesn't install a second copy, & checks only the root.
Personally, I keep a combined local database & use "update -i" unless I want to view an info file where I use appbrowser. I do like the inclusion of both tcz & tce files to choose from, however.
-
Would it be possible to only have tcz's in the repository but to augment the tcz format such that it could easily be converted by the installer into a tce if desired?
edit: Further to my point, the installer could be made responsible for the installation format per the users instructions once the repository format is settled.
-
This is a slight expansion of this thread, but I think it's relevant to the discussion. I'd be very glad for a flexible mechanism for loading different collections of extensions at boot time. Such a mechanism could also be made to specify which extensions get loaded to ramdisk and which get mounted as a loop file. This could kill multiple birds with one stone and really help those who use their tinycore installs for different purposes, or on different machines with different memory amounts and hardware requirements.
Imagine a plain text config file in which you can list 'modules'---user created collections of extensions. You can have any number of such modules, and each can have any number of .tcz's (assuming that we settle on a single extension type each of which can be either be loaded to ramdis or mounted). For each tcz, you also optionally specify if you want it loaded to ramdisk. By default it gets mounted and symlinked tcz-style.
For example, I have one setup on a low ram computer where I need everything possible mounted as a loop file, but don't need development files or my gtk2 stuff. I'll call this 'skinny.'
Sometimes I'm on a newer machine. If I find that it speeds things up, I'd be glad to load more core functionality to RAM. So I have my basic set of modules, including the gtk2 stuf, with some critical parts loaded to ramdisk. I call this 'fat'.
In either case I might want some functionality which I rarely use, provided by extension X and its dependencies. I set up a new module with those extensions, call it wizbang. And another collection, with, say the wireless drivers, programs and utilities, which I call 'wireless.' I also would have modules called 'devel+x' 'mediaplayer' etc.
Then at boot time, I can specify any number of 'modules' to load with a boot option that takes a comma-separated list
tce=sda1 modules=fat,wireless,wizbang,mediaplayer OR: modules=skinny,somemodule
Later, if I discover I need a module that I didn't load at boot time, that should be possible as well. If I need a group of extensions I haven't yet set up, I can edit the modules configuration file and create the module. Then load it. Ideally of course there should be a script that checks for dependencies. Even more ideally it would let me download dependencies if a connection is available and prompt me to add them to my module-configuration file, or let me force-ignore them if I know what I'm doing.
This should allow all extensions to be kept in a single directory (even the optional ones), and be of a single type. It could allow on-demand use of pre-defined groups of extensions (keeping memory free if you don't need those extensions). It would also minimize the current difficulties associated with keeping different configurations available. Currently I either keep a copy of an extension for each configuration I want (easier, since dependencies are managed, but wastes bandwidth and it's harder to keep multiple configurations up-to-date), or else I create symliked collections from a central repository. I have to manage dependencies manually, which is error-prone.
Thanks for entertaining this rather long-winded idea. It would have some issues to be worked out, but it could be a huge leap forward in usability and flexiblity, and simplify extension provision as well.
-
I already provide the ability to have named tce directories.
You can boot with tce=dev for a development environment, or tce=offrice whereas the office directory contains tce and tcz for office extensions, or tce=games, etc.
-
I already provide the ability to have named tce directories.
You can boot with tce=dev for a development environment, or tce=offrice whereas the office directory contains tce and tcz for office extensions, or tce=games, etc.
I would also like to note that it is possible to use your own custom boot codes. This was discussed way back in January. http://forum.tinycorelinux.net/index.php?topic=460.msg2914#msg2914 (http://forum.tinycorelinux.net/index.php?topic=460.msg2914#msg2914)
I do like the idea of user control "to specify which extensions get loaded to ramdisk and which get mounted as a loop file".
-
If I can voice my opinion as end user, I strongly vote for one repository, which means tcz only. This is to maximize efficiency and memory utilization as well as streamlining maintenance. This makes it also simpler for end user, who does not need to understand the difference to make a choice.
My opinion brings nothing new to this discussion, I am just emphasizing what has been said already and just want to express my end user perspective.
-
I like the idea of a single extension type that could easily be either loaded or mounted. Perhaps an extra button on appbrowser - one for "load selected" and one for "mount selected"? (And equivalent command line operation, of course.)
From an extension builders point of view, my own (very limited) experience has been:
1) I put together the gtm5.tce extension
2) There's no reason (that I know of) why it couldn't be a .tcz as well but
3) I just never got around to making a .tcz out of it.
That's lack of time (or just laziness) on my part. I'm sure its not hard to make a .tcz extension - its just that I've never done it and now it has been so long since I even read about it that it will take me a while when I do. With a single extension type, I'd be done and wouldn't be worrying about it.
As for the idea of "modules" or sets of extensions, that can be done with a pseudo-extension listing the extensions to be included as deps, though I suppose there's plenty of room for increased ease of use and/or other improvements. And isn't this roughly (really roughly) like what runlevels do?
Once the tce directory has been set at boot time (either by scanning or by an explicit "tce=whatever" boot code), is there a convenient way to change it?
-
There are a couple of issues related to the possibility of a 'copy to the system' use of the tcz. One is that the user.tar.gz is being deprecated to using the startup scripts.to copy config files. The user.tar.gz was used to hold writable config files and others that must exist as real files in the system. But by definition the user.tar.gz overwrote any files in it's way, which is not a good idea for config files that already exist in the system. If you have a config file in your backup, and you have not yet loaded the extension the file belongs to, that backed up file will be clobbered when you load the extension. Here is a way that will help avoid that by use of the startup script. Say you have an extension extension.tcz and a config file /usr/local/etc/extension.conf that is part of it, as well as /home/$USER/extension.txt. I would place the config files under /usr/local/share/extension and copy it to /usr/local/etc/ using a test case:
#!/bin/sh
TCUSER=`cat /etc/sysconfig/tcuser`
[ -f /usr/local/etc/extension.conf ] || cp /tmp/tcloop/extension/usr/local/share/extension/extension.conf /usr/local/etc/
[ -f /home/"$TCUSER"/extension.txt ] || cp /tmp/tcloop/extension/usr/local/share/extension/extension.txt /home/"$TCUSER"/
This would prevent clobbering of config files and allow the possible 'copy to system' use of tcz's if that is implemented. This works as well with tce's to prevent the clobbering of writable configs, of course then you would be copying from /usr/local/share/extension. If you are installing _anything_ to the /home directory, it is a must to use the TCUSER variable to allow the script to work both during boot and runtime loading, as well as not to clobber files in the /.home directory.
-
Don't expect the tce repository to go away in the 2.x series, as that would break eariler 2.x versions.
However, I will be introducing new options to "test the waters" in next RC of 2.3.
Please take heed to Jason's advice, as many tczs will likely need to be adjusted during this phase.
-
I already provide the ability to have named tce directories.
You can boot with tce=dev for a development environment, or tce=offrice whereas the office directory contains tce and tcz for office extensions, or tce=games, etc.
I like that feature and use it a lot. I was just musing about a way to expand that idea which might be easier for the user to _maintain_ multiple configurations, and which might also work for modular collections of apps and dependencies which could be combined at boot time or loaded after boot. Presently you have to treat each of your 'tce=xxxxyyy' setups as separate systems and maintain them separately. (Either you keep your extensions in one place and symlink to your 'office and 'dev' directories or you just maintain and update separate systems. The latter is really easier, since you don't have to manage dependencies manually, but it uses more drive space and bandwith). I thought if we could use something else besides the filesystem to group and specify such configurations we might get more flexibility (including the flexibility to specify whether to load a file to the ramdisk or to loop mount it).
Thanks Jason for your helpful summary.
-
There are a couple of issues related to the possibility of a 'copy to the system' use of the tcz. One is that the user.tar.gz is being deprecated to using the startup scripts.to copy config files. The user.tar.gz was used to hold writable config files and others that must exist as real files in the system. But by definition the user.tar.gz overwrote any files in it's way, which is not a good idea for config files that already exist in the system. If you have a config file in your backup, and you have not yet loaded the extension the file belongs to, that backed up file will be clobbered when you load the extension. Here is a way that will help avoid that by use of the startup script. Say you have an extension extension.tcz and a config file /usr/local/etc/extension.conf that is part of it, as well as /home/$USER/extension.txt. I would place the config files under /usr/local/share/extension and copy it to /usr/local/etc/ using a test case:
#!/bin/sh
TCUSER=`cat /etc/sysconfig/tcuser`
[ -f /usr/local/etc/extension.conf ] || cp /tmp/tcloop/extension/usr/local/share/extension/extension.conf /usr/local/etc/
[ -f /home/"$TCUSER"/extension.txt ] || cp /tmp/tcloop/extension/usr/local/share/extension/extension.txt /home/"$TCUSER"/
This would prevent clobbering of config files and allow the possible 'copy to system' use of tcz's if that is implemented. This works as well with tce's to prevent the clobbering of writable configs, of course then you would be copying from /usr/local/share/extension. If you are installing _anything_ to the /home directory, it is a must to use the TCUSER variable to allow the script to work both during boot and runtime loading, as well as not to clobber files in the /.home directory.
It seems that restore will overwrite files installed by the tcz extension, so if the extension loads the config file to /home/tc, I can overwrite it and it will remain after rebooting.
-
Restoring from backup will overwrite any existing file. But loading the tcz will not overwrite an already restored file (or simply a preexisting file) with a new copy using the above approach, and that is the desired behavior.
-
Restoring from backup will overwrite any existing file. But loading the tcz will not overwrite an already restored file (or simply a preexisting file) with a new copy using the above approach, and that is the desired behavior.
That's probably a cleaner approach. My point was that restore seems to occur after tcz loads, so the end result should be the same. (ie, the loaded tcz config file gets overwritten with the desired copy saved in backup).