Tiny Core Linux

Off-Topic => Off-Topic - Tiny Tux's Corner => Topic started by: roberts on March 26, 2010, 08:01:15 AM

Title: Applications and bundled libraries
Post by: roberts on March 26, 2010, 08:01:15 AM
A very interesting article concerning the future direction of Linux applications.
http://lwn.net/Articles/378865/

I was always aware of OSX style of application and bundled libraries.
I believe Gobo Linux also uses this method.

It does mean using much more memory and disk space.

I had seriously considered this model for Tiny Core.
The advantage being easily unmounting self contained applications.
Title: Re: Applications and bundled libraries
Post by: lucky13 on March 26, 2010, 05:59:25 PM
PC-BSD also uses self-contained applications.
http://www.pcbsd.org/content/view/39/30/

It doesn't necessarily need to use a lot more memory/storage. It depends how many libraries are deemed "essential" and therefore in a base (whether in a distro's base or in some package) -- e. g., so that the most commonly needed GTK-related libs are either in the base or in one grand unified package rather than in separate packages. That would mean that some users would end up with certain unnecessary libraries (rather than the piecemeal approach which saves a marginal amount of space) just to use some small application built off some of those combined libraries. I always thought it would make more sense to bundle more of those kinds of things anyway, and IIRC that was one of the suggestions I made if DSL had gone to more of a rox-ish AppDir paradigm to get around the inode issue that could cause to older, smaller drives.
Title: Re: Applications and bundled libraries
Post by: Jason W on March 28, 2010, 05:33:25 AM
Xine-xvesa is built this way as to not interfere with other Xine extensions, and same with Pysolfc.  I needed a certain lib built a certain way for pysol to run so I just put it in the extenison itself to get loaded.  And since pysol remains self contained into it's own directory, it works equally well with either python-2.5 or python-2.6 installed.  When pysol is installed into /usr/local with it's installer, then it can only be used with the version of python it is installed with as it resides then under /usr/local/lib/python-version.  I would rather avoid having to rebuild apps when library versions change when it is possible.

 If you build a self contained application installed into it's own directory, you don't have to worry about it conflicting with other extensions and you can pack in needed libs that will not conflict with other versions of those libs in the repo.  As we have seen the headache of dealing with and updating openssl versions, I am personally going to either build statically against openssl or include the openssl libs I built against into a special directory in the extenison and use LD_LIBRARY_PATH to find them in preference to /usr/local/lib.  At least for small, simple apps.  Not all apps take to upgrading patchlevels of openssl and if you include your ssl libs in your extension in a different directory than /usr/local, then you will never have to worry about openssl version changes.  This works with apps, though I know that with libs like qt it is not so easy and would just have to use the version found in the standard path.   Of course, this is not meaning that openssl will be built into it's own directory again, but that I will include the libs I built against in the extension itself to avoid future conflict and reduce the openssl dependency.    

#Clarification, only apps that will never be used as a dependency of others, or there could be the issue of 2 ssl versions if there were libs bundled with a dependency.  Packing ssl libs into an app that has dependencies that also rely on openssl is perhaps asking for trouble.  Better to use the standard openssl extension as a dep or build statically.

This approach works well with the current tcz extension type, though it could also allow the simple mountable app that does not require it's files to be symlinked to the system.  And to me probably the main advantage is being more independent of other extenisons in the repo and avoid having to rebuild the extension when lib X or Y is updated.
Title: Re: Applications and bundled libraries
Post by: SamK on March 30, 2010, 03:26:43 AM
Related comments on a similar topic:
http://forum.tinycorelinux.net/index.php?topic=5601.0
   
Title: Re: Applications and bundled libraries
Post by: bigpcman on April 11, 2010, 08:06:50 AM
PC-BSD also uses self-contained applications.
http://www.pcbsd.org/content/view/39/30/

It doesn't necessarily need to use a lot more memory/storage. It depends how many libraries are deemed "essential" and therefore in a base (whether in a distro's base or in some package) -- e. g., so that the most commonly needed GTK-related libs are either in the base or in one grand unified package rather than in separate packages. That would mean that some users would end up with certain unnecessary libraries (rather than the piecemeal approach which saves a marginal amount of space) just to use some small application built off some of those combined libraries. I always thought it would make more sense to bundle more of those kinds of things anyway, and IIRC that was one of the suggestions I made if DSL had gone to more of a rox-ish AppDir paradigm to get around the inode issue that could cause to older, smaller drives.

For those of us not completely aware of the linux library system here's some background:

http://www.innovationsts.com/blog/?p=1042
Title: Re: Applications and bundled libraries
Post by: tclfan on April 12, 2010, 06:30:02 AM
A very interesting article concerning the future direction of Linux applications.
http://lwn.net/Articles/378865/

I was always aware of OSX style of application and bundled libraries.
I believe Gobo Linux also uses this method.

It does mean using much more memory and disk space.

I had seriously considered this model for Tiny Core.
The advantage being easily unmounting self contained applications.
Similarly, this trend is very strong on Windows side. Portable applications, which you can put on usb stick and boot from any computer existed for many years, but the real breakthrough was the concept and implementation of application virtualization, where VMware (bought Thinstall) ThinApps and Xenocode allow to virtualize any application, including very complex ones.  Something that was unthinkable two years ago now is happening. You can put the entire Office 2007 as one or two files on usb stick and use anywhere without installing... Applications are self-contained. All the dreaded dll hell and files installed all over the place - this problem is gone...
Title: Re: Applications and bundled libraries
Post by: tclfan on April 12, 2010, 07:26:45 AM
I had seriously considered this model for Tiny Core.
The advantage being easily unmounting self contained applications.
This would bring immensely more value to TinyCore than anything else.
After testing this (ThinApps and Xenocode)  a lot on Windows side and with Igelle distro on Linux side, there is no going back to old messy way...
Title: Re: Applications and bundled libraries
Post by: bmarkus on April 12, 2010, 07:31:22 AM

This would bring immensely more value to TinyCore than anything else.
After testing this (ThinApps and Xenocode)  a lot on Windows side and with Igelle distro on Linux side, there is no going back to old messy way...


You are right, if you don't care hardware resources. And you are right, no way back to an economical system. Long live memory manufacturers...
Title: Re: Applications and bundled libraries
Post by: bigpcman on April 12, 2010, 07:35:25 AM

This would bring immensely more value to TinyCore than anything else.
After testing this (ThinApps and Xenocode)  a lot on Windows side and with Igelle distro on Linux side, there is no going back to old messy way...


You are right, if you don't care hardware resources. And you are right, no way back to an economical system. Long live memory manufacturers...

I don't get why TC could not be made to support both the tcz of today and future "all in one packages".  What's the fundamental technical roadblock? I love the Tinycore Burger King theme "have it your way".
Title: Re: Applications and bundled libraries
Post by: tclfan on April 12, 2010, 07:37:39 AM
You are right, if you don't care hardware resources. And you are right, no way back to an economical system. Long live memory manufacturers...
I do not think so. I certainly care about memory utilization and always memory-optimize my systems. E.g. I was able to run XP in 64M...
However, from the perspective of memory eficient user, the new trend is worth some memory incremental for the sake of freedom from dll hell and system mess, where installed files are all over the place. This desease started on Windows spread to Linux big way. Now the time has come to put an end to this with new architecture running self-contained applications.  It brings back the freedom of installing and deleting applications without causing system rot and conflicts. Un-installing application is basically deleting the application folder and it is gone completely! No more hunting for installation remnants, impossible to track...

This new way is not for really old hardware, such as 486 or 128M , but anything but really old one should be fine... E.g. Igelle requires 256M memory.  Nowadays 1G memory is a very basic amount, although most of my systems have much less than that and I have no problems running this architecture.
Title: Re: Applications and bundled libraries
Post by: curaga on April 12, 2010, 08:50:19 AM
I thought tcz packages were already one-file-uninstalls, if the deps aren't counted of course. I think appsaudit can delete unused deps as well.

Self-contained apps are a security nightmare, and somewhat bigger. They are also bigger in ram, which starts to suck as a thought when one realizes there's 15 copies of gtk2 in the ram :P

What I find frightening, is that ram prices are 3x of what they were ~year ago. Holy crap, the cheapest 2gb ddr2 stick online (in my country, after a quick search) is now 51e.

I don't get why TC could not be made to support both the tcz of today and future "all in one packages".  What's the fundamental technical roadblock? I love the Tinycore Burger King theme "have it your way".
We already do, there's nothing much to add support for. Just either compile things statically or include the libs, and it's a self-contained extension.
Title: Re: Applications and bundled libraries
Post by: bmarkus on April 12, 2010, 09:18:14 AM
For me it just doesn't fit to Tiny Core Linux philosophy, it is something else.
Title: Re: Applications and bundled libraries
Post by: bigpcman on April 12, 2010, 09:33:54 AM
For me it just doesn't fit to Tiny Core Linux philosophy, it is something else.

I suppose you are right in so much as tc does not support the philosophy of dynamically uninstalling applications. The focus has been on maintaining a pristine system through the use of reboot after extensions are removed from the local extension repository (or boot list).
Title: Re: Applications and bundled libraries
Post by: tclfan on April 12, 2010, 10:04:22 AM
For me it just doesn't fit to Tiny Core Linux philosophy, it is something else.
I suppose you are right in so much as tc does not support the philosophy of dynamically uninstalling applications. The focus has been on maintaining a pristine system through the use of reboot after extensions are removed from the local extension repository (or boot list).
TinyCore is a very nice concept as it is now, maintaining a pristine state through reboot.
However after loading a number of applications that users use, at some point memory requirement becomes quite significant.  We come to a point of paradox that some users are reporting TinyCore as using 1G of memory fully loaded with applications they use while Zenwalk or Windows XP can run them in 256M... The capability to unload unused applications from memory without rebooting becomes increasingly important feature to have, while preserving modular architecture and maintaining pristine state.
Title: Re: Applications and bundled libraries
Post by: bmarkus on April 12, 2010, 10:15:35 AM
We come to a point of paradox that some users are reporting TinyCore as using 1G of memory fully loaded with applications they use while Zenwalk or Windows XP can run them in 256M...

Loaded into the RAM or using RAM while mounted? Don't tell me that XP is running applications in RAM!
Title: Re: Applications and bundled libraries
Post by: bigpcman on April 12, 2010, 10:29:26 AM
We come to a point of paradox that some users are reporting TinyCore as using 1G of memory fully loaded with applications they use while Zenwalk or Windows XP can run them in 256M...

Loaded into the RAM or using RAM while mounted? Don't tell me that XP is running applications in RAM!

Seems to me there are many reasons users like to dynamically uninstall applications since this is generally supported in most general purpose OS's (with mixed success). However, from the start tc has been successfully optimized to be a small "nomadic" system that users can count on to bootup in a clean known state.

What I wonder about is can application extensions ever be packaged and installed in a way that enables simple and accurate dynamic removal?
Title: Re: Applications and bundled libraries
Post by: sandras on April 12, 2010, 10:55:56 AM
I thought tcz packages were already one-file-uninstalls, if the deps aren't counted of course. I think appsaudit can delete unused deps as well.

Self-contained apps are a security nightmare, and somewhat bigger. They are also bigger in ram, which starts to suck as a thought when one realizes there's 15 copies of gtk2 in the ram :P

What I find frightening, is that ram prices are 3x of what they were ~year ago. Holy crap, the cheapest 2gb ddr2 stick online (in my country, after a quick search) is now 51e.

I don't get why TC could not be made to support both the tcz of today and future "all in one packages".  What's the fundamental technical roadblock? I love the Tinycore Burger King theme "have it your way".
We already do, there's nothing much to add support for. Just either compile things statically or include the libs, and it's a self-contained extension.

About including lib's to extensions and ram usage. I think, making an extension a self contained package would only take it to include all it's dependencies (lib's binaries and so on) in to it. That means, that when you boot and TC loads extensions, there would be link's made from TC's file system remaining in ram to the needed files (libraries or binaries or anything else) in one or another extension, if there's the same lib/bin/etc in two or more extensions. Basically, this approach would eat more hard drive space and provide a user with that drag'n'drop type of installation. But still, it would require the same amount of ram. Am I right or am I missing something?

One more thing. I wonder what takes more time - the dependency resolution? There would be no more dep's this way so we would save some time on that, but if the links were to overwrite each other that would also add some time. Just my two cents.
Title: Re: Applications and bundled libraries
Post by: tclfan on April 12, 2010, 11:06:45 AM
Loaded into the RAM or using RAM while mounted? Don't tell me that XP is running applications in RAM!
No, XP does not run in ram. it is not TinyCore, nor SliTaz nor Austrumi. But arguments against the self-contained application architecture used ram requirement, so here you have: If this measure is used to evaluate against, rather than combination with merits of running in ram, then these run-in-memory systems will have higher memory requirements than traditionally run such as Ubuntu or Windows...
My point was that measure of memory requirement alone is not deciding factor in self-contained applications architecture, but rather the huge advantage of simplicity and keeping system away from rot resulting from installing and uninstalling applications and other pollution and desease.
TinyCore as it is now achieves this goal very well by preserving pristine state on reboot and is very close to expand this modular architecture to include self-contained apps and unloading applications from memory.

Repository of such self-contained apps exists already and growing. It is used by Igelle. Igelle however does not have the TC advantage of running from memory.
TinyCore being flexible and modular can advance to combine the best ideas in its architecture...
Title: Re: Applications and bundled libraries
Post by: curaga on April 12, 2010, 11:21:13 AM
About including lib's to extensions and ram usage. I think, making an extension a self contained package would only take it to include all it's dependencies (lib's binaries and so on) in to it. That means, that when you boot and TC loads extensions, there would be link's made from TC's file system remaining in ram to the needed files (libraries or binaries or anything else) in one or another extension, if there's the same lib/bin/etc in two or more extensions. Basically, this approach would eat more hard drive space and provide a user with that drag'n'drop type of installation. But still, it would require the same amount of ram. Am I right or am I missing something?

With this approach, the versions included in different extensions would very likely differ. Even if the change was insignificant, copies of the lib are loaded to ram when an app uses them.

Dynamic vs static, having a ton of copies of the same lib dynamically makes no sense. It would just introduce app starting overhead. If going to the self-contained road, linking statically everything not in the base would be the better path.

Quote
One more thing. I wonder what takes more time - the dependency resolution? There would be no more dep's this way so we would save some time on that, but if the links were to overwrite each other that would also add some time. Just my two cents.
Dep resolution time is negligible.
Title: Re: Applications and bundled libraries
Post by: bmarkus on April 12, 2010, 11:23:35 AM
There are cons as well. For example if you are updating a common library, you have to rebuild all extensions, while using shared libs only once. Most of the applications with GUI are GTK+ based. There are many small progs where their dependency is larger than itself in regards size. Just installing few small tools will increase storage size and RAM usage when loaded into RAM unnecesarily. Also, using multiple copies of the same library without virtualization may cause other conflicts and system wise problems.

I can imagine applications and environment when this is a must have, but too much price for a typical TC/MC application suite.
Title: Re: Applications and bundled libraries
Post by: bigpcman on April 12, 2010, 12:08:09 PM
Although there has been a lot of good discussion on library usage problems in an "all in one" application package system, I still wonder if it's possible to handle dynamic application removal of applications that are things like system services or drivers. The problems I can foresee have to do with how to handle applications that modify things like system configuration files. How are the changes tracked back to the source. (I don't mean simple menu entries) This is one of the problems with the windows registry, it ends up filled with useless entries.

Perhaps there has to be classes of applications some of which can be "all in one" and some that can't.
Title: Re: Applications and bundled libraries
Post by: tclfan on April 12, 2010, 12:27:16 PM
There are cons as well. For example if you are updating a common library, you have to rebuild all extensions,...
I do not think so. Not the way I understand this concept of self-contained apps is, at least not the way it is implemented by VMware ThinApp abd Zenwalk. Self-contained apps are self-contained in their own containers. Different extensions (Apps) are using DLLs with which they are designed. If you update a dll someplace, it does not affect other extensions - would not break them with upgrades, etc. You can run together applications that are normally conflicting or have conflicting dlls. E.g. on Windows where previously you could run (have installed) only one version of Internet Explorer, now you can run several versions and even simultaneously! Not that I need to do it, not being a developer, but it shows the power and flexibility.
Thanks to flexible modular architecture and the way extensions are implemented, TinyCore is already close to this concept...
Title: Re: Applications and bundled libraries
Post by: tclfan on April 12, 2010, 12:33:32 PM
Although there has been a lot of good discussion on library usage problems in an "all in one" application package system, I still wonder if it's possible to handle dynamic application removal of applications that are things like system services or drivers. The problems I can foresee have to do with how to handle applications that modify things like system configuration files. How are the changes tracked back to the source. (I don't mean simple menu entries) This is one of the problems with the windows registry, it ends up filled with useless entries.

Perhaps there has to be classes of applications some of which can be "all in one" and some that can't.
The way it is implemented by VMware ThinAPPs and Xenocode is that applications are fully virtualized. Any system changes, such as registry entries are also contained in the application containers. No true such system change occurs.
Title: Re: Applications and bundled libraries
Post by: curaga on April 12, 2010, 01:46:07 PM
If you update a dll someplace, it does not affect other extensions - would not break them with upgrades, etc. You can run together applications that are normally conflicting or have conflicting dlls.

This is a big downside as well. What if there's a security issue in libfoo? Or a bug in the file handling of libxyz?

You'd have to update every single one of the extensions with that lib.
Title: Re: Applications and bundled libraries
Post by: tclfan on April 12, 2010, 02:29:20 PM
In home or experimental environment - yes.
In a non-production environment, where changes can cause impact on other applications can be done without much regression testing. In environment which demands stability, such as managed environment extensive regression testing must be performed before changes like this can be implemented in production. That is why ThinApps (Self-contained apps) are making big news in corporate environment, where maintenance of desktop systems is managed. Upgrading apps will no longer be such a big project, as to test whether this would not cause conflict or impact on other applications.