Tiny Core Linux

Off-Topic => Archive / Obsolete => SCM EXtensions => Topic started by: Jason W on January 28, 2012, 05:18:27 PM

Title: SCM Basics
Post by: Jason W on January 28, 2012, 05:18:27 PM
Scm's install into /apps, like /apps/leafpad.  The main thing that differentiates the scm from the tcz is the scm is self contained and can be mounted and unmounted.  The degree of being self contained can vary, as the scm can still have dependencies.  The binaries get symlinked into /apps/bin, unless a value is set in the scm's spec file. 

Located here:

http://distro.ibiblio.org/tinycorelinux/4.x/x86/scm/ (http://distro.ibiblio.org/tinycorelinux/4.x/x86/scm/)

The load script scm-load is very similar to tce-load in function.  "scm-load -iw audacious-gtk3.scm" will fetch that extension from the repo and load it.  "scm-load -r audacious-gtk3" will unmount (unload) the extension.  The icon and menu entries get added when the extension is loaded and removed when unloaded.   For the user, using the two different extension types will feel about the same. 

Here is the basic extension structure of an scm.  Lets take leafpad for example.  Inside /apps/leafpad you will find the usual directories:

/apps/leafpad/bin
/apps/leafpad/config
/apps/leafpad/etc
/apps/leafpad/lib
/apps/leafpad/share

..and so forth.  The config directory is where the install and spec files are located if they are used.  install is the startup script, just like /usr/local/tce.installed/leafpad.  spec is where variables or flags can be set, the only existing now is the "link_bin=no" if you don't want scm-load to link all the binaries in bin to /apps/bin. 


BUILDING
#####
When building an scm, you do not need to use a DESTDIR variable, just use the ./configure flag "--prefix=/apps/l3afpad"  and your app will install to that as it's base directory.  Certain values need to be set if installing multiple sources that depend on each other into that location.  Basically the below would suffice in most cases:

Code: [Select]
export LDFLAGS="-L/apps/l3afpad/lib -L/apps/gtk3/lib"
export CPPFLAGS="-I/apps/l3afpad/include -I/apps/gtk3/include"
export PATH="/apps/l3afpad/bin:/apps/gtk3/bin:$PATH"
export PKG_CONFIG_PATH="/apps/l3afpad/lib/pkgconfig:/apps/gtk3/lib/pkgconfig:$PKG_CONFIG_PATH"

In the final package, often a wrapper is needed for the binaries so they can find their libraries since the libs are not installed into a path searched by ldconfig.  Here is the one for l3afpad, the gtk3 fork of leafpad.  Essentially you rename l3afpad to l3afpad-bin and create an executable script named l3afpad:

Code: [Select]
#!/bin/sh

export LD_LIBRARY_PATH="/apps/gtk3/lib"
export XDG_DATA_DIRS=/apps/gtk3/share

/apps/l3afpad/bin/l3afpad-bin "$@"

L3afpad depends on gtk3.scm, so we have to specify the LD_LIBRARY_PATH to point to the gtk3 libs, and since it is a gtk3 app, also the XDG_DATA_DIRS.

Freedesktop files are located in /apps/PKGNAME/share/applications, and icons in /apps/PKGNAME/share/pixmaps.  The scm-load script will link and remove those entries into /usr/local/share/applications and /usr/local/share/pixmaps so all desktops can make use of the desktop and icon files. 

send in submissions to scmsubmit@gmail.com

#####

Title: Re: SCM Basics
Post by: gerald_clark on January 28, 2012, 05:30:13 PM
I noticed that /apps/bin is not added to PATH. Is this an oversight, or intentional?
Title: Re: SCM Basics
Post by: Jason W on January 28, 2012, 05:35:45 PM
For now intentional, as perhaps some would like it in the front of PATH, others would like it last. 
Title: Re: SCM Basics
Post by: Lee on January 28, 2012, 05:41:43 PM
So its still a mountable squashfs filesystem, like a .tcz?

And how should we pronounce ".scm"?  Is it dot-ess-see-em or dot-scum?  Or maybe dot-scam?  :)
Title: Re: SCM Basics
Post by: Jason W on January 28, 2012, 06:21:53 PM
lol.  Have not thought much about pronunciation, as we normally type things here rather than speak them.

Yes, a squashfs file just like tcz, made with mksquashfs, in it's file type. 
Title: Re: SCM Basics
Post by: coreplayer2 on January 28, 2012, 07:57:25 PM
I'm not sure what it is but sounds like "scrim"

The idea of cleanly unmounting a app sounds cool
Title: Re: SCM Basics
Post by: Jason W on January 28, 2012, 08:21:24 PM
Scum, like "scuzzy" for scsi, I guess.  lol

But anyway, the scm is just a simple putting back into practice the compressed mountable image - .uci - that Robert pioneered many years ago which I am personally getting once again hooked on.  I like that it can be a safe playground where we can go ahead and use gtk3, python3, other new things, etc, without having to wait on a migration in the tcz area.

Don't get me wrong, I plan to maintain my tcz extensions, and the tcz area remains of utmost importance as it is our 'system', whereas the scm's are self contained and basically affect nothing else outside themselves.  Though Robert (gutmensch) has taken the lead in the tcz repo since I have my hands full right now with scms.
Title: Re: SCM Basics
Post by: coreplayer2 on January 28, 2012, 08:40:39 PM
Let me see if I get this.

The objective is to have both tcz's and scm's installed simultaneously, one with deps and the other may be completely self contained and likely have duplicates of some deps already installed for the tcz's.

Interesting.
Title: Re: SCM Basics
Post by: Jason W on January 28, 2012, 08:48:52 PM
Basically, tcz's can support scms as dependencies, but scms do not interfere with either the base system or the system provided by the installed tcz extensions.

Scm's only install into /apps, and if writable space is needed, then /opt/scm/APPNAME is used and symlinked to from inside the scm.  So scm's do not in any way interfere with the installed system, only affecting /apps (and /opt/scm/APPNAME if needed).

One selling point of course, is the quick, clean-and-pristine, and officially supported uninstalling of extensions.
Title: Re: SCM Basics
Post by: coreplayer2 on January 28, 2012, 09:33:49 PM
scm's can be a tcz dep?  there's an interesting dynamic for sure. 

Important to note that tcz's are not being depreciated :) 

So scm's are supplemental to the system where we can use additional versions of a lib already installed through a tcz dep yet functioning within a sandbox so to speak??

is there going to be a submitscm audit script available soon?  to make sure we get it right.
Title: Re: SCM Basics
Post by: Jason W on January 28, 2012, 09:52:11 PM
No, scm's will not be listed as dependencies of tcz's.

And yes, by their nature scm's are supplemental in that they do not impose on the underlying system, and are free to use what dependencies they want configured like they want, all contained in the scm package. and affecting nothing outside of it's own scm package.
Title: Re: SCM Basics
Post by: coreplayer2 on January 28, 2012, 10:14:36 PM
Cool thanks for explaining :)

More choice , I like that.


and any submitscm soon ??
Title: Re: SCM Basics
Post by: vinnie on January 29, 2012, 02:20:57 AM
these scm are similar to the request for a GoboLinux package system like that someone asked a few months ago?
Title: Re: SCM Basics
Post by: uggla on January 29, 2012, 06:06:19 AM
Theoretically, what system directories could be removed in a scm-only remaster?
Title: Re: SCM Basics
Post by: Jason W on January 29, 2012, 06:21:13 AM
Gobolinux does more like a tcz, installing packages into their own directories and then symlinking them into the system, or at least to certain paths.

As for system directories, I don't think there really is any in base that would be a good idea to remove.
Title: Re: SCM Basics
Post by: V.Krishn on January 30, 2012, 05:34:55 AM
scm is quite similar to sfs modules for puppylinux, except that its more modular/dynamic.
Would it be good ides to have scm-build scripts for these scm instead directly of scm files.
Benefits:
1. Space save on server.
2.  Dynamic build on users end.
So while selecting a package, example, 'php' in appbrowser one can opt to install as usual or select 'build scm'.
Title: Re: SCM Basics
Post by: Jason W on January 30, 2012, 09:32:38 AM
I am not familiar with Puppy and sfs files, but from what I understand the sfs is similar to the tcz in that it is files installed into PKGROOT/usr/, PKGROOT/usr/local/ and such.  And then the sfs is overlayed into the system with unionfs so it is as if they are being coped to the system but without the memory space being taken up. 

That is actually much different than the scm in that the scm apps are specifically compiled with --prefix=/apps/{appname}.  While some extensions would work to copy thier tcz contents to a directory in /apps, most would not work as most paths to config files, share directories, and all are set at compile time. 

I am assuming that by building an scm you mean copying the contents of various tcz's into the scm, which would not work as nothing is overlayed into the system when the scm is mounted, which allows for the clean loading and unloading the scm offers. 
Title: Re: SCM Basics
Post by: V.Krishn on January 30, 2012, 11:36:05 AM
So sfs is overlayed at boot time while scm is sandboxed in /apps.
This is why it makes scm more modular/dynamic.

scm allows for the clean loading and unloading. 
Just to clear, what does it mean by clean unloading?

I have not tried to install, if scm /apps/leafpad is intalled and tcz leafpad also exist, which one gets entry in user menus?

Title: Re: SCM Basics
Post by: gerald_clark on January 30, 2012, 11:39:55 AM
Both.
Title: Re: SCM Basics
Post by: Jason W on January 30, 2012, 02:41:39 PM
Just to clear, what does it mean by clean unloading?

Meaning that nothing is installed outside of /opt or /apps, so no system area is touched.  Well, execpt for the freedesktop files and icons in /usr/local/share/applications and /usr/local/share/pixmaps that get created and removed by scm-load as the app is loaded and unloaded. 
Title: Re: SCM Basics
Post by: V.Krishn on January 30, 2012, 09:43:01 PM
I think it might still be possible to have a 'scm-build' scripts for making scm directly from tcz(dynamically on user end).
Just few more questions:
How are the links to required files in tcz done?
I mean get all files in /tmp/tccloop/<tczA> and perform filesystem equivalent symlink, or something more.
Title: Re: SCM Basics
Post by: bmarkus on January 30, 2012, 11:04:13 PM
I think it might still be possible to have a 'scm-build' scripts for making scm directly from tcz(dynamically on user end).
Just few more questions:
How are the links to required files in tcz done?
I mean get all files in /tmp/tccloop/<tczA> and perform filesystem equivalent symlink, or something more.

Why to convert tcz to scm? I do not see any reason. tcz is not the 'old' and scm is not the 'new' way of extensions, they are for different purposes. tcz is not obsolate and most cases for new extensions still the way is tcz while scm is for special cases.  No need to convert extensions to scm except it give benefits and cant't use tcz due to library conflicts or so.

Just my view on scm.
Title: Re: SCM Basics
Post by: V.Krishn on January 30, 2012, 11:27:03 PM
Thats true. There is no 1 to 1 relation of tcz to scm. Proposed/current scm system would not exist without tcz, so tcz stays.
Example gtk2.scm contains following tcz rolled into one, except with a different method of compile:
atk
cairo
expat
fontconfig
gdk-pixbuf2
glib2
gtk2
hicolor-icon-theme
libffi
libjpeg-turbo
libxft
libxml2
pango
pixman
tiff
Title: Re: SCM Basics
Post by: gerald_clark on January 31, 2012, 10:00:57 AM
Another good target for a scm might be compiletc.
It could be loaded to work on a project, and unloaded when done.

One thing to remember is that with scm, your memory usage may increase.
Libraries can be either statically compiled, linked from the shared libraries in /lib and /usr/local/lib,
or linked from /apps/extension/libs.
You may end up with multiple copies of the same libraries in memory, loaded from different locations.
multiple copies of libraries that could otherwise be shared.
Title: Re: SCM Basics
Post by: bmarkus on January 31, 2012, 10:11:16 AM
One thing to remember is that with scm, your memory usage may increase.
Libraries can be either statically compiled, linked from the shared libraries in /lib and /usr/local/lib,
or linked from /apps/extension/libs.
You may end up with multiple copies of the same libraries in memory, loaded from different locations.
multiple copies of libraries that could otherwise be shared.

Exactly. This way preferred extension type must be .tcz and select carefully when scm is really needed.

Title: Re: SCM Basics
Post by: V.Krishn on January 31, 2012, 10:32:10 AM
I would still like to have such a system though.
I have used microcore "http://insteps.net/pr/a/pmwiki/Apps/PmReader" to build this.
Think the portable document as explained on the webpage to contain Ad hoc module needed to view the document. Eg. Mimetex. Once the document is uploaded along with Mimetex.scm, its gets loaded. The document is viewable and later, document when deleted, its is scripted to be removed.

Does mimetex.tcz exist? :-)
Title: Re: SCM Basics
Post by: Jason W on January 31, 2012, 02:55:50 PM
I have expected some concern over RAM use and duplicated libraries.  I myself wrestled with it as I was getting back involved in using a self contained approach, and I am finding the theory of it paints a much darker picture than what the reality is.  At first, I expected a much larger cost in terms of RAM and disk space than what I have actually been seeing upon experience.

Out of curiosity, I did a quick side by side test today of an scm-only system and a tcz-only one, with some of our popular and larger apps to see just how much difference we can expect when we open up many apps at once.   Both systems were booted with only loading these apps and their deps, as well as icewm.  Here is the list of apps that I installed and launched - Xchat2, gqview2, leafpad, transmission (gtk2 for tcz, gtk3 for scm), opera-11, midori, audacious (gtk2), audacity-1.3, epdfview, emelfm2, gimp2, geany, pidgin, minefield9, gtk-gnutella-gtk2, foxit-reader, and geeqie.  I opened the apps, and then cleared the memory cache.  Nothing scientific, but able to give a general idea.  Transmission-gtk3 is a gtk3 app which also brought in gtk3 to ram for the scm install which made a larger impact on the scm system than the gtk2 version would have, tcz install used gtk2 for transmission which is also shared among its other loaded apps.

Ext type       total RAM          total "du" size of /apps                         
                    used at idle        or /tmp/tcloop

tcz               333mb                     271mb

scm              326mb                    360mb

The total size of the scm files could be trimmed more, but they seem to hold their own in the memory use dept, at least on first launch.  I am not saying anything bad about the tcz, and there are many little factors that go into how much ram an app uses when launched.  But I am just throwing some numbers up to give folks an idea of expected system impact before they rule out an extension type before they try it.
Title: Re: SCM Basics
Post by: gerald_clark on January 31, 2012, 03:21:48 PM
I was not ruling out scm. In fact I am using it presently for xchat and xmms.
I would really like to see a compiletc.scm.
Title: Re: SCM Basics
Post by: Jason W on January 31, 2012, 03:57:53 PM
I didn't think you weren't ruling them out, ruling out was not a good choice of words for me.  But as system efficiency is one of the goals of TC (otherwise why be small) memory usage in particular is a valid concern in any undertaking here. 



Title: Re: SCM Basics
Post by: TheNewbie on January 31, 2012, 07:28:47 PM
Correct me if I'm wrong, but to summarize the basic idea of scm packages, they're basically 100% self-contained binaries. They can require tcz dependencies or package their own libs, but there are no scm interdependencies, so scm packages can be loaded/unloaded cleanly with a Core util.

It sounds like a good idea if one wanted to try different window managers, browsers, etc., which all required the same basic set of deps (which could be loaded in tcz form), so that you could test each one and unload it without hassle. Definitely won't replace common dep packages, though, for the reason of memory efficiency as stated above.
Title: Re: SCM Basics
Post by: Jason W on January 31, 2012, 07:41:01 PM
Memory efficiency?  But look at the scm vs tcz findings above.  There really is not much a difference, if not an advantage of self contained as provided thus far, given the above findings on the existing scm apps.  Sure, some things need to exist in the system as a low level shared app or daemon.  But for gtk2 and other apps as provided thus far, there really is not so much a disadvantage.

Each has their own benefits, but if upon findings of real memory disadvantages, then that can be addressed by changes in approach of the self contained method.
Title: Re: SCM Basics
Post by: gerald_clark on January 31, 2012, 07:46:19 PM
Window managers would require symlinks outside of the /apps directory.
Title: Re: SCM Basics
Post by: Jason W on January 31, 2012, 08:57:36 PM
As Gerald mentioned, WM's require base functions outside of /apps that make them best suited to remain tcz's, and they tend to be fairly small anyway.
Title: Re: SCM Basics
Post by: curaga on February 01, 2012, 04:42:24 AM
I don't think compiletc would work as a scm, it would require a ton of symlinks still.
Title: Re: SCM Basics
Post by: floppy on February 01, 2012, 07:56:08 AM
I did not catch the main target of it and the use target.
run a session with only 256MB RAM, no swap, for installing a system with it..?
What is the main reason?.
Title: Re: SCM Basics
Post by: gerald_clark on February 01, 2012, 08:50:42 AM
Floppy, please start at post #1 and read the whole thread.
Title: Re: SCM Basics
Post by: Jason W on February 01, 2012, 06:42:13 PM
For now, to save download and disk space I have taken the development files out of the gtk2 and gtk3 scm's, leaving only the runtime.  The full packages with both runtime and dev stuff is available in the scm repo as gtk2-dev.tar.xz and gtk3-dev.tar.xz.  To use them, just unload the gtk2.scm, then untar the gtk2-dev.tar.xz file into /apps.  It will untar in a gtk2 directory, the same name as the .oaded scm's directory.  Then you can build against it.  Delete that directory when through compiling and reload the gtk2.scm.  It saves a bit of space this way and I think it is worth the extra steps of using the tar.xz.  Regular users won't need to be concerned with using the tar.xz.  And there are and will be only a few large tookit extensions that will need to be done this way, so these simple manual steps should not add undue complexity.
Title: Re: SCM Basics
Post by: Jason W on February 03, 2012, 07:40:18 PM
As mentioned in the CDE thread, I have made some progress on making a self contained and portable app out of stuff existing tcz extensions.  Audacious available as audacious-portable.tar.ga in the scm source area ,self contained, runs for me when untarred into any directory and booted with the "base norestore" option using only base Xlibs and a WM, also installing OSS.tcz, available for testing here:



This is not the original method used configuring with --prefix=/apps/audacious, but is much more portable and flexible, and allows making packages on TC to use anywhere, and to make packages elsewhere to use on TC.  Making self contained portable packages elsewhere for TC or on TC for elsewhere is not the priority, but may exist as a benefit.
Title: Re: SCM Basics
Post by: vinnie on February 03, 2012, 08:57:18 PM
I tested the package, it works but if I try to open the preferences, it crashes.
I understand but I also tried cde, I noted that captures the libraries as the program requires it.
Open preferences while running cde perhaps avoid the crash.
Title: Re: SCM Basics
Post by: Jason W on February 03, 2012, 09:00:28 PM
Ok, I will test it some more and see.
Title: Re: SCM Basics
Post by: Jason W on February 03, 2012, 09:08:14 PM
Ok, it is not seeing the Skins directory, will see what I can do.  This may not be feasible, but we will see.
Title: Re: SCM Basics
Post by: Jason W on February 04, 2012, 03:21:39 PM
I won't spend much time on trying to make a portable app out of existing packages, since they result in much larger size due to including everything.  And that means larger download/disk requirements and larger runtime memory, both of which are primary concerns in the scm model.  Much better to be custom built and lean, and the portability thing is being done by some other projects if one wants to grab a few portable apps.  But the current scm's are pretty portable as they are, since they normally just need some X libs, I run them on slackware and debian.
Title: Re: SCM Basics
Post by: vinnie on February 04, 2012, 07:01:29 PM
I understand, honestly suspected that it was not possible (although I was hoping otherwise), however jason thank you for trying  :D
Title: Re: SCM Basics
Post by: Jason W on February 04, 2012, 07:40:44 PM
Possible is one thing, feasible and in keeping with the overall scm concept is another. 

I say never say never, but for now compiling specifically for /apps and scm is the way.
Title: Re: SCM Basics
Post by: V.Krishn on February 04, 2012, 11:56:24 PM
What are the inherent problems for scm-build scripts to exist? What I am trying to say is for example, if gtk2.scm did contain the current dependencies for tczs by just rolling into one file.

Can someone point me to url or write the simple steps that takes place in installing a tcz package once it is downloaded in optional dir, ie. mounting+symlinking..etc?
Title: Re: SCM Basics
Post by: Jason W on February 05, 2012, 06:26:55 AM
Though I was frustrated with audacity crashing on the preferences tab, I had no trouble making a portable transmission-gtk.tcz in which all the functions seem to work, available in the scm source area as transmission-portable.tar.gz, should work when booting 'base norestore' with only X and a WM running.  I will try my hand at a few and if they are as simple then it is not a bad thing to be able to convert tczs to portable scms.  For one thing, the tczs are already built and there is no duplication of effort when a tcz is converted rather than building from scratch.  Though I intend to compile scm's and keep with that concept and not let it get lost to portability, nothing is lost if we are able to make mountable apps out of stuff in the repo. 
Title: Re: SCM Basics
Post by: V.Krishn on February 05, 2012, 06:38:04 AM
The initial concept of scm is ok, I guess If I am not mistaken its one step closer to being portable. Only effort is to make tcz repo re-usable.
Title: Re: SCM Basics
Post by: vinnie on February 05, 2012, 06:44:08 AM
I remain alert to every news!
Title: Re: SCM Basics
Post by: V.Krishn on February 07, 2012, 03:49:04 AM
I am outlining a probable method for 'scm-built' script.
Still wondering if it would work.

To Build - gtk2.scm
++++++++++++++++++++
Roll the dependencies in one, with current method as eg gtk2.scm, but build directly from tcz package.
Example:
1.Its built by mounting tcz in tmp/cairo and coping or rsyncing it into tmp/gtk2.
2.Like wise all dependencies.
3.Special scripts that runs when scm is loaded can be root or in folder /tmp/gtk2/config.
4.Folder /tmp/gtk2/scm.installed if needed.
Resulting folder structure:
 /usr
 /bin
 /etc
 ...
AND not
 /atk
 /cairo
 ...

To Install
+++++++++++
Mounted in /tmp/scm/nnn.gtk2, where nnn is last new number in /tmp/scm folders for scms.
Symlinks done similar to when installing any tcz, except the path is now to /tmp/scm/ and not /tmp/tcloop.

Symlink Method (Install)
+++++++++++++++++++++++++
1. Dynamically create 2 file, 010.gtk2.scm.list, 010.xgtk2.scm.list in /tmp/scm folder
2. If link present in any other <009.x<packages>.scm.list do not make a symlink but APPEND it to 010.xgtk2.scm.list.
3. If not present in the system already, make symlink and append it to 010.gtk2.scm.list.
4. If present APPEND it to 010.xgtk2.scm.list

Symlink Method (Un-Install)
++++++++++++++++++++++++++++
1. Unlink links in 010.gtk2.scm.list that are not present in >010.x<packages>.scm.list and NOT having keyword /tmp/tcloop in the symlink.
2. Remove 010.gtk2.scm.list, 010.xgtk2.scm.list files
3. Unmount gtk2.scm.

Hope I have not left any steps.
Title: Re: SCM Basics
Post by: Jason W on February 07, 2012, 08:26:11 AM
V.Krishn -

It sounds like that method is basically like combining tcz's into one extension, thus eliminating having to call on dependencies, and then linking them into the shared system just like the tcz.

One issue with that is that it would cause library file conflicts since various versions of the same library, built perhaps with different options, would be in /usr/local/lib.  And also the main purpose of the scm is to have a clean mount/unmount that does not install/symlink files into the system, therefore not causing any file or library conflicts.   At least not outside of the .desktop and icon files in /usr/local/share that get linked to the system.

####

Bmarkus in the release candidate thread mentioned the existence of dependencies in the scm's in regards to their effect on the self contained format.  Here is my view, that scm's can be as dependent or not dependent on other extensions that the extension maker deems fit.  I first was including gtk2 into each gtk2 app like leafpad, but that does result in about 7mb of ram being occupied when each different scm gtk2 related app is launched.  And also the time of building gtk2 over and over for each app.  Thus the existence of gtk2.scm and gtk3.scm.  And maybe one day qt4.scm.  To me that is a good middle ground to ease the memory requirement of scm's.  And Xorg-7.6-lib is called on as a dep instead of building parts of Xorg into each extension, though those libs could be placed in the scm's lib/ path to not need that dependency. Up to the extension maker as we already host the source for Xorg.

There is a trade off between total self containment and extension size and resource use.  One can even use the gtk2.tcz and it's deps for an scm dependency, but the more dependencies you add the less benefit of being self contained.  But again, it would be up to each extension maker, there is not a hard and fast rule.   Me, I aim for three or less dependencies to keep it simple - Xorg libs, gtk2/gtk3/qt, and perhaps python or similar.  Smaller stuff gets built into the scm.   

I mentioned about making scm's out of tcz's, and I have done that for avidemux, it needs no deps but either OSS or alsa at the user's choice.  It is basically a test concept that could save a bit of work by using tips from portablelinuxapps.org.  The source is already hosted, no having to rebuild.  There are some snags I have run in to, and I don't know how far I will try to go down that path, but it is kind of neat to play with.  I will of course troubleshoot any issues with the converted scm's, and anyone let me know if they have issue with me making a self contained app out of their build.  But for those apps that convert cleanly and work without issue, it is of great help to be able to use them.

Anyway, not to ramble too much, but that is where things are at now.  I think I am taking a sensible road in regards to structure and dependencies, etc, and I am open to any ideas.
Title: Re: SCM Basics
Post by: V.Krishn on February 07, 2012, 09:02:20 AM
Current scm method is ok. It is also intended to run apps in its own Sandbox environment,
But I think  it can still be done through tcz. Sandbox apps in tcz would be in /opt or /apps dir and compiled as per.

Symlink Method (Install) & Symlink Method (Un-Install) does take care of loading and unloading. Let me know if some step is left.

My steps does not concern on number of Dependencies. All dependencies can be clubbed in one scm or mix-matched.
During installation the scripts just tries not to do anything with existing symlinks pointing to /tmp/tcloop.
Example:
/usr/lib/<somefile> ---> /tmp/tcloop/<somepackage>/usr/lib/<somefile>
here while making a link "/usr/lib/<somefile>" the scm install script checks if it already exists and is pointing somewhere in /tmp/tcloop. If yes its a base system installation.
Title: Re: SCM Basics
Post by: Rich on February 07, 2012, 09:32:52 AM
Hi Jason W
So if I understand correctly, an scm is basically the same as how  Xampp  gets installed, except it
gets mounted as a squashed file system. All links to specific library versions and executables are
relative to the installation location to keep them out of the default path.
Title: Re: SCM Basics
Post by: Jason W on February 07, 2012, 09:51:26 AM
Rich,
Yeah, that is exactly the concept, at least in it's purest form.  I did make an Xampp.scm early on, it requires some writable space inside it and to be installed into /opt.  It required a little trickery, but it did work.  I don't have it posted though. 
Title: Re: SCM Basics
Post by: uggla on February 11, 2012, 06:01:17 PM
I would prefer if tczs were built exactly like scms but with (many) dependencies.
Title: Re: SCM Basics
Post by: curaga on February 12, 2012, 12:46:19 AM
uggla, do you mean just with a prefix of /apps? What would be the gain?
Title: Re: SCM Basics
Post by: Jason W on February 12, 2012, 05:33:59 AM
uggla - if you mean building into /usr/local but also including dependencies in the same extension, that is sort of what we did early on as it was an old habit from another distro.  But then you run into file conflicts, and different library versions in the same place over writing each other, and so on.

Actually, as stated in the release candidate thread, we are no longer allowing tcz's to be dependencies of scm's.  It will get a little tricky for things that, say, need dbus, but Apple includes dbus in their self contained apps from what I understand, so it can be done.  In fact, if not allowing tczs as dependencies, we almost don't need a dep file since something like gtk2.scm or gtk3.scm can just be mentioned in the info file. 
Title: Re: SCM Basics
Post by: uggla on February 12, 2012, 05:55:33 AM
uggla, do you mean just with a prefix of /apps? What would be the gain?

If I understand things right, this would make it possible to run different versions of the same library/application at the same time without conflicts. Installed App1 could then use Foolib1.2 while installed App2 uses Foolib2.2.
Title: Re: SCM Basics
Post by: beerstein on February 14, 2012, 04:44:34 AM
Hi:
Please can somebody help me to understand the SCM:

1. We have our extensions .tcz why do we need another extension genre anyway?

2. I read in the posts that the .scm extensions are more "portable" - what does this mean - is there an example where a portable extension is better than a "non-partable" ?

3. Can we use both extensions in one TCL setup?

4. What happens when we load both extensions from the same application?

5. Will the .tcz extensions disappear sooner or later?

Thank you for your help.
Title: Re: SCM Basics
Post by: gerald_clark on February 14, 2012, 07:37:12 AM
Please read this thread from the beginning.
Title: Re: SCM Basics
Post by: Akane on February 18, 2012, 11:39:18 PM
Can I use .tcz extensions with .scm extensions?
Title: Re: SCM Basics
Post by: Jason W on February 19, 2012, 05:47:23 AM
Akane-

Yes, they can be used together, as the scm is designed no to collide with things in the usual filesystem outside of /apps, or /opt if writable space is needed.
Title: Re: SCM Basics
Post by: uggla on February 19, 2012, 10:21:21 AM
Actually, as stated in the release candidate thread, we are no longer allowing tcz's to be dependencies of scm's.  It will get a little tricky for things that, say, need dbus, but Apple includes dbus in their self contained apps from what I understand, so it can be done.  In fact, if not allowing tczs as dependencies, we almost don't need a dep file since something like gtk2.scm or gtk3.scm can just be mentioned in the info file.

Abiword-cde.scm still has cde.tcz as dependency.
Title: Re: SCM Basics
Post by: Jason W on February 19, 2012, 11:14:15 AM
Thanks, I will include the cde binary with them.
Title: Re: SCM Basics
Post by: Jason W on February 19, 2012, 06:32:24 PM
Actually, I will drop the remaining one or two cde scm's as cde is a virtualization and sort of a chroot, great for it's purposes but since I can convert tcz's into scm's natively, I would rather stick with that route.

But the cde.tcz extension of course remains and it's use encouraged if you want to zip up an app and run it on, say, Debian stable.
Title: Re: SCM Basics
Post by: bmarkus on February 21, 2012, 10:57:48 AM
thunderbird installs in its own directory /usr/local/thunderbird so it can be put into an SCM. However it need dependencies like gtk2, dbus, openssl and few others, so not a real SCM can be achieved. I'm not sure what benefits SCM would bring over TCZ.
Title: Re: SCM Basics
Post by: Jason W on February 21, 2012, 11:23:37 AM
I will give thunderbird a shot soon, as openssl, gtk2, and such is no issue to either contain or use the gtk2.scm as a dep.  The problem I am running into with some apps is a self contained dbus.  At least when converting existing tczs, actually building the package from scratch may alleviate that issue.

transmission-gtk3 uses dbus well when dbus is a tcz.  There is a simple way around not having tcz's as deps of scms in this case, perhaps by adding the below to the startup script that would install and start dbus if needed:

if [ ! -f /usr/local/tce.installed/dbus ]; then
  su tc -c "tce-load -i dbus.tcz" || su tc -c "tce-load -iw dbus.tcz"
fi

if  [ "$(/usr/local/etc/init.d/dbus status)" != "1" ]; then
  /usr/local/etc/init.d/dbus start
fi

I am not on TC so I can't verify the correctness of that, but that is a general idea.  I am still pondering on how to handle dbus or similar things in the self contained apps.
Title: Re: SCM Basics
Post by: Jason W on February 22, 2012, 06:41:28 PM
Thunderbird.scm is uploading now, depends on gtk2.scm.  Dbus is optional, so noted in the info file to start it if desired, but the needed libs to run are included.
Title: Re: SCM Basics
Post by: bmarkus on February 28, 2012, 12:37:11 AM
Google's GO Language compiler creates a single selfcontained executable which fits nicely to SCM. Of course for new developments.

Language is interesting and efficient, worth to take a look at http://golang.org/ (http://golang.org/)
Title: Re: SCM Basics
Post by: Jason W on February 28, 2012, 08:15:58 AM
I will take a look at it soon, but of course anyone is welcome to get to it first.  I am unfamiliar with Go, so I would have to learn a little to be able to test the results.
Title: Re: SCM Basics
Post by: bmarkus on February 28, 2012, 08:33:54 AM
Go is an interesting animal. In many aspects it looks an interpreted language like Python with nics data structures and and extensive module library. Of cource there are not onéy similarities. But you will find it familiar in a short time.

Documentation is good generally and mainly it is targeting LINUX, next is WINDOWS. Less developed is the ARM version.

There is a very active community also.
Title: Re: SCM Basics
Post by: Rich on March 01, 2012, 09:35:56 AM
Hi Jason W
In case it's of interest,  lilypond-2.12.2.tcz  is practically an SCM already. Except for a dozen files installed in
/usr/local/bin , everything else is installed in  /home/tc/lilypond  and it has no dependencies. This is not a
request for an extension, just something I happened to notice.
Title: Re: SCM Basics
Post by: Jason W on March 01, 2012, 05:03:39 PM
Thanks Rich, I will look into porting it.
Title: Re: SCM Basics
Post by: vinnie on March 02, 2012, 12:24:38 AM
I know some applications that work this way, but what if it save the data in the same directory where it is?
Title: Re: SCM Basics
Post by: Jason W on March 03, 2012, 05:23:26 PM
Looking at lilypond.tcz, if there ever was a simple and solid conversion of a tcz to scm, this one is it.

I aim to get it uploaded tonight.
Title: Re: SCM Basics
Post by: Lee on March 03, 2012, 05:42:27 PM
Slightly O.T., but speaking of lilypond - I was running UBC Logo a while back and it required some little library that was only to be found (in the repo) in the lilypond extension.  It wasn't prohibitive, but it seemed a shame to download the entire (20 MB ? - I'm not on my Core machine right now to check) just for that one library... any chance of seeing lilypond refactored a bit?  I'll get on my Core machine in an hour or so and get my details straight.
Title: Re: SCM Basics
Post by: Jason W on March 03, 2012, 06:12:41 PM
There should not be a library existing only in a collection like an scm, that is needed by other apps.  Rather, the opposite should normally occur, that common libs are found in an extension colleciton. 

Common libs should be available in common tcz extensions, and no small extension should depend on an scm self contained package.
Title: Re: SCM Basics
Post by: Lee on March 05, 2012, 10:00:12 AM
I may have been overly specific in my original "provides" search.  The library in question was libgmp.so and I was searching for the specific version mentioned when UBC Logo failed - libgmp.so.3.4.1 - which indeed is only "provided" by lilypond.tcz. 

In checking today (in the 4.x repo - the original search may have been in the 3.x repo) I checked for -any- "libgmp.so" and found several providers: gmp.tcz, lilypond.tcz and openssl-1.0.0.tcz.  I don't know what versions are provided in gmp.tcz and openssl-1.0.0.tcz.

 - I suppose this sub-topic should be split out to different thread as it is not really SCM-specific.
Title: Re: SCM Basics
Post by: Jason W on March 05, 2012, 01:40:00 PM
On looking at the list files, lilypond-2.12.2.tcz and openssl-1.0.0.tcz do not have libgmp.so* in a standard library path, so they do not provide libgmp.so* in a way that can be used by other extensions as a shared lib.

Yeah, this thread has ventured off a bit from the original topic a while back, but by now may be difficult to split the separate parts out.
Title: Re: SCM Basics
Post by: cast-fish on March 15, 2012, 06:10:09 AM
Hello

what is the advantage of having the ability to "mount" and "unmount" an scm application? (.....over regular apps that are installed with the .tcz approach?)

I know that there are 4 different ways to deal with .tcz files.  (Download only. On demand etc)...

How are the "scm" applications able to be handled?...is it just one approach of installing the .scm application?

how do you "unmount" and why would you need this?

i heard it's helpful to have "self contained" .scm files because when a TCL system
gets any kind of conflicts they can be easily sorted out?

thanks

V.

Title: Re: SCM Basics
Post by: Guy on March 15, 2012, 06:19:25 AM
Quote
what is the advantage of having the ability to "mount" and "unmount" an scm application?

When you unmount the app, you free up RAM.
Title: Re: SCM Basics
Post by: cast-fish on March 15, 2012, 06:24:13 AM
Ahh right

Having mentioned this "freeing ram" thing earlier , it was not sighted as
the main reason for .scm apps.

maybe there was a missunderstanding on my part.

but thanks for the quik reply.

V.
Title: Re: SCM Basics
Post by: SunBurnt on March 28, 2012, 02:42:29 PM
Hi Jason W; The scm package sounds like what I was suggesting quite awhile back.

Do they use masses of links to make them work, or are they used directly through the mount point?
Title: Re: SCM Basics
Post by: Jason W on March 28, 2012, 02:49:41 PM
Scm uses few symlinks, just what is inside the bin/ directory in the mountpoint, and the .desktop files and icon.
Title: Re: SCM Basics
Post by: SunBurnt on March 31, 2012, 07:24:01 PM
Excellent!!! It does sound like what I call "no-union" Squash apps. Or "SqApps".

Do you have a build script or GUI that automates this?
Title: Re: SCM Basics
Post by: Jason W on March 31, 2012, 07:40:09 PM
I use build scripts for everything that is not converted directory from tcz, which converting from tcz involves a lot of potential for error that does not exist when building outright.  Though some even complicated examples work well with the tcz to scm approach, like VLC.  That is accomplished with some methods taken from portablelinuxapps.org.

The build scripts are in the src directory of the scm area.  I use a general script called "buildscm" that calls on include files, which are easily made for each source and it's dependencies. 

The build process is automated once created, like with tcz. 
Title: Re: SCM Basics
Post by: cast-fish on April 02, 2012, 10:09:51 AM
Hello

do you think that "many more" apps from the app store will end up in the SCM browser?

Scm browser seems so convenient for a tinycore user.

Scm is simple, clean and with it's advantages. For me particularly, scm browser totally simplifies some understanding
of how to use tinycore to my advantage.

i expect it's down
to many factors right?...not least of which is the complexity of shifting tcz's into .scm files (maybe it's not possible for certain apps?)

thanks

V.
Title: Re: SCM Basics
Post by: SunBurnt on April 02, 2012, 01:44:56 PM
I think all new scm builds are a good way to go, following Jason W`s thought that using tcz files can be hazardous.

A relatively complete list of of the best scm apps. in each category should be built to supplement the list of tcz apps.
Ideally over time scm would become Tiny Core`s main and maybe only app. package format, why have many?

The advantages of scm files are well known and they`re obviously superior to any other package type I`m aware of.
Standard packages; .tar.gz, .deb, .rpm, etc., are legacy old school and are relics of a bygone era really.
Puppy Linux`s .sfs files come close to a good package, but use the union file system that`s complex and unneeded.
If it weren`t for the "many Linuxes, many builds" problem, scm type files could work with any Linux distro. ( portable ).
Title: Re: SCM Basics
Post by: cast-fish on April 02, 2012, 03:29:53 PM
yes....if .scm apps can be built right away...why bother with .tcz builds of any new apps?

as for existing .tcz apps getting converted to .scm apps....it would be fantastic to see it happen for
every single .tcz app

i know of newcomers to tinycore who instantly can work with SCM browser no worries....its intuitive "one file" approach is exactly what people want

V.
Title: Re: SCM Basics
Post by: gerald_clark on April 02, 2012, 03:36:59 PM
You assume you speak for everybody.
Scm is not the be-all end-all you claim.
Title: Re: SCM Basics
Post by: cast-fish on April 02, 2012, 03:53:53 PM
sorry you feel that way

V



Title: Re: SCM Basics
Post by: jls on July 24, 2012, 01:03:20 PM
Code: [Select]
export LD_LIBRARY_PATH="/apps/gtk3/lib"
export XDG_DATA_DIRS=/apps/gtk3/share
cannot this two variables be setted by gtk3 itself so we can avoid to make wrappers?


Title: Re: SCM Basics
Post by: gerald_clark on July 24, 2012, 01:11:02 PM
That would break the isolation between SCMs.
Title: Re: SCM Basics
Post by: jls on July 24, 2012, 02:02:12 PM
could you make an example?
Title: Re: SCM Basics
Post by: Jason W on July 24, 2012, 04:48:02 PM
LD_LIBRARY_PATH and XDG_DATA_DIRS is used by the app that is looking for the libs, so nothing can be done from within gtk3.scm that will allow other scm's to find it's libs.   
Title: Re: SCM Basics
Post by: jls on July 24, 2012, 05:01:37 PM
cannot this variables be exported  globally by gtk3.scm?
Title: Re: SCM Basics
Post by: Jason W on July 24, 2012, 05:45:52 PM
I believe I remember with the previous distribution ld.so.conf had the library paths of self contained apps added to it so they would be recognized. 

But one of the goals of the scm is not to interfere with the system, or be interfered with by other extensions by means of using the libs and binaries contained in the scm itself, or the target scm like gtk3.  Which means using wrappers for some binaries.  It can be a pain, but it should make for few if any conflicts between extensions.
Title: Re: SCM Basics
Post by: cast-fish on July 25, 2012, 11:53:50 AM
Hello

that is sounding so ridiculously complex.....


i know from my own experience of win32  that "software conflicts" is a continual ongoing
problem 24 seven when using the OS and APPS   "dll..."   "hell....."     (lots and lots and lots
of the architecture simply does not work right)

You can barely ever have 3 or 4 apps before typical problems arise.....not sure about the

newer multi tasking chips, but perhaps it's even worse with them....?

anyhow,

Tinycore is totally different isn't it.  Light years away

Vince.





Title: Re: SCM Basics
Post by: aus9 on April 02, 2013, 10:11:49 PM
Hi

I can't find a howto in the wiki, if this is the main howto, any chance you can make this a sticky at page one?

ALSO you may wish to make this post a sticky?  allowed dependencies

http://forum.tinycorelinux.net/index.php/topic,12681.0.html

I am thinking of having a go at making a scm but at the moment this is the only post I can find that might be useful.

It seems to me, that those with low spec machines would prefer an SCM to unload to free up RAM. Is that the main use?

Feel free to correct me as I am often wrong, drunk or misguided.

cheers
Title: Re: SCM Basics
Post by: Jason W on April 03, 2013, 02:09:20 AM
Hi aus9,

I stickied this topic.  Allowed deps are as of present gtk2, python, perl and things like that.  There was a discussion on making every scm completely self contained relying on nothing else, though that is a huge overhead in the amount of effort to create each package. 

I think the main use of the scm is the fact it is simpler and less breakable than the standard setup.

But with TC 5.x around the corner, I have been spending time in other areas besides scm.  Sending in scm extensions is always welcome of course, but my own activity has been paused. 
Title: Re: SCM Basics
Post by: aus9 on April 03, 2013, 04:04:56 AM
thanks for the sticky

net issues
http://www.tinycorelinux.net/4.x/x86/scm/  for me borks into just a text listing while
http://distro.ibiblio.org/tinycorelinux/4.x/x86/scm/  behaves like I expect it too.

any chance you can fix?

Title: Re: SCM Basics
Post by: Jason W on April 03, 2013, 06:10:12 AM
I believe the viewing of a list is the expected behavior for the tinycorelinux.net scm and tcz repo pages.  The pages are not really intended to be used for downloading anyway.


Thanks.
Title: Re: SCM Basics
Post by: Rich on April 03, 2013, 07:06:24 AM
Hi aus9
The text listing is intentional, see:
http://forum.tinycorelinux.net/index.php/topic,14438.msg82439.html#msg82439
Title: Re: SCM Basics
Post by: tinypoodle on April 03, 2013, 09:32:57 AM
Unless I am missing something, this intentional behavior has as side effect putting an obstacle in access to source of scm's and tcz's.
Title: Re: SCM Basics
Post by: Rich on April 03, 2013, 11:50:03 AM
Hi tinypoodle
Quote
side effect putting an obstacle in access to source of scm's and tcz's.
Yes, but it's only a minor obstacle. Just add
Code: [Select]
src/to the end of the url.
Title: Re: SCM Basics
Post by: tinypoodle on April 03, 2013, 12:10:19 PM
Agreed on a practical level, and that prevents load from listing of extensions.

Just that it might be considered optimal if access to source is not made more difficult than necessary.
Latter could possibly be achieved by moving the /src dirs out of being subdirs of /tcz and /scm or by directly linking to them.
Title: Re: SCM Basics
Post by: aus9 on April 03, 2013, 05:28:55 PM
Hi  Guys

I am sure you have already dismissed it but I shall re-vomit it

why not have a few email addresses created to gmail then each email address can get free 10G of google sites storage.

That has the effect of sharing the "load" across a number of servers, and even if you dismiss the idea, you could use google sites as a backup?

2) So when you click into alsa (top entry) that currently takes you to
http://www.tinycorelinux.net/4.x/x86/scm/src/alsa/

it will eventually point to (and I shamelessly point to my own backup for example only)
https://sites.google.com/site/gordontinycore/ijs

But I removed all the tczs etc

3) The first weakness in my suggestion is that sites only allows an upload of 25 Megs so if you source is bigger you need to run
split ....upload the splits.....and edit the file to advise users how to download each split and cat it together

4) The second weakness is the tcz repository would need multiple email addresses to cover it all (either as a real substitute or just a backup)


thanks for reading
Title: Re: SCM Basics
Post by: althalus on April 03, 2013, 05:56:41 PM
That has the effect of sharing the "load" across a number of servers, and even if you dismiss the idea, you could use google sites as a backup?
My initial gut reaction is that this is a bad idea - It's most likely against Google's TOS to use sites as a backup, to my knowledge Google make no assurances they won't lose your data, when something goes wrong Google is woefully inadequate at customer support, not to mention they are quite happy to kill of widely used services just because they aren't used widely ENOUGH, or because they don't match Google's vision of the future any more.