importsce "add-on".sces based on "base".sces

[netnomad]

hi roberts, hi jason w,

with microcore i configured a very lean base-system with ondemand-packages that cover almost every task i could need during the year with my micro-tiny-multi-core-stick.
this lean base system can be expanded with some really heavy packages like libreoffice, multimedia-stuff and virtualboxes,
but usually i load only few packages that are really needed.

my dCore-configuration offers only few packages although my mydesktop.sce weights already 250mb.

now i want to configure a similar lean configuration with dCore,
but i experienced that additional, separated sce-packages depend on so many duplicated packages.
in the moment it seems to be problematic to reach my goal with this approach.

please consider following idea:

importsce -f base
# base.sce offers a base-system

importsce -b base -f desktop
# desktop.sce includes all further packages based on the packages and their dependencies
# that are provided by base

importsce -b base,desktop -f multimedia
# multimedia.sce is imported based on the packages
# that are provided by base and desktop

with such an approach dCore could offer a resembling small footprint like the tcz-packages of the other cores.

thank you for improving this exciting project, for your time and your commitment.

[netnomad]

hi friends,

jason w contributed a great tool called sce-merge:
http://forum.tinycorelinux.net/index.php/topic,16199.msg95963.html#msg95963

as i´m looking for a solution with low hardware-requirements,
i think that a modular approach could help.
with the current situation my configuration is getting soon so fat that it could get unusable

thank you for your help and hints.

[netnomad]

hi roberts, hi jason w,

did you miss to read my post?

thank you for your reflections about this topic.

[Jason W]

Hi netnomad,
The use of a list file to import that creates one sce with no duplicate files is the closes thing to what you are talking about that we are able to support. 

[netnomad]

hi jason w,

thank you for your reply.
yes, you are right that i could do everything in one sce and then there are no duplicates in that sce.

i'm looking for a lean, fast and stable system that is so multifuctional that i can use it for all my every day tasks.
so i want to start with a lean basic system and load the seldom used application only when they are needed.

my basic programm-set includes fluxbox, iceweasel, icedove and some basic cli-programs

in some cases i want to load these fat programs, only when i need them:
libreoffice, gimp, xsane, gtkam, isomaster, minitube, linphone, skype, virtualbox, several additional browsers.

all programs in one sce extend the resources of my hardware and my patience (to boot)

on the other side it is not very handy to package additional sces,
cause every separate package needs so many duplicate libraries, especially for X

in my proposed concept these duplicates could be avoided and
the advantages of the tcz-approach could improve the dCore-concept in a lean way.

so please be so kind and consider the suggestions in my post.
thank you for your time and attention

[Jason W]

I see how what you are wanting could be done, simply download and merge what is not already installed by a running sce.  Simply add on another layer.  The problem is that would create a web of dependent sces and once one of them is taken from sceboot.lst, the system is broken.  That, and one would have to know which packages are smaller than others so they could start with the smaller ones first.  And then those sce's would have to be loaded in exactly the same order as they were imported or there would be missing libraries or files possible.

The only advantage I see is saving HD or flash media storage space.  What is your goal, a lean and clean system where nothing is duplicated in RAM or simply smaller storage media space requirements?  Nothing is duplicated in the running system, loading one sce that contains an xorg and then another that also contains it does not require any more RAM.

[netnomad]

The only advantage I see is saving HD or flash media storage space.  What is your goal, a lean and clean system where nothing is duplicated in RAM or simply smaller storage media space requirements?  Nothing is duplicated in the running system, loading one sce that contains an xorg and then another that also contains it does not require any more RAM.

hi jason w,

most of the time i need just X, a browser, a mail-agent, mc and ssh.
for this scenario i need a fast booting, lean system that is easy to update, to clone or to modify.
i think there is a difference between 10s boottime or around 40-60s,
around 300mb of ram or almost all my 1gb that has to swap and so on.

if i need libreoffice, gimp or multimedia than it's o.k. to wait another 10s to load the extension,
then it's o.k. that more ram is needed for that single package,
but the unneeded packages should not bloat my system.

the main reason is that i want to avoid to put everything in one sce
that could be rarely needed once in the next 12 months

That, and one would have to know which packages are smaller than others so they could start with the smaller ones first.  And then those sce's would have to be loaded in exactly the same order as they were imported or there would be missing libraries or files possible.

i think that the users that are interested in this option will learn to understand the way how to use it

by the way some distributions use a similiar approach...
first a core-package is loaded, then a package with the x-windows-infrastructure,
afterwards you can choose between different packages that include windowsmanagers or desktop-environments,
then there is a choice of different browser-packages,
later you can choose between a small office-package like abiword or the big libreoffice-suite,
and at the very end eventually a development-tool-set is loaded.
with this approach the user can decide between a non-X-base system, different desktop-environments, browsers, office-solutions and the option of development without the disadvantages of an full-blown livesystem that includes all choices that could be needed some time.

i think that it's not so difficult to start with a base.sce, then package in the next stage something like desktop.sce based on base.sce and then package something like multimedia.sce or office.sce based on these two.

perhaps it won't be the most common way to use dCore, but it could be a suitable option.

thank you for all your considerations, ideas and help.

[netnomad]

hi jason w,

i want to give you some facts:

i boot with syslinux the actual dCore 5.14.02.11 and my basic micocore 4.7.7:

###
dCore 5.14.02.11:
bootime displayed in the boot-messages 36,69s
real boottime from syslinux to fluxbox:  1:18 min
used ram 538000 from 1gb
loaded sces:
/dev/loop0       14M   14M     0 100% /tmp/tcloop/original-modules-3.8.13-tinycore
/dev/loop1      362M  362M     0 100% /tmp/tcloop/mydesktop

mydesktop includes:
Xprogs
wbar
xorg-all
grep
bash
screen
mc
ssh
iptables
fluxbox
fail2ban
laptop-mode-tools
pm-utils
iceweasel
icedove
cups
cups-driver-gutenprint
hplip-cups
jpilot
gnupg
rsync
epdfview
gthumb
xvnc4viewer
gimp
importupdatecheck
alsa-base
alsa-utils
alsamixergui
cdparanoia
faad
lame
vlc
nmap
lvm2
cryptsetup
xsane

after loading icecat in dCore 5.14.02.11 and doing some actions my ram changed to
used ram 762800 from 1gb
after loading thunderbird
used ram 917000 from 1gb

###
microcore 4.7.7
bootime displayed in the boot-messages 3,79s
real boottime from syslinux to fluxbox: 0:29min
used ram 107300 from 1gb
loaded tczs:
/dev/loop0              288.0K    288.0K         0 100% /tmp/tcloop/fltk-1.1.10
/dev/loop1                3.2M      3.2M         0 100% /tmp/tcloop/Xlibs
/dev/loop2              128.0K    128.0K         0 100% /tmp/tcloop/Xprogs
/dev/loop3              364.0K    364.0K         0 100% /tmp/tcloop/Xvesa
/dev/loop4                1.3M      1.3M         0 100% /tmp/tcloop/libiconv
/dev/loop5               76.0K     76.0K         0 100% /tmp/tcloop/expat2
/dev/loop6              132.0K    132.0K         0 100% /tmp/tcloop/fontconfig
/dev/loop7              656.0K    656.0K         0 100% /tmp/tcloop/Xorg-7.6-lib
/dev/loop8              668.0K    668.0K         0 100% /tmp/tcloop/fluxbox
/dev/loop9               36.0K     36.0K         0 100% /tmp/tcloop/wbar
/dev/loop10               8.0K      8.0K         0 100% /tmp/tcloop/915resolution
/dev/loop11             140.0K    140.0K         0 100% /tmp/tcloop/kmaps
/dev/loop12              12.0K     12.0K         0 100% /tmp/tcloop/actkbd
/dev/loop13             552.0K    552.0K         0 100% /tmp/tcloop/myfonts
/dev/loop14               1.4M      1.4M         0 100% /tmp/tcloop/openssl-1.0.0
/dev/loop15             840.0K    840.0K         0 100% /tmp/tcloop/libssl-0.9.8
/dev/loop16              60.0K     60.0K         0 100% /tmp/tcloop/libssh2
/dev/loop17              12.0K     12.0K         0 100% /tmp/tcloop/libffi
/dev/loop18               1.4M      1.4M         0 100% /tmp/tcloop/glib2
/dev/loop19             396.0K    396.0K         0 100% /tmp/tcloop/slang
/dev/loop20               1.1M      1.1M         0 100% /tmp/tcloop/mc
/dev/loop21              12.0K     12.0K         0 100% /tmp/tcloop/ncurses-common
/dev/loop22             148.0K    148.0K         0 100% /tmp/tcloop/ncurses
/dev/loop23             392.0K    392.0K         0 100% /tmp/tcloop/bash
/dev/loop24             380.0K    380.0K         0 100% /tmp/tcloop/screen
/dev/loop25             304.0K    304.0K         0 100% /tmp/tcloop/gcc_libs
/dev/loop26             876.0K    876.0K         0 100% /tmp/tcloop/openssh
/dev/loop27             316.0K    316.0K         0 100% /tmp/tcloop/netfilter-3.0.21-tinycore
/dev/loop28             448.0K    448.0K         0 100% /tmp/tcloop/iptables
/dev/loop29              28.0K     28.0K         0 100% /tmp/tcloop/bzip2-lib
/dev/loop30             328.0K    328.0K         0 100% /tmp/tcloop/sqlite3
/dev/loop31               8.6M      8.6M         0 100% /tmp/tcloop/python
/dev/loop32              92.0K     92.0K         0 100% /tmp/tcloop/fail2ban
/dev/loop33              28.0K     28.0K         0 100% /tmp/tcloop/cpufrequtils
/dev/loop34              28.0K     28.0K         0 100% /tmp/tcloop/acpid
/dev/loop35              80.0K     80.0K         0 100% /tmp/tcloop/cpufreqd
/dev/loop36              60.0K     60.0K         0 100% /tmp/tcloop/laptop-mode-tools
/dev/loop37             164.0K    164.0K         0 100% /tmp/tcloop/libx86
/dev/loop38              24.0K     24.0K         0 100% /tmp/tcloop/libpci
/dev/loop39             316.0K    316.0K         0 100% /tmp/tcloop/suspend-utils
/dev/loop40               4.0K      4.0K         0 100% /tmp/tcloop/mirrors
/dev/loop41              28.0K     28.0K         0 100% /tmp/tcloop/ipager
/dev/loop42              12.0K     12.0K         0 100% /tmp/tcloop/ntpclient
/dev/loop43              88.0K     88.0K         0 100% /tmp/tcloop/pdnsd
/dev/loop44             220.0K    220.0K         0 100% /tmp/tcloop/libevent
/dev/loop45              36.0K     36.0K         0 100% /tmp/tcloop/redsocks
/dev/loop46              16.0K     16.0K         0 100% /tmp/tcloop/xautolock-2.2
/dev/loop47               8.0K      8.0K         0 100% /tmp/tcloop/xtrlock
/dev/loop48               4.0K      4.0K         0 100% /tmp/tcloop/xautolock_meta

installed tczs:
915resolution
acpid
actkbd
atk
bash
bzip2-lib
cairo
cpufreqd
cpufrequtils
expat2
fail2ban
fltk-1.1.10
fluxbox
fontconfig
gcc_libs
gdk-pixbuf2
glib2
graphics-libs-1
gtk2
icecat
ipager
iptables
kmaps
laptop-mode-tools
libasound
libevent
libffi
libiconv
liblzma
libpci
libssh2
libssl-0.9.8
libx86
libxcb
libxml2
mc
mirrors
myfonts
ncurses
ncurses-common
netfilter-3.0.21-tinycore
nspr
ntpclient
openssh
openssl-1.0.0
pango
pdnsd
pixman
python
redsocks
screen
shared-mime-info
slang
sqlite3
suspend-utils
wbar
xautolock-2.2
xautolock_meta
Xlibs
Xorg-7.6-lib
Xprogs
xtrlock
Xvesa

after loading icecat in 4.7.7 and doing some actions my ram changed to
used ram 362300 from 1gb
after loading thunderbird
used ram 524500 from 1gb

[Jason W]

I am open to options that are obtainable without costing performance for everyone else.  But to make dCore modular is not as simple as it sounds and would basically require a rewrite from scratch.  We did try at first the modular approach, but the performance of the import routine was slow and Debian packages have circular dependencies that lock up our dependency routine.

dCore certainly uses some more resources than standard Core due to the size and dependencies of Debian packages, but please use the "cache-clear" command before comparing RAM between the two systems.  I am sure the results would be more comparable after clearing the memory cache, use cache-clear and post the results again.

As for the boot time, I assure you that if we used the modular approach to dCore that it would be a much longer boot, SCE cut down that time very significantly. 

And you are comparing two different package sets.  The dCore install has the heavyweight VLC , xsane, gthumb with it's large dependency list, epdfview, gimp (another large one) and xorg-all, the full Xorg package.  It is not a fair comparison, the only somewhat large package in your tcz installation is the icecat browser.

Choose a similar package list for both tcz and sce, use cache-clear once booted, and I am very interested in those results in RAM occupation, boot time, and total package size.   

[yoshi314]

I am open to options that are obtainable without costing performance for everyone else. 

what if we could specify just one package as base for another one.  And with no recursive deps.

e.g. i start with bare dCore. and i make xorg-all. It probaby has many libraries in that other packages could reuse.

now i make , e.g. i3 package (the window manager), by doing something like

Code: [Select]

importsce i3 --base xorg-all which will work like import i3 , but will only include files not provided by xorg-all sce.

However, doing

Code: [Select]

importsce something --base i3
should fail with "i3 is not a standard sce, can't be used as dependency".


For simplicity, only one package can be specified as argument for --base , and it has to be standard sce package (doesn't depend on any other sce).

So you would make one basic package, and base arbitrary number of packages on it. But there would be only one level of hierarchy and no circular deps.

For faster package generation, classic sce's could be accompanied by list of deb's that it was made from.


-----------------------------

I use dCore on dell c400 laptop and generation of SCE's takes a while, even though i mostly need i3, geeqie, geany, audacious, dwb browser, elinks and few cli tools.

If i want to add an app to it, rebuild of this one big sce is slow. Making separate sce is also fairly slow and reuses lots of files of original sce.

I would happily switch to clasic tinycore, but i cannot get dwb to build on it with new webkit, and other browsers and not very lightweight in comparison. I would probably have to stick with core 4.x. dCore has just way more software available more easily.

[Jason W]

First, I appreciate any and all interest in dCore and I don't want to come across as defensive or dismissive in the presence of new ideas or requests.   I also think it would be useful if there was an option that reduced duplication among sce packages.

How it is now is basically like PC-BSD, all dependencies are in the package file which is quite large, but only what is not already installed is merged into the live directory tree. 

I have some ideas in this direction I may can prototype, I will think more on it.

 

[thane]

I admit to being in over my head here, but my understanding is that it takes a lot of effort on the part of developers to maintain the Tiny Core tcz repository, not only to package apps as tcz's but to make sure all the dependencies are in sync etc.

I thought an advantage of dCore is that less of that is required. Since each app is packaged separately with its own set of dependencies (regardless of what's packaged with other apps) there's a lot more flexibility in obtaining apps from other repositories and having them work in TC. Not only a lot more apps but less effort needed from developers.

Is this wrong?

[Jason W]

Thane - that is correct.  However I have made some adjustments to importsce and the related files that will allow the use of existing sce's as dependencies.   Like the size listing option, it will not affect those that want to stay with the current self contained sce.  This option would make expanding one's sce collection much more efficient, of course the current routine will remain the same.

Say you have a huge mydesktop.sce that contains all of the deps of gthumb.  Use the following command to make use of mydesktop.sce as a dependency.

importsce -d gthumb

With the -d option, you will be prompted with a select menu listing existing sce's in your sce directory.  They don't have to be mounted.  Select mydesktop as a dep, then proceed.  Multiple dependencies are possible.

The result is that gthumb.sce will contain only the gthumb package if all the required deps exist in mydesktop, that is checked for by the list of debs that mydesktop.sce is made of against what gthumb requires.  A gthumb.sce.dep file is created, containing mydesktop as the entry.  If loadsce sees a sce.dep file when loading, it will check the needed debs versus what the sce in the dep file consists of, alerting the user if the required debs are no longer in the mydesktop.sce.  Just in case one updates mydesktop.sce and removes packages on re-creation of it.  That way, no mysterious brokenness, missing required packages will be echoed.

I have the code in place and will test more this evening. 

[netnomad]

hi jason w,

great news... i'm looking forward and help testing

thank you for your commitment.

[yoshi314]

The result is that gthumb.sce will contain only the gthumb package if all the required deps exist in mydesktop, that is checked for by the list of debs that mydesktop.sce is made of against what gthumb requires.

What will happen if there are some dependencies missing? Will they be included into gthumb, or will a rebuild of mydesktop.sce will be required?