sce-import modification

[ulfr]

   just a question

   could the sce-import command be modified
   to individually package each app and dependency into a tcz 
   (or an sce that is similar to a tcz)
   instead of loading them into one big sce
   i know it would take longer to make extensions initially 
   but then the benefits of tinycore and dcore could be combined
   such as loading and updating apps individually 
 
   anyway
 
   thanks for making and maintaining this masterpiece
   its the best designed distro ever 

   ulfr
 

[Jason W]

Hi ulfr,
This was considered early in dCore, but the SCE design was settled on for various reasons.  Each has its advantages, but with the larger number of packages that Debian splits apps and libraries into, SCE simplifies things.  I recall circular dependencies in the Debian repo killed the TCZ approach, but we deal with those circular deps in sce-import.  I will do some private testing of allowing TCZ alongside SCE would be neat to be able to choose for those that prefer TCZ.  Now that things have settled a bit i can ponder it.  Probably not TCZ but  one SCE per Debian package and a dep file created  as an option of sce-import.  That way the SCE concept and structure, loading, updates,  stays the same.  Thanks for your input.

[ulfr]

   thanks jason for taking the time to explain

   i dont want to introduce more work or complexity
   i make the tcz and sce format interchangeable by just making a handmade sce
   as you outlined on the wiki
   (and thanks a lot for taking the time to provide that info as well) 
   
   i notice that the sce-load script finds and merges packages
   i was just thinking of squashing each package instead of merging it
   and just adding the name to a deps list for loading
     
   thanks again for all your hard work
   and attention to our annoying questions
   i want you to know that we appreciate the genius of this creation

   regards
   
   ulfr 

[Jason W]

Thanks ulfr, but the wiki info has been provided and maintained by others.

On further thought and looking at the code, the SCE design is fairly complex from a code standpoint and quite featured, so will require further thought to weave in a fully modular option.   In the early days would have been simple, but even then one reason to turn to the self contained SCE was for performance.  I copied in a terminal the contents of a a 2GB mounted SCE with 1215 packages as symlinks with the command in sce-load to another place in RAM, took a little over 2 seconds whether the files existed already or not.  Now, loading the SCE from scratch took about 2 minutes, but after that loading an SCE with the same package contents and only the addition of icewm took about 30 seconds.  The main time factor is the startup scripts.  But if loading 1215 packages was done one extension per package, I am sure performance would suffer with the parsing massive number of individual dep files with their large contents, if that were the only factor which it is not.  Each SCE that uses another SCE as a dependency has the list of packages of each dependency SCE contained in it, recursively, for use by sce-update and other functions. 

Granted, sce-update will make one large SCE get updated for one little startup script or package update.  But the SCE approach is as modular as one wants it to be.  Say an SCE named base-libs, one named server, one named python, one gtk2, and one xorg, etc, etc, with each depending on the others so no duplication.  Would make updating them take less time the more modulear it is. 

Thanks for your questions and ideas, all are taken into consideration and vetted, and most features have been the result of a feature request.  Not saying supporting a fully modular usage of the SCE conecpt will never happen, I will think more on it and how it can be done in the framework of the existing structure.

JW

[ulfr]

    thanks very much for taking the time to answer my questions
    i hope i did not distract you too much from your work

    after a little thought i agree with you 
    that there are many advantages to a combined sce
    and that the system you have created is both very powerful and complex
    and should not be modified

    since a homemade sce is easy to make
    that is clearly the way to go
    if one wants an interchangable tcz/sce
    your system provides a wealth of information
    that can be used to direct its construction 

   thanks again for all your work and patience
  i will now go back to playing with your masterpiece

   best regards

   ulfr

     


   
     

[Misalf]

Just dropping my two cents.
I haven't yet tried dCore. Even though I find the combination of Core's extensions management and Debians repo quite appealing, I'm quite limited when it comes to downloading from the internet.
If I get it right, there is much more network traffic involved when is comes to installing, downloading or updating extensions on dCore.
I understand the performance problem regarding big numbers of extensions, even on default Core, and I try to workaround this by using initrds containing the extracted extensions I'd load anyway. Didn't tried to combine TCZ extensions yet but that might be an option as well.
If, maybe one day, dCore provides a similar extension structure as default Core, I'll be happy to hop in the boat.

[ulfr]

   hi misalf

   having a combo sce automatically downloaded from debian is awesome 
 
   in addition
   i have found it is easy to make an extension that can function as both a tcz and a sce
   the only thing that is needed for it to be an sce is the md5sum file and naming it sce
   the only thing that is needed for it to be an tcz is the deps file (if needed) and naming it tcz
   i tried it with very simple tcz/sce first
   e3 which has no deps
   and then gtkedit which only has gtk1 and gtk1lib as deps (cant remember the exact names now)
   its very useful because you can make the exact extension combo you want
   and then load them when and if you want them in either tinycore or dcore 

   you might try this for fun

   i would also like to thank whoever writes the wiki
   its great having such fantastic documentation

   ulfr

[Jason W]

After some very detailed examining of the code and features of the sce format, it has become clear to me that we can't weave in a modular option as the changes would be like starting over and then the bug testing and fixes, etc.  Things are very stable now, and I wish we could have this option but it appears we need to stay the current course.   It was worth considering, and thanks for everyone's input.

[ulfr]

 
  thanks Jason

  for what its worth i agree
  dont mess with a masterpiece

  the way to introduce compatibility between the 2 systems
  is by making a handmade sce/tcz
 
  and thanks again for your awesome system 
  dcore and tinycore are the greatest

  ulfr

[nitram]

Glad you find the dCore wiki useful ulfr. IIRC forum member sm8ps laid the wiki groundwork, others provided minor contributions, i've been the primary contributor for the last few months. There's only a few more things planned then most of my wiki work will be done. Please let me know if there are new topics that would be useful. Feel free to contribute too if you wish. All contributions welcome, even if just posting a dCore screenshot, documenting system specs and any special installation tips in the installation section or adding newly trialed software to the desktop applications section,

Just wanted to clarify some things that may help potential dCore users. Converting *.tcz to a dCore *.sce is easy, the other way around not so much. Would be nice if dCore could be used as a tool to build complex packages to use in Tiny Core but it's not that simple. TC is still the leanest but dCore builds complex extensions with ease that would typically be very difficult to build in TC (missing dev files, wrong package versions, multiple dependencies, uncooperative build scripts, etc).

Using the sce-import -d or -l (dependency or list) options greatly minimize duplicate dependencies between extensions, outlined in the advanced extension management section of the dCore wiki.

Regarding bandwidth, dCore starts with a small base like TC. When importing extensions the *.deb files remain archived on the system, unless purged by the user, and remain available for update re-imports or importing new extensions. So once a system is built the only downloads required are base file system updates (~13MB following RC release), updated *.deb files only (not the entire SCE extension library) and the DEBINX (Debian index database files few MB). The extension update process is CPU not download intensive. During updates there are usually only a couple/few *.deb files that require downloading at a time, similar to TC. The CPU just needs to rebuild the SCE using a combination of the newly updated and archived *.deb files.

To save bandwidth a dCore user could run stable release, install leaner software, stretch out SCE update checks (eg. monthly vs weekly), use import -p (preserve) to avoid updating DEBINX between every import and minimize purging *.deb files.

Misalf if you ever have extra bandwidth give dCore a try, your feedback and expertise would be appreciated for testing, bug fixes and enhancements

[Misalf]

So I don't have to be too frightened to waste my bandwidth data plan. Just need extra storage. Doesn't sound that bad.

[curaga]

I believe most dCore parts use zsync like Core, ie delta downloads, which saves a lot of bandwidth.

[ulfr]

 thanks nitram curaga and jason for all your work 
 and for providing such great systems and documentation
 they are certainly sophisticated and powerful
 dare i say elegant
 they are the only ones i use
 now i can have some fun digesting all the possibilities

 regards

 ulfr
 
 
 

[jls]

I believe most dCore parts use zsync like Core, ie delta downloads, which saves a lot of bandwidth.

Hi
I don't think zsynch is used