Let's make the best Core for Linux, and the rest by SCE's! Unifying Linux!

[LichenSymbiont]

Let's discuss things for making the best core -- things that need no GUI programs.
Like hardening the security (can be done through just a script). Like having a kiosk-script, which limits the user just for the session.
Just for wishlists, and scripts and other practical ideas.

But first for an install script:
I thought that I'd contribute a script for building a minimal graphical dCore, making one core SCE with the script, but I would like some feedback and discussion first!

I really, really, really, really ... really... don't get the point of forking dCore and other cores!
It's obvious that we should just create the best damn Linux Core distro that we can, together!
All other distros could be created from such a core, and all Linux developers could contribute to such a core!

This was my initial motivation for getting into Tiny Core -- it made sense, unlike all other forking distros!
But also for speed and for having a simple live distro, of course.

We have the best people for the job right here in the forum! Don't we? I really hope so...

And then we just create install scripts, with the resulting system made into an iso!
But not even an iso at first, a core graphical SCE would be wonderful, to just add into your core!
This SCE should also be provided as a torrent (I can provide my Raspi for this).
And then we can have a minimal CLI torrent client / daemon, in the core, for a speedy download of your favorite base.

This way we can truly create a minimal Linux distro, that can also be fitted for any purpose, just by downloading an SCE for your use-case.
But it should be: Core.sce -> Use-case.sce

Why is cooperation and dialogue not the standard way?
I'm asking myself this as well, as I should have posted this earlier, and gotten a discussion going...

Mockup script:

Code: (bash) [Select]

#!/bin/bash
#well.. would be nice to have full bash functionality,
#as it has aritmetic and other nice scripting functionality I've come to expect.
#Or is the ash in Busybox as complete as bash? as I see most/all dCore scripts
#runs on it.

#Questions at the start! Then it will install it by the provided answers! No interruptions!
echo "Want a development environment? y/n or just press enter for no."
read ans
if [ $ans == "y" ]; then
dev="y"
fi

echo "Automate the Xorg configuration process, and start X after install?"
read ans
if [ $ans == "y" ]; then
autox="y"
fi

echo "Where do you live?"
read ans
set-main-mirror $ans
#Wow, I'm so ignorant of dCore yet... but I will learn.
#But there should be a script for splicing in Debian mirrors from your location, at the top of the mirror-list.

echo "Minimal X-server?"
read ans
if [ $ans == "y" ]; then
importsce -br Xvesa
fi
#I know this is a much bulkier build than TC's own! Perhaps we can get a TinyX or NanoX package?
#This is one of the reasons I don't write a proper install script, before getting feedback.
#But clearly a minimal X-server is required, in case the graphics fails.


#import whatever package contains lspci: (this is important for hw detection, and is a small utility, so it should probably be added to the core)
importsce -rb lspci

graphics=$(lspci | grep -m 1 "VGA")
if [[ $graphics == *Intel* ]]; then
graphics="intel"
else
if [[ $graphics == *NVIDIA* ]]; then
graphics="nv"
else
if [[ $graphics == *ATI* ]]; then
graphics="ati"
else
graphics="all"
fi
fi
fi

importsce -br Xorg-$graphics


importsce -br Xprogs
importsce -br gtk2
#and so on... until it contains the libraries that most GTK programs uses.
#And for a minimal WM, which isn't a crazily huge package in Debian:
importsce -br aewm++
importsce -br wbar
#wbar, as dCore has it configured.

if [ $dev == "y" ]; then
importsce -br gcc
#and other tools...
fi
echo "* * * All done! * * *"

if [ $autox == "y" ]; then
sudo Xorg -configure
sudo mv /newconfigfile /etc/X11/um...
#could we just use Puppy's xorg startup script?
startx
else
echo "Now you can configure Xorg yourself! Have fun!"
fi

Well, don't run this one, I just felt like providing a more practical example.
I could just as well have just listed what packages the core SCE should contain.
And this script uses importsce like usual... while it should be something like importinto /place/yourcore.sce packagename
Or just run "merge-all-mounted-SCEs" or something, after the script...

The purpose of this install script is to be able to set up a dCore from scratch in a few minutes.
This is something that would help Linux-adoption greatly, by having a live distro that can run anywhere, and also function as your main distro. And be able to install it quickly and easily.

So let's get a more cooperative atmosphere. Let's make such an awesome core, with GUI additions through SCE's, that it will be much more useful right away.
Let's discuss what's needed for Linux developers and all developers to embrace such an effort!

I have such a project going myself, for creating a unified GUI for all GUI-based applications, in FLTK (which can be made to look and behave just as well or better than Qt!).
But I want to add all the essential things for it before publishing it, as GPL code, to create a development environment that wouldn't be possible in any other OS.
I will easily be able to create a package manager in it, but I'm working with the low-level stuff still.
I hope I will be able to get it publishable in two weeks time.
Then we can have community-contributions right from inside the OS, through git, and IRC, and other means.

But the effort here is not a distro-creation effort, we should create a system for creating your own merged SCE's, and have it evolve by just regular users adding packages, and merging them into SCE's.
However, I don't think copying all data from such an SCE into RAM would be wise...

Let's get focused! Let's make TC/dCore a more focused development effort, but without the limiting focus of other distros. With this core you can create everything else!

It's only because the TC project doesn't have this focus of creating the best core for Linux, and not having simple ways of contributing, and not having a proper focus in the community, that I've even thought about forking my own TC-like project, for unifying Linux.

But let's do it here, let's create a layered core, where we have clear goals in every layer!
Elimination of conflicts through proper layers!

But having an install-script, for creating a basic graphical SCE is essential.
Look at the Puppy Linux community... that's just because it's graphical, and reasonably fully-featured...
Let's create a better community than that! Let's integrate all Linux communities!

Well, let's discuss ^^

Edit:
I might as well write about my wishlist.
Once we have a proper live distro that can be molded into whatever other distros do for people, we can start focusing on the actual software-stack of Linux.
Essential components:
*The kernel.
*Drivers (especially OSS graphics).
*X-server.
*Sound-layer.

These are the things I would be working with right now, if we already had a proper Linux distro, and a united community for the core Linux-system. Where people stopped acting like monkeys, and just cooperated on one, layered system!

Even in the kernel, if we have proper layers for things, the goals becomes clear, and there is no conflict.
Conflicts are the result of basic design flaws.

But first we need:
*A proper window manager (something I'm working on).
*A proper development environment (working on it). An IDE, where you have access to the source of all the extensions, or packages in your system.
*Community integration (WIP).
Well, I might as well say that I'm working on the whole user-experience, and I will establish the whole basic architecture, for all future (F)OSS development. ^^

Once we have such a system, with a superior development environment, it might become synonymous with Linux.
So people just refer to Core as *the* Linux distro, like "Here is Linux, make it into whatever you want!".
And GNU or other official Linux sites need not refer to bulky distros, but *the* Linux Core!
Of course, it needs to adhere to the software freedom principles, but you may still get binary blobs, but be informed clearly that it's a binblob.
I think I've made my point...

[LichenSymbiont]

As this thread is eerily silent, I will post my new idea for constructing the install script here:
I have been experimenting with SquashFS on my regular Arch Linux computer, and I see you can easily add new files to it! It's not a read-write file-system, but using the regular squash tool (mksquashfs), you can add files by just adding to your sfs file!

So I was about to propose a new CLI option for importsce, to have packages merged into one .sce, instead of one named after the package.
But there is already:
http://forum.tinycorelinux.net/index.php/topic,16199.msg95963.html#msg95963
Jason's script for merging sce files.

But sce's contains duplicate libraries, if you don't use the -d option well, so it's not the optimal solution.
What I wanted to propose was this:
http://forum.tinycorelinux.net/index.php/topic,16197.0.html
Wow, a historical thread indeed: the -d option was created there!

Now let's implement what was actually, really, the point netnomad had: to have a base.sce, which you use the -d option for all the other graphical packages.
Of course, you could have also have other super-sce's to merge to, like term-core, and base/core.sce, just by providing --use-this-sce file.sce, when using importsce.
I don't know what letter should be used, perhaps -f as now -l is being used instead.

So when you use that option, you import your packages to that .sce, and not to the sce named after the package.
Really simple!
Though it would have to contain packages that don't change too often, but then you can just add the new package, and it will be updated (I would expect this is how squashFS works).

This would be a LOT more practical to use, as we can then have a trusted sce repo, with the checksums, and then torrents, for other dCore users to share some bandwidth (as torrents is the quickest and easiest way to handle country-specific mirrors).
As sometimes I really just want to have a nice, simple graphical install, which I could install for someone non-technical, in a few minutes, and just leave it...
This is a good feature of an OS, is it not?

Oh, and the .dep file for these super-packages could just be written to in the script with >>, to just add deps from the package that is being merged.
Or perhaps just read in the .dep file, add the missing dep's, and rewrite it with the new list.

So, let's discuss!

I will now create this option, and experiment with it, so as to know that it would really work...
And then expand, and fix my install script, to at least merge the most basic things into this core.sce.

edit:
Well, now I see how the -f (future -l ?) option works, so that should be used by the install script.
But continuous integration of packages would be nice.

[netnomad]

hi LichenSymbiont,

i want to add one aspect to your ideas:
what about the option to update single packages in a sce-container?
sometimes there is just a little package out of date
and the reimport of a huge sce takes too much time and resources
with your approach it could be possilble to update single packages in a sce?

keep on, thank you for sharing your indeas.

[LichenSymbiont]

Hi! Great that the one that initially posted this idea comes here! ^^
And thank you for your support!

When you just merge a directory/file into an SquashFS file, and files have the same name, they get _# appended to them (ie _1, _2 ...).
Ideally, you'd just want to just remove the old package, and replace it with the new.

So can you remove files with mksquashfs, or other tool...?
I can't see any way with mksquashfs... or even change the duplicate-files behaviour to just overwrite...
So we would need a special tool to be able to manipulate the squashfs data, without having to rewrite everything.

But that option is there: to just load the sce, load the new package into the new sce, and import all the other packages contained in the sce, into the new one.
Though, you would have to know exacly what files belongs to what packages... so it would be just as well to import the .deb files that are contained in the sce (in the .md5sum file, somewhere, which is a strange file to put such info).

But this makes me wonder if .deb files could just be extracted and placed in their own directories in the sce file, so you know what package contains what... I mean, what would be the purpose of .deb files if we could just have them extracted neatly into SquashFS files?
I would like to make an experimental script which just copies all .deb files right into an sce file.
Which you can then have mounted (but not copied to RAM), and then if there is a conflicting package (directory with the same name as one already contained), it will place it in a new packages#.sfs file.

Then you can just copy anything you like from this sce into your file-system.

Well, I will think some more about it... it's an interesting idea at least.

Edit:
I think I missunderstood your question, which was not just if you could merge an updated package, but also import it.
Well, with the idea with having .deb files extracted into sfs files, that would be trivial: just remove the files extracted from the old package, and copy the files from the new package to the file-system.

[Jason W]

LichenSymbiont, it would be much easier to read and understand your posts if they were shorter and more to the point.   I do see some points that are brought up in this thread, one of them being a much quicker means of updating an sce without having to totally import the whole thing again.  I have a 2 GB sce that I use and it does take a while to re-import even though perhaps only a few packages have changed. 

I am now polishing off a "one command update" function for one's SCE directory.  But I do have an idea I can work on before I further test that one.  I have thought of a way to quickly update an sce with only things that have changed using the existing functions.  Basically, unpack the sce in to the standard working/merging directory, then compare md5sums with the sce contents and the Packages file.  Then, download new startup scripts as is normally done, and then only download and merge any changed debs.  If there is only one changed deb, then only one deb is downloaded and merged into the new sce, along with fresh startup scripts.   In fact, to save a hit on the server for every deb package checking for a startup script, I may pack then into a tar.gz that is downloaded and then handled locally.  Would save time and server hits. 

For things like sharing general sces and hosting them, or a script to create them, that is outside the scope of what I am dealing with.  But scripts are welcome in the scripting section.  As always, if a script or an idea is something we can use in base dCore, it will be considered.

[LichenSymbiont]

"it would be much easier to read and understand your posts if they were shorter and more to the point."
Thank you for the feedback! I shall definitely try to make my points clearer, and organize things better.
But I don't think I really repeat myself, only re-express an important point.

"As always, if a script or an idea is something we can use in base dCore, it will be considered."
That's what I like to hear 
Though at the moment, it's good enough to just publish on this forum.
And I think most of my scripting ideas I'm working with at the moment, are upstreamable (if that's a word).

Oh, and the dCore team shouldn't have to manage sce super-packages, through it would be nice if we had a graphical sce for a basic graphical dCore, just like a graphical TC.
I think that's an essential piece, and I would have jumped on this project sooner if it was as straight-forward as a graphical TC.

As one of the major benefits of a live distro is after all to require no installation -- perhaps just a little configuration.
I mostly just want to have the community engage in an exchange of scripts, as that's after all one of the major benefits of using Linux (open source code).
So you'd just have to manage the check-sums, for things shared through a torrent.

Nice to hear you are implementing such an update script! great work!

As in http://en.wikipedia.org/wiki/Great_Work : "Within Thelema, the Great Work is generally defined as those spiritual practices leading to the mystical union of the Self and the All.".
The Great Work of the software world is just the same... to crystallize the Philosophers Stone of software, hehe... But a package management system is a package management system, and can only be made more solid, simplified and optimized. But it's a Great Work, in that it will help in the unification of Linux.

Well, down the line... but even my direct effort of unifying all the basic GUI programs in the simplest and lightest GUI framework (FLTK), will just be a ripple, and it might just be another tool at first.
But I hope it will become the core GUI of TC, as it integrates things better, and presents them better (or however the user chooses to arrange it). As I've already integrated most tools a graphical TC comes with. Just not the more advanced stuff, like the tool that lists processes, mounted files, and so on.
But it will make IDE's obsolete almost from the start, as you can easily set up a development envrionment, and then extend it yourself, to make it your ideal IDE and work environment.

Well, the GW can't easily be described in words ^^
But I will publish my terminal unification tool within a week, which I hope will clarify what I'm working on.
I would have published it sooner, if NDK++ (the C++ library for ncurses), was programmed better. So I'm working on my own classes for ncurses, as well as the system for encapsulating modules in the program.
Well, even this simpler tool, only dealing with text is not easy to explain...

And today I will experiment with making my simple change to the import script, to merge packages in one core sce (or other super-sce), and have packages in their own directories within (so it's easy to find what belongs to what package).
And from there, perhaps try the way of integrating all the debs in one sfs file-system.
Too bad you can't easily remove files, or in this case, a directory... or to just replace the merged files.

[LichenSymbiont]

I just published the installation script as it's right now, but there is a lot more to do, and then a lot of testing!
https://github.com/LichenSymbiont/linux-scripts/blob/master/dCore/dcore-install.sh
Please inspect it, and give feedback and suggestions.

I will clean up the first message of this topic, and add the link there later. But for tonight I will just read the Arch Install Scripts (https://github.com/helmuthdu/aui), so I can start integrating more advanced functionalities.

Edit: I guess I can't edit the first message. I thought my inability to edit my posts was due to forgetting to allow tinycore.net in NoScript...
Edit2: I see now that this thread should have remained inactive. But it was here I first proposed the idea of an installation script... but I've been far too verbose here...

[hal9king]

Anything BUT: Debian, Ubuntu, Arch, RedHat/Fedora ...  there is a reason people still use Slackware ... it still works!

[Jason W]

I did make a prototype of the Arch Rollback Machine, which lets you choose a repo snapshot to pull from.  The catch is it is x86 only, and only one server which gets slow after a couple of imports.   But arch packages are more simply built than Debian, kinda like Slackware with dependency logic.  So if there were more servers it would be worth having around.  Tcz was the extension type I prototyped with.  Dep files were created, tcz-load used, etc.  Kinda neat for the couple afternoons I put into it.  Xorg and fluxbox worked.  They had pulled the server at one point, so I did not go further since another pull would mean dead in the water and all effort for nothing.