libre kernel

[FilthyJack]

Hello everyone,

I've been playing with corepure64 for quite some time and it works perfectly, I've been able to test the kernel compilation with in-ram compiletc suite and my own version of distcc which was a pleasure. Now I'd like to reproduce the steps leading to a working ramfs and would need some tips on how to proceed.

I'd like to use libre-kernel 3.16.6 (is the current one deblob already?)
Are the devs working on a base initramfs and compile BysyBox+LibC statically into it?
Can you share the base initramfs tree and give some details on compilation?
I guess I'd have to statically recompile other TC specific tools, is there a base tree showing what's added after basic busybox+libc?

Thanks for the work you put in TinyCore, happy coding!

[curaga]

Our kernel is vanilla, ie it supports firmware blobs. Only ldconfig is statically linked (and from uclibc), everything else is dynamic.

The base initramfs is what you have in the ISOs or download as core.gz. Reproducing it all from scratch would be quite involved though.

[FilthyJack]

Thanks for the answer!

I'm not really trying to reinvent the whole tinycore concept, I'm sure you've been through a lot to get it straight. What I'm after is the possibility to have a libre kernel + corepure64.gz + compiletc and then to be able to recompile everything from (full) source.

This is why I mentioned a pre-made initramfs , a template, before you add any pre-compiled stuff to it. Treating the binaries inside core.gz just like any package.
With that we could:

-Compile kernel.
-Get a fresh core.gz (populated with default stuff and portable scripts from TC).
-Compile ldconfig/busybox and all binary components. (creating a new core.gz)
-Compile any extension for the target system.

The whole OS could compile itself from full source and port itself to different platforms pretty easily. [/dream]

[curaga]

The TC scripts are stored in git, but that's not really a complete template, there's various config files, etc. So if you want core.gz without binaries and libs, you'd need to take it and remove them, for example with the help of find, file, and xargs.

There aren't that many packages involved, we've had various ports, though now only ARM is active. There isn't documentation on what packages exactly, though - this is something you can help with, if you'd like. You can log in to the wiki with your forum credentials (after 5 posts, IIRC), and create a new page for this.

[FilthyJack]

I have been able to uncompress core.gz, modify it and recompress it countless times. Everything works fine, which is strange since the bzip/cpio/advdef process never produces the same output size.

Code: [Select]

# zcat core.gz | cpio -i -H newc -d
# find | cpio -o -H newc | gzip -2 > new_core.gz

When I uncompress it again the custom binaries I put in there are clean and the core itself runs fine.
Assuming there is a random component in the compression then fine but this is bothering me.

Otherwise yes I'd love to share the script I made with some docs on the wiki when I reach 5 posts.
Assuming you have already compiled replacement bins/libs:

Code: [Select]

sudo ./core_patch.sh <path to core.gz> <search path for replacement>
 (must be executed from its own directory)
 
-Unpack core.gz
-List every binary/lib excluding kernel objects and charmaps
-Check if you have a replacement for them in <custom path>
-Replace them and repack everything in new_core.gz
http://www.lotusrevenge.fr/PublicProjects/core_patch.sh

[bmarkus]

Just a note, -H is nor needed when uncompressing.

[curaga]

cpio is an archive format, as such if you replace the files with the same contents, the time stamp changes, and this then changes the output file. If you recompress the same tree twice in a row, without touching it, the result will be byte identical.

[FilthyJack]

I see! That makes sense, but it was confusing when I needed to check if the resulting core was identical.

Well, there is also that static ldconfig version, before I try to run it I wanted to ask if this was the appropriate way to compile it:
I guess it will simply panic or throw a bunch of errors if not but...

Code: [Select]

//from uClibc-0.9.33.2
make headers && make -C utils
The result binary being slightly bigger than the one you compiled in corepure64.gz I guess it is statically linked.

The default from uClibc doc was:

Code: [Select]

make headers && make CC="gcc -Wl,--dynamic-linker,${ldso_dir}/ld-uClibc.so.0 ${ldso_dir}/libc.so.0" -C utils

Again this is a lil confusing, it looks like the second option is correct because it does specify a lib to compile against even tho its name is dynamic-linker.

Thanks a lot for the tips, I'll remove the -H option as well!

[curaga]

I don't remember, it's been years. Probably I built everything and just copied the binary. It's strictly a size optimization - glibc's ldconfig works perfectly, it's just 10x as large.

[FilthyJack]

It was actually the bigger executable size which was static so no problems, my version of uClibc is different but the exe can be stripped and reduced in size to almost match the one you had in corepure64.gz. I also had to recompile kernel modules when I changed the toolchain but I think this is normal.

I'm not really used to publishing stuff on wikis, I guess I could post some kind of easy step by step to expose components present in a core release. With the scripts I use to manipulate core/kernel/deblob/extensions. Is this what you had in mind?

Thanks for your help curaga, really appreciate.
bmarkus: Removed "-H newc" from the uncompressing command and it still works perfectly, thanks

[nick65go]

Thanks for the answer!

I'm not really trying to reinvent the whole tinycore concept, I'm sure you've been through a lot to get it straight. What I'm after is the possibility to have a libre kernel + corepure64.gz + compiletc and then to be able to recompile everything from (full) source.

This is why I mentioned a pre-made initramfs , a template, before you add any pre-compiled stuff to it. Treating the binaries inside core.gz just like any package.
With that we could:

-Compile kernel.
-Get a fresh core.gz (populated with default stuff and portable scripts from TC).
-Compile ldconfig/busybox and all binary components. (creating a new core.gz)
-Compile any extension for the target system.

The whole OS could compile itself from full source and port itself to different platforms pretty easily. [/dream]

I was thinking about the same process...
Uclib will be history, soon. Have a look at https://www.landley.net/toybox/about.html
Most of the apps from busybox are ready to be replaced, no need for libc/uclib etc.
BTW, see also a good environment to rebuild Tinycore from scratch http://landley.net/aboriginal/about.html
All in all, still Tinycore is the winner (until now) against other linux distros.

[curaga]

Why? Toybox is a license-motivated competitor to busybox with less functionality. Its license is the only advantage, and being BSD it is not an advantage vs GPL to us.

[nick65go]

Hi curaga, sorry, I can not modify my previous post
I am not concerned about license thing. I am interested in smaller size, modular, less dependency, build itself from sources by scripts.
I agree, maybe for now toybox is not a FULL replacement of busybox, but for sure is smaller, simpler that busybox.
All the functions in toybox are new code, not clones of old code borrowed from busybox. Focus is on security (google accepted the code).
My idea was to replace, in steps, one by one the busybox functions. Having a init ram template without binaries were the first step...

I would like to add that it is good to remember that emeritus Robert, Tinycore "inventor", started with init-ram/tempfs and busybox. And Rob Landley (see my previous posted links), author of toybox  and aboriginal linux, was the pioneer of init-ram / in ram file system and developer/maintainer of busybox. IMHO, I think that is a good think to keep an eye on his developing software, not ignore his work.
Building ITSELF from sources, modular, using scripts, in the best thing to happen to tinycore.

[FilthyJack]

I've tried aboriginal, open-embedded and a bunch of other cross-compiling suite but I agree TinyCore is my fav and as a developper I'm really interested in audit and source control.
The closest I've come to reproduce TC 7.1 from source is:

-compile kernel-libre-gnu-4.2.7.
-depmod/install kernel modules to the rootfs.
-compile glibc 2.22/binutils/busybox/udev etc (buildroot can help too).
-copy all the boot/setup/config scripts related to TC.

Then you have full control over what is going in your rootfs and can compile any part of your OS from source or modify a root component like bysybox/toybox/*libc.

[mocore]

The whole OS could compile itself from full source and port itself to different platforms pretty easily. [/dream]

I was thinking about the same process...

I have been considering attempting similar ,
.. but thaught it might be novel to try and build/package the core base using nix or guix package manager / 'expressions' [/dream]