WelcomeWelcome | FAQFAQ | DownloadsDownloads | WikiWiki

Author Topic: libre kernel  (Read 7598 times)

Offline FilthyJack

  • Newbie
  • *
  • Posts: 6
libre kernel
« on: July 21, 2016, 08:40:26 AM »
Hello everyone,

I've been playing with corepure64 for quite some time and it works perfectly, I've been able to test the kernel compilation with in-ram compiletc suite and my own version of distcc which was a pleasure. Now I'd like to reproduce the steps leading to a working ramfs and would need some tips on how to proceed.

I'd like to use libre-kernel 3.16.6 (is the current one deblob already?)
Are the devs working on a base initramfs and compile BysyBox+LibC statically into it?
Can you share the base initramfs tree and give some details on compilation?
I guess I'd have to statically recompile other TC specific tools, is there a base tree showing what's added after basic busybox+libc?

Thanks for the work you put in TinyCore, happy coding!

Offline curaga

  • Administrator
  • Hero Member
  • *****
  • Posts: 11048
Re: libre kernel
« Reply #1 on: July 21, 2016, 01:55:37 PM »
Our kernel is vanilla, ie it supports firmware blobs. Only ldconfig is statically linked (and from uclibc), everything else is dynamic.

The base initramfs is what you have in the ISOs or download as core.gz. Reproducing it all from scratch would be quite involved though.
The only barriers that can stop you are the ones you create yourself.

Offline FilthyJack

  • Newbie
  • *
  • Posts: 6
Re: libre kernel
« Reply #2 on: July 21, 2016, 03:26:45 PM »
Thanks for the answer!

I'm not really trying to reinvent the whole tinycore concept, I'm sure you've been through a lot to get it straight. What I'm after is the possibility to have a libre kernel + corepure64.gz + compiletc and then to be able to recompile everything from (full) source.

This is why I mentioned a pre-made initramfs , a template, before you add any pre-compiled stuff to it. Treating the binaries inside core.gz just like any package.
With that we could:

-Compile kernel.
-Get a fresh core.gz (populated with default stuff and portable scripts from TC).
-Compile ldconfig/busybox and all binary components. (creating a new core.gz)
-Compile any extension for the target system.

The whole OS could compile itself from full source and port itself to different platforms pretty easily. [/dream]




Offline curaga

  • Administrator
  • Hero Member
  • *****
  • Posts: 11048
Re: libre kernel
« Reply #3 on: July 22, 2016, 04:27:01 AM »
The TC scripts are stored in git, but that's not really a complete template, there's various config files, etc. So if you want core.gz without binaries and libs, you'd need to take it and remove them, for example with the help of find, file, and xargs.

There aren't that many packages involved, we've had various ports, though now only ARM is active. There isn't documentation on what packages exactly, though - this is something you can help with, if you'd like. You can log in to the wiki with your forum credentials (after 5 posts, IIRC), and create a new page for this.
The only barriers that can stop you are the ones you create yourself.

Offline FilthyJack

  • Newbie
  • *
  • Posts: 6
Re: libre kernel
« Reply #4 on: July 22, 2016, 02:13:17 PM »
I have been able to uncompress core.gz, modify it and recompress it countless times. Everything works fine, which is strange since the bzip/cpio/advdef process never produces the same output size.

Code: [Select]
# zcat core.gz | cpio -i -H newc -d
# find | cpio -o -H newc | gzip -2 > new_core.gz

When I uncompress it again the custom binaries I put in there are clean and the core itself runs fine.
Assuming there is a random component in the compression then fine but this is bothering me.

Otherwise yes I'd love to share the script I made with some docs on the wiki when I reach 5 posts.
Assuming you have already compiled replacement bins/libs:

Code: [Select]
sudo ./core_patch.sh <path to core.gz> <search path for replacement>
 (must be executed from its own directory)
 
-Unpack core.gz
-List every binary/lib excluding kernel objects and charmaps
-Check if you have a replacement for them in <custom path>
-Replace them and repack everything in new_core.gz
http://www.lotusrevenge.fr/PublicProjects/core_patch.sh
« Last Edit: July 22, 2016, 02:32:14 PM by FilthyJack »

Offline bmarkus

  • Administrator
  • Hero Member
  • *****
  • Posts: 7183
    • My Community Forum
Re: libre kernel
« Reply #5 on: July 22, 2016, 03:27:33 PM »
Just a note, -H is nor needed when uncompressing.
Béla
Ham Radio callsign: HA5DI

"Amateur Radio: The First Technology-Based Social Network."

Offline curaga

  • Administrator
  • Hero Member
  • *****
  • Posts: 11048
Re: libre kernel
« Reply #6 on: July 22, 2016, 03:52:36 PM »
cpio is an archive format, as such if you replace the files with the same contents, the time stamp changes, and this then changes the output file. If you recompress the same tree twice in a row, without touching it, the result will be byte identical.
The only barriers that can stop you are the ones you create yourself.

Offline FilthyJack

  • Newbie
  • *
  • Posts: 6
Re: libre kernel
« Reply #7 on: July 22, 2016, 06:11:21 PM »
I see! That makes sense, but it was confusing when I needed to check if the resulting core was identical.

Well, there is also that static ldconfig version, before I try to run it I wanted to ask if this was the appropriate way to compile it:
I guess it will simply panic or throw a bunch of errors if not but...

Code: [Select]
//from uClibc-0.9.33.2
make headers && make -C utils

The result binary being slightly bigger than the one you compiled in corepure64.gz I guess it is statically linked.

The default from uClibc doc was:
Code: [Select]
make headers && make CC="gcc -Wl,--dynamic-linker,${ldso_dir}/ld-uClibc.so.0 ${ldso_dir}/libc.so.0" -C utils

Again this is a lil confusing, it looks like the second option is correct because it does specify a lib to compile against even tho its name is dynamic-linker.

Thanks a lot for the tips, I'll remove the -H option as well!
« Last Edit: July 22, 2016, 06:22:10 PM by FilthyJack »

Offline curaga

  • Administrator
  • Hero Member
  • *****
  • Posts: 11048
Re: libre kernel
« Reply #8 on: July 23, 2016, 05:48:37 AM »
I don't remember, it's been years. Probably I built everything and just copied the binary. It's strictly a size optimization - glibc's ldconfig works perfectly, it's just 10x as large.
The only barriers that can stop you are the ones you create yourself.

Offline FilthyJack

  • Newbie
  • *
  • Posts: 6
Re: libre kernel
« Reply #9 on: July 23, 2016, 06:44:30 AM »
It was actually the bigger executable size which was static so no problems, my version of uClibc is different but the exe can be stripped and reduced in size to almost match the one you had in corepure64.gz. I also had to recompile kernel modules when I changed the toolchain but I think this is normal.

I'm not really used to publishing stuff on wikis, I guess I could post some kind of easy step by step to expose components present in a core release. With the scripts I use to manipulate core/kernel/deblob/extensions. Is this what you had in mind?

Thanks for your help curaga, really appreciate.
bmarkus: Removed "-H newc" from the uncompressing command and it still works perfectly, thanks :)
« Last Edit: July 23, 2016, 06:47:40 AM by FilthyJack »

Offline nick65go

  • Hero Member
  • *****
  • Posts: 839
Re: libre kernel
« Reply #10 on: July 30, 2016, 01:17:49 PM »
Thanks for the answer!

I'm not really trying to reinvent the whole tinycore concept, I'm sure you've been through a lot to get it straight. What I'm after is the possibility to have a libre kernel + corepure64.gz + compiletc and then to be able to recompile everything from (full) source.

This is why I mentioned a pre-made initramfs , a template, before you add any pre-compiled stuff to it. Treating the binaries inside core.gz just like any package.
With that we could:

-Compile kernel.
-Get a fresh core.gz (populated with default stuff and portable scripts from TC).
-Compile ldconfig/busybox and all binary components. (creating a new core.gz)
-Compile any extension for the target system.

The whole OS could compile itself from full source and port itself to different platforms pretty easily. [/dream]

I was thinking about the same process...
Uclib will be history, soon. Have a look at https://www.landley.net/toybox/about.html
Most of the apps from busybox are ready to be replaced, no need for libc/uclib etc.
BTW, see also a good environment to rebuild Tinycore from scratch http://landley.net/aboriginal/about.html
All in all, still Tinycore is the winner (until now) against other linux distros.


Offline curaga

  • Administrator
  • Hero Member
  • *****
  • Posts: 11048
Re: libre kernel
« Reply #11 on: July 30, 2016, 02:07:56 PM »
Why? Toybox is a license-motivated competitor to busybox with less functionality. Its license is the only advantage, and being BSD it is not an advantage vs GPL to us.
The only barriers that can stop you are the ones you create yourself.

Offline nick65go

  • Hero Member
  • *****
  • Posts: 839
Re: libre kernel
« Reply #12 on: July 30, 2016, 02:37:40 PM »
Hi curaga, sorry, I can not modify my previous post
I am not concerned about license thing. I am interested in smaller size, modular, less dependency, build itself from sources by scripts.
I agree, maybe for now toybox is not a FULL replacement of busybox, but for sure is smaller, simpler that busybox.
All the functions in toybox are new code, not clones of old code borrowed from busybox. Focus is on security (google accepted the code).
My idea was to replace, in steps, one by one the busybox functions. Having a init ram template without binaries were the first step...

I would like to add that it is good to remember that emeritus Robert, Tinycore "inventor", started with init-ram/tempfs and busybox. And Rob Landley (see my previous posted links), author of toybox  and aboriginal linux, was the pioneer of init-ram / in ram file system and developer/maintainer of busybox. IMHO, I think that is a good think to keep an eye on his developing software, not ignore his work.
Building ITSELF from sources, modular, using scripts, in the best thing to happen to tinycore.

Offline FilthyJack

  • Newbie
  • *
  • Posts: 6
Re: libre kernel
« Reply #13 on: July 31, 2016, 01:53:55 PM »
I've tried aboriginal, open-embedded and a bunch of other cross-compiling suite but I agree TinyCore is my fav and as a developper I'm really interested in audit and source control.
The closest I've come to reproduce TC 7.1 from source is:

-compile kernel-libre-gnu-4.2.7.
-depmod/install kernel modules to the rootfs.
-compile glibc 2.22/binutils/busybox/udev etc (buildroot can help too).
-copy all the boot/setup/config scripts related to TC.

Then you have full control over what is going in your rootfs and can compile any part of your OS from source or modify a root component like bysybox/toybox/*libc.

Offline mocore

  • Hero Member
  • *****
  • Posts: 647
  • ~.~
Re: libre kernel
« Reply #14 on: August 02, 2016, 08:04:10 AM »
The whole OS could compile itself from full source and port itself to different platforms pretty easily. [/dream]

I was thinking about the same process...

I have been considering attempting similar ,
.. but thaught it might be novel to try and build/package the core base using nix or guix package manager / 'expressions' [/dream]

« Last Edit: August 02, 2016, 08:08:13 AM by mocore »