Kernel Panic - Not Syncing Error

[Frank]

fellow, that was an amazing story to read. I completely agree with your analysis: the source of the problem is not the 2.6.29 kernel by itself, but "something" in the initramfs ( = tinycore.gz).

Here is my take. Everyone reports the same error message: "Attempted to kill init!" That looks very familiar. I get that message when I play with custom initramfs files and introduce a typo or "thinko" whereby the kernel cannot pass control to the init process after it (the kernel) is done with its own stuff.

That makes me think that the "init" program in TinyCore 2.0 is (somehow) the culprit. "init" in TinyCore 2.0 is provided by Busybox 1.13.3; TinyCore 2.1 upgraded to Busybox 1.13.4. Looking at the Busybox website, major bugfixes have been applied between 1.13.3 and 1.13.4, including this one: "init: major improvement in documentation and signal handling. Lots of nasty, but hard to trip, races are fixed."

I therefore suggest that you stop concentrating on TCL2.0, and try a more recent version instead. I know, the Wiki says that 48 MB RAM are required for TinyCore, and your machine has only 40 MB. It's worth a try nevertheless: first, using the "embed" boot code, I can boot even TinyCore 2.7 in a virtual machine (QEMU) with 40 MB; second, you could use MicroCore instead of TinyCore, simply to see whether the Busybox version is a factor or not.

[gerald_clark]

I have an eBox 2300 with the SiS processor that has the same problem. Here is what Itried.
Boot 2.8rc2 - Error
Boot 2.8 with 1.4 initrd - boots
Boot 2.8 with 1.4 initrd with 2.8 modules added - boots
Boot 2.8 with 1.4 initrd with 2.8 modules and 2.8 busybox - boots.

So it apparently is not the 2.8 kernel, the 2.8 modules, or the 2.8 busybox.

-------------------------------

After further investigation I found the problem to be with libc.so and ld-linux.so.
Copying those 2 libraries ( and symlinks ) from V 1.4 to V 2.8 allows V2.8 to finish loading.
Copying libresolv.so allows nfs to mount, and all my extensions to load.
There are still GLIBC errors reported, but that is expected.

The problem appears to be some change in libc and/or ld, or in the way they were compiled.

[curaga]

Using the bootcodes "base norestore embed nofstab noswap" you can really drop the minimum memory requirement, at the expense of functionality.

Hmm, thanks for the info gerald_clark. Let's see what Juanito says.

[Juanito]

I compiled libc in tc-1.x in pretty much the same way as in tc-2.x - the version of libc and gcc is different in both cases, so it's kind of hard to know what exactly the problem is.

I know in other threads some users have found errors in exception handling with apps compiled with the tc-2.x version of gcc, but this seems to be quite rare. The are no doubt some bugs with the libc version used in tc-2.x, but they don't show up on any hardware I have.

Let me think about this a while.

[Juanito]

When you mention libc.so, ld-linux.so and libresolv.so, I take it you're speaking of:

/lib/libc.so.6 -> /lib/libc-2.9.so
/lib/ld-linux.so.2 -> /lib/ld-2.9.so
/lib/libresolv.so.2 -> /lib/libresolv-2.9.so

Looking at /usr/lib/libc.so, there is a small difference between the versions:

Code: [Select]

[new]
$ cat /usr/lib/libc.so       
/* GNU ld script
   Use the shared library, but some functions are only in
   the static library, so try that secondarily.  */
OUTPUT_FORMAT(elf32-i386)
GROUP ( /lib/libc.so.6 /usr/lib/libc_nonshared.a  AS_NEEDED ( /lib/ld-linux.so.2 ) )

[old]
$ cat /usr/lib/libc.so
/* GNU ld script
   Use the shared library, but some functions are only in
   the static library, so try that secondarily.  */
OUTPUT_FORMAT(elf32-i386)
GROUP ( /lib/libc.so.6 /usr/lib/libc_nonshared.a )
..but this is not in the base, and I can't imagine it would make a difference anyway.

I double-checked my notes and libc was compiled for use on i486 and up.

[curaga]

It's possible it's a gcc bug as well, gcc might have let 686 code in.

[fellow]

Many thanks for all your precious replies !

Frank and gerald_clark with your input I can now confirm that the panic error does _NOT_ come from the 2.6.29 kernel by itself :

I've customized both initramfs files of MC 2.7 and TC 2.7 by copying  libc.so ,  ld-linux.so  and  libresolv.so  INCLuding the respective symbolic links from TC 1.4.3
(thereby learning from gerald_clark that one must not always create a new symlink but may just COPY over :-)

Then I just remotely edited my notebook's GRUB  menu.lst  "on the fly" sitting in front of my "serialized/terminalized" Ubuntu desktop (remember my previous post ;-)
( having learned from Frank that the boot option 'embed' had been made available to stay on initramfs,
  i.e. to skip the usage of tmpfs in the init script preceding BusyBox' init )

FIRST trial with fixed _M_C 2.7 initramfs

   kernel (hd0,0)/boot/bzImage-2.7 embed noscsi nousb noagp nofirewire acpi=off pci=off S debug loglevel=7 initcall_debug console=tty0 console=ttyS0
   initrd (hd0,0)/boot/microcore-2.7-with-TC1.4.3-libs.gz

   ==>  BOOOOOOOTed and worked like a charm !!!

SECOND trial with fixed _T_C 2.7 initramfs

   kernel (hd0,0)/boot/bzImage-2.7 embed noscsi nousb noagp nofirewire acpi=off pci=off S debug loglevel=7 initcall_debug console=tty0 console=ttyS0
   initrd (hd0,0)/boot/tinycore-2.7-with-TC1.4.3-libs.gz

   ==>  Booted "almost" up to the end ... but finally experienced segfault errors

   Setting keymap to us Done.
   login[1769]: root login on 'tty1'
   esetroot[1843]: segfault at 43434704 ip b7fbde5b sp bfdc77f0 error 4 in ld-2.3.6.so[b7fb2000+14000]
   flwm_topside[1841]: segfault at 43434704 ip b7f77e5b sp bf87f240 error 4 in ld-2.3.6.so[b7f6c000+14000]
   wbar[1846]: segfault at 43434704 ip b7fdbe5b sp bfde29d0 error 4 in ld-2.3.6.so[b7fd0000+14000]

   ==>  with Ctrl-Alt-F1 I can reach the text console and continue "normal" working in the _C_UI

   ==>  so the old loader ld-2.3.6.so stemming from TC 1.4.3 seems not to be in harmony with _G_UI stuff

   ==>  Questions to gerald_clark
               (1) Is it this what you meant by "There are still GLIBC errors reported, but that is expected." ?
               (2) Could you please give me/us some hints of your systematic approach in finding out what libs the culprits are ?

THIRD trial with fixed _T_C 2.7 initramfs  and  a CUSTOMIZED and even NEWER kernel 2.6.30.10 from kernel.org + patched with SLICES of the TC-patch

   kernel (hd0,0)/boot/bzImage-2.6.30.10-tc-patch-adapted embed noscsi nousb noagp nofirewire acpi=off pci=off S debug loglevel=7 initcall_debug console=tty0 console=ttyS0
   initrd (hd0,0)/boot/tinycore-2.7-with-TC1.4.3-libs.gz

   ==>  Booted "almost" up to the end ... SAME result as in SECOND trial (as expected ;-)

It's fantastic to see once again the Core Team members, in this case curaga and Juanito, doing a perfect work, caring that much about the users and being accompanied by such an invalueable community !
It's a real pleasure being with you on board !

[gerald_clark]

To find the libraries causing the problem, I ran ldd on busybox ( the suspect program ).
I then replaced the libs in various combinations from 1.4.
The libc and ld-linux were both needed to allow the system to complete booting.
The libresolv was needed to get some of the other utilities not to core dump.
There are still some library incompatibilities that cause some programs to fail, but this
was just an exercise to determine the root cause of the boot failure.
We now know it is not a kernel problem, but a problem in the libraries.
Whether it is a library source or a compiler bug is not yet known.

[fellow]

gerald_clark, I just want to say thank you for explaining your evident procedure (finally with wikipedia's help I even remembered the meaning of core dumps  :-))

I'm eagerly looking forward to what the investigators will discover ... thanks in advance !

[fellow]

Dear members of the TinyCore Team

would you be so kind and give us, i.e. the folks being unfortunately somewhat locked out by this (not kernel but ;-) lib or gcc caused panic, some prospect of what you plan or even are about to do in curing us from this issue ...

I hope that you will take this request not as some kind of demand but as our hope that we too are further-on able to participate and contribute to this unique TC/MC-project and its lovely community.

Is there perhaps any additional information and/or examinations you would need to be delivered by the affected users ... of course just let us know.

To be honest I'd like to append :
Though I've already succeeded a couple of times to build a customized kernel it seems to me being a far more challenging enterprise to cope with the specialties of the various envolved libraries or the last touches of the compiler options (at least I got this impression when looking into a huge book belonging to those topics ;-) ...

Thank you very much in advance for considering this issue !
 

[Juanito]

About the best thing I can think of at the moment is to recompile libc and/or gcc when we get to tc-3.

I'm a bit stuck as I do not have an i486 or i586 to test with.

[curaga]

Is there perhaps any additional information and/or examinations you would need to be delivered by the affected users ... of course just let us know.

Thanks, but your efforts have already pinpointed where the issue is.

[fellow]

Juanito, curaga, thanks for your replies.

Just to assure that I got your points right ( and my learning curve may go ahead ... :-) :

(1)  Recompiling the core's libs can not be done under MC/TC itself ( + compiletc.tcz etc. )  but only on your specific IDE which regularly generates MC/TC  ?

(2)  To find out the correct set of options to be fed into the gcc command line for recompiling the libs may not be sufficient ...  ?

(3)  ... even the gcc source package _itself_ may have to be recompiled ( again in your IDE ? ) to get rid of possibly sneaked-in i686 code  ?

(4)  I hope you will indulge me in posing the ( unavoidable ;-) question, if a GROSS prospect could be given of the appearance time of tc-3  ?
 

[curaga]

1 - no, they are done under TC/MC and should be. It's just that you cannot upgrade in place, you have to store them somewhere, upgrade offline, and boot with the upgraded files to test.

2 - sometimes it requires magic incantations and sacrifices

4 - there have been some hints dropped at the fora

[fellow]

Okay ... I got that with the base' libs.

Aside the TC Team's professional work there seems to be some need for outside mysterious support    ... I hope the "sacrifices" will not be that sort of bloody ones    ... because we can't afford to lose any of our precious administrators 

But nevertheless the new word "incantation" will further elaborate my english code.

I'm still somewhat confused concerning gcc's role in this issue :
One needs gcc.tcz, gcc_libs.tczl, and quite a few more tcz to re-build the base libs ... but what if the anomaly originates from gcc(_libs) itself ... 

Unfortunately up to now I couldn't find any of those revealing "hints dropped" ... you don't accidentally remember a respective link ...