Custom kernel: frozen black screen; used to work

[theYinYeti]

Hi! I’m running a fully up-to-date TC 4.7.7.
For a month or so, I haven’t been able to compile a working kernel, whereas my script worked before.
The script actually follows the wiki (http://wiki.tinycorelinux.net/wiki:custom_kernel):

Code: [Select]

# prepare
tce-load -il /etc/sysconfig/tcedir/optional/compiletc.tcz
tce-load -il /etc/sysconfig/tcedir/optional/perl5.tcz
cd /work/path
wget "http://repo.tinycorelinux.net/4.x/x86/release/src/kernel/linux-3.0.21-patched.txz"
wget "http://repo.tinycorelinux.net/4.x/x86/release/src/kernel/config-3.0.21-tinycore"
mkdir mod_path
xz -d <linux-3.0.21-patched.txz | tar -xf -
cd linux-3.0.21
cp -f ../config-3.0.21-tinycore .config

# patch
sed -i '/^#define[[:blank:]]*HID_MAX_USAGES[[:blank:]]/ s/12288/65535/' include/linux/hid.h

# make
make oldconfig
make bzImage
make modules
make INSTALL_MOD_PATH="/work/path/mod_path" modules_install firmware_install
mv arch/x86/boot/bzImage /work/path/vmlinuz-new
Then I take care of the core.gz file:

Code: [Select]

# prepare
tce-load -il /etc/sysconfig/tcedir/optional/advcomp.tcz
cd /work/path
wget "http://repo.tinycorelinux.net/4.x/x86/release/distribution_files/core.gz"
sudo mkdir "core-new"
cd "core-new"
gzip -dc ../core.gz | sudo cpio -i

# patch
find lib/modules/3.0.21-tinycore/ ! -type d -print \
| while read f; do if [ -f ../mod_path/"${f%.gz}"* ]; then
sudo rm -f "$f"
sudo cp -fa ../mod_path/"${f%.gz}"* "$(dirname "$f")/"
fi; done

# make
sudo chroot . depmod -a
find . | sudo cpio -H newc -o | gzip -9 >../core-new.gz
cd ..
advdef -z4 core-new.gz
First time I compiled my own kernel, I was very surprised to find out that my modules did not look the same as those from the standard TC core.gz (not the same files, compressed on one side, not on the other…), so I went what I thought was the safe route, considering the fact that my kernel is still very similar to the standard one: I pick from my new modules tree only those that already exist in the original modules tree and then do the replacement, taking the difference of compression into account where the filenames are concerned.

This worked well for me for several months. Not anymore, even though the base files (patched kernel and config) did not change: now, such a core-new.gz leads to a plain black screen: in Grub2 I select TinyCore with my new boot files, and then the screen simply goes black and does not respond to either CtrlAltDel, nor AltSysRq… I only can switch the power off :-(
Note that my -new kernel with TC’s standard core.gz works, but then I loose the effect of my patch…

Can someone tell me the right way to apply such a simple patch, and end-up with a working kernel of similar size and modules?

[theYinYeti]

I just tried simply copying the whole tree over instead of filtering, like this:

Code: [Select]

# patch
sudo rm -rf lib/modules/3.0.21-tinycore
sudo cp -r ../mod_path/lib/modules/3.0.21-tinycore lib/modules/but I end-up with a core-new.gz file almost 3 times bigger than standard core.gz!!
And I’m not even sure it works (I haven’t tried yet).

[theYinYeti]

After some comparisons, I think I understand better: some modules are kept into the initrd, and some are put into extensions. I did the same, and my vmlinuz-new and core-new.gz files are roughly the same size as those from TC. Yeah!
Yet my own kernel+initrd fails to boot: I just get a black screen without a single letter or pixel on it… And once again, vmlinuz-new does boot correctly when paired with an older proven-to-work core.gz file.
So the problem seems to be with core-new.gz.

So, here’s the full script, with all the gory details, where core-new.gz is concerned:

Code: [Select]

KVER=3.0.21
BITS=$(uname -m | grep -o 64) # empty in my case (32bits)
TCZs='ntfs-3g'
INST=/usr/local/tce.installed

tce-load -il /etc/sysconfig/tcedir/optional/advcomp.tcz
tce-load -il /etc/sysconfig/tcedir/optional/squashfs-tools-4.x.tcz
tce-load -il /etc/sysconfig/tcedir/optional/tar.tcz

sudo mkdir tcz
sudo mkdir -p firmware-new/usr/local/lib
sudo mkdir -p modules-new/usr/local/lib
sudo mkdir "core${BITS}-new"

# build core-new.gz file

cd "core${BITS}-new"
gzip -dc ../"core${BITS}.gz" | sudo cpio -i
sudo rm -rf lib/modules/${KVER}-tinycore
sudo cp -r ../mod_path/lib/modules/${KVER}-tinycore lib/modules/
sudo tar -cf - --remove-files \
lib/modules/${KVER}-tinycore/kernel/arch/x86/kvm \
lib/modules/${KVER}-tinycore/kernel/drivers/bluetooth \
lib/modules/${KVER}-tinycore/kernel/drivers/char/agp \
lib/modules/${KVER}-tinycore/kernel/drivers/firewire \
lib/modules/${KVER}-tinycore/kernel/drivers/gpu \
lib/modules/${KVER}-tinycore/kernel/drivers/hwmon \
lib/modules/${KVER}-tinycore/kernel/drivers/i2c \
lib/modules/${KVER}-tinycore/kernel/drivers/input/gameport \
lib/modules/${KVER}-tinycore/kernel/drivers/input/joydev.ko \
lib/modules/${KVER}-tinycore/kernel/drivers/input/joystick \
lib/modules/${KVER}-tinycore/kernel/drivers/input/tablet \
lib/modules/${KVER}-tinycore/kernel/drivers/input/touchscreen \
lib/modules/${KVER}-tinycore/kernel/drivers/leds \
lib/modules/${KVER}-tinycore/kernel/drivers/md \
lib/modules/${KVER}-tinycore/kernel/drivers/media \
lib/modules/${KVER}-tinycore/kernel/drivers/message \
lib/modules/${KVER}-tinycore/kernel/drivers/mtd \
lib/modules/${KVER}-tinycore/kernel/drivers/net/hamradio \
lib/modules/${KVER}-tinycore/kernel/drivers/net/irda \
lib/modules/${KVER}-tinycore/kernel/drivers/net/pcmcia/xirc2ps_cs.ko \
lib/modules/${KVER}-tinycore/kernel/drivers/net/wimax \
lib/modules/${KVER}-tinycore/kernel/drivers/net/wireless \
lib/modules/${KVER}-tinycore/kernel/drivers/pci/hotplug \
lib/modules/${KVER}-tinycore/kernel/drivers/platform/x86/thinkpad_acpi.ko \
lib/modules/${KVER}-tinycore/kernel/drivers/scsi \
lib/modules/${KVER}-tinycore/kernel/drivers/usb/misc \
lib/modules/${KVER}-tinycore/kernel/drivers/usb/serial \
lib/modules/${KVER}-tinycore/kernel/drivers/watchdog \
lib/modules/${KVER}-tinycore/kernel/fs/hfsplus \
lib/modules/${KVER}-tinycore/kernel/fs/jffs2 \
lib/modules/${KVER}-tinycore/kernel/fs/jfs \
lib/modules/${KVER}-tinycore/kernel/fs/minix \
lib/modules/${KVER}-tinycore/kernel/fs/nfsd \
lib/modules/${KVER}-tinycore/kernel/fs/reiserfs \
lib/modules/${KVER}-tinycore/kernel/fs/ubifs \
lib/modules/${KVER}-tinycore/kernel/fs/udf \
lib/modules/${KVER}-tinycore/kernel/fs/xfs \
lib/modules/${KVER}-tinycore/kernel/lib/raid6 \
lib/modules/${KVER}-tinycore/kernel/net/ax25 \
lib/modules/${KVER}-tinycore/kernel/net/bluetooth \
lib/modules/${KVER}-tinycore/kernel/net/bridge \
lib/modules/${KVER}-tinycore/kernel/net/ipv4/netfilter \
lib/modules/${KVER}-tinycore/kernel/net/ipv6 \
lib/modules/${KVER}-tinycore/kernel/net/irda \
lib/modules/${KVER}-tinycore/kernel/net/mac80211 \
lib/modules/${KVER}-tinycore/kernel/net/netfilter \
lib/modules/${KVER}-tinycore/kernel/net/netrom \
lib/modules/${KVER}-tinycore/kernel/net/rose \
lib/modules/${KVER}-tinycore/kernel/net/sched \
lib/modules/${KVER}-tinycore/kernel/net/wimax \
lib/modules/${KVER}-tinycore/kernel/net/wireless \
lib/modules/${KVER}-tinycore/kernel/sound \
| (cd ../modules-new/usr/local && sudo tar -xf -)
sudo ln -s /usr/local/lib/modules/${KVER}-tinycore/kernel lib/modules/${KVER}-tinycore/kernel.tclocal
sudo chroot . depmod -a
sudo mkdir -p "${INST:1}"
sudo chown ${USER}:staff "${INST:1}"
for t in $TCZs; do
sudo mount -o loop,ro /etc/sysconfig/tcedir/optional/${t}.tcz ../tcz
(cd ../tcz; find * ! -type d -print0 | xargs -0 tar cf -) | sudo tar xpf -
sudo touch "${INST:1}/$t"
sudo umount ../tcz
done
sudo sed -i 's/\$(blkid \$1)/$(blkid $1|sed s@TYPE=\\"ntfs\\"@TYPE=\\"ntfs-3g\\"@g)/' usr/sbin/fstype
sudo sed -i 's#\(ntfs\|msdos\|fat\))#\0 grep -qi "lang=[^ ]*utf8" /proc/cmdline \&\& OPTIONS="${OPTIONS},iocharset=utf8"; #' usr/sbin/rebuildfstab
sudo sed -i '
/lst=\*)/ a\
mlst=*) MODULAR_LSTS=${i#*=} ;;
/Process regular extensions/ {
a\
for modlst in $(echo $MODULAR_LSTS | tr , " "); do if [ -f "$TCEDIR/$modlst" ]; then
a\
for FILE in $(cat "$TCEDIR/$modlst"); do FILE="$TCEDIR/optional/$FILE" && process; done
a\
fi; done
}
' usr/bin/tce-setup
sudo chroot . ldconfig
find . | sudo cpio -H newc -o | gzip -9 >../"core${BITS}-new.gz"
cd ..
advdef -z4 "core${BITS}-new.gz"

# build the modules-new extension

cd modules-new
sudo chown -R root:root .
sudo find . -type d -exec chmod 755 {} \;
sudo find . ! -type d -exec chmod 644 {} \;
sudo mkdir "${INST:1}"
sudo sh -c "printf '#!/bin/sh\nsudo depmod -a\n' >${INST:1}/modules-new"
sudo chown -R root:staff "${INST:1}"
sudo chmod -R 775 "${INST:1}"
mksquashfs . ../modules-new.tcz
cd ..
md5sum -b modules-new.tcz >modules-new.tcz.md5.txt

# build the firmware-new extension

cd firmware-new
sudo cp -r ../mod_path/lib/firmware usr/local/lib/
sudo chown -R root:root .
sudo find . -type d -exec chmod 755 {} \;
sudo find . ! -type d -exec chmod 644 {} \;
sudo mkdir "${INST:1}"
sudo sh -c "printf '#!/bin/sh\nsudo depmod -a\n' >${INST:1}/firmware-new"
sudo chown -R root:staff "${INST:1}"
sudo chmod -R 775 "${INST:1}"
mksquashfs . ../firmware-new.tcz
cd ..
md5sum -b firmware-new.tcz >firmware-new.tcz.md5.txtJust so you know, my currently running TC has the misc. “sed” commands above applied, and ntfs-3G inside as well. So I’m rather confident that the problem lies with the handling of kernel modules and/or firmware.
Also, currently in my tests, I don’t load the above 2 extensions that I have created.

Where’s my error? I have a feeling that the “rm -rf lib/modules/${KVER}-tinycore” removes too much, but I don’t know what to write there instead…

[curaga]

Even with a completely corrupted initrd you should get some kernel output. Try adding the "debug" param to your boot line.

[theYinYeti]

Thank you curaga. I’ll let you know when I have the chance to try.

[theYinYeti]

Well, there sure is something strange going on… I tried with “debug”, and with “debug nomodeset”, to no avail. Still a plain black screen without a single non-black pixel on it, the moment I leave Grub2.
I don’t know if it is any help, but I just saw that when I boot with the working core.gz, once on the desktop, the “firmware-radeon” extension is loaded, even though I did not load it myself (onboot).

Is something wrong with the way I handle modules and firmware above?

[tinypoodle]

What do you see (or not) if you boot kernel with out any initrd - just for testing, kernel will panic as soon as it tries to hand control to /init.

Are you using KVM or not?

[theYinYeti]

I tried this:
— vmlinuz-new debug + core.gz: all is fine, graphical desktop included.
— vmlinuz-new debug: TC logo, some boot text, and then kernel panic.
— vmlinuz-new debug + core-new.gz: nothing… all black… frozen.

It seems utterly illogical! My feeling is that I don’t actually leave Grub2 in this third setup. The only logical explanation I see is that the new core-new.gz file somehow sends the boot-loader into an infinite loop, or appears as an “infinite file”. Yet it is just a file, and I tried many times with many different “core-new.gz” files.
Interestingly, if I take “core.gz” and only apply the miscellaneous “sed” commands on it, but I leave out any kernel-related tampering, then it works.

Hence my question: what did I do wrong?
(and because I’m curious: how could “doing wrong” lead to such a result!?)

[tinypoodle]

A possible conclusion is that kernel never even starts to boot in scenario 3.

More to test:

1. boot with new initrd and "nomodeset vga=ask" to see if you get as far as to the vga selection screen.

2. Try with different bootloaders.

[theYinYeti]

Thank you for the ideas, tinypoodle. I’ll try both your suggestions as soon as I can.

Meanwhile, I launched the script and captured the output (both stdout and stderr) in a log file. Here it is:
http://yeti.selfip.net/public/www/TC.u2.log

I notice these errors at lines 12772 and 12773:

Code: [Select]

+ sudo chroot . depmod -a                                                                                       
depmod: ./build: No such file or directory
depmod: ./source: No such file or directoryDoes it matter?

[Edit: BTW, it’s a real PC, not a virtual machine. I boot from a FAT flash drive]

[theYinYeti]

Hi tinypoodle! I did not try your first suggestion yet (couldn’t remember what it was…) but I did manage to use syslinux (your second idea).
On the bright side, I do get a result! Now, the other side of things is that it still does not work: I barely have time to see syslinux loading vmlinuz-new and core-new.gz, and then I see a lot of text scrolling during a split second, and everything stops at this:
http://yeti.selfip.net/public/www/TC-new+syslinux.png

I would say that neither Grub2 nor syslinux is at fault. So something is wrong in TinyCore tools, or with the way I do things? Which is it?

[tinypoodle]

Search web and forum for the "Fix recursive fault..." msg, this has come up before.

[theYinYeti]

Indeed, I found a few threads, that all come down to these:
http://forum.tinycorelinux.net/index.php/topic,12525.msg68636.html#msg68636
http://forum.tinycorelinux.net/index.php/topic,14302.msg80647.html#msg80647

I ended up trying the following:
— “cgroup_disable=memory” ⇒ no change.
— “noapic” ⇒ no change.
— “nolapic” ⇒ no change.
— “noapic nolapic” ⇒ everything is working all right, except I only have 1 CPU…
I was hoping to avoid the “maxcpus=0” option and keep working with 4 CPUs. In the end, only one CPU works.

Following the hint there:
http://forum.tinycorelinux.net/index.php/topic,14378.msg81176.html#msg81176
I also tried creating core-new.gz again, but this time, I forced /usr/bin and /usr/sbin to be first in the $PATH, and I also declared this:

Code: [Select]

CHROOT="busybox chroot"
CPIO="busybox cpio"
DEPMOD="busybox depmod"
GZIP="busybox gzip"
XZ="busybox xz"and forced any usage of chroot, cpio, depmod, gzip, and xz to use busybox, using the above variables.
⇒ No change

Considering the fact that the process used to work, and that TC’s own core.gz works correctly not only in my case but also in others’ cases, my conclusion is that something must differ between the tools/environment as they used to be some time ago (at most at the time when core.gz was created: 2013-May-10), and now. Or, maybe the platform used to generate core.gz is not up-to-date and thus not subject to this bug?

Can someone familiar with the process of creating TinyCore think of something significant that changed in the toolset or anything, in the last month? Maybe more if TC’s building platform is not up-to-date…

[edit: can an admin please move this topic into CORE, instead of TCE?]