Redirecting console messages over the network while booting

[Rich]

This involves using the netconsole driver which is part of the base system.

From the kernel docs:

This module logs kernel printk messages over UDP allowing debugging of
problem where disk logging fails and serial consoles are impractical.

Found here:
https://mjmwired.net/kernel/Documentation/networking/netconsole.rst

Since the point is catching errors, we want to start this as early as possible
in the boot process. Since I couldn't find a way for the kernel to start it while
booting, I went after the first process outside of the kernel, /init.

Here are the first few lines of a modified init:

Code: [Select]

#!/bin/sh
modprobe sky2
modprobe netconsole netconsole=6665@192.168.1.49/eth0,6666@192.168.1.45/78:AC:C0:44:28:7B
mount proc
 ----- Snip -----The modprobe lines are my additions.
The first loads my network driver. Without it, I get a netpoll failure for netconsole.
The second loads netconsole and its settings.

We take the modified init and package it into its own initrd:

Code: [Select]

mkdir tmp
cp init tmp
cd tmp
sudo find . | sudo cpio -o -H newc | gzip > ../netconsole.gz
cd ..
cp netconsole.gz to your boot directory
Modify the bootloader to load the second initrd. I use grub2 and it looks like this:

Code: [Select]

initrd /boot/core.gz /boot/netconsole.gzI think some bootloaders use a comma instead of a space for a separator.

These are the settings for netconsole:

Code: [Select]

netconsole=[+][r][src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr]

   where
+             if present, enable extended console support
r             if present, prepend kernel version (release) to the message
src-port      source for UDP packets (defaults to 6665)
src-ip        source IP to use (interface address)
dev           network interface (eth0)
tgt-port      port for logging agent (6666)
tgt-ip        IP address for logging agent
tgt-macaddr   ethernet MAC address for logging agent (broadcast)I did not use the + or r options. I recommend using all of the others.
For src-ip you can use the same IP address as you normally set your NIC
to. But you need to set it so you can tell the listener who to listen to.

To listen, run this on the other computer:

Code: [Select]

tc@box:~$ nc -v -l -u -p 6666 192.168.1.49 > dmesg.txt
listening on 0.0.0.0:6666 ...
connect to 192.168.1.45:6666 from 192.168.1.49:6665 (192.168.1.49:6665)
^Cpunt!The  connect  message comes up when it starts receiving data.
Here's the best part. When it connects, it prints the entire dmesg
buffer, from time zero, to the present, and beyond.
When it's done, hit Ctrl-C to stop receiving and close the file.

If you want to see the data scrolling while it's being saved:

Code: [Select]

nc -v -l -u -p 6666 192.168.1.49 | tee dmesg.txt
Start your listener, then boot the machine under test.
From what I've read, this requires a wired connection, and will not
work with wifi.

Attached is the dmesg.txt I received from a TC16 x86 machine
booting to the command line. You can see the NIC driver and
netconsole starting up at time stamp 1.775229.

[Rich]

... Modify the bootloader to load the second initrd. I use grub2 and it looks like this:

Code: [Select]

initrd /boot/core.gz /boot/netconsole.gzI think some bootloaders use a comma instead of a space for a separator. ...

I forgot to mention, while modifying the bootloader, change  quiet  to  debug  on the
kernel parameters line.