Issues with reboot not rebooting

[ToasterKing]

I'm seeing a very odd problem on an older installation (3.6) that until now has been working perfectly.

A process on the box stopped responding and could not be killed, so we tried to use the reboot command to reboot the box.  The reboot command just hangs and cannot be interrupted.

If I log in via a separate ssh session, I can use ps and see that the reboot command is sitting there in the process table (multiple times now), but it cannot be killed either.

I saw something sort of like this once before.  That time I was running a disk load test on the box and reboot typed while the load test was running would not return.  Once the load test was killed, reboot would complete.  Not sure this is related though, nothing that was running should have been accessing the file system other than having a log file open (but not growing) in /tmp.

I am under the impression that the busybox reboot command really just tells init to do a restart.  According to the inittab file that should run rc.shutdown (which checks with what usually happens).  This time around, the system never seemed to get to the running rc.shutdown point.

Any thoughts? This particular box is located an hour away from anyone, and although someone was able to drive out and restart it, I'm catching some grief that anyone had to.

[Rich]

Hi ToasterKing
While I can't vouch for this technique, a little Googling turned up:
http://www.akamit.com/blog/2012/03/rebooting-linux-remotely-when-reboot-command-is-not-working/
I suspect you need to execute these commands as root.

[tinypoodle]

And in extension to above:

http://julien.danjou.info/projects/sysrqd

http://www.vanheusden.com/tcpconsole/

To diagnose PIDs not easy to kill you can use "fuser" (in base) and "lsof".

"fuser -k" can be used to kill processes (use with caution!!).

[ToasterKing]

You know, I like to think I'm pretty knowledgeable about Linux, but the /proc/sysreq-trigger trick (and the associated magic sysrq key commands (http://en.wikipedia.org/wiki/Magic_SysRq_key) are something I didn't even know existed.

tcpconsole is a new one too, and I like the idea but I'm going to have to think about how I could actually set it up since it uses telnet and not ssh (for very good reasons).

You can bet I've added these to my toolbox now.  I just need to wait for a system to lock up again...



Tinypoodle,

Are you thinking that there was a file being held open preventing the file systems from being unmounted?  I used lsof to look for that, and didn't see anything obvious other than the extensions  themselves and some log files in /tmp.

[tinypoodle]

I'd rather use "fuser" on PIDs in question first.