WelcomeWelcome | FAQFAQ | DownloadsDownloads | WikiWiki

Author Topic: Issues with reboot not rebooting  (Read 4022 times)

Offline ToasterKing

  • Newbie
  • *
  • Posts: 14
Issues with reboot not rebooting
« on: July 31, 2012, 07:32:07 PM »
I'm seeing a very odd problem on an older installation (3.6) that until now has been working perfectly.

A process on the box stopped responding and could not be killed, so we tried to use the reboot command to reboot the box.  The reboot command just hangs and cannot be interrupted.

If I log in via a separate ssh session, I can use ps and see that the reboot command is sitting there in the process table (multiple times now), but it cannot be killed either.

I saw something sort of like this once before.  That time I was running a disk load test on the box and reboot typed while the load test was running would not return.  Once the load test was killed, reboot would complete.  Not sure this is related though, nothing that was running should have been accessing the file system other than having a log file open (but not growing) in /tmp.

I am under the impression that the busybox reboot command really just tells init to do a restart.  According to the inittab file that should run rc.shutdown (which checks with what usually happens).  This time around, the system never seemed to get to the running rc.shutdown point.

Any thoughts? This particular box is located an hour away from anyone, and although someone was able to drive out and restart it, I'm catching some grief that anyone had to.

Offline Rich

  • Administrator
  • Hero Member
  • *****
  • Posts: 11581
Re: Issues with reboot not rebooting
« Reply #1 on: July 31, 2012, 08:31:40 PM »
Hi ToasterKing
While I can't vouch for this technique, a little Googling turned up:
http://www.akamit.com/blog/2012/03/rebooting-linux-remotely-when-reboot-command-is-not-working/
I suspect you need to execute these commands as root.

Offline tinypoodle

  • Hero Member
  • *****
  • Posts: 3857
Re: Issues with reboot not rebooting
« Reply #2 on: August 01, 2012, 06:17:09 AM »
And in extension to above:

http://julien.danjou.info/projects/sysrqd

http://www.vanheusden.com/tcpconsole/

To diagnose PIDs not easy to kill you can use "fuser" (in base) and "lsof".

"fuser -k" can be used to kill processes (use with caution!!).
"Software gets slower faster than hardware gets faster." Niklaus Wirth - A Plea for Lean Software (1995)

Offline ToasterKing

  • Newbie
  • *
  • Posts: 14
Re: Issues with reboot not rebooting
« Reply #3 on: August 01, 2012, 10:50:37 AM »
You know, I like to think I'm pretty knowledgeable about Linux, but the /proc/sysreq-trigger trick (and the associated magic sysrq key commands (http://en.wikipedia.org/wiki/Magic_SysRq_key) are something I didn't even know existed.

tcpconsole is a new one too, and I like the idea but I'm going to have to think about how I could actually set it up since it uses telnet and not ssh (for very good reasons).

You can bet I've added these to my toolbox now.  I just need to wait for a system to lock up again...



Tinypoodle,

Are you thinking that there was a file being held open preventing the file systems from being unmounted?  I used lsof to look for that, and didn't see anything obvious other than the extensions  themselves and some log files in /tmp.

Offline tinypoodle

  • Hero Member
  • *****
  • Posts: 3857
Re: Issues with reboot not rebooting
« Reply #4 on: August 01, 2012, 11:12:02 AM »
I'd rather use "fuser" on PIDs in question first.
"Software gets slower faster than hardware gets faster." Niklaus Wirth - A Plea for Lean Software (1995)