My server is freezing up

[remus]

Hi guys.

My backup server running microcore 3.8.4 has started freezing up when staff are performing a backup to it. The server has been working fine for years without a problem.

When I get emails from staff about failed backup, and get to the server the the screen is dark, pressing keys on keyboard does not seem to wake up the system. I have to kill power, and reboot to get the system up and running.

The system boots from a microcore 3.8.4 CD
tce file is located on a 500gb ext usb hd, this same usb hd contains a few folders which are shared.
also attached is an 500gb pata hd which contains a few folders which are shared.

I'm wondering if its possible to get he messages file to be written to disk in some simple way, so I can review it for helpful hints as to the problem ?

Any comments and suggestions welcome.

Thanks

[Pengo]

Judging from the symptoms this looks very much like a hardware issue.
I would check  the servers'
- powersupply,
- RAM
- USB interfaces and connected devices

Regards,
Pengo

[curaga]

Yeah, two ways, but it might not be a sw issue, and even if it is, sudden failure like that might not make it to the logs.

1) Log over the network (rsyslogd for example, not sure if the bb one can do it)
2) Mount a partition on the ide disk with "-o sync", and configure syslogd to log there

Sync slows down writes a lot, but in case of a power or system failure it means more data will reach the disk. Normally writes are cached for several seconds, and any data there would be lost.

[remus]

Sorry for the late reply.

I never got notified about further post's on this topic

Curaga, your suggestion to change where syslogd logs to sounds like what I need.

Any chance you could give me some guidance on how to achieve that ?

P.S.
After eliminating hardware items one at a time, I decided to detach my ext usb 500gb hd (I had my tce folder and a few other shares on it, just had to move the tce folder to my internal drive, and disable shares that were on the ext hd) rebooted the system, and it hasn't crashed since, and is accepting backups via rdiff-backup just fine.

I really need a way to get my logs saved to an accessible media incase of system freeze in the future, perhaps there was a hint in the log file that would of saved me a lot of time.

Thanks

[remus]

I may be onto something.

searching these forums found this.

http://forum.tinycorelinux.net/index.php/topic,18062.msg109419.html#msg109419

$ syslogd -O /mnt/sda1/messages
If I add that to my /opt/bootlocal.sh file I might be onto something.

I'm wondering if I could get this to happen from bootup in some way, that would be cool. I will have to try and remember to test that. maybe I could copy the existing messages file to /mnt/sda1 then run the command syslogd -O /mnt/sda1/messages and it will append new data, instead of wiping the file.

[Rich]

Hi remus
Maybe something like this:

Code: [Select]

FILENAME=messages_`date "+%m%d%H%M"`
syslogd -O /mnt/sda1/$FILENAME
The filename will have the format  messages_12282357. The date function appends month, day, hour, minutes to the messages
part of the filename. This way you can tell when you rebooted by the filename and the old file won't get overwritten.

[remus]

So I get back from Christmas break. And the server has frozen again.

Unresponsive to keyboard, or via putty.

I should note that the server is a microcore 3.8.4 boot cd and the tce folder is on an internal 500GB PATA HD. I've moved the hd and boot cd to another computer, so new motherboard, and all the normal bits now. I've done 2 extended WD Diag scans on the hd and found no error's. I an't think of anything else.... could the server freeze from a bad network cable ?

The server is connected to monitor and keyboard via KVM switch. While the server is frozzen, the KVM switch does not allow me to access the other sever connected to the KVM switch. Once the frozen server is unplugged from the KVM switch the good server is accessible from the KVM switch.

Has anyone ever heard of a KVM switch being the culprit ?

[remus]

Have added

Code: [Select]

syslogd -O /mnt/sda1/logs.messages/messages to my /opt/bootlocal.sh file.

So I've decided its probably the hard drive.

I get a new one setup and start copying data across to the replacement.

I start seeing HD errors occurring on the screen while I'm sitting in front of the machine to see them, however these errors are not being replicated into the log file saved to hd.
Is the most likely reason for this that the error's are printed to screen, and fail to save to file on hd as the hd has failed to operate at that critical moment ?

I've attached some image's of the hd errors.

System refuses to boot now, I bet the drive is dead.

[curaga]

Yeah, dead disk.