Linux and usb drive stability

[remus]

Hi all,

I've had some bad luck with linux and external hard drives. A while back I was recovering a computer with a PING IS NOT GHOST partition image, and got some errors at the end of the job. I was using a PING boot cd and the partition image was stored on a 500GB ext USB Hard Drive.

The image to partition dump failed at 96% with an UNKNOWN ERROR.

So I rebooted and tried again, to discover that all data on the ext hard drive was now un accessible.

Turns out that the Partition Table was corrupted, so tried every trick I could find to repair the drives partitions and recovery data, but nothing worked.

After 3 days of googling and tech forums I gave up, formated the drive and it was ready to go again, I lost a huge amount of data.

Does tinycore linux have any apps that monitor for ext usb drive connection stability and gives a warning or does something to protect the data if any error is detected during a read/write ?

[bmarkus]

Does tinycore linux have any apps that monitor for ext usb drive connection stability and gives a warning or does something to protect the data if any error is detected during a read/write ?

Monitoring: Instead of monitoring fix root cause or drop external USB drive. Reporting a dead hard drive is nearly useless.

Protection: Use a realible system setup. And of cource a good journaling file system like ext4 may help.

[Juanito]

Does tinycore linux have any apps that monitor for ext usb drive connection stability and gives a warning or does something to protect the data if any error is detected during a read/write ?

You don't mention if you're using linux or windows file systems on the external drives?

I've been using linux file systems on many usb sticks for over 5 years - and been abusing some of them by installing in "scatter mode" for a limited period of time - and never lost any data.

If you periodically monitor dmesg or similar to check for messages relating to the usb drives/sticks and use fsck when recommended, I doubt you'll experience many problems.

[bmarkus]

Well, he mentioned external USB drive and not USB stick. I have never head any problem with stick, I have few 4-5 years old used with journaling and still tehy are OK.

A hard disk may be different, specially if engine is stopped after a period. It requires high current when spinning up which may be an issue when powered from USB or other hardware related problems may occure. But no details on the setup.

[curaga]

If the hw is getting flaky, you will get warnings in dmesg. Same when any errors are discovered on the file system, or when it is recommended to run a fsck just in case.

But this sounds like the usb disk screwed up itself after disconnecting itself. Nothing the computer could do there.

[remus]

Thanks for all the great input all

The drive file system is fat32.
Its attached to a computer running windows xp, and is used as a backup share for 5 computers also running windows xp. Its never given us any trouble in this setup.

So I can test the drive with a tinycore boot disc, and check the dmesg for problems.

[curaga]

Wait, fat32 as a destination for disk images? Does ping split them to 4gb chunks?

[genec]

I'd lean towards issues outside of Linux.  Your USB controller, the USB-HDD controller or the HDD itself is likely heating up and failing.  I'd lean towards the USB-HDD controller is faulty unless there's a chance it's been hit/dropped/bumped while in operation.  As a result, I'm guessing you're getting corruption.  smartctl from smartmontools may help to see if the HDD has recorded errors but you'll likely need to remove the USB-HDD controller from the equation as it most likely won't pass the SMART commands.

Since you're starting from scratch, I'd recommend starting with running the long test and the offline test against the drive to attempt to detect errors before you reuse the drive at all.  I'd also consider running badblocks on the entire drive in write mode (which destroys everything).

[remus]

PING spans images at 630mb limit by default.

I have not found any software that does a good job of checking a ext usb hd for errors. So yeah, I assumed the usb controllers were filtering out the hd diagnostic tools I tried.

We had this problem with the drive about 6 months ago actually, and since formatting and keeping it attached to a windows xp machine, its given us ZERO problems since. I have considered taking the drive apart to access the internal hard drive unit, but if its a sata drive, I do not have any connectors or adapters that could let me connect it to a ata laptop that I own.

I'm a volunteer for a non profit Community Center, and we have a very limited budget. Pretty much every computer in the building was donated to us, or rescued from the dump

I would like to setup a tinycore/samba/ssh linux file server with a remastered tinycore boot cd with all samba and required conf integrated. Which I've learnt how to do thx to this forum
I'd like to attach an ext usb hd to make it easy to swap it out incase of a failure, but I'm starting to think its not worth the trouble. Its probably better to use an internal drive that will work with proper hard drive diagnostic software.

Thx again to the most responsive and helpful linux forum i've ever joined

[Rich]

Hi remus
I'd like to make a few suggestions. Go with an internal drive (or 2 or more), your data transfer rates
will be far superior over a USB external drive. Plan on some persistent storage for the operating
system. I recommend a USB thumb drive to keep system files separate from the internal data drives.
You'll want to backup system settings like /usr/local/etc/samba/smb.conf to it. You might as well also
put extensions on it to allow for easy upgrades. If the machine supports booting from a thumb drive,
you could remove the CD drive thus freeing up an IDE channel.

[genec]

1) It's a matter of finding a USB-HDD controller that passes these commands through.  Most, if not all filter it.

2) This sounds like whatever failed before is doing it again.

3) USB tops out around 25-35MB/s and I've found IEEE 1394 FireWire more reliable and faster (even though the raw bit rate is slower) with lower CPU overhead.  However, USB is more ubiquitous.  And now, eSATA is the gold standard for performance in the consumer market.

[Rich]

Hi genec

3) USB tops out around 25-35MB/s and I've found IEEE 1394 FireWire more reliable and faster (even though the raw bit rate is slower) with lower CPU overhead.  However, USB is more ubiquitous.  And now, eSATA is the gold standard for performance in the consumer market.

Assuming you are referring to USB2, I think you meant 25-35Mb/s (bits not bytes) since it is a serial
interface.
Since remus is working with donated and salvaged equipment, he may not have the luxury of picking
and choosing among "gold standard" technologies.

[remus]

Hi Rich
My main objective in putting everything onto a boot cd is based on the idea that if there is any glitch in the system or someone logs in as root and plays with the server and corrupts the system, all problems can be fixed by rebooting the computer, and it will load the same every time.
I understand that this means that I won't be able to get updates.

I've considered setting up a raid, but spare hard drives are not available.

[Rich]

Hi remus

I've considered setting up a raid, but spare hard drives are not available.

I guess that was in response to this:

Go with an internal drive (or 2 or more)

Actually what I was suggesting was not RAID, but that you could install multiple drives for sharing if
you have them available. For instance, using several smaller drives if you have a larger drive that
would be better used elsewhere. You can create a common share point and mount multiple drives
to it.
You might still want to start with persistent storage for  /usr/local/etc/samba/smb.conf  until you get
it properly configured and then transfer it to CD.

[genec]

Assuming you are referring to USB2, I think you meant 25-35Mb/s (bits not bytes) since it is a serial interface.

Nope.  Raw data rate of USB 2.0 High Speed is 480Mbps but 25-35 MB/s is the data rate you'd probably see with a mass storage device on USB 2.0 HS.