Bend in the Weather

In a not to distant past Russell Coker wrote about RAID Issues and referred in part to a report containing data from 1,530,000 disks running at NetApp customer sites. (also available in PDF or Postscript)

Interesting reading, for sure - particularly if you run any large dataset and want to ensure it stays intact!

It prompted in the recesses of my memory a report compiled in February 2007 by Google. The report, 'Failure Trends in a Large Disk Drive Population' was a report presented at the USENIX (FAST '07) Conference.

The Google report looked at actual hardware failures of disk Google saw over several years. The numbers were crunched (Is there anything else Google does besides crunch large datasets?). Some interesting results popped out:

• The disks studied where either SATA or PATA consumer-grade disks that were either 5400RPM or 7200RPM ranging in size from 80G to 400G and been commissioned from any time after 2001. - Interesting this is the same disks that many of us will find in our own machines. No Enterprise disks, SCSI or SAS disks in the study.
• HDDs had a higher tendancy to fail at the start of their life or anything beyond >3 years of use.
• Low or Heavy Utilization of the HDD resulted in greater loss then 'Medium use'.
• Disks that had surface scan errors had a greater result of failure over the next 60 days.
• HDDs in operating in cool temperature (15-30^oC) had much greater failure rate in the first 3 years. Whilst disks > 3 years had a greater failure rate with the higher the operating temperature. The ideal rate for running disks to minimise failure rate was 30-35^oC.
• A disk that spends more than 50% of it's powered on time > 40^oC is a good indication of a possible problem.
• SMART data analysis revealed that it is not a reliable way to determine if a disk is about to fail. 36% of all disk failure had no SMART errors. The disks that had SMART errors the majority where seek errors (~72%). So basically, expect to see seek errors, beyond that you appear to be running blind with SMART.

Looking at both the Google Report and the NetApp Storage Report some 'best practices' become apparent to ensure you minimise your data loss:

• HDDs are mechanical devices. Expect failure and plan for it.
• Attempt to operate disks in the 30 - 35^oC temperature range to extend their life.
• Monitor disk temperatures. Extended periods where a disk temperature rises unexpectedly (ie: not under any additional load than normal), is often an early sign that failure is close at hand.
• Attempt to purchase disks not from the same batch. This will avoid a common manufacturing fault taking your disks out at the same time. (Many storage companies will do that for you as part of their service.)
• HDD failure follows the Bathtub curve. The 'right side' of the bath kicks in around 3 years. Getting life out of your HDDs beyond that is a bonus, treat it as such!
• HDDs are cheap these days. Don't be cheap -- implement RAID-6 over RAID-5 as a matter of course. Ensure you use Double-parity on your RAID-6 implementation. You'll find most recent versions of RAID-6 implement double-parity as 'standard'. (NB: If using NetApp - it's coined as RAID-DP.) Some vendors even allow you to upgrade the storage firmware online if using RAID-6 with Double Parity (NetApp for example has this feature).
• Hot-Spares in your RAID-6 array is a very good idea. For the cost of the array consider it an insurance policy against the dreaded multiple disk failure which could potentially toast your array.
• How important is your data? Can you put a cost on it? If the cost of replacement is extreme, consider redundancy options. This could include: Implementing RAID-60 (or RAID-6+0), archive/backup solutions, or even a total Disaster Recovery (DR) solution.
• Air flow around an array unit is critical. Don't cram your arrays in a fully populated rack, as minimal air-flow will ensure. This will add to the HDD temperatures and general storage enclosure. Remember you're aiming to keep your disks at 30-35^oC.
• Keep a logbook of when each drive was added/replaced. You know that 18 month disk then is less likely to fail than that disk that has been whirring away for 5 years. Record size, manufacturer and model/run. You may see some 'patterns' emerge in your own failure rates that will help with additional purchase decisions (ie: particular makes/models to avoid!)
• Perform regular 'scans' or 'checks' of your HDD health, knowing the current state of disks allows you to plan for the inevitable failure.

Other tips I've picked up over the years looking after Enterprise systems attached to large storage arrays:

• There is a reason that the ASX demands that publicly listed companies with 'mission critical' services for the public (items such as water supply, electricity/gas, and telecommunications) MUST ensure their 'mission criticial' applications have full Disaster Recovery operation on hot-standy.
• Don't cut corners and not implement items like HotSpares and Double Parity. When you have a drive failure (not if), you'll be glade you spent that little bit extra on it.
• A well planned storage solution should have drive failures, it shouldn't have storage failures. (Don't tell Sun/Internode that!)
• Ensure you have clean-filtered power. Don't assume it. Power voltage fluctuations and disk writes don't play nicely.
• Don't put all your eggs in one basket. Bad things do happen (I won't mention Internode again). Spreading your data out over multiple storage arrays helps add another level of redundancy. This is a GoodThing^TM
• It's never a bad idea to have spare disks in storage waiting for that inevitable failure, rather than relying on a vendor to have your disk capacity/model available. Every day that dead disk isn't replaced you are putting your array at risk.
• Look after your disks and arrays, and they generally will look after you (ie: ensuring you don't spend countless hours in the early mornings attempting to recover the unrecoverable!)

Above all, may your next disk failure not be a complete failure.

I'm interested to hear others stories, experiences and ideas that they have put in place to keep their data nice and cosy and their HDDs whirring in a constant and reliable state. Feel free to drop me a comment.

As a side note: I've been putting RAID-1 in place on my desktop machines as disk prices are so cheap now over the past 18 months. For the price of a few hundred dollars why bother the stress of a hard-disk failure? Having said that, it's no replacement for backing up your 'important' information. RAID-1 is still susceptible to the dreaded multiple disk failure issue. Having said that, I've been lucky at that hasn't occurred. (Touch wood!)

I've owned a Maxtor Shared Storage Plus NAS (MSS+) device for quite a while. It was reasonably cheap when we picked it up and it's well and truly paid for itself saving me on several occasions!

It's basically a 500GB disk inside an enclosure that you network (10/100Mbps), and contains 2 additional USB ports so you either plug additional USB storage or a USB printer into (to make it a networked printer).

In our instance we use it mainly for backing up. For this purpose it serves well.

We ran into a problem today when the device filled.
That was unusual, as we traditionally have loads of space left on it (we only backup our content/configurations), not the OS or applications... and we purge old backups.

A closer inspection showed that the backup software we were running on the Windows machine was actually not removing old archive files. That explained the lack of disk space.

It seemed under certain conditions that the Windows machine was unable to delete the files it created. This now appears more an issue with Windows then the actual backup software. (I could delete the mentioned files using Nautilus fine under Linux, just not through Explorer!)

Whilst the device itself has worked flawlessly (it basically creates SMB based shares), it did become a problem when large amounts of files needed to be deleted (it was dreadfully slow over SMB - as it appeared to require a full stat of the files to be deleted, prior to even commencing the delete ). This could take hours before one file was ever deleted.

It always was frustrating that besides the web-interface to manage the device, that SMB appeared to be the only option available for file management.

Many people have used SLUGs (aka NSLU2 - LinkSys's NAS device). This allowed people to use modified firmware to create a nice embedded Linux device, allowing them to install additional software and expand on the base functionality.

How nice would this be with my MSS+

I thought I would take a peep at the support website, as I realized that this device too is likely to use OSS software such as Samba to create it. Low and behold, the source code was available.

This got my creative juices running! Could I use this to create a modded version myself and add items like NFS, FTP and SSH to the device? Surely someone else has already thought the same thing.

Apparently they have! OpenMSS is just that. Not only does it put down a telnet server by default (so you can login, they have got BusyBox working on it as well.

From my reading it appears NFS is there, just was never enabled on the original firmware image. It also allows you to install additional software, as well as a bunch of software from the ports made available through the work by the guys over as nslu2-linux.org.

It looks like with little work I'll be able to have an FTP, NFS and rsync all as options, as well as SSH to login to the box.
The list of packages available is breath-taking, and I can see a lot of work has gone into the porting.

What's even better, the web-interface and existing data stay intact when migrating over to OpenMSS -- just the ability to get into the heart of the box is opened up.

So I think it's now a case of having a play around and actually getting some 'smarts' into this device. I'm doing away with the Windows backup software and we'll look at running a central backup from the device itself to pull in backups from other hosts using rsync. (much nicer). I see ccollect is one of the packages available, so this may indeed do the trick... just through on vixie cron at the same time and I've got a nice simple and effective solution.

I felt a little cheated initially when I heard of the SLUG (after my MSS+ purchase) now I'm feeling quite happy that I can play in the same league!

It's a job for later in the week/weekend (when time permits), but it appears I may have just solved some of the issues that has been burning me for ages (access to the device and trouble free backups from Windows to it). I'll keep you posted on the outcome.

Don't ya just love OSS -- putting the power back in the users hands!