Bend in the Weather (Entries tagged as google)

RAIDing the RAID

mbottrell@gmail.com (Matt) — Sat, 18 Oct 2008 03:44:00 +1100

In a not to distant past Russell Coker wrote about RAID Issues and referred in part to a report containing data from 1,530,000 disks running at NetApp customer sites. (also available in PDF or Postscript)

Interesting reading, for sure - particularly if you run any large dataset and want to ensure it stays intact!

It prompted in the recesses of my memory a report compiled in February 2007 by Google. The report, 'Failure Trends in a Large Disk Drive Population' was a report presented at the USENIX (FAST '07) Conference.

The Google report looked at actual hardware failures of disk Google saw over several years. The numbers were crunched (Is there anything else Google does besides crunch large datasets?). Some interesting results popped out:

The disks studied where either SATA or PATA consumer-grade disks that were either 5400RPM or 7200RPM ranging in size from 80G to 400G and been commissioned from any time after 2001. - Interesting this is the same disks that many of us will find in our own machines. No Enterprise disks, SCSI or SAS disks in the study.
HDDs had a higher tendancy to fail at the start of their life or anything beyond >3 years of use.
Low or Heavy Utilization of the HDD resulted in greater loss then 'Medium use'.
Disks that had surface scan errors had a greater result of failure over the next 60 days.
HDDs in operating in cool temperature (15-30^oC) had much greater failure rate in the first 3 years. Whilst disks > 3 years had a greater failure rate with the higher the operating temperature. The ideal rate for running disks to minimise failure rate was 30-35^oC.
A disk that spends more than 50% of it's powered on time > 40^oC is a good indication of a possible problem.
SMART data analysis revealed that it is not a reliable way to determine if a disk is about to fail. 36% of all disk failure had no SMART errors. The disks that had SMART errors the majority where seek errors (~72%). So basically, expect to see seek errors, beyond that you appear to be running blind with SMART.

Looking at both the Google Report and the NetApp Storage Report some 'best practices' become apparent to ensure you minimise your data loss:

HDDs are mechanical devices. Expect failure and plan for it.
Attempt to operate disks in the 30 - 35^oC temperature range to extend their life.
Monitor disk temperatures. Extended periods where a disk temperature rises unexpectedly (ie: not under any additional load than normal), is often an early sign that failure is close at hand.
Attempt to purchase disks not from the same batch. This will avoid a common manufacturing fault taking your disks out at the same time. (Many storage companies will do that for you as part of their service.)
HDD failure follows the Bathtub curve. The 'right side' of the bath kicks in around 3 years. Getting life out of your HDDs beyond that is a bonus, treat it as such!
HDDs are cheap these days. Don't be cheap -- implement RAID-6 over RAID-5 as a matter of course. Ensure you use Double-parity on your RAID-6 implementation. You'll find most recent versions of RAID-6 implement double-parity as 'standard'. (NB: If using NetApp - it's coined as RAID-DP.) Some vendors even allow you to upgrade the storage firmware online if using RAID-6 with Double Parity (NetApp for example has this feature).
Hot-Spares in your RAID-6 array is a very good idea. For the cost of the array consider it an insurance policy against the dreaded multiple disk failure which could potentially toast your array.
How important is your data? Can you put a cost on it? If the cost of replacement is extreme, consider redundancy options. This could include: Implementing RAID-60 (or RAID-6+0), archive/backup solutions, or even a total Disaster Recovery (DR) solution.
Air flow around an array unit is critical. Don't cram your arrays in a fully populated rack, as minimal air-flow will ensure. This will add to the HDD temperatures and general storage enclosure. Remember you're aiming to keep your disks at 30-35^oC.
Keep a logbook of when each drive was added/replaced. You know that 18 month disk then is less likely to fail than that disk that has been whirring away for 5 years. Record size, manufacturer and model/run. You may see some 'patterns' emerge in your own failure rates that will help with additional purchase decisions (ie: particular makes/models to avoid!)
Perform regular 'scans' or 'checks' of your HDD health, knowing the current state of disks allows you to plan for the inevitable failure.

Other tips I've picked up over the years looking after Enterprise systems attached to large storage arrays:

There is a reason that the ASX demands that publicly listed companies with 'mission critical' services for the public (items such as water supply, electricity/gas, and telecommunications) MUST ensure their 'mission criticial' applications have full Disaster Recovery operation on hot-standy.
Don't cut corners and not implement items like HotSpares and Double Parity. When you have a drive failure (not if), you'll be glade you spent that little bit extra on it.
A well planned storage solution should have drive failures, it shouldn't have storage failures. (Don't tell Sun/Internode that!)
Ensure you have clean-filtered power. Don't assume it. Power voltage fluctuations and disk writes don't play nicely.
Don't put all your eggs in one basket. Bad things do happen (I won't mention Internode again). Spreading your data out over multiple storage arrays helps add another level of redundancy. This is a GoodThing^TM
It's never a bad idea to have spare disks in storage waiting for that inevitable failure, rather than relying on a vendor to have your disk capacity/model available. Every day that dead disk isn't replaced you are putting your array at risk.
Look after your disks and arrays, and they generally will look after you (ie: ensuring you don't spend countless hours in the early mornings attempting to recover the unrecoverable!)

Above all, may your next disk failure not be a complete failure.

I'm interested to hear others stories, experiences and ideas that they have put in place to keep their data nice and cosy and their HDDs whirring in a constant and reliable state. Feel free to drop me a comment.

As a side note: I've been putting RAID-1 in place on my desktop machines as disk prices are so cheap now over the past 18 months. For the price of a few hundred dollars why bother the stress of a hard-disk failure? Having said that, it's no replacement for backing up your 'important' information. RAID-1 is still susceptible to the dreaded multiple disk failure issue. Having said that, I've been lucky at that hasn't occurred. (Touch wood!)

Making Google Chrome more private

mbottrell@gmail.com (Matt) — Sun, 28 Sep 2008 02:43:00 +1000

Most people are aware the Google's Chrome Browser is based on the great work of the Chromium Browser Project.

CodeWeavers have even created a free 'port' of Google Chrome for Linux users, aptly named CrossOver Chromium. This means Linux people don't need to feel left out of all the Google lovin'.

There have been a few people concerned with privacy issues (also here and here) when using the Google Chrome browser. Even the EFF has raised concerns. Google has reacted and posted some information regarding privacy in Chrome.

There has been some applications produced that remove one of the largest concerns people have (a unique ID assigned to every installation of Chrome). In particular Chrome Privacy Guard will strip this unique ID out for you. Turning off some of the features (like Google Suggest) via the Browser preferences will also limit the amount of information sent back to the mothership (Google).

Buoyed by all these privacy concerns, SRWare a German software company have released 'Iron', a browser also based on Chromium source but without all the 'Googleness' privacy issues.

If you like Chrome, but are concerned about the privacy aspect, Iron might be your solution.

For those on Linux and really just wanting to use a WebKit based browser, take a peek at Midori.

Visit Disney World - in the luxury of your own home

mbottrell@gmail.com (Matt) — Mon, 09 Jun 2008 02:28:00 +1000

We all enjoy a good holiday. Getting the time to take one always seems to be the problem for me however. I think I may have just found the answer...

If you're like me you grew up watching The Wonderful World of Disney at 6:30pm every Sunday. Here we marveled at cartoons of Mickey, Goofy and other great Disney characters. (So much so, many the hostnames on my home network are now named after Disney characters! ).

It also introduced us to classic 'childrens' movies like Herbie, 101 Dalmatians, Mary Poppins and Alice in Wonderland.

It's probably the dream of every child to visit the magical place of Disneyland or DisneyWorld.

With the help of Google you can now explore DisneyLand without leaving your chair!
Google has produced a 3D interactive representation of Disneyland inside Google Earth.

If you've got Google Earth installed try the following:

Once Google Earth loads, search for "Disney World".
Click on the golden Disney Mickey Ears - This will load an information window.
In the info window click on Show Disney World in 3D.

The 3D world contains over 1,500 3D models and will let you explore right through the park. The models are highly detailed and a sight to behold!

It's nearly as good as being there!
Just think - no more queues for rides, no long-haul flight getting there, and the park is now open 24 hours for you.

Have fun exploring!

Personalize your Google Experience

mbottrell@gmail.com (Matt) — Tue, 20 Dec 2005 12:16:01 +1100

I use a personalized Google homepage on my work computer.

It provides me access to a help of information very quickly with just a glance. From my gmail inbox, latest news, weather and date/time. Above that, I can also quickly search google.

What's pretty funky is that you can search for a particular topic and groups for it. You can also rearrange the page so your most important information is up on the top.

If you haven't tried it out... it's well worth looking at. Just go to this google link.

Hype and Fizzle

mbottrell@gmail.com (Matt) — Wed, 05 Oct 2005 12:58:58 +1000

Well late yesterday the news and forums were abuzz with news that Google and Sun were about to make a big announcement.

Many thought it would be "Google Office", an office suite based on OpenOffice and using AJAX similar to Gmail.

This would of been hailed by many as a breakthrough and help break the shackles of MS Office.

Sadly... it was pure speculation, and nothing really eventuated. Instead... we now hear that Sun will be bundling the Google Toolbar with Java.. and the Google bar will help promote OpenOffice... Not really 'ground breaking'.

The market hasn't taken kindly to the news... particular after Sun stock rose on the idea of an alliance for a "Google Office" and Microsoft's stock price fell 2%.
I expect the markets will now right themselves and may even swing against Sun... as it appears Sun has done much of the beat up pre-announcement.

What do you think? Could a web-based Office really knock off Microsoft Office -- the biggest cash-cow Microsoft has?