Bend in the Weather (Entries tagged as ups)

RAIDing the RAID

mbottrell@gmail.com (Matt) — Sat, 18 Oct 2008 03:44:00 +1100

In a not to distant past Russell Coker wrote about RAID Issues and referred in part to a report containing data from 1,530,000 disks running at NetApp customer sites. (also available in PDF or Postscript)

Interesting reading, for sure - particularly if you run any large dataset and want to ensure it stays intact!

It prompted in the recesses of my memory a report compiled in February 2007 by Google. The report, 'Failure Trends in a Large Disk Drive Population' was a report presented at the USENIX (FAST '07) Conference.

The Google report looked at actual hardware failures of disk Google saw over several years. The numbers were crunched (Is there anything else Google does besides crunch large datasets?). Some interesting results popped out:

The disks studied where either SATA or PATA consumer-grade disks that were either 5400RPM or 7200RPM ranging in size from 80G to 400G and been commissioned from any time after 2001. - Interesting this is the same disks that many of us will find in our own machines. No Enterprise disks, SCSI or SAS disks in the study.
HDDs had a higher tendancy to fail at the start of their life or anything beyond >3 years of use.
Low or Heavy Utilization of the HDD resulted in greater loss then 'Medium use'.
Disks that had surface scan errors had a greater result of failure over the next 60 days.
HDDs in operating in cool temperature (15-30^oC) had much greater failure rate in the first 3 years. Whilst disks > 3 years had a greater failure rate with the higher the operating temperature. The ideal rate for running disks to minimise failure rate was 30-35^oC.
A disk that spends more than 50% of it's powered on time > 40^oC is a good indication of a possible problem.
SMART data analysis revealed that it is not a reliable way to determine if a disk is about to fail. 36% of all disk failure had no SMART errors. The disks that had SMART errors the majority where seek errors (~72%). So basically, expect to see seek errors, beyond that you appear to be running blind with SMART.

Looking at both the Google Report and the NetApp Storage Report some 'best practices' become apparent to ensure you minimise your data loss:

HDDs are mechanical devices. Expect failure and plan for it.
Attempt to operate disks in the 30 - 35^oC temperature range to extend their life.
Monitor disk temperatures. Extended periods where a disk temperature rises unexpectedly (ie: not under any additional load than normal), is often an early sign that failure is close at hand.
Attempt to purchase disks not from the same batch. This will avoid a common manufacturing fault taking your disks out at the same time. (Many storage companies will do that for you as part of their service.)
HDD failure follows the Bathtub curve. The 'right side' of the bath kicks in around 3 years. Getting life out of your HDDs beyond that is a bonus, treat it as such!
HDDs are cheap these days. Don't be cheap -- implement RAID-6 over RAID-5 as a matter of course. Ensure you use Double-parity on your RAID-6 implementation. You'll find most recent versions of RAID-6 implement double-parity as 'standard'. (NB: If using NetApp - it's coined as RAID-DP.) Some vendors even allow you to upgrade the storage firmware online if using RAID-6 with Double Parity (NetApp for example has this feature).
Hot-Spares in your RAID-6 array is a very good idea. For the cost of the array consider it an insurance policy against the dreaded multiple disk failure which could potentially toast your array.
How important is your data? Can you put a cost on it? If the cost of replacement is extreme, consider redundancy options. This could include: Implementing RAID-60 (or RAID-6+0), archive/backup solutions, or even a total Disaster Recovery (DR) solution.
Air flow around an array unit is critical. Don't cram your arrays in a fully populated rack, as minimal air-flow will ensure. This will add to the HDD temperatures and general storage enclosure. Remember you're aiming to keep your disks at 30-35^oC.
Keep a logbook of when each drive was added/replaced. You know that 18 month disk then is less likely to fail than that disk that has been whirring away for 5 years. Record size, manufacturer and model/run. You may see some 'patterns' emerge in your own failure rates that will help with additional purchase decisions (ie: particular makes/models to avoid!)
Perform regular 'scans' or 'checks' of your HDD health, knowing the current state of disks allows you to plan for the inevitable failure.

Other tips I've picked up over the years looking after Enterprise systems attached to large storage arrays:

There is a reason that the ASX demands that publicly listed companies with 'mission critical' services for the public (items such as water supply, electricity/gas, and telecommunications) MUST ensure their 'mission criticial' applications have full Disaster Recovery operation on hot-standy.
Don't cut corners and not implement items like HotSpares and Double Parity. When you have a drive failure (not if), you'll be glade you spent that little bit extra on it.
A well planned storage solution should have drive failures, it shouldn't have storage failures. (Don't tell Sun/Internode that!)
Ensure you have clean-filtered power. Don't assume it. Power voltage fluctuations and disk writes don't play nicely.
Don't put all your eggs in one basket. Bad things do happen (I won't mention Internode again). Spreading your data out over multiple storage arrays helps add another level of redundancy. This is a GoodThing^TM
It's never a bad idea to have spare disks in storage waiting for that inevitable failure, rather than relying on a vendor to have your disk capacity/model available. Every day that dead disk isn't replaced you are putting your array at risk.
Look after your disks and arrays, and they generally will look after you (ie: ensuring you don't spend countless hours in the early mornings attempting to recover the unrecoverable!)

Above all, may your next disk failure not be a complete failure.

I'm interested to hear others stories, experiences and ideas that they have put in place to keep their data nice and cosy and their HDDs whirring in a constant and reliable state. Feel free to drop me a comment.

As a side note: I've been putting RAID-1 in place on my desktop machines as disk prices are so cheap now over the past 18 months. For the price of a few hundred dollars why bother the stress of a hard-disk failure? Having said that, it's no replacement for backing up your 'important' information. RAID-1 is still susceptible to the dreaded multiple disk failure issue. Having said that, I've been lucky at that hasn't occurred. (Touch wood!)

UPS monitoring under Linux

mbottrell@gmail.com (Matt) — Wed, 28 May 2008 00:23:00 +1000

I recently made reference to the fact I purchased some UPSes.

For our desktop machines I purchased 2 x Powerware 5110 1500VA units. We also purchased a 1 x Powerware 5125 2200VA (15A socket) to run our 19" racks. (This will also include some networking gear, a 8-port KVM with 15" LCD monitor, and a reasonable server and Disk Arrays).

I make it a habit when sizing a UPS of ensuring I don't load it too high. Whilst I can run many things on each UPS, I would refer a relatively low load, so that if required I can hold the box up long enough to shutdown everything safely. There is nothing worse than hard crashing a server because the UPS load sat at 70-80% load and couldn't hold up for 5-10 minutes whilst machines started shutting down. This is particularly true of a busy database server.

I generally try to run a UPS load of around < 50% and ideally between 10-30% if I can. I would rather have a few minutes up my sleeve then spending time worrying whether I will be able to recover from backup.

LanSafe v5 for Windows

The software that comes with the Powerware 5110 runs under Windows. (They do provide 'Linux' software but it's pretty shoddy and a PITA to try and get working).
LanSafe - does work well under Windows.

Pauline runs Windows as her default OS (mainly due to her need for MYOB and ATO's ECI Client Software both only native to Windows. There is also the occasional business website she requires that is IE only - much to our disgust...)

We will probably look at moving her over to a Linux desktop in the future and running Windows in either a dual boot or virtual machine. In fact, we'll probably do that sooner rather than later, for a number of reasons.

End of Life for Windows XP Professional has been announced, for early next year. (if you don't get 'Extended Support' which most consumers won't be offered - it's mainly for business.).
We've purchased 2 x 1TB SATA2 disks for her machine
A reinstall of Windows may resolve the long delay at startup she faces - upto 5 minutes before the machine becomes 'usable'. Yes, we've cleaned the registery, removed any unwanted startup programs, defragged the disk, and even cleaned out old pre-fetch data. It's still a mystery.

nut-cgi showing my Powerware 5110 UPS

KDE Client for Nut

I wanted something that easy under my Linux desktop (Ubuntu in this case).

That's where NUT comes in.

NUT supports a wide range of UPS makes, models and connectivity options. Feature-wise it is quite impressive.

Reading through the documentation for NUT it's quite clear that you can extend on the notifications you can generate. From items like sending wall messages, Emails, syslog alerts, to even Email messages. You can even get an OSD display if the UPS state changes. I'll probably make use of the SMS daemon I wrote about previously to send me an SMS.

Apparently Gnome Power Manager is supposed to pickup a UPS (it is just a battery after all) is attached. Seems it doesn't in this instance. (I'm happy to be corrected -- but I'm yet to find anything on the web that shows so much for this UPS).

There has been a bit of discussion in recent times of getting an independent system together that relies on D-BUS/HAL so that other Window Managers can also hook into it. A lot like what we're seeing now in Network Manager (love or hate it!) (See: BetterPowerManager and the Power Management Specification

For the record, you can also use check_ups from Nagios to actively monitor your UPS that is being managed by NUT

Those using an APC UPS should look at apcupsd (a new stable version was just released on the 20^th of May 2008). There is also native Windows versions available.

Asterisk, Snom-300 VOIP phone and Power Over Ethernet.

mbottrell@gmail.com (Matt) — Tue, 13 May 2008 18:16:00 +1000

Well it's coming up to the end of financial year, and like many businesses we needed to make a few purchases. (That or pay more corporate tax -- hmmm let me think.. more toys or more tax? )
Generally we don't make many purchases throughout the year and then buy up large near the end of the financial year. (well unless something fails!)

As such we decided to get a decent VOIP phone. We currently use a LinkSys SPA3102 with our cordless phone, but wanted to get a phone with a decent headset so we could talk hands free easily.

Snom-300. An excellent VOIP phone, with fantastic voice quality and a solidly built professional phone.

We spoke to a few people, did the research and settled on a Snom VOIP phone. In particular, we elected the Snom-300 as the phone of choice for us. Whilst it claims it's the 'base phone', unless you're mega-boss or a telemarketing guru I doubt the snom-320 or snom-360 phones are really required. (They basically have a lot of additional buttons and few features added to the base functionality).

You can configure the phone via the LCD menu system, or use the built in web based configuration. The web-based configuration is very well laid out and easy to navigate and locate what you need.

Upgrading the phones firmware is extremely easy and quite painless. You can watch the progress of the phone being flashed, and it's quite novel to see items like chroot, mount appear on your phones LCD.

Chroot, mount you say?! Yup, at the heart of every Snom is a Linux kernel. Whilst this wasn't the sole reason for buying the device, it's handy to know that with a bit of time I could in theory hack the phone.

The Snom-300 HS-MM3 headset is both comfortable and functional. Sound is clear and crisp through it.

I unfortunately get stuck is long boring teleconferences far too often, so I wanted to ensure I also had a phone that was able to plugin in a headset. Not all IP VOIP phones we looked at seemed to have that feature.

The Snom phones do, however they use an RJ-11 headset connection instead of a 2.5mm or 3.5mm jack that people are most familar with. (Many cordless phones also use the 2.5/3.5mm jacks).

As such, we decided to purchase them. They don't come cheap but are worth it. It's a much better solution than getting 'neck cramp' attempting to hold the phone under your chin, or even speaker phone (which seems to pickup more background noise), which can annoy anyone else within earshot.

With the headset, I'm able to easily keep working on my computer; punching away on the keyboard, and those on the other end are none the wiser! (Well besides being too distracted to really keep up with the conversation; dang! I think I just let my secret out....and to think -- people just thought I was vague on the phone!)

Our current setup has me registering both the ATA and the Snoms devices registering directly to our VOIP provider. This works fine, and allows multiple out going calls at the same time, but means only our ATA rings for incoming calls (we have two DIDs).

It makes sense, as the ATA picks up the incoming request and responds... well before the Snoms do. How to resolve?

Easily -- Asterisk to the rescue!

I can setup Asterisk as our local PBX (or PABX for those in Australia!), and have all devices register to it. Then the Asterisk server registers with our Voice Provider.

A bit of Asterisk config and I should be able to get all the phones to ring. In addition, it will allow me to use T.38, add MOH, and voice mail. I also like the ability to have local extensions so we can forward calls between various devices. Currently both Pauline and I share an office, but in our new house we'll be up other ends of the house. So call transfers make sense. I think it would be great to also create an IVR application that puts telemarketers into a loop of endless questions. (randomly generated of course -- so they aren't aware!) Though I think Paul Dwerryhouse's idea is probably the solution.

I'm a newbie to Asterisk (though I've been aware of it for a while -- I just haven't had a need for it)... so I imagine it might take me some time to configure it all up as we desire.

Once done, the only thing Telstra will be getting out of us will be the cheapest line rental we can pay. (They overcharge for that as well!) Though, there is not much getting around it whilst stuck on an ADSL1 connection.

Another purchase we made for the upcoming house was a nice 48-port managed Gigabit switch that contains 12 POE (Power over Ethernet) ports. These will be used to power our phones in the future, saving the need for the current power-pack.

The switch we went for is a D-Link DGS-3100-48P. Like most large switches it's full of fans and sounds like a DC-7 taking off. As a result, it's safely packed back in it's box and will be used after the house move!

We also purchased to UPSes (the old ones had dead batteries) -- strangely it's cheaper buying new UPSes than batteries.
So we ended up with 1 x 1500VA UPS and a 1 x 2200VA UPS.
It appears we'll probably need one more 1500VA.
I went with Powerware UPSes as their is reasonably good local support (and hopefully battery replacements!), and nut seems to work well.

All in all it's been a good month for hardware toys to play with.
Shame a fair bit of this new gear is a few months off until we get into the new house!

Now if only those Intel 8-core Xeons were out.... (my desired replacement server that will consolidate everything down!) If you wish to view a demo of what's coming.. take a look at the Intel demonstration.