Bend in the Weather

I think we can all attest to the phenomenon known as the Teddy Bear troubleshooting.

I think we all probably need our own Teddy Bears in each of our human office box cubicles or work desks.

So next time you need to do some serious troubleshooting or some heavy lifting when debugging - try pulling out the Teddy Bear.
Even better, you can hug something after it's solved!

Even as a Linux desktop user, I still have a need to run various pieces of legacy software under Windows.
Ideally, I would love to find a Linux alternative, but whilst the itch remains, I still need to scratch it.

As such, I run various Virtualbox Virtual Machines running Windows XP.

This gives me the added benefit of still being able to run my preferred operating system as my main desktop (Linux), whilst still using some legacy applications under Windows.

This is nothing new, and most people do this all the time. With most modern CPU chipsets from both AMD (AMD-V) and Intel (VT-x) providing hardware assisted virtualization solutions - it has become a simple task for many people.

One thing that has annoyed me, and any user that has ever had to reinstall a Windows machine (it happens to the best of us -- Windows often decides to eat it's own registry), is that whilst you might install straight from CD, it also means you have to then spend a good few hours applying all the service packs and then all the updates from the Microsoft website (and sit back and watch a dozen or so reboots between multiple updates). It's enough to send any sane person around the bend.

Once you've done it - you vow never to do it again, yet as Murphy would have it... it's bound to happen again. This is partly due to the fact that Windows Deteriorates over time, until one day the performance is so shoddy you can swear you saw a snail push the cursor on your screen. Once it comes to that, it's time to reinstall again.

Microsoft also recognised it's a problem, particularly in corporate environments that run many desktop PCs. As such they released WSUS. That's fine and well -- if you run a Microsoft Windows 2003/2008 Server. Unfortunately most home users don't.

Don't dispair! Some geeky German's have released a GPLv3 licensed utility that will pretty much undertake the task for you without the need for an expensive Microsoft Windows Server anywhere in sight. c't Offline Update, allows you to use either Linux or Windows to download all the updates for both Microsoft Office and Microsoft Windows platforms. In addition it can handle different language versions, and even build a nice ISO off all the updates so you can take it on the road, hand it to friends, etc.

Why on earth would they code it to download the Windows Updates on Linux? Well, it makes it easier to install in a central location -- say on a Samba server so that all your desktop machines can then install the updates from a central location over your LAN. (Call it a poor man's WSUS!)

To run under Linux it couldn't be easier. Once you have downloaded c't Offline Update and extracted it... change into the sh subdirectory. It's then just a matter of running the DownloadUpdates.sh shell script with the right arguments:

$ ./DownloadUpdates.sh -help

**********************************************************
*** c't Offline Update Downloader ***
*** for Linux Systems ***
*** ***
*** http://www.heise.de/ct/projekte/offlineupdate/ ***
*** Authors: Tobias Breitling, Stefan Joehnke ***
**********************************************************

Usage: ./DownloadUpdates.sh [system] [language] [parameter]

Supported systems:
w2k, wxp, wxp-x64, w2k3, w2k3-x64, w60, w60-x64, oxp, o2k, o2k3, o2k7, all-x64, all-x86

Supported languages:
enu, deu, nld, esn, fra, ptg, ptb, ita, rus, plk, ell, csy
dan, nor, sve, fin, jpn, kor, chs, cht, hun, trk, ara, heb

Parameter:
/excludesp - do not download servicepacks
/makeiso - create ISO-Image
/dotnet - download .NET-Framework
/nocleanup - do not cleanup client directory
/proxy - define proxyserver( /proxy http://[username:password@]:)

Example: ./DownloadUpdates.sh wxp deu /dotnet /makeiso

$

c't Update also provides a simple Windows GUI to automate the task:

Installing the updates on the client.
Once you've downloaded the updates and either burnt them to a CD/DVD (for easy portability) or made them available on a network share (Tip: share the sub-directory called client) Windows users can then easily apply the updates.

A user can run UpdateInstaller.exe (in the root directory of the CD or the share). The UpdateInstaller will determine what you have installed already (and grey these options out). It's then just a case of selecting the options you want.Now for those family members who are in dire need of a 'computer fix' and need me to reinstall Windows for them, they'll be given burnt copies of the latest ISOs, so they can in future undertake the task themselves. I would rather educate someone how to undertake the rebuild themselves for future use. (That, and I really don't have the spare time/desire to rebuild dozens of Windows machines a year! )

Finally it's also worthwhile downloading and running the free Microsoft Baseline Security Analyzer which will ensure you have all the updates applied (and inform you if you are missing any). It's also a good idea to disable any non-essential services the tool recommends.

Hopefully this tip will speed up the time it takes to fully patch a Windows desktop after a reinstall.

In Australia Internet users are limited to the amount they may download. Each Internet Service plan normally involves a speed and download limit. Failing to keep an eye on this and exceed your limit normally effects the user in question either by: Paying exorbitant excess fees that can be as much as $150/GB! Being 'throttled' by your ISP, in effect turning your lovely xDSL/Cable connection into something slower than dial-up. As you can image, most people wish to avoid these. As a result ISPs provide a usage meter normally on their site, however you have to remember to view it, which unless you're diligent in doing so, you can still run over your quota. Previously I've relied on the fabulous Firefox extension NetUsage, however this meant you had to have a browser open. This isn't always the case, and I found it frustrating to have to load a browser just to check my usage. (I could just login to the ISP's usage meter if I had a browser open). I've seen others produce mrtg graphs and even Cacti graphs to display their usage; however they too also rely on having a browser open. Ideally, I want to have my usage available, when I wanted, and without loading a browser. I use conky on my desktop to display a range of information. It seemed a logical step would be to integrate a meter into my conky setup. As you can see from the graph in the picture to the left, it displays a raw percentage of what has been used, as well as a visual bar graph. Below that is the raw data, showing exactly how much I've downloaded (~11.5GB), what my ISP believes my quota to be (55GB in this case), and when this billing period ends (the 13th of every month). I can now view my current Internet usage at a glance, without the need to load any additional program. So how did I achieve it? Like many ISPs, my ISP provides an interface to the raw data. This is used by 3rd party utilities like NetUsage and others to parse and generate the information. Internode (my ISP) has such an interface. A quick and dirty script and I was able to spit out the graph in a format conky liked. Whilst this works directly with Internode, similiar ISPs have such interfaces and you will be able to hack the script to suit your own environment.

Below is the simple bash shell script used to generate the data:
#!/bin/bash

# Get usage information from Internode's web page.
# Setup to run once every half hour.
#

# Set the username and password
Username=INTERNODE_USERNAME
Password=INTERNODE_PASSWORD

# Grab the details
Temp=`wget -q -O - --post-data "username=${Username}&password=${Password}" --user-agent="Conky ISP Quota Grabber 0.1" https://customer-webtools-api.internode.on.net/cgi-bin/padsl-usage`

# Generate the percent used.
echo $Temp|gawk '{print $1/$2*100}'> /tmp/node-graph.txt
Percent=`cat /tmp/node-graph.txt`
echo $(printf %.0f $Percent) > /tmp/node-percent.txt

# Break up the values into their respective parts
Values=($Temp)
echo "Used: $(printf %.0f ${Values[0]})MB Quota: ${Values[1]}MB Period Ends: ${Values[2]}" > /tmp/node-text.txt

# Time to leave.
exit

Sure it's not the most secure script (It stores your username/password in plain text). For me, the script sits on my desktop machine... and is protected by file permissions. The password I use for my ISP I also don't consider highly sensative. (It's important to note that most people throw around their password in clear-text across the Internet when picking up their mail with POP3! ). It does however do the job.

The above shell script I run every 30 minutes from cron.

It is then up to your ~/.conkyrc file to display the results. Here is the snippet of the appropriate code for the ~/.conkyrc file that will do that for you:

${color orange}INTERNET USAGE ${hr 2}$color
${execi 900 cat /tmp/node-percent.txt}% ${execibar 900 cat /tmp/node-graph.txt}
${execi 900 cat /tmp/node-text.txt}

There you have it, a simple ISP usage meter sitting right on your desktop.
It's not the most elegant or complicated program, however it serves it's purpose extremely well. YMMV.

Leave your IRC client set to away for a day, and then check your private chats later that night.

It's amazing some of the strange banter you get:

[someuser] hi..
[someuser] yesterday we completed the network config change in all servers, and now network part is clear
[someuser] ;P
[someuser] ;P
[someuser] :p
[someuser] please ignore above 3 lines

For the record, I was helping a fellow admin out with configuring a series of bonded interfaces for a server (7 connections - 3 bonded, one not).

I take it the first two lines of the message tells me it's been completed.
The next 3 lines were meant for someone else, whilst the last the user recognised what they had done.

The lesson for today: Check which window you're in before typing.

In a not to distant past Russell Coker wrote about RAID Issues and referred in part to a report containing data from 1,530,000 disks running at NetApp customer sites. (also available in PDF or Postscript)

Interesting reading, for sure - particularly if you run any large dataset and want to ensure it stays intact!

It prompted in the recesses of my memory a report compiled in February 2007 by Google. The report, 'Failure Trends in a Large Disk Drive Population' was a report presented at the USENIX (FAST '07) Conference.

The Google report looked at actual hardware failures of disk Google saw over several years. The numbers were crunched (Is there anything else Google does besides crunch large datasets?). Some interesting results popped out:

• The disks studied where either SATA or PATA consumer-grade disks that were either 5400RPM or 7200RPM ranging in size from 80G to 400G and been commissioned from any time after 2001. - Interesting this is the same disks that many of us will find in our own machines. No Enterprise disks, SCSI or SAS disks in the study.
• HDDs had a higher tendancy to fail at the start of their life or anything beyond >3 years of use.
• Low or Heavy Utilization of the HDD resulted in greater loss then 'Medium use'.
• Disks that had surface scan errors had a greater result of failure over the next 60 days.
• HDDs in operating in cool temperature (15-30^oC) had much greater failure rate in the first 3 years. Whilst disks > 3 years had a greater failure rate with the higher the operating temperature. The ideal rate for running disks to minimise failure rate was 30-35^oC.
• A disk that spends more than 50% of it's powered on time > 40^oC is a good indication of a possible problem.
• SMART data analysis revealed that it is not a reliable way to determine if a disk is about to fail. 36% of all disk failure had no SMART errors. The disks that had SMART errors the majority where seek errors (~72%). So basically, expect to see seek errors, beyond that you appear to be running blind with SMART.

Looking at both the Google Report and the NetApp Storage Report some 'best practices' become apparent to ensure you minimise your data loss:

• HDDs are mechanical devices. Expect failure and plan for it.
• Attempt to operate disks in the 30 - 35^oC temperature range to extend their life.
• Monitor disk temperatures. Extended periods where a disk temperature rises unexpectedly (ie: not under any additional load than normal), is often an early sign that failure is close at hand.
• Attempt to purchase disks not from the same batch. This will avoid a common manufacturing fault taking your disks out at the same time. (Many storage companies will do that for you as part of their service.)
• HDD failure follows the Bathtub curve. The 'right side' of the bath kicks in around 3 years. Getting life out of your HDDs beyond that is a bonus, treat it as such!
• HDDs are cheap these days. Don't be cheap -- implement RAID-6 over RAID-5 as a matter of course. Ensure you use Double-parity on your RAID-6 implementation. You'll find most recent versions of RAID-6 implement double-parity as 'standard'. (NB: If using NetApp - it's coined as RAID-DP.) Some vendors even allow you to upgrade the storage firmware online if using RAID-6 with Double Parity (NetApp for example has this feature).
• Hot-Spares in your RAID-6 array is a very good idea. For the cost of the array consider it an insurance policy against the dreaded multiple disk failure which could potentially toast your array.
• How important is your data? Can you put a cost on it? If the cost of replacement is extreme, consider redundancy options. This could include: Implementing RAID-60 (or RAID-6+0), archive/backup solutions, or even a total Disaster Recovery (DR) solution.
• Air flow around an array unit is critical. Don't cram your arrays in a fully populated rack, as minimal air-flow will ensure. This will add to the HDD temperatures and general storage enclosure. Remember you're aiming to keep your disks at 30-35^oC.
• Keep a logbook of when each drive was added/replaced. You know that 18 month disk then is less likely to fail than that disk that has been whirring away for 5 years. Record size, manufacturer and model/run. You may see some 'patterns' emerge in your own failure rates that will help with additional purchase decisions (ie: particular makes/models to avoid!)
• Perform regular 'scans' or 'checks' of your HDD health, knowing the current state of disks allows you to plan for the inevitable failure.

Other tips I've picked up over the years looking after Enterprise systems attached to large storage arrays:

• There is a reason that the ASX demands that publicly listed companies with 'mission critical' services for the public (items such as water supply, electricity/gas, and telecommunications) MUST ensure their 'mission criticial' applications have full Disaster Recovery operation on hot-standy.
• Don't cut corners and not implement items like HotSpares and Double Parity. When you have a drive failure (not if), you'll be glade you spent that little bit extra on it.
• A well planned storage solution should have drive failures, it shouldn't have storage failures. (Don't tell Sun/Internode that!)
• Ensure you have clean-filtered power. Don't assume it. Power voltage fluctuations and disk writes don't play nicely.
• Don't put all your eggs in one basket. Bad things do happen (I won't mention Internode again). Spreading your data out over multiple storage arrays helps add another level of redundancy. This is a GoodThing^TM
• It's never a bad idea to have spare disks in storage waiting for that inevitable failure, rather than relying on a vendor to have your disk capacity/model available. Every day that dead disk isn't replaced you are putting your array at risk.
• Look after your disks and arrays, and they generally will look after you (ie: ensuring you don't spend countless hours in the early mornings attempting to recover the unrecoverable!)

Above all, may your next disk failure not be a complete failure.

I'm interested to hear others stories, experiences and ideas that they have put in place to keep their data nice and cosy and their HDDs whirring in a constant and reliable state. Feel free to drop me a comment.

As a side note: I've been putting RAID-1 in place on my desktop machines as disk prices are so cheap now over the past 18 months. For the price of a few hundred dollars why bother the stress of a hard-disk failure? Having said that, it's no replacement for backing up your 'important' information. RAID-1 is still susceptible to the dreaded multiple disk failure issue. Having said that, I've been lucky at that hasn't occurred. (Touch wood!)

I was appalled after reading the rather rash statements made by Jim Zemlin the Executive director of the Linux Foundation.

His rather bizarre statements against Sun (who is also a silver member of the Foundation) must have left many execs in Santa Clara scratching their heads and wanting some answers.
What's worse is the story originally ran in InfoWorld, then was picked up by the New York Times. Other notable online sites also ran with it, including Slashdot and LWN to mention just two.

Lets look at some of the statements the Exec Director of the Foundation made:

1. "The future is Linux and Microsoft Windows, it is not Unix or Solaris."


2. Solaris has almost no new deployments and is a legacy operating environment offered by a company with financial difficulties. Original equipment manufacturers also do not see a bright future for Solaris. This was paraphrased and not a direct quote.


3. Sun, he declared, should just move over to Linux.


4. "It's certainly true that Unix is on the decline." - Discussing IBM AIX and HP-UX


5. "The only people I hear talk about DTrace [Solaris's technology for assessing program and OS behaviours] and ZFS [the Zettabyte File System] as competitive features [are] Sun Microsystems sales representatives. It's not something I believe is impacting the market in any way."


6. "...With capabilities such as ZFS and DTrace, Sun is trying to compete based on minor features", Zemlin says. "That's literally like noticing the view from a third-story building as it burns to the ground."


7. Zemlin, on Sun's open-source Solaris as "too little, too late." He also goes on to claim that there is no real open source community around OpenSolaris, arguing that Sun still controls development


8. Open Solaris is no more than an attempt to expand the Solaris user base to drive customers to commercial Sun technology.

Seriously where does the guy get off?

Looking at his profile on the Linux Foundation it doesn't instil a level of confidence.
A former Exec at Free Standards Group (who with the OSDL merged to become the Linux Foundation). He was also VP of Marketing of Colavent Technologies (basically a crowd that sold commercial Apache support amongst other things). In 2000 he was a board member of Corio an ASP, that had an IPO then was sold off in chunks. IBM retains the http://www.corio.com/ URL.

So, from my reading he's got quite a bit of marketing/exec background for web-based application servers.... beyond that, I don't see any real technical prowless. He's a marketing suit that appears to shoot from the hip.

Let's review some of his "quick-draws":

• The future is Linux and Windows?
  
  WTF?! Did you take crack before taking the interview?
  
  Sure as the exec. director of The Linux Foundation we expect you to come out and say "Linux is the future" - but to throw Microsoft in the mix -- you have to be kidding me.
  
  Look at general stall that Microsoft has had with Vista. They have forced OEMs to install it, and even some of the larger ones now have kicked back and offer corporations and high-end consumer devices the option of Windows XP instead.



• No major deployments of Solaris... and disliked by the OEMS.
  
  That must be news to both IBM and Dell, who offer OpenSolaris on their x86 series servers.
  
  As for a company in 'financial trouble' this is an extract of the letter to shareholders attached to their latest Annual Report /10-K filing to the SEC (Read it online here).
  
  I've included a screenshot the opening paragraph of the letter:
  
  
  That doesn't sound like a company in 'financial trouble' to me.
  
  You can see this on their Consolidated Statements Of Operations



• The decline of Unix (Solaris, HP-UX, AIX).
  
  It's interesting to note that all of these make hardware as well as an operating system. These systems normally scale much larger than that of a traditional Linux server (excluding clusters). Take a look at Sun's M9000, IBM's p5 595 and HP's HP-9000 Superdome Server. These computers can replace dozens of Intel based servers and thus are not something companies run out and buy dozens of. (well normally!) Interestingly... both HP and IBM are also Platinum Members of the Linux Foundation. With a Platinum membership of $500,000USD each. How's that for biting the hand that feeds you (literally in Zemlin's case).
  
  The I/O throughput and disk-subsystems available for these machines normally far outstrip that of anything under Intel/Linux range.
  


• ZFS, Sun Zones/Containers and DTrace.
  
  Seriously you jest Mr Zemlin?
  
  Linux virtualization is still a mish-mash under heavy development. (take Xen, KVM, LVS and even Sun's VirtualBox)... I wouldn't class as enterprise ready. Hell, even VMWare that started in Linux roots, has moved over onto their own kernel. Zones and Containers under Solaris are much neater and better suited to an enterprise environment.
  
  As for ZFS -- Linux has no mainstream file-system that competes. The size of volumes and the ease of use leaves ext3 behind. Even the upcoming ext4 has no real claims over ZFS. For large scale filesystems, ZFS has it over native Linux filesystems.
  
  DTrace - ask any administrator of 100's of Linux servers if they think DTrace is a minor thing? Sure Systemtap is nice, but the comparison of Systemtap to DTrace still shows some deficiencies. Being able to trace safely on production systems is a requirement for something that can be used in the real world. Add to it the ability to trace user-space programs and DTrace does indeed have some features many SysAdmins discuss and would love to see under Linux.
  
  


• Sun's open source Solaris 'too little-too late'. No community, still controlled by Sun.
  
  Wow... I wonder if he would make the same statement if Microsoft released their OS as open-source? Open-Solaris is a community site, though Sun still controls it. There is nothing wrong with that. It's their baby, and they can do with it what they like.
  
  Sun is still a large OSS backer.... remember products like MySQL, Innotek/VirtualBox, and hey, they even OpenSourced Java.
  
  Sun have shareholders to consider, and they need to make a return on their investments / R&D. I don't really have anything bad to say about the CDDL. It makes sense for them... and hey something is better than nothing. What I find hilarious is that many Linux advocates will bag the CDDL, but love Mozilla. Wake up people... the CDDL is based on the MPL! The CDDL is also recognised as a license under the OSI.
  
  


• Open Solaris attempts to drive customers to commercial Sun technology
  
  Actually most of the people I've come across it are already using the commercial Solaris or wish to learn Solaris. OpenSolaris is a nice way they can install it at home and come familiar with the environment outside of work at their own pace. It makes sense that they can utilise the same operating system in their work-place and in the comforts of their own home.
  
  

For me, I'm a fan of Linux. I love the breadth of software available to me, the rich and colourful community around it. I have grown up watching Linux (I first compiled up an 0.54 kernel on a lowly 286). My servers and desktops/laptops all run Linux.

Having said that, I learnt *NIX on SunOS and later Solaris. It has it's place, even in today's world.
I agree, the low-end UNIX servers are often replaced by Linux servers, yet there is nothing really in the Linux space that competes with the high-end UNIX environments.

A lot of what we see in Linux has it's roots in commercial UNIX. (Hell, who has networked file systems in their environment that doesn't use Sun's NFS?)

So take some advice from a Linux admin at the coal face. Linux co-exists well in a heterogeneous environment, sharing the space with commercial UNIXes and even those annoying Windows servers.

If you need to beat up a 'Server OS' next time Zemlin, try attacking Microsoft (Hint: they are not a member of the Linux Foundation).

Linux servers have probably dented more Windows server sales then that of commercial UNIX. Samba has played a large role in that.

Exchange is probably the last bastion of Microsoft dominance. With the amount of Exchange 'replacements' now on the go that run under Linux (ie: PostPath, Zafara, OpenGroupWare, Scalix and Open-Xchange to name just a small fraction of those available. ) It's likely more inroads into the proprietary walls of Microsoft are now showing cracks in many corporations. Many IT managers are struggling to justify the outrageous price they get slugged for Microsoft Exchange, and these cheaper and feature-rich alternatives are often fractions of the cost if not free.

So next time you attempt to speak for the 'Linux Community' Jim Zemlin, try actually speaking to some Linux administrators that live in the Fortune 500 world. We aren't all hippies yelling 'free OS love'. In fact we happily co-exist with commercial *NIX and sometimes even Windows servers (much to our disgust).

Leave your "Us vs Them" mentality at your door, it's not a view held by a large proportion of the Linux community. Indeed many Linux administrators actually also administer other commercial *NIX machines in their day-to-day jobs. There are more similarities between the commercial *NIX flavours and Linux than not.

We do realise it's part of the Foundation's role to 'promote Linux', but keep the mud-slinging and crap out of it. Let Linux stand on it's own technical merits and not at the detriment of dragging another 'cousin' down in the process. We see enough of this style of 'marketing' from Microsoft that we don't need to stoop to the same levels.

---

_{Disclaimer:
• I work commercially as a Linux System Administrator at a Fortune-500 company.
• I have however administered a large range of *NIX based operating systems over the years... including Solaris, HP-UX, AIX, Tru64, and IRIX just to mention a few of the more 'known' variants.
• My home desktop and all my servers run Linux. Many that know me think I'm a one-eyed Linux zealot!
• I have no affiliations with Sun Microsystems at all.
• The views and opinions expressed by some members of The Linux Foundation are not mine.}

I'm eagerly awaiting large amounts of presents for all those demanding users out there that think sleep is a value-added extra for Sysadmins, or that weekends/public holidays/holidays in general are something that don't apply to us.

Today is System Administration Appreciation Day.

So before you ring me up today to fix your urgent problem, ensure to have that present sent over first.
Any calls raised without the appropriate attached gift will be ignored today.

Geek presents earn additional bonus points. Caffeine and chocolate substances are also welcome.