Nagios 2-way alerting via SMS - Part 1

This is a 3 part posting that covers how you can setup 2-way Nagios alerting via SMS.

The series is broken down as such:

1. Part 1: Background and Requirements (this article)
2. Part 2: Sending SMS messages from Nagios
3. Part 3: Acknowledging SMS messages

For those that wish to setup simple, inexpensive monitoring you will find that it is simpler than first thought. Feel free to grab the code-snippets provided over the series and make use of it in your own environment.

The Nagios SMS alert system has been running here now in production for approximately 6 months and works quite well. The Nagios system manages a range of services/hosts and checks approximately 1,500 items (by no means large); it has however cut down the amount of time I require looking at Nagios alert screens.

Background and Requirements

Background
I like to dabble in web development and have now for around 10 years. As a result I undertake hosting in commercial data centers in order to ensure maximum uptime and good responsiveness. To offset the cost, I now provide hosting/email and web-development services to a number of clients.

As a result, it means one cannot easily look at the diagnostic LEDs, or even the console easily to determine when a problem has occurred. (Whilst I can make use of an IP KVM; it is normally a PITA and best avoided!)

Ideally I was after a monitoring system that alerted me prior to a problem so that I can deal with it before it became a bigger issue.

Nagios was chosen as it provided all the features we required, had support for a large number of items we wished to monitor, and was simple to extend to check custom services based on being written in Perl. Nagios comes with a swag of documentation making it easy to write simple checks or extend on as was seen fit. As I've coded a fair amount in Perl over the years for various System Administration tasks, Nagios was seen as a good fit. Being an Open-Source project ensured that it was an added bonus of both being free-to-use, but also I had access to the source code to understand better how the system worked.

I have used Nagios for years now, and historically have relied on web/email alerts. However, being human means I can't always be in front of my computer 24x7. This worked the majority of the time, though as luck would have it we ended up with a system failure that occurred early one mid-week morning. As a result it was decided we needed to put a system in place that alerted when something occurred when I wasn't in front of the computer.

We already had a working Nagios monitoring environment, so we wanted to make use of this system going forward.

Requirements
I wrote down our requirements for a paging system. I wanted to ensure whatever solution was put in place that it could grow with us, but also that we weren't locked into a solution for years or even worse, that we embarked upon a solution that would ultimately restrict any future requirements we might have.

There were several ways we could be alerted:

1. Telephone: Have the system 'call us'. Possibly play a standard recorded message.
2. Pager: Make use of an alpha-numeric pager.
3. SMS: Send a text message.

We ruled out a telephone, as a recorded message wouldn't provide us any real details of what the issue was, unless we actually started recording exactly what the issue was. It was beginning to look too complicated for a simple requirement. I really thought a Voice-IVR was an overkill for what we required. (Whilst it may of indeed been fun to hack/configure!)

A pager was then looked into. Whilst pagers work well in ensuring a message is delivered we saw this as a drawback, in the sense it meant carrying around another device on us. (We already carried a mobile phone and the prospect of carrying a pager didn't appeal). Also pagers traditionally send one-way messages; and we wanted a system where we could acknowledge an alert.

We settled on a alert delivery system to be used via SMS. Whilst SMS doesn't have the guaranteed delivery system of that of a pager, it did open up the possibility of being able to respond to an alert. It had the added bonus that we already carried our mobiles close by 24x7 so there was no need to carry an additional device. Using a SMS to deliver a text based message also ensured we could then place useful information in the message to assist in the problem diagnosis.

We had now determined our delivery system (SMS), it was time to find technical solutions that would fit in with the existing Nagios monitoring platform.

Nagios by default has the ability to alert via various means. Pagers, SMS and Emails are handled quite easily within the standard configuration that ships with the software.

It would make the integration of SMS quite easily. However, it was only by default a 1-way system. There was no provision to handle a 2-way message. ( Alert Message then acknowledgment message)

A 2-way SMS alert system was preferred as it could be setup to stop alerting us continually via SMS once the alert had been acknowledged. It also had the added benefit that whilst the message was sent to other parties, it could also ensure that they were aware the problem was being investigated and avoid duplication of work.

We also wanted to run the SMS alert system as close as possible to the Nagios monitoring host. This would ensure that if we had a physical network or IP link down, we could still get messages out. This cut out using a 3^rd party SMS service.

Coming Up In the next part of the series...
In the next part, I'll describe the components that make up our 2-way SMS solution using OSS software and some of the scripts we hacked together to 'glue' the solution together.

It's now been running without a hitch now for about 6 months.

We also use it as a Web2SMS gateway locally, as well as being open enough that I can utilize it via the CLI to send out alerts.

Looking back the solution was quite simple, and met all our requirements.