Bend in the Weather (Entries tagged as nagios)

Nagios 2-way alerting via SMS - Part 3

mbottrell@gmail.com (Matt) — Sat, 24 May 2008 02:54:00 +1000

This is a 3 part posting that covers how you can setup 2-way Nagios alerting via SMS.

The series is broken down as such:

Part 1: Background and Requirements
Part 2: Sending SMS messages from Nagios
Part 3: Acknowledging SMS messages (this article)

Review

The SMS message Nagios generates when there is a problem.

As we saw in Part 2 we covered setting up Nagios to send out alerts.

Whilst this is great, it's not of great value if we can't do anything with these SMS messages. In addition, if no-one is sitting in front of the actual Nagios web console to acknowledge the issue, then Nagios will continue sending the SMS messages and will quickly fill up your phone's message memory. Not ideal!

The message is quite straight forward as well to read. A service problem message contains:

The type of incident: PROBLEM
The Host affected
The Service affected
The Severity of the incident
Additional Information about the incident.
The Time of the incident
A field to Respond/acknowledge the incident

A host problem message is quite similar but does not have a Service or Severity line.

Here is a Host problem message for comparison:

PROBLEM
Host 'MyServer' is DOWN
I: CRITICAL - Host Unreachable (10.0.0.74)
T: 2008-05-22 17:52:33
R:

Replying to the Nagios Problem SMS

Sending a response back to Nagios via SMS.

When a problem from Nagios is sent to you via SMS it's quite easy to respond.

All you need to do is reply to the message (and include the text - which most mobile phones on the market provide a feature to do).

Whilst not required, you can append a simple message to your reply. This will be used in the acknowledgment response and add a comment to the service.

Comments don't need to be long, but they do help your other team members or interested parties know what is happening with the issue.

Be careful what you write!
The response you send will also be used in a follow-up SMS to everyone notifying them that someone is looking at the issue. You don't want to send something that might not be appreciated others that recipients

Acknowledging SMS messages
Acknowledging messages from Nagios is quite simple now we have our SMS Daemon setup (see Part 2 for details)
As a result, any incoming SMS messages get stored into our MySQL database.

We just need a process now to read them from the database, interpret them and then update Nagios appropriately.

Continue reading "Nagios 2-way alerting via SMS - Part 3"

Nagios 2-way alerting via SMS - Part 2

mbottrell@gmail.com (Matt) — Fri, 23 May 2008 01:25:00 +1000

This is a 3 part posting that covers how you can setup 2-way Nagios alerting via SMS.

The series is broken down as such:

Part 1: Background and Requirements
Part 2: Sending SMS messages from Nagios (this article)
Part 3: Acknowledging SMS messages

Sending SMS messages from Nagios

Introduction

To send messages from Nagios, I'm going to assume you already have a working Nagios environment.

2-way SMS messaging refers to the fact that you can reply to an SMS message and action is taken based on your response to the sender.
In this instance, Nagios will send a SMS (a Nagios alert), and you can reply to the SMS (a Nagios alert acknowledgment).

This part will cover sending out the SMS from Nagios once a host or service problem occurs. Part 3 will deal with how to send the reply and process it within Nagios.

Keep reading!

Due to the large size of this posting (it contains step-by-step instructions on setting up the SMS gateway), you may find your feed reader only contains the post up to here.

If that is the case continue reading the post here.

Continue reading "Nagios 2-way alerting via SMS - Part 2"

Nagios 2-way alerting via SMS - Part 1

mbottrell@gmail.com (Matt) — Mon, 19 May 2008 18:09:00 +1000

This is a 3 part posting that covers how you can setup 2-way Nagios alerting via SMS.

The series is broken down as such:

Part 1: Background and Requirements (this article)
Part 2: Sending SMS messages from Nagios
Part 3: Acknowledging SMS messages

For those that wish to setup simple, inexpensive monitoring you will find that it is simpler than first thought. Feel free to grab the code-snippets provided over the series and make use of it in your own environment.

The Nagios SMS alert system has been running here now in production for approximately 6 months and works quite well. The Nagios system manages a range of services/hosts and checks approximately 1,500 items (by no means large); it has however cut down the amount of time I require looking at Nagios alert screens.

Background and Requirements

Background
I like to dabble in web development and have now for around 10 years. As a result I undertake hosting in commercial data centers in order to ensure maximum uptime and good responsiveness. To offset the cost, I now provide hosting/email and web-development services to a number of clients.

As a result, it means one cannot easily look at the diagnostic LEDs, or even the console easily to determine when a problem has occurred. (Whilst I can make use of an IP KVM; it is normally a PITA and best avoided!)

Ideally I was after a monitoring system that alerted me prior to a problem so that I can deal with it before it became a bigger issue.

Nagios was chosen as it provided all the features we required, had support for a large number of items we wished to monitor, and was simple to extend to check custom services based on being written in Perl. Nagios comes with a swag of documentation making it easy to write simple checks or extend on as was seen fit. As I've coded a fair amount in Perl over the years for various System Administration tasks, Nagios was seen as a good fit. Being an Open-Source project ensured that it was an added bonus of both being free-to-use, but also I had access to the source code to understand better how the system worked.

I have used Nagios for years now, and historically have relied on web/email alerts. However, being human means I can't always be in front of my computer 24x7. This worked the majority of the time, though as luck would have it we ended up with a system failure that occurred early one mid-week morning. As a result it was decided we needed to put a system in place that alerted when something occurred when I wasn't in front of the computer.

We already had a working Nagios monitoring environment, so we wanted to make use of this system going forward.

Requirements
I wrote down our requirements for a paging system. I wanted to ensure whatever solution was put in place that it could grow with us, but also that we weren't locked into a solution for years or even worse, that we embarked upon a solution that would ultimately restrict any future requirements we might have.

There were several ways we could be alerted:

Telephone: Have the system 'call us'. Possibly play a standard recorded message.
Pager: Make use of an alpha-numeric pager.
SMS: Send a text message.

We ruled out a telephone, as a recorded message wouldn't provide us any real details of what the issue was, unless we actually started recording exactly what the issue was. It was beginning to look too complicated for a simple requirement. I really thought a Voice-IVR was an overkill for what we required. (Whilst it may of indeed been fun to hack/configure!)

A pager was then looked into. Whilst pagers work well in ensuring a message is delivered we saw this as a drawback, in the sense it meant carrying around another device on us. (We already carried a mobile phone and the prospect of carrying a pager didn't appeal). Also pagers traditionally send one-way messages; and we wanted a system where we could acknowledge an alert.

We settled on a alert delivery system to be used via SMS. Whilst SMS doesn't have the guaranteed delivery system of that of a pager, it did open up the possibility of being able to respond to an alert. It had the added bonus that we already carried our mobiles close by 24x7 so there was no need to carry an additional device. Using a SMS to deliver a text based message also ensured we could then place useful information in the message to assist in the problem diagnosis.

We had now determined our delivery system (SMS), it was time to find technical solutions that would fit in with the existing Nagios monitoring platform.

Nagios by default has the ability to alert via various means. Pagers, SMS and Emails are handled quite easily within the standard configuration that ships with the software.

It would make the integration of SMS quite easily. However, it was only by default a 1-way system. There was no provision to handle a 2-way message. ( Alert Message then acknowledgment message)

A 2-way SMS alert system was preferred as it could be setup to stop alerting us continually via SMS once the alert had been acknowledged. It also had the added benefit that whilst the message was sent to other parties, it could also ensure that they were aware the problem was being investigated and avoid duplication of work.

We also wanted to run the SMS alert system as close as possible to the Nagios monitoring host. This would ensure that if we had a physical network or IP link down, we could still get messages out. This cut out using a 3^rd party SMS service.

Coming Up In the next part of the series...
In the next part, I'll describe the components that make up our 2-way SMS solution using OSS software and some of the scripts we hacked together to 'glue' the solution together.

It's now been running without a hitch now for about 6 months.

We also use it as a Web2SMS gateway locally, as well as being open enough that I can utilize it via the CLI to send out alerts.

Looking back the solution was quite simple, and met all our requirements.