We are running about two dozen of Apache servers and from time to time they tend to croak. They are normally extremely stable and reliable, however about once a month, sometimes more often, they refuse to serve pages and hurl out status code 500. I did some tracing and tinkering around and found that this behavior is sometimes associated with a surge in traffic, other times it still remains a mystery to me. Apache is one hell of a beast...

Quick fix to this problem is a simple restart of the Apache web service. This somehow fixes Apache and the servers are good for another month or so. The problem is that when this problem happens, I am usually away from my desk and unable to issue a restart command. Sometimes I won't notice for hours that something went wrong, other times I get angry calls. Well, I am sure you get the picture... So using Mango Monitoring Service I created a simple automation tool that takes care of this and I have one less thing to worry about. Other people out there might be facing the same or similar problem, so this article might come in handy.

Mango Monitoring Service will remotely monitor various services on your servers and let you know if something goes wrong. It also gives you an access to useful web APIs you can use to create your own automation tools. I have used the XML export web service and a simple Perl script to create a simple tool that makes my life easier. Here is a short how-to:

Create a file on your server in your preferred script directory called mango.pl. I have put mine into /scripts/mango.pl. Paste the following code into it and run chmod 755 on it:

#!/usr/bin/perl
#PURPOSE: Restart apache on specific status code

# use modules
use XML::Simple;
use LWP::UserAgent;
use strict;

# create variables
my ($host_id, $key, $window, $status_code, $type_code, $url, $data, $xml, $ua, $req, $res, $event);
$host_id = 1111; # set to your host id
$key = "5555-abcdef"; # set to your API key
$window = 12; # how far into the past you want to go (in hours)
$url = "http://www.numango.com/web/manage_servers/".
"export/format/xml/host_id/$host_id/".
"window/$window/key/$key";
$status_code = 500; # your trigger code
$type_code = 3; # code 3 are HTTP codes

# create objects
$xml = new XML::Simple(forcearray => 1);
$ua = new LWP::UserAgent;
$req = new HTTP::Request GET => $url;

# check if we got data
print "Fetching XML data...\n";
$res = $ua->request($req);
if ($res->content eq "Invalid key!")
{
  die("Invalid key!");
}

# read XML file
print "Reading in XML data...\n";
$data = $xml->XMLin($res->content);

# process XML data
print "Processing XML data...\n";
foreach $event (@{$data->{event}})
{
  if (($event->{type_code}[0] == $type_code) && # we are only interested in HTTP codes
  ($event->{alert_note}[0] == $status_code) && # we want to trip on this status code
  ((time()-$event->{date_unix}[0]) <= (10*60)) # we only care about events <= 10 minutes old
  )
  {
    print "Status code $status_code detected @ ".localtime($event->{date_unix}[0])."\n";
    print "Restarting Apache server...\n";
    system("/usr/sbin/apache2ctl restart");

    # we only care about the most recent one, so quit the loop after first restart
    last;
  }
}

# exit
print "Done!\n";
exit(0);

You will need XML::Simple and LWP::UserAgent for this to work. You can get both easily if you are using apt-get (apt-get install libxml-simple-perl libwww-perl). Modify $host_id to match the ID of the server you want to monitor, $key is your webAPI key that can be found under My Account. $window defines how far into the past you want to look (in hours). I wouldn't worry about anything older than couple of hours, so I put 12. $status_code contains the code you want the server to restart on. This can be easily expanded to an array and you can restart on more than one status code (e.g. 501 and 502). And that's it. Now run it and you should get something like this:

[root@host: /scripts]# ./mango.pl
Fetching XML data...
Reading in XML data...
Processing XML data...
Done!

If error was detected then you will get this:

[root@host: /scripts]# ./mango.pl
Fetching XML data...
Reading in XML data...
Processing XML data...
Status code 500 detected @ Mon Nov 19 17:20:42 2007
Restarting Apache server...
Done!

The last piece of the puzzle is to have this script run periodically, I opted for it to run every five minutes. Thus I edited /etc/crontab and added the following line. You can select interval that fits your needs. Note: you should match the polling interval that you have selected for your server on the Mango website.

0-59/5 * * * * root /scripts/mango.pl

Voila! I hope this little how-to helps you. You can expand this to reboot any service like SMTP, DNS or anything that Mango lets you monitor.