Monitor Apache Processes

Alert Yourself of Cyber Attacks

Ok, this article will not alert you to all cyber attacks, but I wrote this script to help me identify huge changes in traffic to my websites and while it isn’t perfect, it has done a pretty good job for me.  Essentially, I am going to show you how I monitor Apache processes.  It has alerted me to huge changes in traffic on my websites.  These changes were due to unexpected traffic from brute force attacks, competitors screen scraping my content, rogue affiliates, etc…  These types of crazy traffic patterns affect performance of your good traffic and can use up all of your available configured server processes.

Apache Prefork Server Configuration

By default, Apache is configured to use a prefork configuration which allows you to configure how many servers Apache will start with and how many it will spin up when it is out of available server processes.  The Apache Process behavior is configured in the httpd.conf file.  You should be able to find the prefork configuration area that looks like this:

The options above tell Apache to startup with 20 processes and to fire up more server processes if we have less than 5 unused server processes up to a maximum of 256 total Apache server processes.  It will also shutdown unused Apache processes should there be 20 or more unused.  Each of these processes takes up server memory and processing time.

So, before we give you the script, let’s discuss what it does.  Below is an example of the output of the file.  You can see that the Apache server is crusing along at 80-90 concurrent processes and then suddenly we see that Apache gets hit by over 400 more processes within a minute.

You can see that the scripts tries to remedy the situation by shutting down and restarting Apache.  What is not shown here is that I was alerted of the situation and was able to login, see where the traffic was coming from and shut it down.

#!/bin/bashLOGFILE=/data/counthttpd/httpdcountreport

RESTART_HTTP()
{
  echo "restarting Apache..."
  /sbin/service httpd stop
  sleep 5
  /sbin/service httpd start
  /sbin/service httpd status
}

 

# Get real load for report
LOAD=`cat /proc/loadavg | cut -d ' ' -f1`

OLDCOUNT=`cat .httpcount`
COUNT=`/bin/ps -ef | /bin/grep httpd | /usr/bin/wc -l`
echo ${COUNT} > .httpcount

# Get Difference - Are we going up or down?
if [ ${OLDCOUNT} -gt ${COUNT} ]
  then
    HTTPDIFF=`expr ${OLDCOUNT} - ${COUNT}`
    DIRECTION="-"
  else
    HTTPDIFF=`expr ${COUNT} - ${OLDCOUNT}`
    DIRECTION="+"
    if [ ${HTTPDIFF} -gt 400 ]
      then
      # Alert Sysadmin
        tail -10 $LOGFILE > .alertfile
        /usr/bin/mutt -s "uptimemadeeasy.com Jumped by ${DIRECTION}${HTTPDIFF}" <<myemailaddresshere>> < .alertfile
        rm -f .alertfile
    fi
fi

# Decide if we are highload
if [ $COUNT -gt 450 ]
  then
    echo `date +%a-%D-%T` Httpds: ${COUNT} Direction: ${DIRECTION}${HTTPDIFF} Load: ${LOAD} `/bin/grep MemFree /proc/meminfo` >> $LOGFILE
    echo "Too many httpds.  Must restart!" >> $LOGFILE
    netstat -an | grep tcp | awk -F" " '{ print $5 }'  | sed 's/ffff//g' | sed 's/^:*//g' | sort | uniq -c | sort -rn |  head -25 >> $LOGFILE
    RESTART_HTTP >> $LOGFILE
  else
    echo `date +%a-%D-%T` Httpds: $COUNT Direction: ${DIRECTION}${HTTPDIFF} Load: ${LOAD} `/bin/grep MemFree /proc/meminfo` >> $LOGFILE
fi

So, modify the above script and then schedule it to run with crontab every minute or so.

The following two tabs change content below.

• Bio
• Latest Posts

Jeff Staten

Jeff has 20 years of professional IT experience, having done nearly everything in his roles of IT consultant, Systems Integrator, Systems Engineer, CNOC Engineer, Systems Administrator, Network Systems Administrator, and IT Director. If there is one thing he knows for sure, it is that there is always a simple answer to every IT problem and that downtime begins with complexity. Seasoned IT professional by day, Jeff hopes to help other IT professionals by blogging about his experiences at night on his blog: http://uptimemadeeasy.com. You can find Jeff on Google+ or LinkedIn at: LinkedIn or Twitter at: Twitter

Latest posts by Jeff Staten (see all)

• Configure Your HP Procurve Switch with SNTP - May 5, 2015
• Configuring HP Procurve 2920 Switches - May 1, 2015
• Troubleshooting Sendmail - November 28, 2014

Related posts:

Benchmark Your Webpage with Siege Install WordPress in a Few Quick Steps Apache File Cache Module Make Easy, Memorable URLs for Your Users