Monitor Apache Processes

Alert Yourself of Cyber Attacks

Ok, this article will not alert you to all cyber attacks, but I wrote this script to help me identify huge changes in traffic to my websites and while it isn’t perfect, it has done a pretty good job for me.  Essentially, I am going to show you how I monitor Apache processes.  It has alerted me to huge changes in traffic on my websites.  These changes were due to unexpected traffic from brute force attacks, competitors screen scraping my content, rogue affiliates, etc…  These types of crazy traffic patterns affect performance of your good traffic and can use up all of your available configured server processes.


Apache Prefork Server Configuration

By default, Apache is configured to use a prefork configuration which allows you to configure how many servers Apache will start with and how many it will spin up when it is out of available server processes.  The Apache Process behavior is configured in the httpd.conf file.  You should be able to find the prefork configuration area that looks like this:

<IfModule prefork.c>
StartServers       20
MinSpareServers    5
MaxSpareServers   20
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  4000

The options above tell Apache to startup with 20 processes and to fire up more server processes if we have less than 5 unused server processes up to a maximum of 256 total Apache server processes.  It will also shutdown unused Apache processes should there be 20 or more unused.  Each of these processes takes up server memory and processing time.

So, before we give you the script, let’s discuss what it does.  Below is an example of the output of the file.  You can see that the Apache server is crusing along at 80-90 concurrent processes and then suddenly we see that Apache gets hit by over 400 more processes within a minute.

Thu-11/07/13-12:10:01 Httpds: 82 Direction: -3 Load: 1.30 MemFree: 16857568 kB
Thu-11/07/13-12:11:01 Httpds: 83 Direction: +1 Load: 1.07 MemFree: 16813168 kB
Thu-11/07/13-12:12:01 Httpds: 83 Direction: +0 Load: 1.18 MemFree: 16764684 kB
Thu-11/07/13-12:13:01 Httpds: 80 Direction: -3 Load: 0.84 MemFree: 16771128 kB
Thu-11/07/13-11:51:01 Httpds: 91 Direction: +14 Load: 2.32 MemFree: 16528272 kB
Thu-11/07/13-11:52:01 Httpds: 91 Direction: +0 Load: 2.06 MemFree: 16317276 kB
Thu-11/07/13-11:53:01 Httpds: 88 Direction: -3 Load: 1.42 MemFree: 16321012 kB
Thu-11/07/13-11:54:01 Httpds: 99 Direction: +11 Load: 0.83 MemFree: 16314872 kB
Thu-11/07/13-11:55:01 Httpds: 90 Direction: -9 Load: 1.62 MemFree: 16333540 kB
Thu-11/07/13-11:56:01 Httpds: 88 Direction: -2 Load: 1.26 MemFree: 16339332 kB
Thu-11/07/13-11:57:01 Httpds: 85 Direction: -3 Load: 1.53 MemFree: 16342412 kB
Thu-11/07/13-11:58:01 Httpds: 82 Direction: -3 Load: 1.72 MemFree: 16736692 kB
Thu-11/07/13-11:59:01 Httpds: 145 Direction: +63 Load: 3.95 MemFree: 16393184 kB
Thu-11/07/13-12:00:01 Httpds: 106 Direction: -39 Load: 3.93 MemFree: 16618016 kB
Thu-11/07/13-12:01:12 Httpds: 555 Direction: +449 Load: 26.45 MemFree: 14101208 kB
not OK
Thu-11/07/13-12:01:12 Httpds: 555 Direction: +449 Load: 26.45 MemFree: 14101540 kB
CPU Load is too high, Must restart!
Thu-11/07/13-12:02:01 Httpds: 495 Direction: -60 Load: 26.41 MemFree: 14303128 kB
not OK
Thu-11/07/13-12:02:01 Httpds: 495 Direction: -60 Load: 26.41 MemFree: 14303236 kB
CPU Load is too high, Must restart!
Thu-11/07/13-12:03:01 Httpds: 435 Direction: -60 Load: 11.75 MemFree: 14663312 kB
Thu-11/07/13-12:04:01 Httpds: 375 Direction: -60 Load: 5.22 MemFree: 15033052 kB
Thu-11/07/13-12:05:01 Httpds: 314 Direction: -61 Load: 2.67 MemFree: 15387040 kB
Thu-11/07/13-12:06:01 Httpds: 253 Direction: -61 Load: 1.55 MemFree: 15797404 kB
Thu-11/07/13-12:07:01 Httpds: 193 Direction: -60 Load: 1.39 MemFree: 16175708 kB
Thu-11/07/13-12:08:01 Httpds: 133 Direction: -60 Load: 1.19 MemFree: 16581888 kB
Thu-11/07/13-12:09:01 Httpds: 85 Direction: -48 Load: 1.18 MemFree: 16894028 kB
Thu-11/07/13-12:10:01 Httpds: 82 Direction: -3 Load: 1.30 MemFree: 16857568 kB
Thu-11/07/13-12:11:01 Httpds: 83 Direction: +1 Load: 1.07 MemFree: 16813168 kB

You can see that the scripts tries to remedy the situation by shutting down and restarting Apache.  What is not shown here is that I was alerted of the situation and was able to login, see where the traffic was coming from and shut it down.


  echo "restarting Apache..."
  /sbin/service httpd stop
  sleep 5
  /sbin/service httpd start
  /sbin/service httpd status


# Get real load for report
LOAD=`cat /proc/loadavg | cut -d ' ' -f1`

OLDCOUNT=`cat .httpcount`
COUNT=`/bin/ps -ef | /bin/grep httpd | /usr/bin/wc -l`
echo ${COUNT} > .httpcount

# Get Difference - Are we going up or down?
if [ ${OLDCOUNT} -gt ${COUNT} ]
    HTTPDIFF=`expr ${OLDCOUNT} - ${COUNT}`
    HTTPDIFF=`expr ${COUNT} - ${OLDCOUNT}`
    if [ ${HTTPDIFF} -gt 400 ]
      # Alert Sysadmin
        tail -10 $LOGFILE > .alertfile
        /usr/bin/mutt -s " Jumped by ${DIRECTION}${HTTPDIFF}" <<myemailaddresshere>> < .alertfile
        rm -f .alertfile

# Decide if we are highload
if [ $COUNT -gt 450 ]
    echo `date +%a-%D-%T` Httpds: ${COUNT} Direction: ${DIRECTION}${HTTPDIFF} Load: ${LOAD} `/bin/grep MemFree /proc/meminfo` >> $LOGFILE
    echo "Too many httpds.  Must restart!" >> $LOGFILE
    netstat -an | grep tcp | awk -F" " '{ print $5 }'  | sed 's/ffff//g' | sed 's/^:*//g' | sort | uniq -c | sort -rn |  head -25 >> $LOGFILE
    echo `date +%a-%D-%T` Httpds: $COUNT Direction: ${DIRECTION}${HTTPDIFF} Load: ${LOAD} `/bin/grep MemFree /proc/meminfo` >> $LOGFILE

So, modify the above script and then schedule it to run with crontab every minute or so.


The following two tabs change content below.
Jeff has 20 years of professional IT experience, having done nearly everything in his roles of IT consultant, Systems Integrator, Systems Engineer, CNOC Engineer, Systems Administrator, Network Systems Administrator, and IT Director. If there is one thing he knows for sure, it is that there is always a simple answer to every IT problem and that downtime begins with complexity. Seasoned IT professional by day, Jeff hopes to help other IT professionals by blogging about his experiences at night on his blog: You can find Jeff on or LinkedIn at: LinkedIn or Twitter at: Twitter

Latest posts by Jeff Staten (see all)

Leave a Reply

Your email address will not be published. Required fields are marked *