Alert Yourself of Cyber Attacks
Ok, this article will not alert you to all cyber attacks, but I wrote this script to help me identify huge changes in traffic to my websites and while it isn’t perfect, it has done a pretty good job for me. Essentially, I am going to show you how I monitor Apache processes. It has alerted me to huge changes in traffic on my websites. These changes were due to unexpected traffic from brute force attacks, competitors screen scraping my content, rogue affiliates, etc… These types of crazy traffic patterns affect performance of your good traffic and can use up all of your available configured server processes.
Apache Prefork Server Configuration
By default, Apache is configured to use a prefork configuration which allows you to configure how many servers Apache will start with and how many it will spin up when it is out of available server processes. The Apache Process behavior is configured in the httpd.conf file. You should be able to find the prefork configuration area that looks like this:
<IfModule prefork.c> StartServers 20 MinSpareServers 5 MaxSpareServers 20 ServerLimit 256 MaxClients 256 MaxRequestsPerChild 4000 </IfModule> |
The options above tell Apache to startup with 20 processes and to fire up more server processes if we have less than 5 unused server processes up to a maximum of 256 total Apache server processes. It will also shutdown unused Apache processes should there be 20 or more unused. Each of these processes takes up server memory and processing time.
So, before we give you the script, let’s discuss what it does. Below is an example of the output of the file. You can see that the Apache server is crusing along at 80-90 concurrent processes and then suddenly we see that Apache gets hit by over 400 more processes within a minute.
Thu-11/07/13-12:10:01 Httpds: 82 Direction: -3 Load: 1.30 MemFree: 16857568 kB Thu-11/07/13-12:11:01 Httpds: 83 Direction: +1 Load: 1.07 MemFree: 16813168 kB Thu-11/07/13-12:12:01 Httpds: 83 Direction: +0 Load: 1.18 MemFree: 16764684 kB Thu-11/07/13-12:13:01 Httpds: 80 Direction: -3 Load: 0.84 MemFree: 16771128 kB Thu-11/07/13-11:51:01 Httpds: 91 Direction: +14 Load: 2.32 MemFree: 16528272 kB Thu-11/07/13-11:52:01 Httpds: 91 Direction: +0 Load: 2.06 MemFree: 16317276 kB Thu-11/07/13-11:53:01 Httpds: 88 Direction: -3 Load: 1.42 MemFree: 16321012 kB Thu-11/07/13-11:54:01 Httpds: 99 Direction: +11 Load: 0.83 MemFree: 16314872 kB Thu-11/07/13-11:55:01 Httpds: 90 Direction: -9 Load: 1.62 MemFree: 16333540 kB Thu-11/07/13-11:56:01 Httpds: 88 Direction: -2 Load: 1.26 MemFree: 16339332 kB Thu-11/07/13-11:57:01 Httpds: 85 Direction: -3 Load: 1.53 MemFree: 16342412 kB Thu-11/07/13-11:58:01 Httpds: 82 Direction: -3 Load: 1.72 MemFree: 16736692 kB Thu-11/07/13-11:59:01 Httpds: 145 Direction: +63 Load: 3.95 MemFree: 16393184 kB Thu-11/07/13-12:00:01 Httpds: 106 Direction: -39 Load: 3.93 MemFree: 16618016 kB Thu-11/07/13-12:01:12 Httpds: 555 Direction: +449 Load: 26.45 MemFree: 14101208 kB not OK Thu-11/07/13-12:01:12 Httpds: 555 Direction: +449 Load: 26.45 MemFree: 14101540 kB CPU Load is too high, Must restart! Thu-11/07/13-12:02:01 Httpds: 495 Direction: -60 Load: 26.41 MemFree: 14303128 kB not OK Thu-11/07/13-12:02:01 Httpds: 495 Direction: -60 Load: 26.41 MemFree: 14303236 kB CPU Load is too high, Must restart! Thu-11/07/13-12:03:01 Httpds: 435 Direction: -60 Load: 11.75 MemFree: 14663312 kB Thu-11/07/13-12:04:01 Httpds: 375 Direction: -60 Load: 5.22 MemFree: 15033052 kB Thu-11/07/13-12:05:01 Httpds: 314 Direction: -61 Load: 2.67 MemFree: 15387040 kB Thu-11/07/13-12:06:01 Httpds: 253 Direction: -61 Load: 1.55 MemFree: 15797404 kB Thu-11/07/13-12:07:01 Httpds: 193 Direction: -60 Load: 1.39 MemFree: 16175708 kB Thu-11/07/13-12:08:01 Httpds: 133 Direction: -60 Load: 1.19 MemFree: 16581888 kB Thu-11/07/13-12:09:01 Httpds: 85 Direction: -48 Load: 1.18 MemFree: 16894028 kB Thu-11/07/13-12:10:01 Httpds: 82 Direction: -3 Load: 1.30 MemFree: 16857568 kB Thu-11/07/13-12:11:01 Httpds: 83 Direction: +1 Load: 1.07 MemFree: 16813168 kB |
You can see that the scripts tries to remedy the situation by shutting down and restarting Apache. What is not shown here is that I was alerted of the situation and was able to login, see where the traffic was coming from and shut it down.
#!/bin/bashLOGFILE=/data/counthttpd/httpdcountreport RESTART_HTTP() { echo "restarting Apache..." /sbin/service httpd stop sleep 5 /sbin/service httpd start /sbin/service httpd status } # Get real load for report LOAD=`cat /proc/loadavg | cut -d ' ' -f1` OLDCOUNT=`cat .httpcount` COUNT=`/bin/ps -ef | /bin/grep httpd | /usr/bin/wc -l` echo ${COUNT} > .httpcount # Get Difference - Are we going up or down? if [ ${OLDCOUNT} -gt ${COUNT} ] then HTTPDIFF=`expr ${OLDCOUNT} - ${COUNT}` DIRECTION="-" else HTTPDIFF=`expr ${COUNT} - ${OLDCOUNT}` DIRECTION="+" if [ ${HTTPDIFF} -gt 400 ] then # Alert Sysadmin tail -10 $LOGFILE > .alertfile /usr/bin/mutt -s "uptimemadeeasy.com Jumped by ${DIRECTION}${HTTPDIFF}" <<myemailaddresshere>> < .alertfile rm -f .alertfile fi fi # Decide if we are highload if [ $COUNT -gt 450 ] then echo `date +%a-%D-%T` Httpds: ${COUNT} Direction: ${DIRECTION}${HTTPDIFF} Load: ${LOAD} `/bin/grep MemFree /proc/meminfo` >> $LOGFILE echo "Too many httpds. Must restart!" >> $LOGFILE netstat -an | grep tcp | awk -F" " '{ print $5 }' | sed 's/ffff//g' | sed 's/^:*//g' | sort | uniq -c | sort -rn | head -25 >> $LOGFILE RESTART_HTTP >> $LOGFILE else echo `date +%a-%D-%T` Httpds: $COUNT Direction: ${DIRECTION}${HTTPDIFF} Load: ${LOAD} `/bin/grep MemFree /proc/meminfo` >> $LOGFILE fi
So, modify the above script and then schedule it to run with crontab every minute or so.
Latest posts by Jeff Staten (see all)
- Configure Your HP Procurve Switch with SNTP - May 5, 2015
- Configuring HP Procurve 2920 Switches - May 1, 2015
- Troubleshooting Sendmail - November 28, 2014