Errors in the Apache access log

Issue

How do I use the Apache access log to troubleshoot?

Resolution

The Apache access log is different from the other logs in this series. Instead of errors and warnings, it keeps track of site requests and can help you figure out who is visiting your site and what they're getting in response. The tools covered in the Tools article are very useful here. The access log is more helpful if you already know that you're looking for a specific problem.

Note

On some versions of UNIX, access.log can be named access_log.

HTTP errors

If, for example, your users are getting 503 errors, you can filter them out and display the last five. You can use any HTTP code in place of the 503 in these examples if you want to check for other issues.

free-2012$ grep -E '" 503 ' access.log | tail -n
510.126.254.118 - - [31/May/2013:17:47:25 +0000] "GET /index.php HTTP/1.0" 503 323 "-" "ApacheBench/2.3" vhost=test.acquia-sites.com host=test.acquia-sites.com hosting_site=test request_time=6465002110.126.254.118 - - [31/May/2013:17:47:25 +0000] "GET /index.php HTTP/1.0" 503 323 "-" "ApacheBench/2.3" vhost=test.acquia-sites.com host=test.acquia-sites.com hosting_site=test request_time=64650021

This example counts the 503 errors in the current log file:

free-2012$ grep -Ec '" 503 ' access.log

It will return a number of matches, like this:

91

You can search for 404 errors as well; if you're hosted on Acquia Cloud, you'll have to download your access logs from the Cloud UI. HTTP errors can have a wide array of causes, from the failure of a particular piece of site functionality, to missing files, to non-functional software or hardware. These kinds of messages indicate that you have a problem, but not necessarily what the specific problem is.

Access by IP address

You may need to find where the majority of your website's traffic is coming from, or you might suspect a DDoS attack. How do you track most access IP addresses that are hitting the access logs? The following command can help you track the IP addresses that hit the access logs most frequently:

awk '{print $1}' access.log |sort |uniq -c | sort -rn | head -n 20

Done correctly, you'll get a result like this:

  30201 10.0.0.113
  16923 10.0.0.135
  16615 10.0.0.21
   4091 91.236.74.135
   2619 10.0.0.233
   1154 71.80.116.164
   1144 10.0.0.229

From these results, we can see that this website had over 30,000 hits from the IP address 10.0.0.113.

Access by user agent

If you suspect a DDoS attack, or some other malicious attempt to harm your website, you can also try to see what user agents are accessing your site. Attacks from agents like Typhoeus can cause serious website problems. This can help you figure out the top user agents visiting your website in a day:

awk -F'"' '{print $6}' access.log | sort | uniq -c | sort -rg | head

You should see results that look similar to this:

   3486 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36 OPR/26.0.1656.60
   1805 Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
   1389 Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
   1373 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
   1335 Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)
   1061 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
    717 Mozilla/5.0 (compatible; spbot/4.4.2; +http://OpenLinkProfiler.org/bot )
    639 check_http/v1.4.16 (nagios-plugins 1.4.16)
    635 check_http/v1.4.15 (nagios-plugins 1.4.15)
    591 Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)