Searching the error logs to troubleshoot problems

About log files

Acquia provides access to a number of log files that can help you troubleshoot various problems. For a full list of log files, locations, and descriptions, see About Acquia Cloud logging.

Acquia Cloud customers can live stream their log files from Acquia Insight. Read Streaming log entries in real time for details.

If you are having issues on a non-Acquia site, you can spot many errors by using the tail and grep commands on one of the log files. For example, tail -f lets you see errors in real time as they're logged:

You can directly access the log files for each of your web servers using SSH. If you are hosted on Acquia Cloud, you cannot access the aggregated access and error logs on the load balancers.

Here's a short list of examples of how to find problems using these tools.

Using the tail command

You can use the UNIX tail command in a shell to view the last ten lines or so of the log. The command has a number of options, including the very helpful -f, which streams the log file as it is updated. The action looks like this:

tail -f access.log[perform an action in your web browser to make the error happen]ctrl-ctail -f error.log[perform an action in your web browser to make the error happen]ctrl-c

If you do not use the -f option, opting for tail error.log, your results will look something like this:

[Fri May 31 17:48:29 2013] [warn] [client 10.126.254.118] mod_fcgid: can't apply process slot for /var/www/html/test/docroot/index.php[Fri May 31 17:48:29 2013] [warn] [client 10.126.254.118] mod_fcgid: can't apply process slot for /var/www/html/test/docroot/index.php[Fri May 31 17:48:29 2013] [warn] [client 10.126.254.118] mod_fcgid: can't apply process slot for /var/www/html/test/docroot/index.php

Using the grep command

The grep command provides powerful pattern matching tools for searching for strings in one or more files at a time. It has options that enable the automatic count of pattern matches.

To count the number of these errors in the most recent log, run the grep command to count the matching errors:

grep -c slot error.log

This should return a number, like this:

91

If you want to count the errors across all the logs, you can do that with zgrep, and now you can see the trend over time:

 zgrep -c slot error.log*

This returns a list of logs, and the number of times that string appeared in each:

error.log:91error.log-20130301.gz:587error.log-20130305.gz:0error.log-20130518.gz:775error.log-20130525.gz:96

To count all the errors just for May 2013, you can add some date numbers to the target filename:

zgrep -c slot error.log-201305*

This returns a list of logs, and the number of times that string appeared in each:

error.log-20130518.gz:775error.log-20130525.gz:96

This example counts the 503 errors in the current log file:

free-2012$ grep -Ec "" 503 " access.log

returning a number:

91

You can use any keyword(s) in your grep command, depending on the error message you are looking for.

Handling large log files

If your log file is very large, but the majority of what it contains isn't relevant to your search, you can also grep for lines that don't match your string, like this:

grep -v needle error.log

This command displays only the lines in the error.log that do not contain the string needle.

Combining the tools

If, for example, your users are getting 503 errors, you can filter them out and display the last five. To do this, you can use any HTTP code.

grep -E "" 503 " access.log | tail -n5

Generally, multiple lines of information are returned for each request:

10.126.254.118 - - [31/May/2013:17:47:25 +0000] "GET /index.php HTTP/1.0" 503 323 "-" "ApacheBench/2.3" vhost=test.acquia-sites.com host=test.acquia-sites.com hosting_site=test request_time=6465002110.126.254.118 - - [31/May/2013:17:47:25 +0000] "GET /index.php HTTP/1.0" 503 323 "-" "ApacheBench/2.3" vhost=test.acquia-sites.com host=test.acquia-sites.com hosting_site=test request_time=6465002110.126.254.118 - - [31/May/2013:17:47:25 +0000] "GET /index.php HTTP/1.0" 503 323 "-" "ApacheBench/2.3" vhost=test.acquia-sites.com host=test.acquia-sites.com hosting_site=test request_time=64650021

Using other tools

You can use a number of other tools, such as awk, sort, and so on, to search for specific strings with great results.

Using the tools to find 404 errors

You can search for 404 errors. If you're hosted on Acquia Cloud, you'll have to download your access logs from the Cloud UI. However, if you have direct access to your logs, you can try either of these commands:

cat access.log | awk '{ if ($9 == 404) print $7}' | sort | uniq -c | sort -n
tail -n 20000 (apachelogfile) | egrep " 404 " | awk -F" '{ print $2}' | awk '{print $2}' | sort | uniq -c | sort -n

These specify the files that are returning a 404 error and the number of times that they've been returned.

71 /sites/all/themes/record2/non-existing.jpg80 /sites/all/themes/nokia_mobile/css/none84 /movil/node/sites/all/themes/nokia_mobile/jquery.cookie.js145 /sites/default/files/imagecache/641x310/rotavl12_230312okmapa.jpg146 /sites/default/files/imagecache/641x310/mahbub_rota23mz.jpg147 /sites/default/files/imagecache/641x310/rot__jericho_23_03___12.jpg149/sites/default/files/imagecache/641x310/rot__fabian_23_03___12.jpg162 /twitter/images/R.png

Access by IP address

You may need to find where the majority of your website's traffic is coming from, or you might suspect a DDoS attack. How do you track most access IP addresses that are hitting the access logs? The following command can help you track the IP addresses that hit the access logs most frequently:

awk '{print $1}' access.log |sort |uniq -c | sort -rn | head -n 20

Done correctly, you'll get a result like this:

  30201 10.0.0.113  16923 10.0.0.135  16615 10.0.0.21   4091 91.236.74.135   2619 10.0.0.233   1154 71.80.116.164   1144 10.0.0.229

From these results, we can see that this website had over 30,000 hits from the IP address 10.0.0.113.

Running a command on multiple servers

If you want to run the same command on multiple servers at the same time, you can also create a loop command that will run in sequence on the servers.

for i in server-a server-b; do echo $i; ssh [email protected]$i.prod.hosting.acquia.com uptime; done

This provides a list of servers and their uptimes:

 16:37:47 up 8 days, 21:47,  0 users,  load average: 0.00, 0.01, 0.00server-b 16:37:50 up 9 days,  3:00,  0 users,  load average: 0.00, 0.01, 0.00

Contact supportStill need assistance? Contact Acquia Support