Using AWK to analyze the drupal-requests.log

Warning

The scripts shown are examples to be used as a source of inspiration to be extended upon and modified.

Acquia provides access to a number of log files that can help you troubleshoot various problems. For a full list of log files, locations, and descriptions, see About Acquia Cloud logging.

You can directly access the log files for each of your web servers using SSH. You can also download the most recent logs from the Acquia Cloud UI. If you are hosted on Acquia Cloud, you cannot access the aggregated access and error logs on the load balancers.

AWK can be a powerful log analysis tool when used correctly, in this article we will explore some of the ways to parse potentially useful information out of the drupal-requests.log.

Find the most frequent user uids

Using the below script we can find the most frequent Drupal uids in the log:

awk -F 'uid=| php_pid' '{print $2}'  drupal-requests.log | sort | uniq -c | sort -nr | head

To find more information on the users you can always follow the above up with drush uinf [uid] --uri=[example.com] to find details such as the users name and email.

Count the number of requests per minute

Using AWK we can quickly print a by minute analysis of the number of requests served by Drupal:

cat drupal-requests.log | \
awk -F ' ' \
 '{ \
    time=substr($1, 2, 17); \
    times[time]=time; \
    numtot[time]++; \
  } \
  END \
  { \
    print "Date/Time #Reqs"; \
    for (t in times) \
    { \
      printf ("%s %6d\n", t, numtot[t]); \
    } \
  }' | sort -n | column -t

Compare Authenticated vs Anonymous traffic

Leveraging more AWK functionality we can easily compare the number of authenticated vs anonymous requests served per minute:

cat drupal-requests.log | \
awk -F ' ' \
 '{ \
    posu=index($0,"uid="); \
    posp=index($0,"php_pid="); \
    uid=substr($0, posu +4, posp-(posu+4)-1 ); \
    if (uid == 0) \
    { \
      time=substr($1, 2, 17); \
      times[time]=time; \
      numtot[time]++; \
      uid_anon[time]++; \
    } \
    if (uid > 0) \
    { \
      time=substr($1, 2, 17); \
      times[time]=time; \
      numtot[time]++; \
      uid_auth[time]++; \
    } \
 } \
 END \
 { \
   print "Date/Time #Reqs Anonymous Authenticated" ; \
   for (t in times) \
   { \
      printf ("%s %6d\t%6d\t%6d\n", t, numtot[t], uid_anon[t], uid_auth[t]); \
    } \
 }'|sort -n |column -t