How to verify a User Agent block in the webserver logs

Introduction

When you have added some blocking code to your .htaccess file, you may want to find out if it's working correctly. Here are some tips that can help

Steps

(1) ssh into a live webserver

(2) cd to the logs directory

(3) To block a certain User Agent string, such as

Mozilla/5.0 (compatible; SemrushBot/6~bl; +http://www.semrush.com/bot.html)

Copy this code snippet into the shell:

cat access.log | awk '{if ($8="403") {print $0}}' |
 grep 'Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)' | head -1 | awk '{print $4,$5}'

Interpretation of output

The snippet outputs the timestamp of the first block in today's Apache log, access.log

An output like [04/Dec/2020:15:12:30 +0000] means the time of the first block today was 15:12 UTC. Generally, you will want to confirm that this is the time you pushed the block code live.

If there is no output, the User Agent was not found in today's logs. To verify that this is a real finding, try removing the filters in the command and determine what happened.

If the time is something like `06:25 UTC` the block was in place before today.