Avoiding 404 error messages in your logs

Issue

How do I avoid 404 messages in the logs?

Resolution

Many Drupal websites have a problem in which one or a few file paths are repeatedly requested by a remote client or another server. This can fill your website's Drupal log with 404 error messages for files (including autodiscover.xml). Having your website's logs filled with these errors can also impose a serious load on your server, because by default Drupal does a full bootstrap to serve the 404 page.

Some approaches to avoid these 404 error messages in your website's logs include the following:

• Create an empty file
• Block all requests via .htaccess
• Block all requests via a WAF
• Use the Fast 404 module
• Configure Apache to handle the file requests

Create an empty file

This approach is the simplest to implement.

Adding a zero-length file to your code repository at that corresponding file location is a simple solution. It has the advantage of allowing the response to be cached by clients and proxies, including the Varnish server Acquia provides, so the request will reach your actual https servers less often.

To do this, complete the following steps:

• Verify that the path to the missing file (/PATH/TO/FILE/EXAMPLE.txt) exists. If it doesn't exist, you will need to create the folder at that location off of your docroot to contain the file; for example, if you are seeing a lot of request for /PATH/TO/FILE/EXAMPLE.txt, you will need to create the path to /PATH/TO/FILE folder:
  copy

  mkdir docroot/PATH/TO/FILE

• Create the file using the following command:
  copy

  touch docroot/PATH/TO/FILE/EXAMPLE.txt

• Commit your changes to the server based on your code repository.

  If you're using Git, use the following commands:

  copy

  git add docroot/PATH/TO/FILE
  git commit docroot/PATH/TO/FILE
  git push origin

Block all requests via .htaccess

Blocking traffic to /autodiscover/autodiscover.xml via .htaccess will block both GET and POST requests, but must be checked into your code repository. Below is an example code snippet of such a block:

RewriteCond %{REQUEST_URI} /autodiscover/autodiscover.xml [NC]
RewriteRule ^ - [F,L]

Block all requests via a WAF

A WAF (Web Application Firewall), such as Acquia Cloud Edge Protect or Cloudflare, will often afford the ability to block specific paths or User Agents at the outermost edge of the cache layer. Setting a block to the /autodiscover/autodiscover.xml paths or the associated Microsoft User Agent prevents the request from accessing the application directly. 

Use the Fast 404 module

If you can't create an empty file to avoid logging 404 error messages, we suggest that you install the Fast 404 module (for Drupal 7 only).

The Fast 404 module can minimize 404 logging and server load if you are seeing a lot of 404 errors, and it resolves the issue in a similar manner as the later Drupal-based patches for that problem. For help using Fast 404, see Using Fast 404 in Drupal.

Configure Apache to handle the file requests

You can also use Files and FilesMatch directives in Apache to attempt to avoid logging 404 errors. This method can be difficult to accomplish, however, as you must make the style of rewrite shown in the initial patches work when the path includes a non-existent directory. You also have to create an autodiscover directory in your docroot for this approach.