Using Fast 404 in Drupal

Whenever visitors attempt to access a URL that doesn't exist on your website, your Drupal website will display an HTTP 404 Not Found webpage. Processing these messages effectively can have a significant impact on website performance. Requests resulting in an HTTP 404 bypass Varnish caching and request data from the server directly.

Depending on your installed version of Drupal, you have several methods to control the display of the 404 Not Found page.

 

Method Drupal 7 Drupal 6 Pressflow
Drupal core Yes    
Fast 404 module Yes Yes Yes

Selecting the best method to use can be helpful both with increasing the 404 webpage's display speed and lessening the website's workload. Fast 404, when configured properly, bypasses Drupal's bootstrap process and eliminates database queries and CPU load.

Drupal core

When Drupal 7 responds to page requests, it runs through the standard bootstrap and, before rendering the page out to the user, Drupal returns a simple 404 page before exiting. This prevents Drupal from executing an expensive page render for all 404 pages, saving between 20-100MB per page load.

While this is good behavior, it can be better. The Drupal 7 core includes the ability to run a 404 check earlier in the bootstrap process. To do this:

  1. Edit your website's settings.php file.
  2. Uncomment the drupal_fast_404(); line, by changing this:
    # drupal_fast_404(); 

    to this:

    drupal_fast_404(); 
  3. Save the settings.php file.

Drupal checks the path prior to bootstrapping the database, leading to faster page returns for most 404 responses as well as lessening the overall memory usage of the website.

This was a new feature in Drupal 7 core. However, with the creation of the Fast 404 module on drupal.org, there's a way to not only replicate, but improve on Drupal 7’s fast 404 functionality in both Drupal 6 and Drupal 7.

Fast 404 module

Drupal core's fast 404 option avoids expensive page rendering, but the Fast 404 module improves on this. The Fast 404 module implements hook_boot, which enables it to process requests earlier in the bootstrap stage, improving performance.

To use the Fast 404 module with your website:

  1. Download and install the Fast 404 module from drupal.org. For information about how to install a module, see Installing modules and themes on drupal.org.
  2. Edit your website's settings.php file, and then add the following lines to the end of the file.
    /**
    * Fast 404 settings:
    */
    // This path may need to be changed if the fast 404 module is in a different location.
    include_once('./sites/all/modules/fast_404/fast_404.inc');
    
    # Disallowed extensions. Any extension in here will not be served by Drupal and
    # will get a fast 404.
    $conf['fast_404_exts'] = '/^(?!robots).*\.(txt|png|gif|jpe?g|css|js|ico|swf|flv|cgi|bat|pl|dll|exe|asp)$/i';
    
    # Array of whitelisted URL fragment strings that conflict with fast_404.
    $conf['fast_404_string_whitelisting'] = array('cdn/farfuture', '/advagg_');
    
    # Default fast 404 error message.
    $conf['fast_404_html'] = '<html xmlns="http://www.w3.org/1999/xhtml"><head><title>404 Not Found</title></head><body><h1>Not Found</h1><p>The requested URL "@path" was not found on this server.</p></body></html>';
    
    # Call the extension checking now. This will skip any logging of 404s.
    fast_404_ext_check();
  3. Save the settings.php file.

 

The Fast 404 module will begin to handle all webpage requests that result in a 404 Not Found webpage.

Additional Fast 404 settings

The Fast 404 module includes an option to undertake some aggressive path checking in the settings.php file for websites on MySQL, allowing for fast 404s on all paths (rather than just those with extensions). This technique creates its own database connection to determine if the path exists before either letting Drupal progress with its bootstrap, or exiting early.

Add the following after the recommended Fast 404 settings at the bottom of your settings.php file.


# Check paths during bootstrap and see if they are legitimate.
# By default, a very plain 404 page is used, because usually errors like this are shown to browsers that only look at the headers. Uncomment this link to use a custom 404 page; this page needs to be in your docroot.
#$conf['fast_404_HTML_error_page'] = './page_not_found.html';

# By default, the custom 404 page is loaded only for path checking. Load it for all 404s with the below option set to TRUE, especially if you are only using fast_404_ext_check().
$conf['fast_404_HTML_error_all_paths'] = FALSE;

$conf['fast_404_path_check'] = TRUE;
fast_404_path_check();

Ensuring Fast404 is working

When Fast404 is working, fewer HTTP responses are returned than when it is not. The biggest indicator is this:

X-Drupal-Cache: MISS 
Expires: Sun, 19 Nov 1978 05:00:00 GMT 
Last-Modified: Fri, 10 Apr 2015 18:19:37 GMT 
Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0 

If this group of responses is present, then Drupal's cache is being hit, indicating that Drupal's bootstrap process came up at least far enough to return a cache response. If these are missing, then it was bypassed.

robots.txt file conflicts

Using a specialized robots.txt file on your website may cause conflicts with the Fast 404 module. For information about excluding the robots.txt file and resolving the conflict, see Serving alternate robots.txt files from a multisite.

Private file systems

The Fast 404 module may have problems with private file systems using system/files or perhaps sites/default/private. If you're using a private file system, you may need to make some configuration changes.

Instead of this code:

$conf['fast_404_exts'] = '/^(?!robots).*\.(txt|png|gif|jpe?g|css|js|ico|swf|flv|cgi|bat|pl|dll|exe|asp)$/i'; 

you may need to use something similar to:

# If you use a private file system use the conf variable below and change the # 'sites/default/private' to your actual private files path 
$conf['fast_404_exts'] = '/^(?!robots)^(?!sites/default/private).*\.(txt|png|gif|jpe?g|css|js|ico|swf|flv|cgi|bat|pl|dll|exe|asp)$/i';

OR

$conf['fast_404_exts'] = '/^(?!robots)^(?!\/system\/files).*\.(txt|png|gif|jpe?g|css|js|ico|swf|flv|cgi|bat|pl|dll|exe|asp)$/i'; 

 

Whitelists

Fast404 has a whitelisting capability that can 404 any requested path, like .php and only allow the core drupal php files like index.php and install.php. Unless sites are serving content to .php paths, there's no reason to even allow these non-Drupal requests to come through and bootstrap Drupal and possibly access backdoor web shells.

Here's an example of how this can be done:

Download this file

Contact supportStill need assistance? Contact Acquia Support