Optimizing file paths: Organizing files in subfolders

Issue

If you're starting a new website, and don't have many files uploaded yet, you can Proactively organize your files in subfolders. If you've got a website that has already been in use for some time, the following information may be more helpful to you.

Website performance can severely degrade when too many files are added and made available to the system in a single directory without any subdirectory structure. For example, on Acquia Cloud, we have found that over 2500 files in any single directory in the files structure, or a total of 250,000 files across all directories, can seriously impact a server's performance and potentially its stability.

Resolution

To avoid this problem, you should:

• Split the files into subfolders that are identified by some characteristic of the files. You can use dates as tokens for these directories. The File (Field) Paths and File Entity Paths modules allow you to use node tokens to select the directory where the images or other uploaded files should reside. Simply select the paths for files attached either through file fields or by way of the Drupal core upload module.
• Ensure no single directory contains more than 2500 files.
• Limit the total number of files across all directories to 250,000 or fewer.

Drush Script

An unruly directory is a common problem that is realized only when the file directory becomes too large and has too many files on any single level of a directory. Then it's necessary to relocate not only new files, but also existing ones. To help new clients with this problem, Acquia developed a Drush script for Drupal 7 that moves existing uploaded files into a subfolder structure.

Important

The script in this section is explicitly for Drupal 7.

The provided script processes a target files folder, moves the files directly into that folder (it is not recursively looking into subfolders), and then moves them to a target location. Once there, it establishes a [year]/[month]/[day] directory structure for the files; the date specifies when the file was uploaded.

Instructions

• Download the script file https://cskb.acquia.com/sites/default/files/2019-08/d7_migration_filepath.drush_.inc_.txt into your $HOME/.drush folder.
• Rename the file migration_filepath.drush.inc
• You may need to issue a Drush cache clear with the command: drush cache-clear drush

To start the script, simply issue the following Drush command from the active website's docroot:

drush migration_filepath

There are two optional parameters, --source (the directory under sites/*/files where the files should be moved from), and --target (the directory where the files will be moved to), in case the source and target are not the default files folder, but are for example, a subdirectory underneath.

The script also updates the database and after the file migration, all content should correctly reference the new file locations.

By default, the script moves only certain file types, but you can easily change this by editing the $extensions variable. The script is case sensitive, so .JPG and .jpg files are treated separately.

After running the script, be sure that the file path is using the token structure; otherwise after performing this cleanup, the website will continue to place new files into a flat structure, leading again to a problem after it grows too large.

The runtime depends on the number of files in the folder and the speed of the filesystem. For a comparison, around 50,000 files were successfully migrated in about two minutes on Acquia Cloud.