Handling large files
Images are an important component of many, if not most, websites. To ensure the proper display, images often require some level of processing by whatever content management system (CMS) is in use, including Drupal.
Processing can span from a single, basic image resize to fit a standard, to multiple versions of an image at various sizes and resolutions. Image processing can be complex, and can put a major hit on website resources — especially memory. Underestimating the memory usage of these processes can have severe impacts on the responsiveness of your website. Other types of files can also have an impact, but they are generally not as severe as image processing.
There are several ways to remediate potential issues due to large images, including the following:
- Use the right image package
- Use a content delivery network (CDN)
- Plan your uploads
- Download with SFTP, scp, or rsync
- Offload the derivative generation
Use the right image package
Consider how your website is going to deal with images. There are two major packages used for image handling with Drupal.
- ImageMagick - ImageMagick doesn't use Apache's processes, and is installed on all Acquia servers. Acquia recommends ImageMagick over GD library.
- GD library - GD library uses PHP memory, which adds to overall PHP needs. It has a larger array of tools for processing images, but can lead to a need for larger servers because it uses more server resources for a longer period.
Use a content delivery network (CDN)
When dealing with an image- or file-heavy website, especially one that has most of its traffic come from anonymous users, Acquia strongly recommends using a content delivery network (CDN) such as Akamai, Acquia Cloud Edge, or Fastly. A CDN can serve images at a significantly higher speed, and easily store multiple versions of an image, with minimal impact to the originating website after the various image versions are generated.
Plan your uploads
Acquia also suggests a thorough consideration of how image and file uploads will be managed. If you are regularly uploading large numbers of images, the processing capabilities of your server may not be able to keep up with traffic demands.
Download with SFTP, scp, or rsync
Acquia does not support file downloads of greater than one gigabyte. To download large files, use SFTP, scp, or rsync to access your Acquia environments.
Offload the derivative generation
To offload the generation of image derivatives, use one of the following methods:
- Separate your production domain from your content creation domain through the use of an edit domain. You can use various methods of file synchronization, depending on the needs of your website, which includes available bandwidth.
- Use a secondary server to store and build images. With this method, all of your website paths need to be set accordingly so that any image requests are redirected to the correct server.
- Use a service (such as Cloudinary) that is designed for large-scale image handling.
Resizing images into the various types that Drupal uses can consume a large amount of memory, even if you're just using Drupal's pre-set image styles (thumbnail, medium, and large). An image that is 2000 pixels by 2000 pixels is nearly 3 megapixels, a size easily within the capabilities of most smartphones.
The following formulas calculate the memory required to open an 8-bit RGB image using the GD and ImageMagick libraries:
- GD library in PHP
width * height * 3 / 1024^2 MB
There is a memory limit calculator available at PHP memory_limit calculator for image resizing if you want to experiment.
- ImageMagick library
(5 * Rows * Columns) / 1024^2 MB
Based on these formulas, the memory footprint required by PHP to open one 2000 x 2000 pixel image using GD is 11.4 MB. Every time an imagecache preset is triggered, it will consume 11.4 MB of memory just loading the image. If there is a smaller image preset, that will consume additional memory. This is on top of any other images being processed at the time, as well as the standard memory usage of the website. An image dumped right from a phone can easily consume 22+ MB of memory per image.
The following Drupal modules can help you with image processing. They may be useful to your website, but they should be tested thoroughly.
|Module||Description||Drupal 6||Drupal 7|
|Media||Provides an extensible framework for managing files and multimedia assets, regardless of whether they are hosted on your own website or a third-party website|
|Imageinfo Cache||Generates image styles both after an image is uploaded and during an entity save|
|Image Style Pregenerate||Generates all the images for a particular image style in bulk||(In dev)|
|Rules Image Styles||Pre-generates styles using rules||(In beta)|
If you suspect large images may be causing a problem, you can look to see how many there are and where — this example will list the top 10 files, by size, in the current directory. If you have not set up your website to vary the storage directory, you can run this from the
ls -alhS | grep -ve ^total | head
As an example, this request might return a response like this:
-rw-rw-r-- 1 mysite_d7 www-data 24M 2013-01-22 05:34 image1.jpg -rw-rw-r-- 1 mysite_d7 www-data 24M 2013-01-22 05:34 wedgeofdestiny.jpg -rw-rw-r-- 1 mysite_d7 www-data 18M 2013-01-02 23:30 image2.png-rw-rw-r-- 1 mysite_d7 www-data 18M 2012-11-09 16:17 image3.png-rw-rw-r-- 1 mysite_d7 www-data 18M 2012-11-11 22:11 testimage_0.jpg -rw-rw-r-- 1 mysite_d7 www-data 18M 2012-11-11 22:17 testimage_1.jpg -rw-rw-r-- 1 mysite_d7 www-data 18M 2012-11-11 22:00 testimage3.jpg -rw-rw-r-- 1 mysite_d7 www-data 18M 2013-01-02 23:30 sadsplinters.png -rw-rw-r-- 1 mysite_d7 www-data 17M 2013-01-28 05:26 bucketlist2.png-rw-rw-r-- 1 mysite_d7 www-data 16M 2013-04-01 13:37 yahboosnubs.png
After you've located a file that is big, and everything on the proceeding list is, you can use the
identify command (part of ImageMagick) to learn specific data about it:
wedgeofdestiny.jpg JPEG 4860x5184 4860x5184+0+0 DirectClass 8-bit 25.02mb 1.410u 0:02
From this, you can determine that the image is 25 megapixels, and requires 72 MB of memory just to open the one file using GD, and 120 MB if using ImageMagick.