Maintaining Your Installed Drupal Distro

Drupal.org provides a number of pre-packaged distributions (e.g., Drupal Commons, DKAN, etc.) that allow users get a fully-featured Drupal installation up and running in no time, but maintaining an installed distribution can be tricky. You may need to juggle distribution updates with contrib module updates, core updates, and your own customizations. If you aren't careful, it can be come a maintenance nightmare!

The Drupal community has a few tools for dealing with common maintenance problems, but you'll be hardpressed to find comprehensive documentation on the matter. This blog post will make an attempt to codify best practices for maintenance of an installed distribution.

Let's start by establishing some goals. In maintaining an installed distribution, I'd like to:

• Easily update to new releases of distro.
• Update Drupal core before distro does (e.g., security release).
• Update contributed modules that are packaged with distro before distro does (e.g., security release).
• Modify distro directly (e.g., patch distro.profile file) in a maintainable way
• Modify contributed modules packaged with distro
• Add additional modules, themes, and libraries to the installed distro

Furthermore, I'd like to avoid these problems:

• Accidentally overwriting my customizations when updating to a new distro release
• Accidentally overwriting my customizations when contributed modules are updated
• Forgetting to contribute patches to contrib modules

We've got our work cut out for us, so lets get down to it.

Distribution anatomy

For the sake of this post, I'm going to choose to use DKAN as an example distribution. DKAN follows Drupal's best practices for managing drush make files, and has roughly the following directory structure:

• dkan
  • modules
    • contrib (empty)
    • dkan (empty)
  • themes (empty)
  • dkan.profile
  • drupal-org.make
  • drupal-org-core.make
  • build-dkan.make

Note that many directories in this tree are empty. This is because contributed modules and themes are NOT packaged directly with distributions, and must be subsequently downloaded via Drush Make.

If you're using a distribution that doesn't follow best practices, it may be worth your while to post an issue in the issue queue.

Drush Make

For our purposes, the '.make' files are the most important aspects of the distribution anatomy. They act as recipes that describe the various modules, themes, and libraries that comprise the distro. Using the Drush make utility, they can be used to automatically build the distribution piece by piece, leaving you with a ready-to-use Drupal code base.

Let's walk through an example build.

Initial build of the distro

Create the following tree of empty directories:

• my_special_site
  • projects
  • scripts

Here are a few handy commands to get you started:

Initialize a git repository for this installed distro.

Download a copy of the unpackaged distribution repository to projects/[distro].

Note that we have intentionally avoided creating a git submodule by removing `projects/dkan/.git.' This is because we will use git subtree for managing the dkan distro later. Your directory tree should now look roughly like this:

• my_special_site
  • .git
  • projects
    • dkan
      • modules
        • contrib (empty)
        • dkan (empty)
      • themes (empty)
      • dkan.profile
      • drupal-org.make
      • drupal-org-core.make
      • build-dkan.make
  • scripts

We can now use the recipe in build-dkan.make to build a fully-fledged Drupal codebase to the docroot folder, containing all of the necessary contrib modules.

The ~/my_special_site/docroot directory should now be populated, leaving you with a directory structure somewhat like this:

• my_special_site (git root)
  • docroot
    • .htaccess
    • includes
    • misc
    • modules
    • profiles
      • dkan
        • modules
          • contrib (not empty)
          • dkan (not empty)
        • themes
        • dkan.profile
        • drupal-org.make
        • drupal-org-core.make
        • build-dkan.make
    • robots.txt
    • scripts
    • sites
    • themes
  • projects
    • dkan
      • modules
        • contrib (empty)
        • dkan (empty)
      • themes (empty)
      • dkan.profile
      • drupal-org.make
      • drupal-org-core.make
      • build-dkan.make
  • scripts
  • sites

Please note that I've intentionally omitted a number of core files and folders for the sake of brevity. Don't panic if you have additional files and folders in ~/my_special_site/docroot.

You may have noticed that we now have two copies of the DKAN distributionâ€one pristine copy of the un-built distro, and one copy of the built distro. Preserving the pristine, canonical copy of the distro will simplify the maintenance process later.

Subsequent re-builds

Great! We've got everything we need to run a Drupal site from docroot. You can now feel free to point an apache vhost to ~/my_special_site/docroot and install Drupal. Next, let's tweak the code base in a typical way.

The scenario

Assume that we make the following changes:

• modified .htaccess
• modified robots.txt
• added new modules, themes, and/or libraries to sites/all
• applied a patch to the DKAN distribution
• applied a patch to a module that is packaged with DKAN in docroot/profiles/dkan/modules (field_group)
• updated a module that is packaged with DKAN (og)

A few weeks later, an update is released for DKAN. The update essentially modifies the '.make' files that ship with DKAN so that newer versions of core and contributed modules are required. How do we integrate these changes into the existing docroot?

Pulling in distro updates

First we need to pull in the new version of DKAN to get those new make files. We will update our canonical copy of DKAN located at my_special_site/projects/dkan. Since we've left that folder entirely pristine, we can easily do this using git subtree.

This just pulled in the latest version of the DKAN profile to ~/my_special_site/projects/dkan, which can be used for our rebuild. It does this in a single, nicely contained commit.

Preserving our changes

Unfortunately, we can't simply rerun the drush make command that worked so well for us the first time, because it would wipe out our changes! What's the best way to preserve our changes?

First, let's think about the changes that we made to DKAN and its included modules:

• applied a patch to the DKAN distribution
• applied a patch to a module that is packaged with DKAN in docroot/profiles/dkan/modules (field_group)
• updated a module that is packaged with DKAN (og)

We're going to need to re-apply these patches and updates after DKAN is rebuilt. The best way to manage this is to simply create our own make file which will inherit DKAN's make file and override specified values.

This file combines DKAN's make definitions with our custom patches and module versions. Let's save this to ~/my_special_site/scripts/rebuild-dkan.make.

Next, let's take care of .htaccess, robots.txt, and those new projects that we added to docroot/sites. We start by moving those files outside of the docroot and replacing them with symlinks in a later step.

Now that those files are safely moved, let's remove the docroot and rebuild it with the updated DKAN make file:

Now let's restore the moved files and directories via a symlink:

The directory structure would then look like this:

• my_special_site
  • .htaccess
  • docroot
    • @.htaccess --> ../.htaccess
    • includes
    • misc
    • modules
    • profiles
    • @robots.txt --> ../robots.txt
    • scripts
    • @sites --> ../sites
    • themes
  • robots.txt
  • projects
  • scripts
  • sites

Awesome! That wasn't too tough, but who wants to do all of that typing every time that DKAN is updated?

Automating the process

Let's wrap this in a nice bash script that will automate the process for us. Here's the gist of it.

First, let's create a script that takes care of the git subtree pull:

Running this script (from anywhere) will pull in the latest revision of 7.x-1.x to ~/my_special_site/projects/dkan.

Second, let's create a wrapper around the rebuild process:

I prefer to store these files in cd ~/my_special_site/scripts so that the final directory structure looks like this:

• my_special_site
  • .htaccess
  • projects
  • docroot
  • robots.txt
  • scripts
    • distro.pull.sh
    • distro.rebuild.sh
  • sites

Whenever you need to update your distribution, this is all you will need to do:

Voila! A fresh copy of the distro is pulled down and rebuilt using the fresh make files.

Gotchyas

There are a few 'gotchyas' that you may run into. Here's a brief list of the ones that I've encountered:

• Patch failure
• The distribution maintainer relies on recursive make files to build the distribution

Patch Failure

You may find that one of your patches fails to apply on a subsequent build. This is a normal part of the update and integration process. Your patch may have been merged into the upstream repo, or a new release may have refactored the code that you're patching. In either case, some low-level gitfoo and manual review will be needed.

Recursive make files

This one can be quite tricky, and I could easily write an entire post on this subject alone. I'll try to boil it down to the essentials.

Drush make allows for other make files to be called recursively. E.g., your makefile download a project that itself has a makefile, which is also called. This presents a problem for our maintenance strategy because you can't override recursively called make files. There are a few workarounds for this, none of which I like.

The best solution is to open an issue in the distributions issue queue and request that all packaged projects be defined in a single, flat build-[distro].make file.. Failing that, you have three options:

• Download a second copy of project and store it in sites/all/modules.
  • Drupal's directory precedence will prioritize the module stored there rather than use the one in profiles/[distro]/modules.
• Use recursive patching.
  • E.g., define a patch in rebuild-[distro].make that actually patches the recursively called makefile with another patch.
  • Yikes. I'm not personally a fan of this solution, but some people are.
• Maintain your own fork of the distro's make file.

Further workflow optimizations

This strategy should nicely optimize the process of maintaining all projects packaged with the distribution, but anything outside of that scope is left out. For additional modules and patches, I'd suggest creating a separate make file that can be built to ~/my_special_site/sites/all. This will aid a team of developers in keeping track of all other projects and customizations.