Best Practices for Site Factory Deployments

Every customer has a unique development workflow and deployment process; however, the following best practice guidelines will apply to most Site Factory deployments. 

Before any changes are made to the production environment, site data should be copied to the staging environment where the deployments can be end-to-end tested against a representative sample of sites. This is the time to review and test deployment related Factory Hooks including the db-update hook.

Then, for production deployments, incorporate the following best practices into your releases:

1. Back up all production sites. Before deploying new code, ensure the release engineer has access to a fresh backup for each and every site. 
2. Document the roll-back plan. Provide a step-by-step process to follow if the databases or files need to be restored from backup. (Test the process to verify that it works.)
3. Document the communication plan. Ensure that a standard operation protocol exists for update business and technical stakeholders. (For example, provide an "all clear" when the production release is complete, if that is appropriate.)
4. Establish an up-to-date inventory of all sites. You may use this inventory as a checklist to validate the successful status of each and every site, following deployments. Consider adding per-site checks (such as ping checks) which can be enabled using New Relic Synthetics (Acquia Academy registration required).
5. Ensure functional drush aliases are available for targeted remediation (if necessary). If you have 100 production sites, you may have 95  update successfully but five fail with errors. In this scenario, you should seek to "fail forward" by remediating only the 5 sites that failed. Use drush to update these sites selectively.
6. If you are using Acquia BLT with your Site Factory, update to the latest appropriate version.

Back up all production sites

There are multiple methods to accomplish this task and no "one size fits all" method. The simplest approach is to visit https://cloud.acquia.com/a/applications/all, navigate to the "01live" production environment, select "Databases" from the sidebar menu, and back up each database manually. Alternatively, you may utilize Site Actions within your Site Factory Management Console; first select "All My Sites", choose the "Back Up Site" option, and repeat this action for each site. (These backups will be queued and will appear under "All My Site Backups" when they are completed.)

If you wish to automate the process, consider using this script or write your own script that implements implements the Site Factory API. The ACSF Tools library, on Github, provides another option. 

Document the roll-back plan

Reverting deployment is a last resort, but it is important to have a written process in place before you need it. List roles and responsibilities appropriate to your organization, and note the steps to be owned by each individual. We recommend you file an Acquia Support ticket immediately if your deployment was not successful (and before taking action to revert database or roll back a deployment).

Document the communication plan

There are many technical and non-technical stakeholders associated with Site Factory applications. Maintenance windows should be communicated to stakeholders when appropriate. For example,

• Will a content freeze be in effect? 
• Will content editors be awaiting an "all clear" notification?
• How should post-deployment issues or bugs be reported by stakeholders?
• Will release notes be prepared for each release to document the new features and functionality? (See an example.)

Establish an up-to-date inventory of all sites

The Site Factory Management Console includes an "Export CSV" button that can be used to generate a simple listing of all sites. Additional organization-specific details (such as internal and external site URLs) may need to be added. Customers add new sites all the time, so it may be necessary to update your inventory ahead of every Site Factory deployment.

Ensure functional drush aliases are available for targeted remediation

In the event that a small number of sites fail to update successfully, you should "fail forward" by correcting the affected sites, as this is often preferable to rolling back a production deployment. Reviewing any errors in the WiP logs and apply appropriate drush commands targeting the affected sites. Similarly, database backups can be restored to selected sites individually, and configuration imports or database updates can be re-applied. Your inventory of production sites should be used to prioritize remediation and keep track of any post-deployment issues. Prevent these sites from blocking the next deployment by correcting the underlying issue with your application.