Build scripts need to wait for code deployments to finish before running further commands

Applications using Travis, Jenkins scripts, GitLab pipelines or other external scripts that push code changes to a branch which is currently deployed onto Acquia Cloud MUST wait for code deployment to finish before running any SSH, drush or other commands that interact with the new code.

If the scripts do not wait for deployment to finish, these symptoms may appear:

• Script commands can fail when attempting to connect to the remote Acquia environment, since the SSH service might not be available for a short time during the Acquia Cloud deployment process. These show up as drush "Exit code: 255" errors, or SSH connectivity errors.
• If SSH or Drush commands do fire successfully, they may be using the previous code version and trigger problems unexpected behaviours including causing a site down event.

On Acquia Cloud Platform, the incidence of this problem is higher, due to the longer code deployment times (of around 2-5 minutes),  that come with the safer Atomic Deployment method of Cloud Platform, as compared to Cloud Classic (usually around 1 minute).

Here is an example scenario that triggers this problem:

• Your Acquia Cloud "Dev" environment, mysitedev.prod.acquia-sites.com, currently has the "develop" Git branch deployed.
• Your external script pushes a change into the "develop" Git branch. This causes Acquia Cloud to initiate a code reployment to the Dev environment.
• Your external script, without waiting, runs "drush @mysite.dev cache-clear". This will very likely clear caches on the site using the old code.
• After a few minutes, the Acquia Cloud code deployment finishes, the new code is in place.
• If the new code required the above drush command to function correctly ... someone or something would now need to run this drush command manually, because drush ran on the old deployment and not the new one.

Resolution

You have several options:

1) Use Cloud Hooks when possible.

Cloud Hooks are scripts that automatically trigger on certain events. For example, if your build process requires to run some commands (like drush cache-rebuild) that can run independent of your script, you could add a post-code-update Cloud Hook to run these commands. These are guaranteed to fire after the right code is in place.

2) Add a delay to your external script or pipeline

If your script does require to interact with the code changes (like, running a test after deployment, importing configuration with drush config:import, etc.), then adding a delay with the "sleep" command or similar will help ensure the code is available before running the next steps.

For Cloud Platform, we recommend a delay of around 5 to 7 minutes; this depends on the historic performance of your code deploys, and may need tweaking after you add it to your scripts.
 

Example scenario showing the problem and resolution

Let's imagine an external script containing this pseudocode:

# BEGIN SCRIPT
 ... snip ...
composer install [or other command that changes your code]

git push [remote-id] [branch-that-is-deployed]

drush @site.env config:import

drush @site.env cr

# END SCRIPT

Then, to avoid the problem you could insert a "sleep [number-of-seconds]" command between the git push and your commands. For example:

sleep 500

On the other hand, since in the above case the script does not seem to interact with the results from the drush commands, you could remove these lines from this script, and transfer them to a Cloud Hook script.