Configuring scheduled jobs for import and export¶

Important points¶

• In Cloud Platform Enterprise and Site Factory, you can configure scheduled jobs for each environment in the Cloud Platform user interface. If you are using ODE environments, you can set scheduled jobs by using the Acquia Cloud API or Drush.
• If you are using multisite in Cloud Platform Enterprise, ensure that you use –uri or -l to identify the correct site from the mutisite fleet.
• Drupal queues differ based on their nature:
  • Drupal queues for export are highly CPU intensive as they calculate dependencies on multiple entities for export.
  • Drupal queues for import are highly database intensive as they save multiple entities in the database.
• The content structure and dependency tree in a site affect the load of queues during processing. If the content structure is highly complex or the number of dependencies is high, queues take more time to calculate dependencies during export and to save a single head node during import. The queues must save all the dependencies.
• To configure multiple queue runners or a dedicated cron job server, contact your Account Manager.

Frequency of scheduled jobs¶

For Content Hub, you can set scheduled jobs by using flock. Flock ensures that only one process runs at a time for every scheduled job. Therefore, you can run a schedule job every minute. You can define the frequency to run the queues according to your needs.

If the queues take long to run or get slow down because of multiple items produced, consider the following options to ensure that queue items run faster:

• Create more queue runners per site. Add more scheduled jobs pointing to the same queue but different lock files. You can add queue runners to different web servers in the Cloud Platform user interface. The number of queue runners added per server depends on the server size, availability of PHP procs, and complexity of the site content structure. You must analyze all these parameters to determine if it is correct to include multiple queue runners per queue. Typically, this keeps the queues empty, unless the number of items produced is extremely high. In such a case, you can use a dedicated cron server.
• Define at least one queue runner for every queue. If the queue runner performs poorly, add more queue runners and monitor performance of the server to ensure that the server has enough resources for additional queue runners. The examples of poor queue runner can be processing of fewer items per hour (less than 100) or consistent growth of the queue. While monitoring the performance and resources availability, you can add queue runners.
• If necessary, create a dedicated cron server. One of the webs in the load balancer can serve exclusively as a cron server, running queues all the time. This dedicated cron server can run the queues in different sites. The creation of scheduled jobs in this server is same as in any other case. However, this server won’t be serving web traffic and all its capacity is dedicated to process queues.

If the scheduled jobs are created using flock in the following format, you can set the job to execute every minute. Every instance running at any time has a single process.

Assumptions¶

Assuming the following:

• Publisher URL: https://publisher-site.com

• Subscriber URL: https://subscriber-site.com

• Acquia Cloud SiteGroup: <account>

• Environment: <env> (For example, 01dev, 01test, 01live)

• The number in the lock files and log files in each command must be different between scheduled jobs in the same environment. If not, some jobs might not run. For example, achim0.lck, achex0.log, achi1.log.

• In some cases, multiple environments run on the same web node, and there can be conflicts of lock files and log files. For example, the dev and stage environments run on the same server.

  In such a case, you might use the following script:

  sitegroup=mysitegroup
  for env in dev stg prod; do
    echo "cron list for $env:"
    acli api:environments:cron-job-list $sitegroup.$env | jq '.[] | .command'|grep acquia_contenthub
  done
  

  Acquia recommends assigning different lock and log IDs to each environment running on the same hardware. To use the preceding script, leverage Acquia CLI.

Scheduled jobs for import queues¶

flock -xn /tmp/achim0.lck -c "drush -l https://mysite.com/ -r /var/www/html/${AH_SITE_NAME}/docroot -dv queue:run acquia_contenthub_subscriber_import" &>> /var/log/sites/${AH_SITE_NAME}/logs/$(hostname -s)/achim0-`date "+\%F"`.log

Here, achim stands for Acquia Content Hub import.

Scheduled jobs for export queues¶

flock -xn /tmp/achex0.lck -c "drush -l https://mysite.com/ -r /var/www/html/${AH_SITE_NAME}/docroot -dv queue:run acquia_contenthub_publish_export" &>> /var/log/sites/${AH_SITE_NAME}/logs/$(hostname -s)/achex0-`date "+\%F"`.log

Here, achex stands for Acquia Content Hub export.

Verifying Content Hub queue processing¶

To verify that your Content Hub queues are processed, you can view the Drush logs by using the command:

cat /var/log/sites/${AH_SITE_NAME}/logs/$(hostname -s)/achex0-`date "+%F"`.log | grep -v "using pcntl_exec" | grep -v "Processed 0 items from"

Sample output:

[success] Processed 1 items from the acquia_contenthub_publish_export queue in 0.84 sec.
[success] Processed 4 items from the acquia_contenthub_publish_export queue in 1.23 sec.