Are the packages version installed on Acquia Cloud secure?

Issue

You have performed a vulnerability scan on your website and the report specifies that some one or more packages (e.g. OpenSSH / Apache) are not secure as vulnerabilities were fixed in later versions than the ones Acquia has installed.

Resolution

Note

Many scanning tools may falsely report a problem. These tools depend mostly on the self-reported version of a package or tool which does not account for security patching.

On the other hand, it is possible (but unlikely) that you have indeed found a problem. Please read on to understand how a security scan could show a false result.

About security patching

Acquia uses software packages (like OpenSSH, Apache, etc.) from the Ubuntu repositories. Canonical (our "upstream" vendor) which provides most of these packages is always monitoring and fixing security vulnerabilities, by maintaining their own patched versions of code, which are based on an original "base" version. They then provide these packages to Acquia (and anyone using Ubuntu) who then deploys them into the server fleet which houses your applications.

The thing that can confuse vulnerability scanners, is that looking at version numbers alone can suggest that Acquia servers are running an old version which has vulnerabilities. 

For example: if a tool flags a problem due to Acquia using "OpenSSH_7.2p2 and prior", it turns out this is incorrect. The Canonical Ubuntu version (which Acquia uses), while based on an older version (7.2p2) actually has additional patches which fix bugs and vulnerabilities.

To illustrate, in the code below, you can see how some methods of getting a software version show only the "base" version and not the patched version. These commands were run with root access inside an Acquia SSH session. 

Note: The below examples were run on Cloud Next. On Classic, this may require root access which is not available for customers.

########## OpenSSL# "Base version" of Open SSL .. does not show the patched version number.
$ openssl version  OpenSSL 1.0.2g  1 Mar 2016

# Show the Ubuntu-patched version
$ dpkg-query -W -f='${Package} ${Version}\n' openssl

  openssl 1.0.2g-1ubuntu4.20+esm9

The above outputs shows the OpenSSL is patched with version 1ubuntu4.20+esm9.  Note how we needed to run a special command (dpkg-query) to get that Ubuntu-patched version.

From the original version number, you would think that a package from 2016 is extremely outdated... however the Ubuntu version shows it has been patched to fix bugs and security issues. 

In contrast, getting the OpenSSH patch version was easier as running ssh -V directly shows it. As an Acquia customer, you can check the Ubuntu-patched version by yourself for OpenSSH but not for OpenSSL, running the below after connecting through the ssh connection.

########## OpenSSH
$ ssh -V

  OpenSSH_7.2p2 Ubuntu-4ubuntu2.10+esm2, OpenSSL 1.0.2g  1 Mar 2016

Here's what the pieces of the version name/numbers mean for the above openssl example:

• openssl: The package name.
• 1.0.2g: Original (or "upstream") software version.
• -1ubuntu4.20: Ubuntu-specific modifications to the package.
• +esm9: Indicates an update as part of Extended Security Maintenance.

Canonical maintains a Common Vulnerabilities and Exposures (CVE) tracker which can be used to check the status of patching for specific vulnerabilities, where you can then get the Ubuntu patch-version number that fixes them. For example, current openSSH CVEs are tracked here : https://ubuntu.com/security/cve?package=openssh

Finding whether a CVE has been resolved in a package

You can always look for CVEs that have been fixed in an Ubuntu package version via a URL like this:

https://launchpad.net/bugs/cve/CVE-2021-41617

An alternative method of finding the patched versions is using the installed documentation files on the Acquia servers. For example, you can use grep to find a certain CVE in the installed man pages like this:

# Run these within an Acquia SSH session.
# EDIT THESE VALUES depending on what you're searching for
$ package=openssh-server
$ searchstring=CVE-2021-41617 

# Run a search against the Debian Changelog ...
$ zgrep -C3 --color "${searchstring}" /usr/share/doc/${package}/changelog.Debian.gz 

openssh (1:7.2p2-4ubuntu2.10+esm2) xenial-security; urgency=medium

  * SECURITY UPDATE: Supplemental groups not initialized
    - debian/patches/CVE-2021-41617.patch: add initgroups()
      before setresgid() in auth2-pubkey.c.
    - CVE-2021-41617

If your command finds a CVE mentioned in the changelog.Debian.gz file, it means that CVE has been patched in the installed package.

Consulting the Ubuntu Official package version lists

One last option is to just consult the official Ubuntu package versions for each OS release. For example, here are the installed version(s) of the libssh2 package for both Ubuntu 16.04 (Xenial) and 20.04 (Focal). (Look under the "Published Versions" section):

https://launchpad.net/ubuntu/xenial/+source/libssh2

https://launchpad.net/ubuntu/focal/+package/libssh2-1

You can use the above sites 

If you still think you found an issue

You should ensure you have properly found out the patched version of anything you suspect is an issue, and use the above Ubuntu CVE tracker to determine if a particular vulnerability has already been fixed by Canonical. Sometimes, a fix may have not been released, and thus Acquia would still need to wait for Canonical to release it  before taking any action.

We take security very seriously! If you do see an open CVE at the above Ubuntu tracker that applies to the Ubuntu versions in use by our platform, our fleet's packages following the cadence documented in the Security Patch Management section. Higher-impact vulnerabilities are fixed faster (as fast as within 7 days of a patch existing), but lower-impact ones may take several months. This practice is within industry standards and approved by the various bodies that have certified Acquia's platform. You can read more about our security certification in our Compliance with standards and regulations page.