Shields

Yesterday, the Drupal Security team announced that all Drupal 7 sites are highly vulnerable to attack. Acquia deployed a platform-wide "shield" which protects all our customer sites, while still keeping them 100% functional for visitors and content editors. These sites can now upgrade to 7.32 in a more calm, controlled timeline.

Collaboration - 7 days before release

The Drupal Security team includes representatives from several Drupal hosting companies and Drupal distributions. This is really important, since those folks are pivotal in updating sites when a Security Advisory is released. While discussing the best fix for Drupal, many of us made plans to protect the websites and products under our care.

A week before the announcement, I started collaborating with fellow Security Team members, Damien Tournoud (Platform.sh) and David Strauss (Pantheon). We weighed possible platform-wide remedies and shared our views on the benefits and risks of each. This isn't novel; we recently collaborated on CMI implementations and Drush improvements. I'm proud to report that the Open Source way is embraced even among commercial competitors! Ultimately, Acquia, Platform.sh, and Pantheon all implemented different platform-wide protections for our respective customers. This blog details Acquia's strategy.

Acquia pursued two independent strategies for mitigating the problem. That way, if one solution didn't pan out, our customers would still be protected. As it turned out, this redundancy paid off handsomely. The two strategies were:

The Acquia MySQL Database Driver

This strategy takes advantage of a seldom used feature of Drupal - custom database drivers. We wrote an Acquia driver that inherits all the behavior of the core MySQL driver, and implements a fixed expandArguments() method. Our expandArguments() is identical to the fixed one in 7.32. The advantage of this strategy is that it fixes the root problem, just as Drupal itself has done. When you use this driver, you are safe.

In order to coax Drupal to use this driver, we take advantage of the fact that Acquia Cloud controls settings.php for its clients in order to inject database details. So we change the ‘driver' item in $databases to reference our custom driver and then Drupal carries on with serving the request. For the code monkeys out there:

/**
 * Use Acquia's Drupal MySQL database driver if it exists, but only for
 * web requests, and only up to Drupal 7.31.
 */
function acquia_hosting_use_db_driver($databases) {
 global $conf;
 if (empty($conf['acquia_hosting_disable_sa_2014_005_fix']) & defined('VERSION') && version_compare(VERSION, '7.32', ' $driver_src = '/path/to/drupal-db-drivers/ahmysqldseven';
 $driver_dst = DRUPAL_ROOT . '/includes/database/ahmysqldseven';
 $incs = array("$driver_dst/database.inc", "$driver_dst/install.inc", "$driver_dst/schema.inc");

 // In Live Development mode, create a symlink to the driver, since the
 // the system won't.
 if (is_writeable(DRUPAL_ROOT) & is_dir(dirname($driver_dst)) && !is_link($driver_dst)) {
 @symlink($driver_src, $driver_dst);
 }

 // If all the include files exist, use the driver.
 if (count(array_filter($incs, 'is_readable')) == count($incs)) {
 foreach ($databases as $name => $info) {
 if ($databases[$name]['default']['driver'] == 'mysql') {
 $databases[$name]['default']['driver'] = 'ahmysqldseven';
 }
 }
 }
}

return $databases;
}

Sanitizing the HTTP Request

Our initial idea with this strategy was to block incoming web requests that contained malicious keys in the GET or POST data. Doing this requires knowing what valid, non-malicious keys can look like, so we could avoid false positives which would block a valid request, breaking the website. We could not just consider what keys Drupal core uses, because our customers run an enormous variety of contributed and custom modules. We therefore deployed some additional logging across our entire platform to log all unusual-looking keys for later human inspection.

We had hoped that there would be some fairly simple rules we could apply, such as blocking requests with keys containing certain characters. As soon as we started looking at the data, we realized that would never work. One site, for example, had POST keys that looked like taxonomy terms or perhaps page titles, such as Real estate news and advice. So much for disallowing whitespace! What about just blocking MySQL keywords? Nope - we found sites with POST keys such as select all.

This meant that if we wanted to block MySQL keywords without false positives, we'd have to construct regular expressions matching more of SQL's syntax, such as SELECT\s.*FROM or INSERT\s.*INTO. However, SQL has a rich syntax, with many potentially dangerous commands: TRUNCATE, GRANT, CHANGE MASTER, etc. We also found sites with legitimate POST keys containing semicolons (the MySQL query separator) and double-hyphen (the MySQL comment character). Bottom line, there was just no reasonable way to correctly identify attacks by inspecting the GET/POST data without a lot of false positives.

Finally, we realized that even if we could correctly detect attacks, it would be insufficient. GET/POST keys were only one possible vector to exploit the underlying vulnerability. For all we knew, there were other vectors that had nothing to do with the payload of web requests at all.

In the end, we decided that a mitigation strategy based on blocking web requests was worse than useless---it would provide a false sense of security while also breaking customer web sites. We abandoned the approach entirely.

Go Live - Wednesday

On Wednesday, we brought up our shields by enabling the database driver. Since the code was already deployed, we just ran chmod 755 /path/to/db-driver on all our servers. This command makes the DB driver readable, and causes us to pass the final is_readable() check in acquia_hosting_use_db_driver() [see code above]. Running this command is a very quick process, as opposed to a full Puppet deployment which takes a few hours, giving hackers a brief window for attack.

Acquia Customer Success then mobilized, sending out emails, tickets, tweets, and so on. The Acquia Remote Administration team kicked off its mass notification and updating of codebases.

Arrival of the attacks - Wednesday night

So far, the Acquia MySQL driver is performing flawlessly. Our customer sites are protected and still functional for all site visitors and content editors. Our logging showed no malicious requests before the SA was released, but within 12 hours we were seeing boatloads of them. For the safety of non-Acquia hosted sites, I won't share the malicious requests. But I can share their payloads. Here are some clever queries that attackers are trying to run.

UPDATE users SET name='admin' , pass = '$S$CTo9G7Lx2rJENglhirA8oi7v9LtLYWFrGm.F.0Jurx3aJAmSJ53g' WHERE uid = '1';
INSERT INTO `menu_router` (`path`, `load_functions`, `to_arg_functions`, `description`, `access_callback`, `access_arguments`) VALUES ('mziogj', '', '', 'mziogj', 'file_put_contents','[TROJAN]');
UPDATE {users} SET mail='[EMAIL_ADDRESS]' WHERE uid=1; SET @a=(SELECT MAX(uid) FROM users)+1;
INSERT INTO users SET uid=@a,status=1,name='n0n0x' , pass = '$S$CTo9G7Lx2jmHrpHDdKDR0R8X/q4H9PXo02REYap3z2t8UE3F0DfC';
INSERT INTO users_roles SET uid=@a,rid=3;

Thanks

Many thanks to the all-volunteer Drupal Security Team. A big part of the web is dependent on your generosity, skill, and professionalism.