--- title: "Mutation Testing in Drupal" date: "2023-03-10T14:49:29+00:00" summary: image: type: "article" url: "/acquia-cloud-platform/help/89476-mutation-testing-drupal" id: "f7778513-82cd-455e-99ef-31f3f069674b" --- > Quis custodiet ipsos custodes (who watches the watchers?) Automated unit tests are one of the best ways of ensuring code quality. But how do you measure and ensure the quality of your unit tests? The [coverage](https://en.wikipedia.org/wiki/Code_coverage) metric is the traditional way of measuring test quality. Coverage measures what percentage of a codebase is executed during unit tests. For instance, if your codebase has 100 lines of code and only 90 of them run during unit tests, it is 90% covered. ### Coverage alone isn't enough Coverage alone is a poor measure of test quality because it doesn't ensure the tests actually _test_ your code, only that they run it. Having 100% coverage only guarantees that your code doesn't fatally error. But saying that your code won't burn down, fall over, and [sink into the swamp](https://greenelibrary.bibliocommons.com/v2/quotation/95890991) is hardly a measure of quality. [Mutation testing](https://en.wikipedia.org/wiki/Mutation_testing) complements coverage by ensuring that your tests... well, actually _test_ something! 🤯 It does this by mutating (i.e., intentionally breaking) your codebase and then checking that your tests catch the breaking change. If the change is caught, the mutant is killed. If the change is not caught, the mutant escapes. The ratio of killed mutants to total mutants, expressed as a percentage, is the mutation score indicator, (MSI). Like coverage, a higher MSI is better! ### Mutation testing by example > https://github.com/danepowell/mutation-example Fortunately, mutation testing is easy to implement in PHP using the [Infection framework](https://infection.github.io/). Consider [this codebase](https://github.com/danepowell/mutation-example) that implements the [card game War](https://en.wikipedia.org/wiki/War_\(card_game\)). Follow along by cloning it yourself (you'll just need PHP 8.1 and the [PHP pcov extension](https://github.com/krakjoe/pcov) installed): pecl install pcov # If you don't already have pcov installed git clone https://github.com/danepowell/mutation-example.git --branch coverage-only cd mutation-example composer install ./vendor/bin/phpunit --coverage-text If you check out the `coverage-only` branch and run `./vendor/bin/phpunit --coverage-text`, you'll see it has 100% coverage. Sounds great, right? But look more closely at the [test cases](https://github.com/danepowell/mutation-example/blob/coverage-only/tests/Danepowell/MutationExample/WarTest.php). What do you notice? public function testAnnounceWinner($card1, $card2, $expectedWinner): void { $war = new War($card1, $card2); $this->assertStringContainsString('!', $war->announceWinner()); } None of the test cases have meaningful assertions. Imagine what would happen if we "accidentally" flip this [inequality on line 36](https://github.com/danepowell/mutation-example/blob/470ab49fd3326de38b6dd97fe9ff4566fd2c1700/src/War.php#L36): public function announceWinner(): string { $card1value = self::getCardValue($this->card1); $card2value = self::getCardValue($this->card2); if ($card1value > $card2value) { return "Player 1 wins!"; } } Our program would return the wrong winner 100% of the time. So much for great test coverage! This thought experiment is exactly what mutation testing implements as a practice. To see this in action, run `./vendor/bin/infection --show-mutations`: $ ./vendor/bin/infection --show-mutations ... 20 mutations were generated: 8 mutants were killed 0 mutants were configured to be ignored 0 mutants were not covered by tests 12 covered mutants were not detected 0 errors were encountered 0 syntax errors were encountered 0 time outs were encountered 0 mutants required more time than configured Metrics: Mutation Score Indicator (MSI): 40% Mutation Code Coverage: 100% Covered Code MSI: 40% Notice the 40% MSI alongside 100% code coverage. This tells a fuller story of our test "quality" (such as it is). Now look at the escaped mutants. 10) /Users/dane.powell/src/danepowell/mutation-example/src/War.php:36 [M] GreaterThanNegotiation --- Original +++ New @@ @@ { $card1value = self::getCardValue($this->card1); $card2value = self::getCardValue($this->card2); - if ($card1value > $card2value) { + if ($card1value return "Player 1 wins!"; } if ($card1value These mutants demonstrate the exact scenario we mentioned, i.e., changing the inequality so that the wrong winner is returned. The fact that our tests didn't fail in response to these mutations makes them **escaped mutants**. Now for the fun part: killing those mutants! The best way to kill a mutant is to make the same change as the mutation locally, i.e., in your IDE. Run your tests to ensure they still pass after making the "breaking" change. Now add an assertion such as this (or check out the `main` branch, which includes all assertions): public function testAnnounceWinner($card1, $card2, $expectedWinner): void { $war = new War($card1, $card2); $this->assertSame($expectedWinner, $war->announceWinner()); } Run your unit tests again to ensure they fail, as expected. Then re-run the mutation tests. $ git checkout main $ ./vendor/bin/infection --show-mutations 20 mutations were generated: 20 mutants were killed 0 mutants were configured to be ignored 0 mutants were not covered by tests 0 covered mutants were not detected 0 errors were encountered 0 syntax errors were encountered 0 time outs were encountered 0 mutants required more time than configured Metrics: Mutation Score Indicator (MSI): 100% Mutation Code Coverage: 100% Covered Code MSI: 100% You'll see that by simply adding a few assert statements, we've increased the MSI to 100% and, more importantly, significantly improved the quality of our tests. > Don't be discouraged if you see a large number of mutants. Remember that Infection can mutate a single line of code multiple times (once per applicable mutator); a single assertion can often kill a half dozen mutants! ### Mutators Going back to the `coverage-only` branch, let's look at how the mutations were generated. You'll see a "mutator" name printed to the right of each mutation, such as `GreaterThan` and `LessThan`. 11) src/War.php:40 [M] LessThan --- Original +++ New @@ @@ if ($card1value > $card2value) { return "Player 1 wins!"; } - if ($card1value + if ($card1value return "Player 2 wins!"; } return "It's a war!"; } } The [Infection library has dozens of mutators built-in](https://infection.github.io/guide/mutators.html), each of which trying to break your code in a unique way. In addition to `GreaterThan` and `Lessthan`, which flip inequalities, you'll see, among others, `DecrementInteger` and `IncrementInteger` mutators, which work by changing the value of any integer they find. 8) src/War.php:28 [M] IncrementInteger --- Original +++ New @@ @@ 'J' => 11, 'Q' => 12, 'K' => 13, - 'A' => 14, + 'A' => 15, }; } public function announceWinner() : string ### Implementing mutation testing Keep reading for some tips and best practices, and when you're ready, refer to the thorough [Infection documentation](https://infection.github.io/guide/installation.html) to get started. The best way to implement mutation testing is via continuous integration. For a brand-new codebase, implement it from the start and require a minimum MSI for all pull requests. When adding mutation testing to an existing codebase, consider running infection with the `--git-diff-lines` option so that only changed lines are mutated. These practices ensure that test quality gradually improves over time without requiring a major up-front investment in mutant-killing. If you use GitHub Actions, make sure the [annotations logger](https://infection.github.io/guide/command-line-options.html#logger-github) (enabled by default) is working so you can see escaped mutants right in the PR! ![Code review interface showing mutation testing results with highlighted changes in a PHP file.](https://acquia.widen.net/content/72daf0ca-eeb2-4223-a4ab-ed0174117438/web/url_4bfaef8b35e63254d0d6fa9b233d6f13.png) Also consider setting up a Stryker Dashboard account to see mutation results beautifully rendered and navigable, as in this example from [Acquia CLI](https://dashboard.stryker-mutator.io/reports/github.com/acquia/cli/main#mutant). ![Stryker Dashboard showing mutation testing results for various files, including mutation scores, killed, survived, and error metrics in a tabular format.](https://acquia.widen.net/content/a37e2c2c-6297-4258-ac2f-1a3700620ccb/web/url_863d8f6965299b07cc6c0ecec3deb5f0.png) Finally, keep in mind that Infection simply runs your existing PHPUnit tests under the hood. Furthermore, it runs these tests in parallel, dozens of times each, and in random order. This means that your PHPUnit tests need to be idempotent and thread-safe, and any underlying stability or performance issues will be exacerbated by mutation testing. ### Happy hunting! Mutation testing is one of the best ways to ensure test quality, and it takes just a few minutes to set up if your tests already follow best practices. Get started and get support by referring to the [Infection docs](https://infection.github.io/) and [Infection GitHub page](https://github.com/infection/infection). As a reference, also check out [Acquia CLI](https://github.com/acquia/cli), an open-source Acquia product which implements mutation testing on its own codebase [via a GitHub Actions workflow](https://github.com/acquia/cli/blob/main/.github/workflows/mutation.yml). Now go forth, kill some mutants, and improve your tests. Happy hunting!