muttest 0.3.0: Turn surviving mutants into a to-do list

muttest so far only printed to the console which mutants were killed and which survived.

That’s the minimal setup you need to act on mutation testing findings, but it’s far from optimal. Clear the terminal, close the IDE, and the results are gone unless you saved the output.

muttest@0.3.0 fixes that with an HTML report.

1. Run your tests with new JSON reporter.
2. Render the report.
3. Open it in your browser, and every mutant is laid over your source code.

Click the ones that survived and see exactly what slipped past your suite. That list of survivors is your to-do list: every one is a candidate test. You decide which are worth writing and which are just noise.

📝 Read the full changelog here.

New to mutation testing?

Coverage tells you which lines ran. It says nothing about whether your tests would notice if those lines were wrong. You can delete every assertion in your suite, run covr, and still see 100%.

Mutation testing asks the harder question: if this code were subtly broken, would a test fail? It makes small changes to your source (> becomes >=, TRUE becomes FALSE) and reruns your tests against each version. Each changed version is a mutant. If a test fails, the mutant is killed. If every test still passes, the mutant survived, and you’ve found a gap.

A surviving mutant isn’t a bug to fix. It’s a missing test that could hide an expensive bug later.

If you want to dive deeper, the 0.2.0 post walks through it with an example.

The report

Run your mutation tests with JSONMutationReporter, call report(), and you get a self-contained HTML file.

See the full example report here.

Here’s what the report does that the console can’t do:

• Your source code annotated with mutants. Every file is shown with its actual code, and each mutated line is marked inline. You read the survivor in context instead of reconstructing where a given line was.
• Per-file scores. The overall number is a starting point, not a diagnosis. The report breaks the score down by file, so you can see your parser sitting at 95% while the validation module is at 40%. That tells you which file to open first.
• Per-line mutant diffs. Click a mutant and see exactly what changed: the original line, the mutated line, and whether your tests killed it. When something survived, the diff tells you what kind of a potential bug slipped through.
• Filter by status. The report opens focused on survivors, since the killed mutants already did their job. You can bring the others back when you want the full picture, but the default is the view you actually came for.
• Keyboard navigation. Step through mutants without scrolling through the whole file.

That’s the workflow – filter to survivors, walk the list top to bottom, and each one is a decision: write the test, or call it noise and move on.

A checklist you can work through.

JSON output

The new JSONMutationReporter writes results to a file that conforms to the mutation-testing-elements schema. That schema isn’t something I invented. It’s the format shared by StrykerJS, Stryker.NET, Stryker for Scala, and others. If other teams in your organization already use Stryker and its reporting tools, you can plug muttest results straight into the same dashboards.

library(muttest)

plan <- muttest_plan(
  source_files = "R/shipping.R",
  mutators = comparison_operators()
)

muttest(plan, reporter = JSONMutationReporter$new())

report()

JSONMutationReporter writes to muttest.json by default, and report() reads that same file and writes muttest.html next to it, so with the defaults there’s nothing to wire up.

If you don’t like the built-in report, the JSON file is there to build your own. I plan to keep this schema stable so don’t worry about your report breaking with every next release.

Watch it run, save it for later

You usually want both: a live progress table while the run happens, and a saved file to open afterward. Before, you picked one reporter and got one behavior.

MultiReporter runs several reporters at once. Feed it a ProgressMutationReporter and a JSONMutationReporter together. The console fills up as mutants are tested, and the JSON file is waiting when it’s done.

muttest(
  plan,
  reporter = MultiReporter$new(
    ProgressMutationReporter$new(),
    JSONMutationReporter$new()
  )
)

It’s exactly the same idea as testthat’s MultiReporter.

Scoring changes worth knowing about

Two changes in this release will move your score:

• Errored mutants now count as killed. Some mutations don’t just fail an expectation; they make the code throw before a test can even assert anything. A mutation that produces an error is still a mutation your suite detected, so 0.3.0 scores it as killed. This matches PIT, Stryker, and mutmut. Previously muttest counted these as survived, which was too harsh: the test did catch the change, it just caught it by blowing up.
• Uncovered code gets its own category. Under FileTestStrategy, a mutant in a source file with no matching test file used to look identical to a mutant your tests ran and missed. Those are different problems. A survived mutant means your tests are weak; an uncovered mutant means there are no tests at all. 0.3.0 reports uncovered mutants as a separate no coverage category and leaves them out of the score, so untested code stops masquerading as escaped mutants and dragging your number down for the wrong reason.

If your score shifts after upgrading, this is why. It’s more honest this way.

Tell me about your experience of using muttest

I hope this release makes it easier to analyze results.

muttest is still young and the interface may shift. If the report is missing something you’d want to see, if a mutation you care about has no way to be expressed, or if the new scoring surprises you in a way that feels wrong, open an issue on GitHub.

Feature requests are just as welcome as bug reports.