Differential Testing

ELI5 - The Vibe Check

Differential testing is running the same input through two implementations that are supposed to do the same job, an old one and a new one, or your version and a reference version, and just diffing the outputs. You don't need to know the correct answer in advance, you just need the two to disagree, and that disagreement is your bug report.

Real Talk

Differential testing feeds identical inputs to two or more implementations of the same functionality and compares their outputs to surface discrepancies, without requiring a predefined expected result. It's especially useful for validating rewrites, compiler optimizations, and migrations where correctness is defined as matching a known-good implementation.

Show Me The Code

import { legacyParser } from './legacy';
import { newParser } from './rewrite';

test.each(sampleInputs)('parsers agree on %s', (input) => {
  expect(newParser(input)).toEqual(legacyParser(input));
});

When You'll Hear This

"We ran differential testing between the old and new pricing engine on a year of real orders before cutting over." / "Differential testing against the previous parser caught three edge cases our unit tests never covered."