The tests that pass because they never run
A test with no matcher, no annotation or a duplicate name stays green forever. I went looking for them in well-known projects, and some were hiding real bugs.
6 min read
Vue's runtime-core had a test called SVG as a mount container. Its last line was this:
expect((app.mount(svgContainer).$.subTree as ...).el instanceof SVGElement)There is no matcher on that expect. Vitest builds the expectation object, nothing calls .toBe(true) on it, and the boolean is thrown away. The test cannot fail.
When I added the matcher, the test failed. document.createElement('svg') does not make an SVG element. It makes an HTMLUnknownElement in the XHTML namespace, so the container was never SVG and neither was the g rendered into it. A test named for SVG mounting had never involved SVG. Creating the container with the SVG namespace fixed it, and Vue handled it correctly. The same PR added the missing matcher to a $watch test whose second line was the behaviour the test is named after. That is vuejs/core#16123, merged.
Here is the general pattern. A test that can never fail looks the same as a test that passes. The runner reports green, the file reads like a check, and a reviewer skims past it. Coverage does not help either: in the Vue test the mount ran, so every line of the code under test was covered. Only the check was missing.
I built dead-tests to find these, then ran it against projects I use. Below are the shapes it finds, each with a fix that went upstream.
A test the runner never collects
JUnit 3 ran any public void testFoo() in a class extending TestCase. JUnit 5 runs only methods annotated @Test. When a project migrates and changes its test base class in the same move, every method that did not get the annotation keeps compiling and never runs again.
jackson-databind had 27 of these across 12 files, left over from moving from a base class that extended TestCase to one that did not. Four classes ran no tests at all. Two of those were DeepJsonParsingTest and DeepJsonTreeTraversingTest, the guards for the nesting-depth limit, so StreamReadConstraints.maxNestingDepth had nothing exercising it. Adding the annotations took the suite from 7203 tests to 7230, and all 27 passed: the limit worked, the check on it was gone. That is jackson-databind#6266, merged.
Python has its own version: pytest collects by name, so a function missing its test_ prefix sits among real tests and never runs. pydantic had three that passed once collected (#13921, merged). It had four more that did not pass, because the behaviour they described had changed since they last ran. Those needed their expectations updated, which went in as #14165.
C# is the same again. A method missing [Test] or [TestMethod] compiles and runs nowhere. I checked on .NET 10 with one such method per framework: xUnit warns about it, and NUnit and MSTest print nothing.
A test replaced by its own name
In Python, a second def test_first_cache in the same file rebinds the name. pytest collects the later one, and the earlier one is gone without a warning.
fsspec had two of these pairs, and neither pair was a duplicate. In test_caches.py the shadowed test_first_cache was the detailed one, the test that checked the cache's hit and miss counters. So every counter assertion in that file was dead. Renaming the later definition put six tests back in the run, and they all passed. That is filesystem_spec#2192, merged.
An assertion after the call that throws
This one is easy to write and hard to see:
with pytest.raises(ValidationError) as exc_info:
validate(bad_input)
assert exc_info.value.errors() == [...]The call raises, the block exits, and the assert below it never runs. It could not have worked anyway, because pytest fills in exc_info.value only after the block exits.
pydantic had a batch of these. Moving the assertions out of the blocks showed three of them were wrong:
- an expected error was missing a
ctxkey the library now returns - an input meant to be JSON was not valid JSON, so it failed for a different reason
- an expected message had been copied from a neighbouring case
Fixed in #14201, merged.
cbor2 had six. Every one expected a message the library no longer emitted, and the suite passed. When I checked against the previous major version, the dead lines matched how 5.x behaved before the 6.0 rewrite. One of them assumed CBORDecodeEOF was a subclass of EOFError, which the type stubs still declare and which is no longer true at runtime. That is cbor2#363, merged. An assertion that never runs never gets updated, so it keeps describing the version where someone last wrote it.
An assertion that is never invoked
mermaid had a test for a linkStyle index past the last edge. The matcher was in the wrong place:
expect(() =>
flow.parser.parse(`...`).toThrow('The index 1 for linkStyle is out of bounds. ...')
);.toThrow is chained onto the parse result inside the arrow function, and expect(fn) with no matcher never calls fn. Neither the parse nor the check ran.
Moving the matcher out made the test fail, and this time the bug was in the product. The code had a guard that throws that message when typeof pos === 'number'. But the grammar passed the index through as the raw string from the lexer, so the guard could never match, and users got Cannot set properties of undefined (setting 'style') instead of the message telling them the valid range. The fix is one line in the grammar so the index is a number. That is mermaid#8675, open as I write this.
What happens when you revive one
Of the first 11 dead tests I revived, in four C# projects, 2 failed on their first run. A test that has not run in years has had nothing keeping it in step with the code. Small sample, but the lesson held everywhere after it: run the revived test before you file anything.
Most of the failures were stale expectations, like pydantic's three and cbor2's six. Those are worth fixing on their own, since the test now checks what the code does. A few were the code being wrong, like mermaid's unreachable guard. The test had the right expectation the whole time and a green tick over a real bug.
Run it
With Docker, against the current directory:
docker run --rm -v "$PWD:/src" ghcr.io/dylanpulver/dead-testsscan detects the languages in the tree and runs every check that applies. There are checks for Python, Java, Kotlin, C#, PHP, Ruby, Go, Rust and JS/TS, each a single stdlib-only file you can read in a sitting.
In CI, the GitHub Action flags dead tests inline on the pull request diff, only in files the PR changes, so an old finding never lands on someone else's PR:
on: pull_request
jobs:
dead-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: dylanpulver/dead-tests@v1It can also write SARIF for GitHub code scanning.
Limits
These are lead generators, not oracles. Read every hit.
- It is static. Nothing gets executed, so it cannot tell you whether a revived test passes. That is your next step.
- Some checks are text scans. The Python and Go checks parse the syntax tree. The Java, Kotlin, C#, PHP, Ruby, Rust and JS checks read lines, so unusual formatting can hide a finding or make a false one.
- False positives exist. A method with no annotation can be a case someone parked on purpose. A value compared to itself can be a determinism test on a property that recomputes. Each scanner's docstring lists the false-positive shapes it has met in real code and which ones it filters.
- It finds dead tests, not weak ones. A test that runs and asserts something too loose is out of scope.
The code is at github.com/dylanpulver/dead-tests. Point it at a repo you know well. If it finds something, I would like to hear what it was.