Mutation Gate Lab
What if your tests are green only because the code path was never exercised? Break a protection on purpose and find out whether the tests notice.
1. From a Git commit to a CI result
A repository holds source files and their Git history. A developer works in a branch or separate worktree, saves changes in a commit and pushes that branch to GitHub. GitHub Actions reads a workflow file (.github/workflows/tests.yml) and starts isolated runners to execute the configured jobs. A successful push is not a merge to main, and a green pipeline is not the same as independent approval.
Edit and test in a separate branch and worktree.
Record a particular version and publish it to a review branch.
GitHub's runners execute tests in new, controlled environments.
Interpret what passed, what was skipped, and what still needs human or independent review.
The full commit hash (SHA) identifies exactly which code was tested. The workflow is configured in version control; it is not an invisible or magical action by an AI model.
2. Real CI evidence: all five jobs green
This is a historical, source-pinned snapshot from a software project, tachylite-mcp, on . It is not a live dashboard. The underlying source repository and run evidence were private when recorded, so some source links require access.
Python 3.11 and 3.13 each reported 309 passed and 2 skipped. The 37 browser smoke tests bring the distinct passing outcomes to 346; do not add the two Python-version runs together as though they were different tests. The 73 structural anchor checks are a different control from the 73 executable mutation checks.
Evidence boundary: This is evidence for checks that actually ran on that commit. It does not establish that the app has no defects. A prior independent adversarial review found 11 serious issues even when earlier automated checks were green.
3. Why Mutation Gate exists
A normal assertion asks whether a feature produces the expected result. Mutation testing asks the harder question: would that assertion fail if we deliberately removed the protection that supposedly caused the result?
In this project's earlier testing, a path-traversal protection was deliberately removed, yet its test stayed green because a separate file-extension check blocked the test payload first. The test was not observing the protection it claimed to verify. That is a false-green test, not evidence that the removed guard worked.
The named test must run and pass against the original code.
A tiny targeted change disables one named protection in an isolated scratch copy.
A real assertion must fail in the test body for the intended property.
CAUGHT is good; ESCAPED is a test gap; BROKEN is inconclusive.
CAUGHT
Baseline passes; the intentionally damaged version fails its own assertion. The test can detect this specific fault.
ESCAPED
The damaged version stays green. The test missed the control, or another guard masked the problem.
BROKEN
Setup, anchoring or execution failed. It would be dishonest to count the result as a detected bug.
Before any mutation is scored, the harness checks that its scratch copy is actually used, that the source anchor matches exactly once, and that the edit genuinely changed the code. A setup ERROR or skipped test cannot masquerade as a CAUGHT assertion failure.
4. Interactive experiment: break one of 13 protections
These 13 scenarios map to the 13 new M1 mutation checks registered for the October 2026 repair. Choose a protection, read why it matters and switch between four teaching outcomes. The example code is simplified pseudocode; no code is modified in your browser or on GitHub.
Opt-in is required
What the real control is supposed to protect
Deliberate sabotage
Expected assertion
Before: representative protected logic
After: representative sabotaged logic
Illustrative pseudocode, not a verbatim source listing.
Real registry ID: . Named test: . The observed historical outcome of all 13 registered mutations was CAUGHT.
Choose what happens to the test
5. What the second review actually uncovered
Eleven serious findings across three adversarial lenses led to the Option-A repair and the 13 new mutations above. Some findings are different descriptions of the same root cause. They are not the same thing as 11 mutation tests.
Findings 2, 5 and 7: group ID values
The group key was validated but the internal group.id value was used in rendered handlers without equivalent validation. A new rule checks both values and their agreement.
Finding 3: group node lists
A collapsed group could display malformed group.nodes data as HTML. The input validator now requires a proper array of node IDs.
Findings 4 and 8: rollback was unreachable
Rendering happened before the try/catch. If drawing failed, the old graph was not restored. The repair moves risky rendering inside the recovery boundary.
Finding 6: lost unsaved edits
A pushed update closed the inspector and discarded what the user was still typing. The repair preserves the inspector when its node is unchanged.
Finding 9: shadowed test
A malformed ID test failed for a different reason than intended. The revised case isolates its ID-shape rule and requires a specific refusal reason.
Finding 10: incorrect test isolation claim
Additional file:// browser tests had the live bridge active. The harness now disconnects file-based tests and the changelog was corrected.
Findings 1 and 11: unresolved HTML sinks
Some node and edge text can still reach unsafe HTML insertion paths. This broader escaping sweep remains separate work, so live state mirroring is OFF by default.
Two kinds of evidence
Mutation Gate confirms a selected test detects selected sabotage. An independent code review searches for unknown paths, false assumptions and missing controls. Both are needed.
6. Quick challenge: when can you trust the result?
Choose an option and check your reasoning.
A non-vacuous test also proves a positive case works. For example, after rejecting malformed WebSocket messages it must still accept a valid message through the same connection. Five selected new tests were additionally shown to fail for the intended reasons against the earlier, unrepaired canvas.
7. What green CI cannot certify
The real mutation run recorded 73 CAUGHT, 0 ESCAPED, 0 BROKEN. That supports the named safety controls, not every possible input, interaction or security risk. In particular, the approximately 53 still-unsafe HTML insertion sites were not all removed by Option A, and enabling live mirroring remained a user opt-in with a warning.
The third independent security review is tracked separately. This lab freezes a dated learning example; its review status must not be mistaken for current deployment state. Passing CI, passing independent review, merging a PR and releasing a product are four different claims, each needing its own evidence.
8. Learn the vocabulary as you explore
Software-engineering terms in the explanatory paragraphs link to the existing From Blueprint to Application glossary. The glossary includes English definitions and plain-language Swedish explanations. Select a term below to jump straight to its entry.
9. Sources and next steps
Read the testing architecture lab for testing layers, then the AI evaluation and CI/CD lab for scheduling, budget and sampling tradeoffs.