Choose Evidence, Not a Shape

"What is our testing strategy?" sounds like one question.

It is at least four.

• How do we build confidence that the product is ready?
• At what boundary should executable code be tested?
• How do we judge output that is probabilistic rather than exact?
• When the tests themselves are generated, what establishes that they assert the right thing?

Treating all four as one pyramid produces familiar arguments: QA versus developers, unit versus integration, pyramid versus trophy, tests versus evals, "the AI wrote us 400 tests" versus "we still ship the same bugs." Most of them are arguments across different axes.

Note what is not on that list. Pyramid, trophy, and honeycomb are not a fifth question. They are a consequence — the shape you end up with once the four are answered. Picking the shape first and back-filling the tests to match it is the mistake this whole post is about.

Quality assurance is a discipline, not a department

The claim worth arguing with is not "QA is the layer above E2E." Almost nobody defends that out loud. The claim actually in circulation is quieter and much more common: QA is the testing team — a group that owns quality, receives the build, and returns a verdict.

That framing is what produces the org chart where developers write features, "QA" writes tests, and quality becomes a stage rather than a property.

ISTQB draws the distinction the framing loses. Testing is product-oriented and corrective — it examines the thing being built, and is a major form of quality control. Quality assurance is process-oriented and preventive: it improves how the work is done, so that fewer defects exist to find. And the Foundation Level syllabus (§1.2.2) adds the part the org chart drops — QA "applies to both the development and testing processes, and is the responsibility of everyone on a project." The standards body most often invoked to justify a QA department does not treat QA as a department. A team can staff testing entirely and still do no quality assurance; a team with no tester at all is doing quality assurance every time it sharpens an acceptance criterion before writing code.

The process work is unglamorous and mostly happens before anything is testable: clarifying acceptance criteria, examining risk, exploring behavior, assessing usability and accessibility, deciding what evidence a release needs, and learning from production. None of that becomes someone else's job by hiring a testing team, and the org chart that implies otherwise is transferring responsibility, not adding capability.

The question is not "do we have QA?" Every team does quality work, whether it names it or not.

The question is:

Which quality practices require human judgment or independent evidence, and what risk justifies their cost?

A small service with low blast radius, strong observability, and easy rollback may need acceptance examples and exploratory attention, but not an independent release gate. A payments flow, or anything with a regulator attached, may need formally independent evidence even when its automated tests are excellent.

Unit, integration, and E2E are three questions

The code-level taxonomy is simpler than the arguments about it. A unit test asks whether one piece of behavior works. An integration test asks whether this code interacts correctly with a real boundary. An E2E test asks whether a critical journey survives the whole system. Those are three different questions, and the labels matter far less than which question you are asking and which resources the test has to touch to answer it.

Unit: does the behavior work?

A unit is not necessarily one class.

It can be solitary: every collaborator is replaced. Or it can be sociable: lightweight collaborators run together. The second is often the better default. Mock a collaborator because it is slow, nondeterministic, expensive, unavailable, or produces side effects — not because a diagram contains a box boundary.

Thick domain logic usually benefits from many fast unit tests. Thin orchestration code often does not. A service that mostly maps HTTP to a database may reveal more with one narrow database integration test than with twenty mock-heavy service tests.

Integration: does the boundary work?

"Integration test" does not have to mean "start the whole platform."

A narrow integration test can exercise one real seam:

• a repository against the real database engine;
• serialization against the actual schema;
• an HTTP client against a controlled server;
• a provider against a consumer contract;
• framework configuration through the real request pipeline.

Keep unrelated boundaries fake or absent.

Broad integration becomes justified only when the defect emerges from the interaction of several real components and cannot be reproduced at a narrower seam.

E2E: does the journey work?

E2E is the hardest gate.

It proves something lower levels cannot: deployment, routing, identity, configuration, orchestration, and user flow working together. That can be valuable for a small number of critical journeys.

It is also slow, expensive to diagnose, sensitive to environment and data, and likely to become flaky.

The default is therefore not "no E2E forever." It is:

No E2E until you can name the journey and the failure that only E2E can expose.

A login-to-payment journey may justify one. A small internal CRUD tool may not.

A worked example

Everything above sits at a comfortable altitude. Here is what the accounting looks like with numbers on it.

This is a hypothetical, not a war story. There is no client and no project behind it; the figures are invented. What is not invented is the shape — a suite that grew by accretion, where every layer was added for a good reason and nobody ever priced the whole portfolio.

Take a B2B invoicing service. Its suite:

• 1,400 unit tests, most of them asserting that a service class called its mocked repository with the expected arguments;
• 40 integration tests, all against an in-memory database substitute;
• 120 browser E2E tests, one per screen, accumulated on the rule "every new page gets a smoke test."

CI takes 47 minutes, of which the E2E stage is 31. Roughly one run in fifteen fails without a code cause, so the team has learned to hit rerun before reading the log. Over a quarter, four defects reach production: two from queries that behaved differently against the real database engine than against the substitute, one from a serialization change that broke an older client, one from a payment provider changing a response field.

The interesting number is not 1,560 tests. It is zero — the number of those four escapes that any part of the suite was positioned to catch. The unit tests asserted against mocks that encoded the same wrong assumptions as the code. The integration tests exercised a database that was not the database. The E2E tests covered every screen and no boundary.

A rebalance that follows the failures rather than the diagram:

• keep 6 E2E journeys, each named after a business outcome (sign in, create invoice, take payment, refund, export, invite a colleague) and drop the other 114;
• move the 40 integration tests onto the real database engine and add ~25 more covering the query, migration, and serialization paths the escapes actually came through;
• add one consumer-driven contract test against the payment provider, which would have caught the fourth escape before deploy;
• delete ~300 unit tests that asserted call order against mocks, and keep every unit test that asserts an invoice calculation.

Total test count drops from 1,560 to about 1,170 — barely the headline. CI drops to roughly 12 minutes, the E2E stage to 5, and the rerun reflex loses its justification. More to the point, all four escapes now have a test pointed at the boundary they came through.

The move is not "fewer tests." It is evidence relocated to where the failures were. That reshaping happens to make the suite look less like a pyramid and more like a trophy, but nobody chose a trophy. The shape is the residue of the decisions, which is the only honest way to arrive at one.

The pyramid and trophy are not enemies

Mike Cohn called it the test automation pyramid in Succeeding with Agile (2009), and his layers were Unit, Service, and User Interface — a pyramid of automated tests, not of testing at large. The unit/integration/E2E version everyone recites is a later retelling. Cohn originated the shape; Martin Fowler's 2012 write-up, which credits him, is why anyone knows it. It remains a good default when most complexity sits in deterministic domain logic.

"Write tests. Not too many. Mostly integration." is Guillermo Rauch's, tweeted in December 2016. Kent C. Dodds published his own take on it a year later, and the Testing Trophy — Static, Unit, Integration, End-to-End, weighted toward integration — arrived in February 2018. He calls it "a general guide for the return on investment of the different forms of testing," governed by one principle: "The more your tests resemble the way your software is used, the more confidence they can give you." He leans on integration because tooling has invalidated the pyramid's assumption that higher-level tests are prohibitively slow and brittle — a good correction when most risk lies in how components, frameworks, browsers, and APIs interact.

Which is where the lineage gets funny: making that case, Dodds refers to "Martin's original Testing Pyramid." It is not Martin's, it is Cohn's, and Fowler wrote it up three years later. The substantive point stands — but even the person thinking hardest about the shape inherited its provenance second-hand.

Spotify's Testing Honeycomb — André Schaffer and Rickard Dybeck's "Testing of Microservices", January 2018 — makes a related point for microservices in a vocabulary that is nobody else's: Integrated Tests (the result depends on another live system), Integration Tests (one service tested in isolation at its interaction points), and Implementation Detail Tests (roughly, units). They advise focusing on Integration Tests, with "a few Implementation Detail Tests and even fewer Integrated Tests (ideally none)." Their "integration" is not the trophy's, and neither is the pyramid's "service" — three shapes, three incompatible vocabularies, which is itself evidence that these diagrams do not compose.

Contract testing adds another useful portfolio shape. When independently deployed services keep breaking one another, a consumer/provider contract is usually a better detector of interface drift than a broad E2E suite.

None of these diagrams is the strategy. Each is a summary of where the risk happened to land in the system its author was looking at — rich domain logic, thin glue over integrations that carry the real value, a service understood through the boundaries it owns, a fleet that has to stay compatible. They are observations about other people's risk profiles, which is exactly as much authority as they should have over yours.

Google's often-repeated 80/15/5 mix is explicitly a rough guideline, not a compliance target. A healthy portfolio follows the system. It does not force the system into a diagram.

Data, ML, and agents add another axis

Data pipelines still have code.

Their transformations can be unit tested. Their storage and orchestration can be integration tested. Their critical run can be exercised end to end.

They also fail because the data is late, malformed, duplicated, shifted, or inconsistent with an upstream contract. Those are data-quality and monitoring concerns, not merely another unit-test folder.

ML systems add model behavior, training-serving consistency, slice performance, leakage, reproducibility, and drift. The ML Test Score — Breck, Cai, Nielsen, Salib and Sculley's production-readiness rubric, published by Google at IEEE Big Data in 2017 — is explicit that it is an overlay and not a replacement: it "focuses on issues specific to ML systems, and so does not include generic software engineering best practices such as ensuring good unit test coverage and a well-defined binary release process. Such strategies remain necessary as well." Across the several dozen Google teams they surveyed, none of the rubric's tests was implemented by more than 80% of them.

LLM and agentic systems make the split even clearer. The deterministic scaffold around a model can be tested exactly, and should be: schema validation, tool dispatch, permission checks, state transitions, retry and backoff behavior, persistence, routing, and guardrails are all ordinary code with ordinary failure modes. The model output is the part that is probabilistic, and its quality is evaluated — against cases, metrics, rubrics, and thresholds — rather than asserted.

That is why a thin LLM service may need:

• unit tests for validation and mapping;
• one narrow HTTP or schema integration test;
• a small reviewed evaluation dataset.

It may not need a ceremonially complete pyramid, a browser E2E suite, or an LLM judge.

A model-based judge becomes useful only when the required quality is genuinely semantic, deterministic checks are insufficient, and the judge has been compared with human-reviewed cases. Temperature zero can reduce variation. It does not turn a model into a deterministic assertion engine.

AI-generated tests need an independent oracle

AI can generate a lot of tests quickly.

That is not the same as generating evidence.

A model that sees only the current implementation is likely to reproduce current behavior — including behavior that is wrong. It can also over-mock, assert implementation details, and produce impressive coverage with weak fault detection. This is the fourth question from the opening, and it is the one the older testing literature has the least to say about, because when the pyramid was drawn a test was expensive enough that someone had to want it.

The guardrail is simple:

A generated test needs a source of intended behavior outside the code it is testing.

That source can be a requirement, acceptance example, invariant, protocol, known-good case, reviewed golden file, or metamorphic relation. What it cannot be is the implementation, because then the test is a mirror and a mirror never disagrees with you.

Ask one review question:

What meaningful breakage would make this test fail?

If there is no clear answer, the test is coverage theater.

Where this framing breaks down

The wizard at the end of this post produces a recommendation. I would rather you argue with it than adopt it, so here is me arguing with the framing that generates it.

The case for a dedicated QA function is stronger than "regulated industries." The real argument is independence, and Glenford Myers made it in The Art of Software Testing in 1979: a programmer should avoid testing their own program, and an organization should not test its own programs. The rationale is not skill or diligence. Testing is destructive work performed by someone holding a constructive stake, and any misunderstanding of the spec propagates identically into the author's own tests — leaving the test blind in exactly the place the code is wrong. A tester who did not write the code, was not in the design meeting, and has no stake in the design being right asks the question the author has already unconsciously ruled out.

ISTQB (§1.5.3) is fairer than either pole, because it prices both sides: independence helps "due to differences between the author's and the tester's cognitive biases," but "independence is not a replacement for familiarity, e.g., developers can efficiently find many defects in their own code," and independent testers can end up isolated from the development team, in an adversarial relationship, with developers losing a sense of responsibility for quality. Its actual recommendation is several levels of independence running at once, not one team — a more interesting position than either pole.

In "Testing and Checking Refined," James Bach and Michael Bolton draw the line most "we don't need QA" arguments elide. Checking is "the mechanistic process of verifying propositions about the product," automatable by definition; testing is "the process of evaluating a product by learning about it through experiencing, exploring, and experimenting." An automated suite is a pile of checks, and it can only encode risks somebody already imagined. That argues for the skill being resourced, not necessarily for a separate reporting line.

"Everyone owns quality" has an unhappy failure mode: a responsibility diffused across a team under deadline pressure is a responsibility nobody has time for, and the exploratory pass is the first thing to go. I concede that fully. My objection is narrower than it usually sounds: I am against quality as a phase — work thrown over a wall at the end — not against the role. An independent tester embedded from the acceptance-criteria conversation onward is one of the higher-leverage people on a team. If your choice is between a QA department and nothing, take the department.

The case for a fat E2E suite is also better than it gets credit for. Every level below E2E tests a model of the system. The unit test asserts against your belief about a collaborator; the integration test asserts against one seam in isolation. Only E2E exercises the artifact that actually deploys, with the configuration that is actually live, over the network path the user actually takes.

The sharpest published version of that argument belongs to the company most often cited for the pyramid. Chapter 14 of Software Engineering at Google lists what unit tests structurally cannot reach: unfaithful doubles, which encode their author's misunderstanding of a dependency and go stale in silence when the real implementation changes; configuration, where "configuration changes are the number one reason for our major outages" — a failure class living entirely outside any unit-testable binary; behavior under load; unanticipated behavior, because "unit tests are limited by the imagination of the engineer writing them"; and behavior that only emerges across scopes. Two of those are the worked example above — mocks encoding the same wrong assumption as the code, and a database substitute that was not the database.

And for a whole class of products the integration surface is the product — a checkout stitching together five vendors, a browser extension living in someone else's DOM, a data platform that is mostly glue. Telling those teams that thin orchestration does not need many tests inverts the advice: the orchestration is the value. The standard objection to E2E is cost and flakiness, and it is worth being honest that those are engineering problems rather than laws of nature. Teams that invest in hermetic environments, deterministic test data, and diagnosable failures run large E2E suites that pay for themselves. Where I land: the cost is real and most teams never pay down the flake, so "few E2E tests" remains the right default — but the size of the E2E budget should scale with how much of your value lives in the seams, and for glue-shaped products that is most of it.

And the three axes are not as separable as I have made them sound. They leak into each other constantly. Whether you need an independent release gate depends on how good your automated tests are — axis one is a function of axis two. Evaluating a stochastic system is partly a quality practice, because a human reviews the golden set, and partly an executable test, because a threshold fails the build. An exploratory session that finds an unhandled empty state has done precisely the job a missing unit test would have done. So the axes are correlated, and I am not going to pretend otherwise. The claim I will defend is weaker than full orthogonality: these decisions have different owners, different cadences, and different failure modes, so budgeting them as a single line item is where teams go wrong. Separating them is a thinking tool, not an ontology. Like the shapes, it earns its place by being useful, and you should drop it the moment it stops being.

Choose evidence, not a shape

A testing strategy should say:

• which risks matter;
• which practice or test catches each one;
• what every layer costs;
• what is deliberately skipped;
• what future signal would reopen the decision.

Choose the smallest reliable evidence portfolio that covers the named risks.

Then draw the shape, if a shape is still useful.

Two ways to run the decision

This post ships alongside a new interactive wizard — the Testing Strategy Builder — where you can click through the decision yourself and walk out with a composable portfolio. You can also point it at a suite you already have, or just browse the pattern catalogue. Treat the result as a starting point to argue with, not a verdict. The section above is the argument I would start with.

The same reasoning is also a Claude Code skill. test-patterns is the fourth skill in arch-crew, added in 0.2.1 alongside decide-architecture, design-patterns, and agentic-patterns.

/plugin marketplace add AdamKrysztopa/architectural-decisions
/plugin install arch-crew

Repo: github.com/AdamKrysztopa/architectural-decisions