Reference · Software quality
Choose what to test — and what not to. QA is a discipline; unit, integration, and E2E answer different questions; data, ML, and agents add evaluation rather than replacing tests.
Testing strategy is usually argued as a single ratio — how much unit, how much E2E — as if every form of evidence sat on one ladder. It does not. Evidence about a software product comes from at least three different dimensions, and a real product needs some of each.
One dimension is quality practice — how a team understands, demonstrates and releases quality at all. A second is executable code tests — where automated checks observe deterministic behaviour, at the boundary that can actually fail. A third is stochastic evaluation — whether probabilistic output is good enough for the intended use. They are not tiers of one pyramid, they have different owners, and flattening them into one list is what produces the two most common failures: a team that automates its way past a requirements problem, and a team that treats "we have a QA person" as a test level.
Read this first — QA is not a test layer
QA describes how a team builds and demonstrates quality. Unit, integration, and E2E describe where executable tests observe deterministic behavior. Data, ML, and LLM systems add quality evaluation and monitoring. Select across these axes; do not force them into one pyramid.
| Dimension | Question | Typical evidence | Primary owners |
|---|---|---|---|
| Quality practice | Are we building the right thing, and is the release evidence sufficient? | Acceptance examples, exploration, usability/accessibility review, UAT, release criteria | Whole team; independent assurance where required |
| Executable code tests | Does deterministic code behave at the right boundary? | Unit, integration, E2E tests | Developers and delivery team |
| Stochastic evaluation | Is probabilistic output good enough for the intended use? | Cases, metrics, rubrics, thresholds, human review, monitoring | ML/agent engineers with domain stakeholders |
The spine of this page. Parts I, II and IV map onto these three rows in order.
Part IQuality practice
None of these is a test level, and none of them can be replaced by writing more tests. They cover ambiguity, human judgment, release evidence and process risk — the failure modes that survive a fully green suite.
Concrete, agreed examples of the expected behaviour, produced before the code is written.
Specification by example. The value is in the conversation that forces business, delivery and testing to state the same expectation in the same words — not in the tooling that later automates it. Teams that adopt the syntax and skip the conversation get a brittle automation layer and none of the benefit. Automate the examples at the cheapest level that can hold them; most belong in unit or narrow-integration tests, not in a browser.
Human investigation that learns about the product while designing and executing tests.
The only technique on this page that finds risks nobody wrote down — which is precisely why it cannot be replaced by automation, and why it is not a fallback for a missing suite. Run it under charters with time boxes and captured notes: that is the difference between exploration and clicking around. Findings that recur become examples, and examples become cheap automated checks.
The artifact that lets an accountable person say "ship it" — and defend the decision afterwards.
Release criteria written before the deadline are a quality practice; release criteria written during the incident review are theatre. This is where the answer to "how do we know it is good enough" gets recorded, along with what was deliberately not covered. It degenerates into ceremony the moment the criteria stop being named and the sign-off becomes a signature on a template.
A reviewer outside the delivery team assesses the evidence, not just the code.
The one place a separate QA function still clearly earns its keep: regulated domains, safety claims, contractual conformance — situations where the party who must accept the risk will not accept self-certification. It is expensive, it queues, and it invites the worst failure mode in this catalog, which is the delivery team outsourcing quality to it. If you cannot name the external party that requires it, you do not need it.
Treat the running system as the last and most honest test environment.
SLOs and error budgets, structured logs and traces, canaries, feature flags, and a rollback that actually works. Pre-release tests answer "does this behave"; production answers "does this behave under real data, real load and real users" — a question no environment reproduces. Where rollback is cheap and fast, buying observability is usually better value than buying another end-to-end test.
Part IIExecutable test levels
These are the primary executable levels. Each detects a different class of defect at a different price, and the level is chosen by which boundary can fail — never by a target ratio. Contract sits alongside them as a level in its own right because it tests a boundary nothing else reaches cheaply.
Fast evidence about a small behavior, invariant, calculation, or state transition.
The split matters more than the level does. A solitary unit test replaces every collaborator with a double; a sociable one lets real collaborators participate and isolates only what is slow, costly or nondeterministic. Sociable is the better default: solitary tests are the ones that harden into mock walls, where a passing suite proves the call sequence is unchanged rather than the behaviour is correct. That is also why unit count is a poor health metric — thousands of solitary tests over a thin domain assert almost nothing.
A test of one real boundary with unrelated dependencies controlled.
A real Postgres in a container and a real query. A real serializer and a real payload. A real HTTP client against a stubbed server. It answers the question a unit test structurally cannot: does this boundary behave the way the calling code assumes? A large share of defects blamed on end-to-end fragility are narrow-integration defects that were simply never tested at the seam — the E2E suite is only where they became visible, at the highest possible cost of diagnosis.
Several real components wired together, short of the full deployed system.
Justified only by a defect class that genuinely emerges from interaction — ordering, shared state, transaction boundaries spanning components, retry storms between two real services. It is not a default middle tier, and it is not "integration testing" as a category. The characteristic failure is ownership: broad tests are slow and ambiguous, so nobody wants to diagnose them, and a chronically red broad suite gets muted rather than fixed. Every failure it finds should be pushed down to the narrowest seam that can reproduce it, and the broad test then retired or kept only as a smoke check.
Executable compatibility evidence between independently evolving consumers and providers.
Consumer-driven contracts replace most of what cross-service end-to-end testing was supposed to deliver, at a fraction of the runtime and with an unambiguous answer to "who broke this". The trade is honest and worth naming: a contract proves the interface still matches, and proves nothing at all about whether the journey through those services works.
A test of a critical journey through production-like system wiring.
The only level that exercises deployment, routing, identity and orchestration together — and the only one whose cost keeps growing after you write it, in runtime, environment ownership, test data and flake triage. It is the most expensive evidence in the catalog per defect found, which makes it a deliberate purchase, not a tier to fill.
Gate hard. Add a small number only when a named critical journey exposes a failure that cheaper boundaries cannot.
| Level | Best at detecting | Main cost | Hard gate |
|---|---|---|---|
| Unit | Logic, invariants, calculations, state transitions | Coupling to design when over-isolated | Meaningful local behavior exists |
| Narrow integration | Query, protocol, serialization, framework, real dependency behavior | Environment and dependency setup | A real boundary can fail despite correct local logic |
| Broad integration | Multi-component interaction defects | Runtime, ownership, diagnosis | Failure cannot be reproduced at a narrower seam |
| E2E | Full journey, deployment, routing, identity, orchestration | Highest runtime and flakiness | A named critical failure is visible only through the whole system |
Read the last column as a precondition, not a suggestion. No gate met, no test at that level.
Part IIIPortfolio shapes
A shape is the consequence of selected risks and boundaries. It is not the input to the decision.
These names are useful for exactly one thing: recognising the distribution you already have, so you can argue about whether the risks justify it. Adopting a shape first and then filling it is how teams end up with thousands of tests that assert nothing and three that everybody fears.
Many unit tests, fewer integration, very few end-to-end.
The shape that falls out of a codebase with substantial in-process logic and stable, well-understood boundaries — algorithms, domain rules, calculations. It stops being appropriate the moment most of the risk lives at I/O boundaries, because then the unit mass is testing glue code around the part that actually breaks.
Integration-weighted, with static checks below and a thin end-to-end cap.
The shape of a thin-logic service where framework, serialization, persistence and wiring carry most of the behaviour — which describes a large share of modern web and API work. The cost is real: a slower suite that demands hermetic environments to stay trustworthy.
Few solitary unit tests, a heavy band of narrow-integration tests, very few end-to-end.
Effectively the trophy with an explicit anti-mocking stance, and the usual outcome in service-heavy systems where a service's job is to coordinate other things. If most of your classes only make sense with a real collaborator attached, this is the shape you already have.
The mass of cross-system evidence sits in consumer-driven contracts.
The shape of an estate with many independently deployed services. It buys most of what cross-service end-to-end promised — drift detection, compatibility, a named owner for every break — at a fraction of the runtime. It still needs a small number of real journeys, because a full set of green contracts does not mean the flow works.
An ordinary shape underneath, plus golden sets, rubrics, thresholds and monitoring over the stochastic part.
Not a fourth shape competing with the others — a layer added on top of whichever shape the deterministic code already has. The mistake this name exists to prevent is treating evaluation as a replacement for tests, which leaves the scaffolding around the model completely uncovered.
Warning shape — the mass of testing sits at the end-to-end and manual level, with little underneath.
Almost never chosen; it accumulates. Each individual decision was reasonable — a bug appeared in the browser, someone added a browser test — and the aggregate is a suite that is slow, flaky, expensive to own, and diagnosed by guesswork. The repair is not "write more unit tests": take each recurring failure and push it down to the cheapest boundary that reproduces it, then delete the E2E test that was standing in for it.
Warning shape — heavy unit coverage and heavy end-to-end coverage, with a hollow middle.
Usually the result of a team that believes in unit tests and a separate team that believes in end-to-end tests, with nobody owning the seams in between. The defects that live at those seams — queries, protocols, serialization, real dependency behaviour — still have to surface somewhere, so they surface in the end-to-end suite at maximum diagnostic cost.
Part IVDomain overlays
Everything in this Part is an overlay on top of ordinary tests, never a substitute for them. A data pipeline still has transformation functions worth unit-testing. An LLM system still has deterministic scaffolding — prompt assembly, tool schemas, parsing, routing, retries, budget limits — that must be tested exactly like any other code. The overlay covers only the part whose output cannot be predicted exactly; if you can assert it deterministically, that is not an evaluation problem, it is a test you have not written.
Unit tests for transformation logic: small handcrafted input frames, explicitly expected output.
The cheapest evidence in any pipeline and the most frequently skipped, because "it needs the cluster" is easier to say than to check. Join fan-out, null handling, timezone arithmetic, deduplication and aggregation errors are all reproducible in a ten-row frame on a laptop.
Runtime checks on the data itself — nullability, ranges, uniqueness, referential integrity, row-count deltas.
These catch the failure class no code test can ever see, because the code did not change: the upstream did. They run against real data in the real pipeline, which makes them the pipeline equivalent of production feedback rather than of a test.
A declared, versioned agreement on schema and semantics between producer and consumer, enforced in production.
The data-plane counterpart of a service contract. It converts silent downstream corruption — the worst failure mode in data work, because it is discovered weeks later in a report — into a named, producer-side violation at the moment of publication. A contract with no enforcement point is documentation.
Metric thresholds, slice performance, invariance and directional-expectation tests on a trained model.
Aggregate accuracy is the number that hides the regression that matters: the slice where the model got worse while the headline went up. Behavioural evaluation names the slices and the expectations in advance. This is also where leakage and reproducibility checks belong — a model that scores well because the target leaked into a feature will pass every ordinary test you own.
Ordinary tests for everything around the model — prompt assembly, tool schemas, parsing, routing, retries, budget limits.
Most agent defects are not model defects. They are a tool schema that does not match the function signature, a parser that breaks on a trailing comma, a retry that loops without a budget, a router that silently falls through. All of it is deterministic, all of it is testable with the model call stubbed at a seam, and it is by far the cheapest evidence in an agentic system. Teams that jump straight to evaluation skip this and then blame the model.
A reviewed set of representative cases used to measure stochastic system behavior over time.
The regression suite for output that cannot be asserted exactly. Its entire value rests on representativeness — a golden set assembled from the cases that were easy to collect measures how well the system handles easy cases.
A model applies a rubric to score another model's output.
The last resort, not the first tool. It is a stochastic measuring instrument pointed at a stochastic system, so it needs its own calibration against human-reviewed cases and its own version pinning — otherwise a judge upgrade silently moves every score you have. Most properties teams reach for a judge to check (valid JSON, required fields present, citation resolves, length and format constraints) are deterministic and belong in an ordinary assertion.
Part VTechniques
Nothing in this Part is a peer of unit, integration or E2E, and none of it belongs on a pyramid. A technique is how you write a test at whichever level you already chose: property-based testing can be applied to a unit or to a narrow-integration test; test doubles can appear at any level; hermetic environments are a property of the environment, not a tier. The same goes for static checks — types, linters, formatters. They are extremely cheap and worth having, and they are not a test level either. Choose the level first from Part II, then pick the technique that makes that test worth reading.
One test body, many cases — a table of inputs and expected results.
The default for any behaviour that must hold across a known set of inputs. The discipline is naming: each case must fail with a label that identifies it, or the table becomes a debugging tax.
Assert an invariant, then let the framework generate inputs that try to break it.
Explores an input space no human enumerates, and shrinks any failure to a minimal reproducing case — which is often the most valuable output. The hard part is that a property is a genuine design artefact: round-trip, idempotence, ordering, conservation, monotonicity. A vague property tests nothing while looking rigorous.
Pin down what legacy code currently does, before you change it.
A scaffold, not an asset. It makes an otherwise unsafe refactor safe without first understanding the code — and it encodes the existing bugs as expectations while doing so. Delete them as the real behaviour becomes specifiable; maintaining them indefinitely means you have promoted the bugs to requirements.
Compare large structured output against a reviewed, committed baseline.
Cheap coverage of outputs too big to assert field by field — rendered documents, API payloads, generated configuration. It only works while a human genuinely reviews the diff, and the tooling makes not reviewing it a single keystroke.
Introduce deliberate faults and check whether the suite notices.
The only honest measure of assertion strength on this page. Line coverage measures execution; mutation score measures detection, which is the thing anybody actually wanted to know. It is also expensive and noisy with equivalent mutants, so it is a diagnostic you run occasionally on the code that matters — not a metric to put in a pipeline gate.
Stubs, fakes, spies and mocks — controlled stand-ins for real collaborators.
Use them for cost, nondeterminism and irreversible side effects; do not use them to avoid understanding a dependency. Prefer a fake — a working in-memory implementation — over a mock — a scripted expectation. A mock asserts the call sequence, which is the design; when the design changes for good reasons, the test fails for no reason. That is the mechanism by which a mock wall makes refactoring impossible.
Builders and object mothers that produce valid test data with only the relevant fields named.
Setup noise is the main reason tests stop being readable, and readability is the main reason they stop being maintained. A factory lets a test state its intent — this order is unpaid — instead of listing twelve fields that do not matter. The counter-risk is a single shared fixture everything depends on, which is coupling wearing a helpful hat.
Every run starts from the same known state, with no shared or external mutable dependency.
Containers per run, seeded databases, frozen clocks, pinned random seeds, no shared staging environment, no test that depends on another test's leftovers. Most of what teams call flaky tests is not a test problem at all — it is a non-hermetic environment, and no amount of retrying fixes it.
Temporary isolation of a nondeterministic test with a named owner, reason, expiry, and repair plan.
All four attributes are load-bearing. Without an owner and an expiry this is not quarantine, it is deletion with extra ceremony — and a quarantine list that only grows is a suite that has already stopped being a gate.
| Technique | Use it for | Warning |
|---|---|---|
| Property-based | Invariants over a wide input space | Poor properties produce false confidence |
| Characterization | Capturing legacy behavior before change | Captured behavior is not automatically desired behavior |
| Snapshot/approval | Reviewing large structured output | Unreviewed updates become rubber stamps |
| Mutation | Checking whether assertions detect deliberate faults | Fix basic reliability before optimizing mutation score |
| Contract | Compatibility across independently deployed boundaries | Does not prove the full journey |
| Test doubles | Controlling cost, nondeterminism, or side effects | Mock walls test implementation rather than behavior |
| Golden evaluation set | Regression evidence for stochastic systems | Dataset quality and representativeness dominate |
Every row above answers "how", not "where". The level was already decided in Part II.
Part VIDecide
Start with the failure you need to catch. Every broader layer requires a force. The result explains what was selected, what it costs, what was skipped, and what would reopen the decision.
The honest takeaway: "choosing a testing strategy" almost never means picking a shape. It means selecting evidence on three independent dimensions, and being able to say out loud what you deliberately did not buy.
A typical modern service ends up with acceptance examples for the contested requirements and production feedback for everything that only real traffic reveals; sociable unit tests where real logic exists and narrow-integration tests at each boundary that can fail on its own; contract tests wherever another team deploys independently; and a very small number of end-to-end journeys — each one attached to a named critical failure that nothing cheaper could expose. If there is a model or an agent in it, that whole portfolio stays, and an evaluation overlay is added on top for the part of the output that cannot be asserted exactly.
Notice what is missing from that list, deliberately: no broad-integration tier by default, no independent QA gate unless an external party requires one, no mutation tool until the suite is already trusted, and no model judge for anything a deterministic assertion can check. A portfolio that cannot name its exclusions has not made any decisions. Treat the result as a starting point to argue with, not a verdict.