mechai · arch-crew← Blog

Reference · Software quality

Test Patterns

Choose what to test — and what not to. QA is a discipline; unit, integration, and E2E answer different questions; data, ML, and agents add evaluation rather than replacing tests.

Quality practice · executable levels · data, ML & agent evaluation

Three dimensions, not one list

Testing strategy is usually argued as a single ratio — how much unit, how much E2E — as if every form of evidence sat on one ladder. It does not. Evidence about a software product comes from at least three different dimensions, and a real product needs some of each.

One dimension is quality practice — how a team understands, demonstrates and releases quality at all. A second is executable code tests — where automated checks observe deterministic behaviour, at the boundary that can actually fail. A third is stochastic evaluation — whether probabilistic output is good enough for the intended use. They are not tiers of one pyramid, they have different owners, and flattening them into one list is what produces the two most common failures: a team that automates its way past a requirements problem, and a team that treats "we have a QA person" as a test level.

Read this first — QA is not a test layer

QA describes how a team builds and demonstrates quality. Unit, integration, and E2E describe where executable tests observe deterministic behavior. Data, ML, and LLM systems add quality evaluation and monitoring. Select across these axes; do not force them into one pyramid.

Dimension Question Typical evidence Primary owners
Quality practice Are we building the right thing, and is the release evidence sufficient? Acceptance examples, exploration, usability/accessibility review, UAT, release criteria Whole team; independent assurance where required
Executable code tests Does deterministic code behave at the right boundary? Unit, integration, E2E tests Developers and delivery team
Stochastic evaluation Is probabilistic output good enough for the intended use? Cases, metrics, rubrics, thresholds, human review, monitoring ML/agent engineers with domain stakeholders

The spine of this page. Parts I, II and IV map onto these three rows in order.

Part IQuality practice

How a team builds and demonstrates quality

None of these is a test level, and none of them can be replaced by writing more tests. They cover ambiguity, human judgment, release evidence and process risk — the failure modes that survive a fully green suite.

1.1Acceptance examples

Concrete, agreed examples of the expected behaviour, produced before the code is written.

Specification by example. The value is in the conversation that forces business, delivery and testing to state the same expectation in the same words — not in the tooling that later automates it. Teams that adopt the syntax and skip the conversation get a brittle automation layer and none of the benefit. Automate the examples at the cheapest level that can hold them; most belong in unit or narrow-integration tests, not in a browser.

StrengthsRemoves ambiguity before implementation; gives one shared vocabulary; the examples double as regression evidence.
CostsWorkshop time across roles; a rotting automation layer if the examples are automated at the wrong level.
Use whenRequirements are contested, or "done" keeps being renegotiated after delivery.

1.2Exploratory testing

Human investigation that learns about the product while designing and executing tests.

The only technique on this page that finds risks nobody wrote down — which is precisely why it cannot be replaced by automation, and why it is not a fallback for a missing suite. Run it under charters with time boxes and captured notes: that is the difference between exploration and clicking around. Findings that recur become examples, and examples become cheap automated checks.

StrengthsFinds interactions, usability problems, and unknown risks that scripted checks did not anticipate.
CostsSkilled time, weaker repeatability, and the need to capture findings.
Use whenBehavior is ambiguous, user interaction matters, or defect risk extends beyond known examples.

1.3User acceptance and release evidence

The artifact that lets an accountable person say "ship it" — and defend the decision afterwards.

Release criteria written before the deadline are a quality practice; release criteria written during the incident review are theatre. This is where the answer to "how do we know it is good enough" gets recorded, along with what was deliberately not covered. It degenerates into ceremony the moment the criteria stop being named and the sign-off becomes a signature on a template.

StrengthsMakes the release decision explicit and auditable; forces criteria to exist before the deadline creates them.
CostsCoordination, documentation, release latency; becomes ceremony when the criteria are unnamed.
Use whenSomeone outside the delivery team must accept the change, or the release has to be defensible later.

1.4Independent assurance

A reviewer outside the delivery team assesses the evidence, not just the code.

The one place a separate QA function still clearly earns its keep: regulated domains, safety claims, contractual conformance — situations where the party who must accept the risk will not accept self-certification. It is expensive, it queues, and it invites the worst failure mode in this catalog, which is the delivery team outsourcing quality to it. If you cannot name the external party that requires it, you do not need it.

StrengthsBreaks author bias; satisfies parties who will not accept the delivery team's own word.
CostsHeadcount, handoffs, queueing, and the standing temptation to make it the only quality gate.
Use whenRegulation, contract, or safety requires an assessment the delivery team cannot self-issue. Otherwise skip it, explicitly.

1.5Production feedback

Treat the running system as the last and most honest test environment.

SLOs and error budgets, structured logs and traces, canaries, feature flags, and a rollback that actually works. Pre-release tests answer "does this behave"; production answers "does this behave under real data, real load and real users" — a question no environment reproduces. Where rollback is cheap and fast, buying observability is usually better value than buying another end-to-end test.

StrengthsCatches what no pre-release environment reproduces; often removes the need for expensive broad tests.
CostsInstrumentation, alert ownership, and a real rollback path; worthless without someone who acts on the signal.
Use whenNearly always — and preferentially over broad tests when rollback is cheap and blast radius is contained.

Part IIExecutable test levels

Where automated tests observe deterministic behaviour

These are the primary executable levels. Each detects a different class of defect at a different price, and the level is chosen by which boundary can fail — never by a target ratio. Contract sits alongside them as a level in its own right because it tests a boundary nothing else reaches cheaply.

2.1Unit — solitary or sociable

Fast evidence about a small behavior, invariant, calculation, or state transition.

The split matters more than the level does. A solitary unit test replaces every collaborator with a double; a sociable one lets real collaborators participate and isolates only what is slow, costly or nondeterministic. Sociable is the better default: solitary tests are the ones that harden into mock walls, where a passing suite proves the call sequence is unchanged rather than the behaviour is correct. That is also why unit count is a poor health metric — thousands of solitary tests over a thin domain assert almost nothing.

StrengthsFast feedback, precise diagnosis, broad input coverage.
CostsCan couple to implementation and encourage mock-heavy designs.
Use whenMeaningful logic can be exercised without a costly environment.

2.2Narrow integration

A test of one real boundary with unrelated dependencies controlled.

A real Postgres in a container and a real query. A real serializer and a real payload. A real HTTP client against a stubbed server. It answers the question a unit test structurally cannot: does this boundary behave the way the calling code assumes? A large share of defects blamed on end-to-end fragility are narrow-integration defects that were simply never tested at the seam — the E2E suite is only where they became visible, at the highest possible cost of diagnosis.

StrengthsHigh fidelity at the seam, better defect localization than broad tests.
CostsDependency setup, slower execution, lifecycle management.
Use whenQuery, protocol, serialization, framework, or real dependency behavior matters.

2.3Broad integration

Several real components wired together, short of the full deployed system.

Justified only by a defect class that genuinely emerges from interaction — ordering, shared state, transaction boundaries spanning components, retry storms between two real services. It is not a default middle tier, and it is not "integration testing" as a category. The characteristic failure is ownership: broad tests are slow and ambiguous, so nobody wants to diagnose them, and a chronically red broad suite gets muted rather than fixed. Every failure it finds should be pushed down to the narrowest seam that can reproduce it, and the broad test then retired or kept only as a smoke check.

StrengthsCatches interaction defects that no single component exhibits on its own.
CostsRuntime, unclear ownership, slow and ambiguous diagnosis.
Use whenA failure demonstrably cannot be reproduced at a narrower seam. Never as a default tier.

2.4Contract

Executable compatibility evidence between independently evolving consumers and providers.

Consumer-driven contracts replace most of what cross-service end-to-end testing was supposed to deliver, at a fraction of the runtime and with an unambiguous answer to "who broke this". The trade is honest and worth naming: a contract proves the interface still matches, and proves nothing at all about whether the journey through those services works.

StrengthsFinds interface drift earlier and more cheaply than broad E2E.
CostsContract ownership, versioning, broker or workflow complexity.
Use whenSeparately deployed systems or teams share an interface.

2.5End-to-end journey

A test of a critical journey through production-like system wiring.

The only level that exercises deployment, routing, identity and orchestration together — and the only one whose cost keeps growing after you write it, in runtime, environment ownership, test data and flake triage. It is the most expensive evidence in the catalog per defect found, which makes it a deliberate purchase, not a tier to fill.

Gate hard. Add a small number only when a named critical journey exposes a failure that cheaper boundaries cannot.

StrengthsCovers deployment, routing, identity, orchestration, and user flow together.
CostsRuntime, flakiness, environment ownership, data, and difficult diagnosis.
Use whenA named critical failure cannot be exposed reliably at a cheaper boundary.
Level Best at detecting Main cost Hard gate
Unit Logic, invariants, calculations, state transitions Coupling to design when over-isolated Meaningful local behavior exists
Narrow integration Query, protocol, serialization, framework, real dependency behavior Environment and dependency setup A real boundary can fail despite correct local logic
Broad integration Multi-component interaction defects Runtime, ownership, diagnosis Failure cannot be reproduced at a narrower seam
E2E Full journey, deployment, routing, identity, orchestration Highest runtime and flakiness A named critical failure is visible only through the whole system

Read the last column as a precondition, not a suggestion. No gate met, no test at that level.

Part IIIPortfolio shapes

Shapes are an output, not an input

A shape is the consequence of selected risks and boundaries. It is not the input to the decision.

These names are useful for exactly one thing: recognising the distribution you already have, so you can argue about whether the risks justify it. Adopting a shape first and then filling it is how teams end up with thousands of tests that assert nothing and three that everybody fears.

3.1Pyramid-like

Many unit tests, fewer integration, very few end-to-end.

The shape that falls out of a codebase with substantial in-process logic and stable, well-understood boundaries — algorithms, domain rules, calculations. It stops being appropriate the moment most of the risk lives at I/O boundaries, because then the unit mass is testing glue code around the part that actually breaks.

Emerges whenReal logic lives in-process and the boundaries are stable and well understood.
Fails whenMost risk is at I/O; the broad unit base then proves very little.

3.2Trophy-like

Integration-weighted, with static checks below and a thin end-to-end cap.

The shape of a thin-logic service where framework, serialization, persistence and wiring carry most of the behaviour — which describes a large share of modern web and API work. The cost is real: a slower suite that demands hermetic environments to stay trustworthy.

Emerges whenBehaviour lives in framework, persistence and serialization rather than in domain logic.
Fails whenEnvironments are not hermetic — the suite becomes slow and flaky at the same time.

3.3Honeycomb-like

Few solitary unit tests, a heavy band of narrow-integration tests, very few end-to-end.

Effectively the trophy with an explicit anti-mocking stance, and the usual outcome in service-heavy systems where a service's job is to coordinate other things. If most of your classes only make sense with a real collaborator attached, this is the shape you already have.

Emerges whenServices mostly coordinate other components; isolating them requires mocks that assert nothing.
Fails whenThere is genuine algorithmic logic being starved of cheap unit coverage.

3.4Contract-centered

The mass of cross-system evidence sits in consumer-driven contracts.

The shape of an estate with many independently deployed services. It buys most of what cross-service end-to-end promised — drift detection, compatibility, a named owner for every break — at a fraction of the runtime. It still needs a small number of real journeys, because a full set of green contracts does not mean the flow works.

Emerges whenMany teams deploy independently against shared interfaces.
Fails whenContracts are treated as journey coverage; the flow still needs one or two real E2E tests.

3.5Evaluation-heavy overlay

An ordinary shape underneath, plus golden sets, rubrics, thresholds and monitoring over the stochastic part.

Not a fourth shape competing with the others — a layer added on top of whichever shape the deterministic code already has. The mistake this name exists to prevent is treating evaluation as a replacement for tests, which leaves the scaffolding around the model completely uncovered.

Emerges whenPart of the output is probabilistic and cannot be asserted exactly.
Fails whenEvaluation displaces ordinary tests instead of sitting on top of them.

3.6Ice-cream cone Warning

Warning shape — the mass of testing sits at the end-to-end and manual level, with little underneath.

Almost never chosen; it accumulates. Each individual decision was reasonable — a bug appeared in the browser, someone added a browser test — and the aggregate is a suite that is slow, flaky, expensive to own, and diagnosed by guesswork. The repair is not "write more unit tests": take each recurring failure and push it down to the cheapest boundary that reproduces it, then delete the E2E test that was standing in for it.

WarningSlowest feedback, highest flake rate, worst diagnosis, and the highest cost per defect found.
RepairPush each recurring failure to the narrowest seam that reproduces it, then retire the E2E test it replaced.

3.7Hourglass Warning

Warning shape — heavy unit coverage and heavy end-to-end coverage, with a hollow middle.

Usually the result of a team that believes in unit tests and a separate team that believes in end-to-end tests, with nobody owning the seams in between. The defects that live at those seams — queries, protocols, serialization, real dependency behaviour — still have to surface somewhere, so they surface in the end-to-end suite at maximum diagnostic cost.

WarningBoundary defects are paid for at E2E prices because nothing tests the boundary directly.
RepairBuild narrow-integration tests at exactly the seams the E2E suite keeps failing on.

Part IVDomain overlays

Evaluation adds a layer; it does not remove one

Everything in this Part is an overlay on top of ordinary tests, never a substitute for them. A data pipeline still has transformation functions worth unit-testing. An LLM system still has deterministic scaffolding — prompt assembly, tool schemas, parsing, routing, retries, budget limits — that must be tested exactly like any other code. The overlay covers only the part whose output cannot be predicted exactly; if you can assert it deterministically, that is not an evaluation problem, it is a test you have not written.

4.1Data transformation tests

Unit tests for transformation logic: small handcrafted input frames, explicitly expected output.

The cheapest evidence in any pipeline and the most frequently skipped, because "it needs the cluster" is easier to say than to check. Join fan-out, null handling, timezone arithmetic, deduplication and aggregation errors are all reproducible in a ten-row frame on a laptop.

StrengthsFast and precise, no cluster required; catches fan-out, null, timezone and aggregation defects at their source.
CostsFixture curation; tiny frames say nothing about scale, skew or performance.
Use whenThe transformation has any non-trivial logic — which is nearly always.

4.2Data-quality assertions

Runtime checks on the data itself — nullability, ranges, uniqueness, referential integrity, row-count deltas.

These catch the failure class no code test can ever see, because the code did not change: the upstream did. They run against real data in the real pipeline, which makes them the pipeline equivalent of production feedback rather than of a test.

StrengthsDetects upstream change that leaves every code test green.
CostsThreshold tuning and alert fatigue; every check needs a named owner or it gets muted.
Use whenThe pipeline consumes data produced by someone you do not control.

4.3Data contracts and monitoring

A declared, versioned agreement on schema and semantics between producer and consumer, enforced in production.

The data-plane counterpart of a service contract. It converts silent downstream corruption — the worst failure mode in data work, because it is discovered weeks later in a report — into a named, producer-side violation at the moment of publication. A contract with no enforcement point is documentation.

StrengthsMoves breakage from silent downstream corruption to an attributable producer-side violation.
CostsGovernance, versioning, and a real enforcement point that can block a publish.
Use whenProducers and consumers ship independently, especially across team boundaries.

4.4ML behavioral evaluation

Metric thresholds, slice performance, invariance and directional-expectation tests on a trained model.

Aggregate accuracy is the number that hides the regression that matters: the slice where the model got worse while the headline went up. Behavioural evaluation names the slices and the expectations in advance. This is also where leakage and reproducibility checks belong — a model that scores well because the target leaked into a feature will pass every ordinary test you own.

StrengthsTurns "the model looks fine" into a release criterion; exposes slice regressions that aggregates conceal.
CostsLabeled data, slice definition, threshold negotiation, and retraining noise to distinguish from real change.
Use whenA model's output drives a decision that someone will have to defend.

4.5Deterministic LLM/agent scaffold tests

Ordinary tests for everything around the model — prompt assembly, tool schemas, parsing, routing, retries, budget limits.

Most agent defects are not model defects. They are a tool schema that does not match the function signature, a parser that breaks on a trailing comma, a retry that loops without a budget, a router that silently falls through. All of it is deterministic, all of it is testable with the model call stubbed at a seam, and it is by far the cheapest evidence in an agentic system. Teams that jump straight to evaluation skip this and then blame the model.

StrengthsCheap, fast and fully deterministic; catches the majority of real agent defects.
CostsAlmost none — provided there is a seam where the model call can be replaced.
Use whenAlways, in any LLM or agentic system. Before any evaluation work.

4.6Golden evaluation datasets

A reviewed set of representative cases used to measure stochastic system behavior over time.

The regression suite for output that cannot be asserted exactly. Its entire value rests on representativeness — a golden set assembled from the cases that were easy to collect measures how well the system handles easy cases.

StrengthsMakes qualitative regression visible and repeatable enough for release decisions.
CostsCuration, labeling, representativeness, and maintenance.
Use whenExact assertions cannot capture output quality.

4.7Model-based judge

A model applies a rubric to score another model's output.

The last resort, not the first tool. It is a stochastic measuring instrument pointed at a stochastic system, so it needs its own calibration against human-reviewed cases and its own version pinning — otherwise a judge upgrade silently moves every score you have. Most properties teams reach for a judge to check (valid JSON, required fields present, citation resolves, length and format constraints) are deterministic and belong in an ordinary assertion.

StrengthsScales semantic review beyond simple deterministic metrics.
CostsModel cost, latency, nondeterminism, bias, calibration work, and judge version drift.
Use whenQuality is semantic, simpler evaluators are insufficient, and human-reviewed calibration cases exist.
Skip whenThe property can be checked deterministically.

Part VTechniques

Techniques cut across levels — they are not levels

Nothing in this Part is a peer of unit, integration or E2E, and none of it belongs on a pyramid. A technique is how you write a test at whichever level you already chose: property-based testing can be applied to a unit or to a narrow-integration test; test doubles can appear at any level; hermetic environments are a property of the environment, not a tier. The same goes for static checks — types, linters, formatters. They are extremely cheap and worth having, and they are not a test level either. Choose the level first from Part II, then pick the technique that makes that test worth reading.

5.1Parameterized examples

One test body, many cases — a table of inputs and expected results.

The default for any behaviour that must hold across a known set of inputs. The discipline is naming: each case must fail with a label that identifies it, or the table becomes a debugging tax.

StrengthsBroad case coverage with no duplicated test code; new cases cost one line.
CostsOver-stuffed tables become unreadable; unnamed cases are painful to diagnose.
Use whenThe same behaviour must hold across an enumerable set of inputs.

5.2Property-based tests

Assert an invariant, then let the framework generate inputs that try to break it.

Explores an input space no human enumerates, and shrinks any failure to a minimal reproducing case — which is often the most valuable output. The hard part is that a property is a genuine design artefact: round-trip, idempotence, ordering, conservation, monotonicity. A vague property tests nothing while looking rigorous.

StrengthsCovers input space nobody would enumerate; shrinks failures to a minimal case.
CostsSlower runs, seed-dependent failures to triage, and genuinely hard property design.
WarningPoor properties produce false confidence.
Use whenA real invariant exists — round-trips, idempotence, ordering, conservation.

5.3Characterization tests

Pin down what legacy code currently does, before you change it.

A scaffold, not an asset. It makes an otherwise unsafe refactor safe without first understanding the code — and it encodes the existing bugs as expectations while doing so. Delete them as the real behaviour becomes specifiable; maintaining them indefinitely means you have promoted the bugs to requirements.

StrengthsMakes an unsafe refactor safe without first understanding the code.
CostsEncodes current bugs as expectations; must be replaced, not maintained.
WarningCaptured behavior is not automatically desired behavior.
Use whenChanging code that nobody can currently specify.

5.4Snapshots and approval tests

Compare large structured output against a reviewed, committed baseline.

Cheap coverage of outputs too big to assert field by field — rendered documents, API payloads, generated configuration. It only works while a human genuinely reviews the diff, and the tooling makes not reviewing it a single keystroke.

StrengthsVery cheap coverage of large structured output; diffs are readable in review.
CostsDiff noise on unrelated change; the update command is one keystroke from erasing the assertion.
WarningUnreviewed updates become rubber stamps.
Use whenOutput is large, structured, and a human can meaningfully review a diff of it.

5.5Mutation testing

Introduce deliberate faults and check whether the suite notices.

The only honest measure of assertion strength on this page. Line coverage measures execution; mutation score measures detection, which is the thing anybody actually wanted to know. It is also expensive and noisy with equivalent mutants, so it is a diagnostic you run occasionally on the code that matters — not a metric to put in a pipeline gate.

StrengthsMeasures whether assertions detect faults, which coverage percentage never does.
CostsExpensive runs; equivalent mutants produce unactionable noise.
WarningFix basic reliability before optimizing mutation score.
Use whenThe suite is already fast, green and trusted, and you want to know whether it means anything.

5.6Test doubles

Stubs, fakes, spies and mocks — controlled stand-ins for real collaborators.

Use them for cost, nondeterminism and irreversible side effects; do not use them to avoid understanding a dependency. Prefer a fake — a working in-memory implementation — over a mock — a scripted expectation. A mock asserts the call sequence, which is the design; when the design changes for good reasons, the test fails for no reason. That is the mechanism by which a mock wall makes refactoring impossible.

StrengthsControls time, network, cost and side effects you cannot take in a test.
CostsDoubles drift from the real dependency and silently stop being true.
WarningMock walls test implementation rather than behavior.
Use whenThe real thing is slow, nondeterministic, costly, or has side effects you cannot undo.

5.7Fixtures and factories

Builders and object mothers that produce valid test data with only the relevant fields named.

Setup noise is the main reason tests stop being readable, and readability is the main reason they stop being maintained. A factory lets a test state its intent — this order is unpaid — instead of listing twelve fields that do not matter. The counter-risk is a single shared fixture everything depends on, which is coupling wearing a helpful hat.

StrengthsTests state their intent rather than their setup; one place to fix when the schema moves.
CostsA universal shared fixture becomes its own coupling problem.
Use whenSetup is obscuring what the test is actually about.

5.8Hermetic environments

Every run starts from the same known state, with no shared or external mutable dependency.

Containers per run, seeded databases, frozen clocks, pinned random seeds, no shared staging environment, no test that depends on another test's leftovers. Most of what teams call flaky tests is not a test problem at all — it is a non-hermetic environment, and no amount of retrying fixes it.

StrengthsRemoves the largest single source of flakiness and makes failures reproducible.
CostsSetup and per-run cost; a few dependencies genuinely cannot be isolated.
Use whenAny test above unit level that you intend to trust.

5.9Flaky-test quarantine

Temporary isolation of a nondeterministic test with a named owner, reason, expiry, and repair plan.

All four attributes are load-bearing. Without an owner and an expiry this is not quarantine, it is deletion with extra ceremony — and a quarantine list that only grows is a suite that has already stopped being a gate.

StrengthsRestores signal without pretending the defect disappeared.
CostsReduced coverage during quarantine and explicit maintenance debt.
Use whenA test is demonstrably nondeterministic and cannot be repaired immediately.
Technique Use it for Warning
Property-based Invariants over a wide input space Poor properties produce false confidence
Characterization Capturing legacy behavior before change Captured behavior is not automatically desired behavior
Snapshot/approval Reviewing large structured output Unreviewed updates become rubber stamps
Mutation Checking whether assertions detect deliberate faults Fix basic reliability before optimizing mutation score
Contract Compatibility across independently deployed boundaries Does not prove the full journey
Test doubles Controlling cost, nondeterminism, or side effects Mock walls test implementation rather than behavior
Golden evaluation set Regression evidence for stochastic systems Dataset quality and representativeness dominate

Every row above answers "how", not "where". The level was already decided in Part II.

Part VIDecide

Build your evidence portfolio

Start with the failure you need to catch. Every broader layer requires a force. The result explains what was selected, what it costs, what was skipped, and what would reopen the decision.

How they compose

The honest takeaway: "choosing a testing strategy" almost never means picking a shape. It means selecting evidence on three independent dimensions, and being able to say out loud what you deliberately did not buy.

A typical modern service ends up with acceptance examples for the contested requirements and production feedback for everything that only real traffic reveals; sociable unit tests where real logic exists and narrow-integration tests at each boundary that can fail on its own; contract tests wherever another team deploys independently; and a very small number of end-to-end journeys — each one attached to a named critical failure that nothing cheaper could expose. If there is a model or an agent in it, that whole portfolio stays, and an evaluation overlay is added on top for the part of the output that cannot be asserted exactly.

Notice what is missing from that list, deliberately: no broad-integration tier by default, no independent QA gate unless an external party requires one, no mutation tool until the suite is already trusted, and no model judge for anything a deterministic assertion can check. A portfolio that cannot name its exclusions has not made any decisions. Treat the result as a starting point to argue with, not a verdict.