QA Is Not a Test Layer

"What is our testing strategy?" sounds like one question.

It is at least three.

• How do we build confidence that the product is ready?
• At what boundary should executable code be tested?
• How do we judge output that is probabilistic rather than exact?

Treating all three as one pyramid produces familiar arguments: QA versus developers, unit versus integration, pyramid versus trophy, tests versus evals. Most of them are arguments across different axes.

QA is a discipline

QA is not the layer above E2E.

It is the process work that makes quality more likely: clarifying acceptance criteria, examining risk, exploring behavior, assessing usability and accessibility, deciding what evidence a release needs, and learning from production.

ISTQB draws a useful distinction. Testing is product-oriented: it examines the thing being built. Quality assurance is process-oriented and preventive: it improves how the work is done. Both are whole-team concerns.

That does not mean every team needs a separate QA phase.

A small service with low blast radius, strong observability, and easy rollback may need acceptance examples and exploratory attention, but not an independent release gate. A regulated or safety-relevant system may need formally independent evidence even when its automated tests are excellent.

The question is not "do we have QA?" Every team does quality work, whether it names it or not.

The question is:

Which quality practices require human judgment or independent evidence, and what risk justifies their cost?

Unit, integration, and E2E are three questions

The code-level taxonomy is simpler. Each level asks its own question:

• Unit — does this piece of behavior work?
• Integration — does this code interact correctly with a real boundary?
• E2E — does this critical journey work through the whole system?

The labels are less important than the question and the resources touched.

Unit: does the behavior work?

A unit is not necessarily one class.

It can be solitary: every collaborator is replaced. Or it can be sociable: lightweight collaborators run together. The second is often the better default. Mock a collaborator because it is slow, nondeterministic, expensive, unavailable, or produces side effects—not because a diagram contains a box boundary.

Thick domain logic usually benefits from many fast unit tests. Thin orchestration code often does not. A service that mostly maps HTTP to a database may reveal more with one narrow database integration test than with twenty mock-heavy service tests.

Integration: does the boundary work?

"Integration test" does not have to mean "start the whole platform."

A narrow integration test can exercise one real seam:

• a repository against the real database engine;
• serialization against the actual schema;
• an HTTP client against a controlled server;
• a provider against a consumer contract;
• framework configuration through the real request pipeline.

Keep unrelated boundaries fake or absent.

Broad integration becomes justified only when the defect emerges from the interaction of several real components and cannot be reproduced at a narrower seam.

E2E: does the journey work?

E2E is the hardest gate.

It proves something lower levels cannot: deployment, routing, identity, configuration, orchestration, and user flow working together. That can be valuable for a small number of critical journeys.

It is also slow, expensive to diagnose, sensitive to environment and data, and likely to become flaky.

The default is therefore not "no E2E forever." It is:

No E2E until you can name the journey and the failure that only E2E can expose.

A login-to-payment journey may justify one. A small internal CRUD tool may not.

The pyramid and trophy are not enemies

The test pyramid is a good default when most complexity is in deterministic domain logic. Many fast unit tests, fewer integration tests, very few E2E tests: that is a sensible economic shape.

The Testing Trophy is a good correction when most risk lies in the way components, frameworks, browsers, and APIs interact. In that system, integration tests may provide more confidence per test than isolated units.

Spotify's Testing Honeycomb makes a related point for microservices: service behavior is often best proven through owned integrations rather than a forest of implementation-detail tests.

Contract testing adds another useful portfolio shape. When independently deployed services keep breaking one another, a consumer/provider contract is usually a better detector of interface drift than a broad E2E suite.

None of these diagrams is the strategy.

They summarize where the risks happened to land:

• Pyramid-like — useful when rich, deterministic domain logic dominates.
• Trophy-like — useful when thin application logic sits over important integrations.
• Honeycomb-like — useful when a microservice is best understood through its owned boundaries.
• Contract-centered — useful when independently deployed consumers and providers must remain compatible.
• Evaluation-heavy overlay — useful when data, ML, or LLM quality is a major source of risk.

Google's often-repeated 80/15/5 mix is explicitly a rough guideline, not a compliance target. A healthy portfolio follows the system. It does not force the system into a diagram.

Data, ML, and agents add another axis

Data pipelines still have code.

Their transformations can be unit tested. Their storage and orchestration can be integration tested. Their critical run can be exercised end to end.

They also fail because the data is late, malformed, duplicated, shifted, or inconsistent with an upstream contract. Those are data-quality and monitoring concerns, not merely another unit-test folder.

ML systems add model behavior, training-serving consistency, slice performance, leakage, reproducibility, and drift. The ML Test Score was created precisely because ordinary code tests remain necessary but are not sufficient.

LLM and agentic systems make the split even clearer.

The deterministic scaffold can be tested exactly:

• schemas;
• tool dispatch;
• permissions;
• state transitions;
• retries;
• persistence;
• routing;
• guardrails.

The model output is probabilistic. Its quality is evaluated against cases, metrics, rubrics, and thresholds.

That is why a thin LLM service may need:

• unit tests for validation and mapping;
• one narrow HTTP or schema integration test;
• a small reviewed evaluation dataset.

It may not need a ceremonially complete pyramid, a browser E2E suite, or an LLM judge.

A model-based judge becomes useful only when the required quality is genuinely semantic, deterministic checks are insufficient, and the judge has been compared with human-reviewed cases. Temperature zero can reduce variation. It does not turn a model into a deterministic assertion engine.

AI-generated tests need an independent oracle

AI can generate a lot of tests quickly.

That is not the same as generating evidence.

A model that sees only the current implementation is likely to reproduce current behavior—including behavior that is wrong. It can also over-mock, assert implementation details, and produce impressive coverage with weak fault detection.

The guardrail is simple:

A generated test needs a source of intended behavior outside the code it is testing.

That source can be a requirement, acceptance example, invariant, protocol, known-good case, reviewed golden file, or metamorphic relation.

Ask one review question:

What meaningful breakage would make this test fail?

If there is no clear answer, the test is coverage theater.

Choose evidence, not a shape

A testing strategy should say:

• which risks matter;
• which practice or test catches each one;
• what every layer costs;
• what is deliberately skipped;
• what future signal would reopen the decision.

Choose the smallest reliable evidence portfolio that covers the named risks.

Then draw the shape, if a shape is still useful.

Two ways to run the decision

This post ships alongside a new interactive wizard — Test Patterns — where you can click through the decision yourself and walk out with a composable testing portfolio. Treat the result as a starting point to argue with, not a verdict.

The same reasoning is also a Claude Code skill. test-patterns is the fourth skill in arch-crew, added in 0.2.1 alongside decide-architecture, design-patterns, and agentic-patterns.

/plugin marketplace add AdamKrysztopa/architectural-decisions
/plugin install arch-crew

Repo: github.com/AdamKrysztopa/architectural-decisions