Ask a team about their testing strategy and you usually get an answer about their framework. Playwright, the page-object layout, how fixtures work, roughly how many tests there are.
None of that is a strategy. A framework answers how tests are written. A strategy answers harder questions: what must be true before a change moves forward, how fast confidence has to arrive, which failures block a release and which do not, and what happens when something gets through anyway.
Those answers already exist in every team that ships. They are in the pipeline configuration, they were mostly written by whoever was on call the week each stage was added, and almost nobody has read the whole thing in one sitting.
Two teams, same test count
Consider two teams with comparable suites and comparable coverage numbers.
The first runs everything on every commit: unit, integration, end-to-end, the lot. Ninety minutes. Failures are common enough that a red build gets a shrug, and reruns are routine. Merges happen when someone has the patience to wait out the flake.
The second runs a fast subset on every commit, a broader set on merge to main, and the full suite nightly. Twelve minutes to a first verdict. A red build means something, because red is uncommon. Nightly failures open a ticket rather than blocking anyone.
Identical coverage on paper. Completely different strategies, and neither team wrote one down. The pipeline wrote it for them.
The second team is not testing less. It has decided when each kind of confidence is needed, which is the decision a strategy exists to make.
Late tests are worth less, and sometimes worth nothing
The most useful reframe here is that a test’s value depends on when it runs, not only on what it checks.
A test that catches a bug thirty seconds after the commit that caused it hands a developer a fault they still have in their head, in code they have not moved on from. The same test catching the same bug three days later during a nightly run hands somebody a fault in a change they have to reconstruct from git history, competing with whatever they are doing now.
Same assertion. Different value, by a wide margin. And below a certain point the value goes to zero and then negative: a test that only fails after the change is deployed is not preventing the incident, it is annotating it — while still costing maintenance, still going flaky, still needing someone to look at it.
This cuts against the instinct to move everything left. Not everything belongs early. A slow, resource-hungry check that catches a rare class of failure does not earn a place in the commit path; it earns a nightly slot and a ticket when it fires. The question is not “how early can this run” but “what is the earliest point where this failing would change what someone does next”.
The gate is not one big suite
The other reframe: a quality gate is a set of decisions, not a single pass-or-fail run.
A gate that is “the regression suite must be green” has exactly one policy for every kind of failure, and that policy is stop. Which means either the bar is too low for the failures that matter or too high for the ones that do not, and in practice it becomes too high, and then people learn to rerun until it passes. A gate that gets bypassed by habit is worse than no gate, because it still costs the time and no longer buys the assurance.
A gate that works usually looks more like a policy table:
| Failure class | What it does |
|---|---|
| Contract or data-integrity break | Blocks, always |
| Auth or permission regression | Blocks, always |
| Core user journey broken | Blocks |
| Secondary journey broken | Blocks on release, not on merge |
| Visual diff below threshold | Warns, records, does not block |
| Known-flaky test | Quarantined, tracked, does not block |
The point is not this table specifically. It is that someone decided each row, which means the decision can be argued with and changed. An undifferentiated suite hides those decisions inside a single boolean, and a decision nobody can see is a decision nobody can improve.
Rollback is a testing capability
The part most quality conversations leave out entirely.
If a bad change can be reverted in two minutes with confidence, the cost of one slipping through is two minutes. If reverting means a coordinated release, a migration to unwind, and a conversation with three teams, the cost of the same slip is a day. The same defect has completely different consequences in the two systems, so the same pre-deployment bar cannot be right for both.
Which means rollback speed belongs in the strategy conversation, and it usually is not there because it lives with a different team. A pipeline that can revert instantly can afford to let more through and catch it in production with fast detection. A pipeline that cannot revert must hold everything up front, and pays for that in cycle time on every change including the safe ones.
You cannot decide where your gate belongs without knowing what a miss costs, and what a miss costs is mostly a function of how fast you can undo it.
More automation can make it worse
An uncomfortable one, and it is why “add more tests” is not a strategy either.
Every automated check has a running cost, a maintenance cost, and a false-positive rate. Past a certain point, adding checks makes the pipeline slower and noisier without making it more informative, and the response to noise is predictable: people stop reading the output. A suite that nobody reads carefully has a detection rate approaching zero regardless of its coverage number.
This is where test selection earns its place as a strategic capability rather than an optimisation. Running the tests that relate to what changed, and running the rest on a different cadence, is not cutting corners. It is the only way a large suite stays fast enough that its verdict still means something.
What the pipeline cannot do
The last piece, and the one that stops this being a closed loop.
Tests check the failures somebody anticipated. Production produces the ones nobody did — the interaction between two features that were each fine, the load pattern nobody modelled, the data shape that only one customer has. No pre-deployment gate finds those, because finding them requires the conditions that only exist after deployment.
That is what observability is for in a quality strategy: not dashboards for their own sake, but the mechanism that turns an unanticipated production failure into a test case for next time. Every incident should end with something added to the suite, and if incidents are not producing new test cases, the suite is frozen at whatever the team could imagine when they wrote it.
That is the same argument as growing an eval set from production rather than imagination. The cases you can invent are the easy half. The ones that matter arrive from the field.
Read your pipeline this week
An hour, one sitting, with the config open:
- How long until a developer gets a first verdict? Not the full suite — the first signal that would change what they do.
- Which failures block, and who decided? If the answer is “all of them” and nobody remembers deciding, that is the finding.
- How often is the gate bypassed? Rerun-until-green is a bypass. Count it.
- How fast can you revert? This sets how strict the front of the pipeline needs to be.
- When did an incident last add a test case? If it has been a while, the loop is open.
The output of that hour is your real test strategy, written down for probably the first time. It is likely to differ from what the team says it is — and the gap between the two is the actual work.