The Bugs Only Deploying Finds

I had a service with fifty-three passing checks. Guards, rate limiting, a budget breaker, health, metrics, memory in Postgres, the whole thing. Every acceptance criterion traced to a test. The suite ran offline in twenty seconds.

That number is the one I had before the afternoon this article is about. The tag carries sixty-nine now: the four defects below, and the independent review that followed them, added sixteen more. If you run the suite at stage-06 you will see sixty-nine, and the difference is the point of the article rather than an error in it.

Then I put it in a container behind a proxy, and it returned 502.

What follows is the four defects that surfaced in the next hour. None of them were visible from the code. Three of them are invisible to unit tests by construction — not because my tests were bad, but because of what a unit test is.

1. The volume belonged to root

First symptom: the proxy served 502, the service log said PermissionError: '/data/traces/service.jsonl'.

The Dockerfile ran the process as an unprivileged user, which is correct. The compose file mounted a named volume at /data/traces, which is also correct. Together they produce a directory owned by root and a process that cannot write to it.

The fix is one line, and the reason is worth knowing:

RUN useradd --create-home --uid 10001 agentic \
    && mkdir -p /data/traces \
    && chown -R agentic /app /data
USER agentic

Docker copies ownership from the image into a fresh named volume, but only on first mount. Create the directory in the image with the right owner and the volume inherits it. Skip that, and the volume is root’s forever — including across image rebuilds, which is why docker compose up --build does not fix it and down -v does.

Why no test sees this: locally the directory belongs to whoever ran the command. The permission model that breaks is one that does not exist on the developer’s machine.

2. Nothing applied the migrations

Second symptom: 503, and health reported store: down, reason: InFailedSqlTransaction.

The table did not exist. Nothing in the stack ran the migrations — I had written them, tested them against a real Postgres, verified they rolled back, and then never wired anything to execute them at deploy time.

That part is embarrassing but ordinary. The interesting part is the second half.

3. A failed query poisoned the connection permanently

InFailedSqlTransaction does not mean “this query failed”. It means “a previous query in this transaction failed, and every statement from now on will be refused until someone rolls back”.

So after I applied the migrations by hand, the service was still broken. The connection it had opened at startup was in an aborted transaction and would stay that way forever. Health kept reporting down with the cause long gone.

def _query(self, sql, params=()):
    try:
        with self._connection.cursor() as cursor:
            cursor.execute(sql, params)
            return list(cursor.fetchall())
    except Exception:
        self._connection.rollback()   # ← this
        raise

This is the one I want to dwell on, because it is a whole category.

It is a defect about memory of the past. The system does not fail because of current conditions; it fails because of a condition that no longer holds. Every unit test in existence starts from a clean state, takes a fresh connection, and therefore cannot observe it. You do not write a test that says “given a connection that failed an hour ago” unless you have already been bitten.

The class is broader than databases. A cached negative DNS result. A circuit breaker that opened and has no half-open state. A client that marked a peer dead and never re-probes. A flag set on first error and never cleared. All of them share the shape: fix the cause, and the symptom stays.

If your service holds any long-lived connection or client, this is worth twenty minutes: find every place a failure can leave that object in a state it never leaves.

4. The safety check refused to start production

My config module refuses to boot in the production profile without real model credentials. It was written months earlier, and the reasoning is sound: a service that reaches production with a fake provider serves fabrications to real users.

It also meant the production profile could not be started at all without a billing account — and the production profile is the only place the real adapters switch on. Postgres instead of a file, a shared store instead of process memory. The most expensive half of the work was unverifiable for want of a credit card.

That is a genuine design tension, not a bug. Three of the four checks in that guard are absolute; only the fourth was in the way. I added a flag:

ALLOW_FAKE_LLM=1

Named so it cannot be set by accident, and — this is the part that makes it acceptable — surfaced in health:

{"status": "up", "dependencies": {...}, "provider": "fake"}

Without a key, because the monitor does not have one. An exception you cannot see from outside is a silent exception, and a silent exception eventually becomes an incident with a very confused post-mortem.

What these four have in common

Three of the four are invisible to unit tests for structural reasons, not for want of diligence:

• Permissions exist between the process and the OS. Unit tests run as you.
• Deployment ordering exists between containers. Unit tests have no containers.
• Aborted-transaction state exists across time. Unit tests have no past.

The fourth — the config guard — was visible in the code the whole time and had simply never been exercised, because nobody had run that profile.

I am not arguing against unit tests. Fifty-three of them caught things a deploy never would: an owner filter running after top-k selection, a counter whose set members collapsed when two events landed in the same microsecond, a rate limit doubling under two workers. Those are logic defects and belong in fast, offline checks.

The argument is narrower: there is a category of defect whose habitat is the boundary between your process and everything else, and the only instrument that finds it is putting the thing where it will actually live.

What I do differently now

Deploy once before you believe the feature is done. Not to a staging environment — locally, in containers, behind the proxy, with the real profile. The whole point is to leave the environment where your assumptions are true.

Write the runbook from real breakage. Mine now has four sections, and each one is a thing that actually happened that afternoon: the 502 with its down -v, the 503 with its aborted transaction, the config refusal, the scheduler restarting where nothing polls it. A runbook written from imagination documents the failures you already know how to avoid.

Ask of every long-lived object: what state can a failure leave it in permanently? That question would have found defect 3 in the code, without deploying, in about a minute. I did not ask it, because nothing prompts you to.

Put the escape hatch in the health endpoint. Any deliberate weakening of a safety check should be visible to whoever is looking at the service from outside. If the only way to know your production is running a fake is to read the environment file on the box, someone will eventually not read it.

The code — nine architecture decisions, eighteen mutation exercises, and a smoke script that runs the same list against localhost and a real domain — is in stage 6 of the course repository, pinned to the tag. The deployment lives in deploy/, and the runbook is the one written that afternoon.