- A 6x Regression Nobody Saw Locally

The same job, same document count, ran in about an hour locally. In a pre-production environment meant to mirror production, it took over 6 hours.

A staging environment that's supposed to be "just like production" almost never is — and the gap between them is exactly where regressions hide that no amount of local testing will ever catch.

The biggest single contributor turned out to be almost embarrassingly simple: a search-index setting meant only for short bursts of writes — how often the index refreshes itself — had been left at its everyday value for a run that lasted hours instead of minutes. That one unchanged setting alone accounted for hundreds of unnecessary index refreshes over the course of the run, each one adding real overhead for no benefit, since nobody was searching the index mid-run anyway.

But the slowdown wasn't the only thing the long run exposed. Running at real scale, for real duration, surfaced problems a short local test simply never triggers:

• A safety threshold on the search cluster tripped under sustained load, and the resulting error type wasn't on the pipeline's list of "retry this" errors — so a batch of records was dropped permanently, silently, before anyone noticed, and without any notification
• A concurrency counter went negative, implying two processes had decremented it at the same time — a race condition that a short run has almost no chance of hitting, but a long one can and did.
• Sensitive credentials appeared in plaintext in the logs multiple times over the run — a leak that a 5-minute smoke test would never generate enough log volume to reveal.

A setting tuned for "occasional writes, mostly reads" behaves fine for a 5-minute local test and quietly compounds into a real cost only once a run stretches past an hour — exactly the difference between a local sanity check and a staging run at production duration and volume.

Note to self: Passing local test proves your logic works, not that your process is configured for production environment.