#lessons learned

Daily writing promptWhat’s a lesson you’ve learned recently that shifted your perspective?View all responses I’m at that age where I am pretty set in my ways with almost all things. It’s hard to change things that you’ve always done or done for a long time. But I think learning to not care as much is such a vibe. (more…)

I found a continuity error QmQ! Character is acting like they already had the realization chapters before it happened. Suppose that's what happems when you write the realization scene first and the go back in time to before the character found out.

But now I have to go back and re-read everything to find where the error began and see how much I subconciously let it affect the plot q_q

Still I fight on!

Sidequest: 49/140

A perspective post from inside the actual mistakes, not the polished after-the-fact framing. Each item below is something we got wrong on a real project, what happened as a result, and the design change that prevents it on every project since. The pattern is that none of the mistakes were exotic; all of them were the kind of "we'll handle that later" decision that compounded into a problem that was harder to fix later than it would have been to address up front.

We share these because we know other teams have made the same calls. The lessons are not unique to 137Foundry, and the fixes are not proprietary; they are the ordinary design discipline that any team can adopt once they have decided to take replay seriously.

Photo by Liam Briese on Unsplash

Mistake 1: trusting the upstream system's event buffer for recovery

Early on, we built an integration that did not have its own event log. The reasoning was: the upstream system buffers events for seven days, so if we go down for anything less than a week, we can just ask the upstream to replay.

This worked until the upstream system, without warning, reduced their buffer retention from seven days to three days. We found out during an incident that exceeded three days. The replayable events were the ones from the last three days; everything older was lost.

The fix: every integration we build now has its own per-event log on our infrastructure, indexed by upstream event ID, with our own retention policy that we control. The upstream system's buffer is a nice-to-have for cross-checking; it is not the source of truth.

Mistake 2: idempotency keys computed over the entire payload

A second early project used a SHA-256 hash of the full event payload as the idempotency key. The reasoning was: the same event should produce the same hash, and the hash detects duplicates.

This worked until the upstream system added a new optional field to the payload. The hash changed for every event that included the field. Our duplicate-detection check returned false. We reprocessed thousands of events that had already been processed during the partial-availability window of an unrelated outage, producing duplicate downstream records that took two weeks to clean up.

The fix: idempotency keys are now computed over a deliberately-chosen subset of fields, with the algorithm and the field set treated as a contract. New optional fields in the upstream payload are ignored at the hash step. Schema-version prefixes (v1:hash(...), v2:hash(...)) handle the case where the contract has to change. The general pattern is the same one that the Stripe API documentation describes for client-provided idempotency keys, applied to the integration's internal key design.

Mistake 3: building a replay tool without a dry-run mode

A third project shipped a replay tool that did not have a dry-run mode. The reasoning was: the team can read the SQL query that drives the replay and confirm what it will do.

The team could indeed read the query. The query was correct. The replay was correct on the first run. The replay was also correct on the second run, where the operator typoed a single character in the time-window argument and replayed the wrong day's events. The downstream impact was a few hundred duplicate records.

The fix: every replay tool now requires a dry-run mode out of the box, and the dry-run runs by default unless the operator explicitly passes a --live flag. The output of the dry-run is the structured summary the operator reads to confirm intent before running for real. The wrong-day typo would have been caught by the dry-run output showing zero unprocessed events in the window, which is a different number than the operator expected.

Mistake 4: defaulting to the integration's normal throughput on replay

A fourth project's replay tool defaulted to the integration's normal steady-state throughput, on the reasoning that "if the integration can handle it at steady state, it can handle it during recovery."

This was wrong because the downstream system had been handling steady-state traffic during the outage, but the burst of replay traffic on top of that pushed it over its connection limit. The replay finished, the downstream fell over, and we now had a second outage to recover from.

The fix: every replay tool now defaults to a rate limit that is conservatively below the integration's normal steady-state rate, often three to five events per second when the integration normally runs at ten. The operator can opt into higher rates if the situation requires it. The default protects against the case where the operator does not have time to think carefully.

Photo by Diego F. Parra on Pexels

Mistake 5: no dashboard for the event log

A fifth project shipped the per-event log and the replay tool but no dashboard. The reasoning was: operations can SQL the log when they need to, and we will build a dashboard if it becomes a problem.

It became a problem during the first incident. The on-call engineer, at 2 AM, did not know the schema of the log table, did not have a saved query for "events with outcome != success in the last six hours," and ended up writing the query inline and getting it wrong twice before the third version returned the right count.

The fix: every integration ships with a small dashboard that surfaces the event log's recent ingest counts, processing outcomes, and any backlog. The dashboard has a saved "find unprocessed events in this time window" query. The on-call engineer does not have to remember the schema; they click a button. The dashboard takes a day to build the first time and most of that day's effort is reusable across projects.

What these five mistakes have in common

Every one of these was a "we'll handle that later" decision at design time. Every one of them turned into a real production incident that cost more to recover from than the original design work would have cost to do correctly.

The pattern is the same across the five: the design decision looked optional at the time, looked like overkill in the absence of an incident, and looked like the obvious right call in the presence of an incident. The discipline we have ended up codifying is to make all five decisions on day one, regardless of how simple the integration appears. The marginal cost is small. The downside protection is large.

The deeper framing here is that reliability work is bought, not earned. Every reliability feature has to be built explicitly; none of them appear automatically because the team "cares about reliability." The teams we work with that consistently have clean recoveries are the teams that built the recovery tooling before they needed it. The teams we work with that have incident-driven cleanups are the teams that built the recovery tooling after they needed it, which is to say after the first incident has already happened.