I Made a Million-Dollar Mistake, Now What?

What it's about

We’ve all shipped something that cost money.

An AWS service is left running. A misconfigured webhook. Test keys in production. Small mistakes, varying blast radius.

But what happens when a system that serves tens of millions of users per month starts leaking revenue without ringing any alarm bells? Not through a massive crash, those are the easy ones to notice. But through partial failures that look “healthy” in most dashboards.

This talk is a case study of a 13-year-old, bootstrapped product that has processed billions of requests, generates tens of millions in annual revenue, and is operated by roughly 10 engineers across the frontend and backend.

In 24 hours, on the frontend alone, we can potentially amass over 120,000 errors, generate more logs than we can afford to retain in Datadog for more than two weeks, and other quirks that come with sheer scale. Failures are rarely total. They are regional, conditional, asynchronous, and financially expensive.

We will analyze real postmortems in which losses accumulated silently over weeks (one even 9 months). We will explore how architecture across the stack, from React boundaries to API contracts to observability pipelines, determines whether you detect risk early or explain it later.

This is not a blame story. It is not a hero story. It is a systems story.

Because at scale, software will break.

The question is whether your architecture tells you fast enough that it is costing you money and forgives you along the way.

What you'll leave with

• How business risk actually emerges in lean teams, beyond individual bugs
• How to reason about impact and loss when the data is messy or contradictory
• How to analyze incidents when dashboards stop making sense
• How everyday engineering practices shape financial risk
• How to move the needle beyond commits and lines of code
• How to improve safety and reliability without slowing the business to a crawl

Who it's for

Engineers and tech leads in startups or scale-ups running user facing systems with small, lean teams.

Where I've given this talk