Highlights from our AMA with Jonah Cohen, Tech Lead for Financial Engineering at OpenAI
For most of SaaS history, two concerns lived in different parts of the codebase. Rate limiting was a reliability problem: stop one tenant from starving everyone else. Billing was a finance problem: count what happened and invoice for it on some cadence. You could build each one in isolation and barely think about the other.
AI products collapse that separation. When every request carries a real, non-trivial inference cost, "should we serve this?" and "what does this cost the customer?" stop being two questions answered in two systems at two different times. They become one question, asked synchronously, inside the request path. That is the problem OpenAI ended up solving with what they call the decision waterfall, described in their February 2026 post "Beyond Rate Limits."
Jonah Cohen architected that system. He joined Stigg for a candid, in-the-weeds session on what it actually took, and what he'd tell engineering teams staring at the same problem now. Here are the highlights.
When you open a blank architecture doc, the temptation is to start with the request path: what checks fire, in what order, backed by what store. But the variable that quietly determines all of those choices is not technical at all. It's who your customers are and how they pay.
Jonah's framing was that this is the clarifying question to answer before you draw a single box. A system built primarily for sales-managed enterprises can bill in arrears: publish asynchronous usage events, aggregate them, invoice on a period. Fraud and uncollectible usage barely register, because there's a contract and a human relationship behind every account. A self-serve product is a different machine entirely, and here he was blunt about what OpenAI learned the expensive way:
"You probably should never bill in arrears for anybody who is not sales managed. At least that has been our experience. We used to have our API bill in arrears even for self serve. And our uncollectible rate, I'm not going to say what it was, but it was breathtaking."
The fix was prepaid. If there's no sales motion and no relationship, no line of credit; payment comes upfront. Even promotional credits for new accounts should be modeled as a positive balance that decrements to zero and cuts off, never as an unbounded negative balance you hope to collect against later.
"Don't write any blank checks."
Most of us carry an intuition inherited from classic SaaS: the marginal cost of one more request is basically nothing, so be generous now and add metering later once it matters. On an inference-backed product, that intuition is a liability, and it's the kind that doesn't announce itself until the compute bill does.
Jonah put the failure mode in about as stark a form as it gets:
"If you are a subscription based business, if you are serving inference, you are going to go bankrupt because inference is expensive. This is not the old world of SaaS where your COGS is tiny and you're all gross margin."
The consequence is architectural, not just financial. If you sell access in fixed intervals of time, you have to bound the inference you serve inside each interval, which means rate limiting is a day-one requirement rather than a growth-stage optimization. You cannot retrofit a ceiling onto a product that was designed to give away unlimited compute for a flat fee.
So you accept it: you'll rate limit, and you'll meter, and you'll bill. The trap is that each of those is easy to build well in isolation, which lulls you into building them as separate systems. A rate limiter is a clean component. A usage meter is a clean component. Then a single plan needs both at once, and the logic that stitches them together does not compose the way you hoped.
That's exactly the path OpenAI walked. The API started as pure usage billing, originally all arrears. ChatGPT was pure rate limits. They were separate, and they worked, right up until one SKU sat in the overlap:
"There's this one plan in the Venn diagram [that needs] both. All the others are one or the other, but there's one that has both. So I got to rebuild the whole system."
The result was the decision waterfall: a single synchronous path that fuses rate limits, credit consumption, and spend controls into one per-request verdict. The branching got complex enough that Jonah couldn't hold it in his head, so he drew it as boxes and arrows in Whimsical and walked the team through the actual decision tree the SKU demanded. He was insistent that this is an endpoint, not a starting point. "This is not the V1 of any of these systems."
Now the hard part. Once metering and billing share the request path, an engineer immediately asks the right question: what does an idempotency key actually guarantee under retries, worker restarts, and concurrent requests racing on the same account? The naive instinct is to make the whole system exactly-once. That's expensive, and in places it's not even coherent.
Jonah's answer was to split the system by how much correctness each part can afford. The asynchronous path, where credit balances get depleted and invoices get generated, runs a very high bar. A series of Kafka streams carry idempotency keys at every layer, dedupe in real time against a finite look-back window, then a sweep re-checks the work asynchronously and issues reversals for anything that slipped through, typically catching it within 24 hours.
"All of our high scale enterprise billing is downstream of our deduping."
The hot path, rate limits, deliberately settles for less. There's in-memory dedupe on the same node, but no attempt at perfect deduplication across every look-back window, and the reason is subtle. Exactly-once on a rate limit is close to a category error:
"You actually cannot guarantee idempotency with a rate limit anyway because the rate limit may change."
The window can reset between the first request and a retry. And if you double-count and want to refund a rate limit but only notice 24 hours later on a five-hour window, the window you'd refund into is long gone. So they accept a bounded tolerance on the hot path and reserve the expensive guarantees for the path where money actually moves. In production, the looser hot-path guarantees have caused no material problems.
Every system has choices that are easy to revise later and choices that calcify the moment you ship them. The data model behind your balance is firmly in the second group, which is why Jonah's answer to "what do you wish you'd known" landed on it.
The balance class in their online store, a sharded key-value store on Cosmos DB, was built as single-entry accounting.
"I'm starting to feel the pain of not going double entry from day one."
For a running tally of usage, single-entry is fine; money goes in, money goes out. The pain surfaces downstream, once a balance has to move through states rather than just accumulate: asynchronous invoicing with terms, partial write-offs, and shuttling value along the pipeline from running tally to uninvoiced receivables to invoiced receivables to cash. Double-entry lets you clear a balance into non-terminal accounts and move it down that assembly line. Single-entry mostly knows about the two ends. As someone who thinks like an accountant even while writing the code, Jonah feels that constraint every time he reaches for an operation the schema won't cleanly express.
Not every team is operating at OpenAI's scale or complexity, so the useful question is what the core is that any team shipping AI products has to get right. A few things came through clearly.
The one closest to the metal: your database has to give you transactions. OpenAI began without transactional writes on Cosmos DB, got by on a janky single-record optimistic lock that didn't even behave correctly, and turned up client bugs in the process. Cross-shard global transactions are a luxury you can usually live without, but single-shard, multi-record transactions are not negotiable.
"You will be in a world of pain if your underlying database doesn't have transactions."
Just as foundational, and another scar carried over from Stripe: unify product usage events and monetization events into one system. At Stripe a usage dataset that didn't power billing meant permanent reconciliation pain. One of the first things Jonah established at OpenAI was that product usage events drive monetization events. They are one system with two pieces, not two systems you later try to reconcile. You trade a little latency for correctness, reconciliation, and auditability, and it's worth it.
There's a payoff most engineers don't think to design for. Under ASC 606, recognizing revenue takes three things: a contract, a reasonable expectation you'll collect, and proof you delivered the service. Usage events satisfy that last one directly.
"Not only did we charge for the thing, we did the thing. And we can prove that we did the thing."
The closing warning was aimed at over-engineering. Building optionality for every imaginable future sounds prudent and is actually the opposite, because real engineering is trade-offs: making one thing easy always costs you somewhere else. Better to name the knobs your company is most likely to need over the next 6 to 12 months, build those with correctness first, trade a little real-time consistency for eventual correctness where it's safe, and ship a V2 when the world moves.
"You don't need to be perfect, but watch out for those trapdoors, and those are where you ask the hard questions."
Huge thanks to Jonah for one of the most detailed sessions we've run in a while. If this is your kind of conversation, we run them every couple of weeks through our HTTP 402 community, a small group digging into exactly these problems. We'd love to have you. If you're not a member of HTTP 402 yet, simply request access here.
