Keeping a payments API online through a DDoS attack

What happened

A payments platform we run started dropping requests during a traffic spike that didn’t match any pattern in its usual load. Within minutes it was clear this wasn’t organic traffic — it was a distributed denial-of-service attack aimed at the public API. Every serious platform that handles real traffic on the open internet eventually sees this. It isn’t a sign that something was built wrong; it’s an external event you have to be ready to detect, contain and absorb. What distinguishes a team isn’t whether an attack happens — it’s how fast it’s recognized, how the response is sequenced under pressure, and what changes afterward so the next one is a smaller problem.

Detection first

The first signal wasn’t a dashboard alert about attack traffic specifically — it was the success-rate and p95 latency panels moving together in a way that didn’t match a normal spike in merchant volume. Distinguishing “we’re suddenly popular” from “we’re under attack” in the first few minutes matters, because the two call for opposite responses: scale up, or filter at the edge. Cross-checking request patterns against the usual shape of legitimate traffic — origin diversity, request paths, payload shapes — confirmed it within minutes as traffic aimed at exhausting the origin rather than a genuine surge in customers trying to charge cards.

Containment before mitigation

The first priority wasn’t blocking traffic, it was confirming the blast radius. Because the architecture already separates the public API from the internal ledger — the API talks to the rest of the system only through the same event-driven interfaces every processor adapter uses — we could confirm within minutes that money movement wasn’t at risk, even while the API itself was under load. That separation is what let the response stay calm: a compromised or overwhelmed edge doesn’t get a shortcut to the parts of the system that move money.

The decision: move the edge, not the origin

The origin had no way to tell attack traffic from a surge of real merchant traffic before both arrived at the same servers. Throwing more origin capacity at that problem doesn’t fix it — it just makes the attack more expensive to run, not impossible. The decision was to move the filtering decision earlier, to an edge network that could absorb and challenge suspicious traffic before it ever reached the application servers processing real charges. We moved the public API behind Cloudflare within the week: DNS and routing changes, WAF rules scoped to the specific request shapes the attack was using, and challenge/rate-limit policies tuned to let legitimate merchant traffic through untouched.

What stayed up

The API’s own request handling degraded during the worst of the spike, but disputes, payouts and reconciliation kept running the entire time. The transactional outbox that already wrote events in the same transaction as the data producing them meant that even the slower minutes at the edge produced delayed side effects, never lost ones — no event was dropped because a request was slow to land. No merchant lost a transaction, and no ledger entry needed manual reconciliation afterward. The API stayed available to customers throughout.

What we’d do differently

The honest gap wasn’t the decision to move to the edge — it was that the edge configuration got built during the incident instead of before it. Two changes came out of that:

• Pre-provisioned edge configuration. The WAF rules, rate-limit policies and DNS failover for every public-facing API we run are now written and tested in advance, sitting inactive until they’re needed, instead of being drafted from scratch while a spike is in progress.
• A rehearsed runbook. Detection thresholds, who makes the call to fail over, and the exact sequence of DNS and routing changes are written down and walked through on systems that haven’t been attacked yet, not authored for the first time on the one that has.

Neither of those is a reaction to something we got wrong during the incident — the response held. They’re the difference between a response that depends on people improvising well under pressure and one that depends on preparation, and preparation is the more reliable of the two.

What it costs now

The edge layer adds a small, fixed monthly cost and a few milliseconds of latency on the first request in a session, plus the standing cost of keeping the runbook and edge configuration current as the system changes. Both are worth it against the alternative: an origin that has to defend itself against everything the internet sends it, on a system where downtime means real transactions failing in real time. Mitigation now runs continuously, not as a response drafted after the fact.