Running an LLM on your own hardware

The default answer is “use the API,” until it isn’t

For most teams building an LLM feature, calling a hosted API is the right default: no GPU fleet to manage, no model to keep updated, someone else’s problem to scale. We reach for a self-hosted model — a Falcon model run on our own infrastructure, in the AML triage workflow of a regulated wallet — only when the specific constraints of the workload push past what an API call comfortably gives you. It’s worth being precise about which constraints those are, because “self-host for control” is not, by itself, a good enough reason.

Data residency: the constraint that doesn’t negotiate

The AML triage workflow reads flagged transaction patterns and account history for a regulated financial product. Even with personal data redacted before it reaches the model, sending that data to a third-party API means it leaves your infrastructure and enters someone else’s, subject to their retention and processing terms, in a domain where regulators expect you to be able to answer exactly where sensitive data has gone. Self-hosting removes that question by construction: the data never crosses a boundary you don’t control, because the model runs on the same infrastructure as everything else in the compliance workflow.

This is the constraint that actually decides the question in this case. Cost and latency are real considerations, but they’re optimization problems; data residency in a regulated workflow is closer to a hard requirement.

Cost at volume: where the math flips

API pricing is per-token, which is a good deal at low volume and a bad one at high, sustained volume. AML triage runs continuously against every flagged case — a steady, predictable load rather than a bursty one, which is exactly the shape of workload where the economics favor owning the hardware instead of renting inference by the token. The comparison that matters isn’t API cost per call versus GPU cost per hour in isolation; it’s API cost at your actual sustained volume versus amortized hardware cost plus the engineering time to run it, and that crossover point arrives sooner for a steady internal workload than it does for a spiky, customer-facing one.

Latency: fewer hops, more control

A self-hosted model removes a network round-trip to a third-party API and lets you control the batch size, the hardware it runs on, and what shares that hardware. For a workflow that’s part of a triage pipeline rather than a live chat interface, latency is a secondary benefit here, not the deciding one — but it compounds with the other two: same infrastructure, same network, fewer external dependencies to be slow or unavailable at the wrong moment.

The price you actually pay

None of this is free, and the operational cost is the part that’s easy to underestimate before you’ve paid it once.

You own the update cycle. A hosted API vendor updates the model behind the scenes, for better and worse; a self-hosted model only gets better when someone on your team decides to retrain, fine-tune, or swap it, tests the change, and rolls it out. That’s ongoing work, not a one-time setup cost.

You own the GPU capacity. Inference at the throughput this workflow needs requires provisioned GPU infrastructure, sized for peak flagged-case volume, sitting idle the rest of the time unless it’s shared with other workloads. Utilization is a real optimization problem once you’re the one holding the hardware.

You own the failure modes. A hosted API failing is someone else’s incident with a status page. A self-hosted model failing — out of memory, a bad deploy, a hardware failure — is your on-call getting paged, and it needs the same monitoring, alerting, and rollback discipline as any other production service:

# excerpt from the inference service's alerting rules
- alert: TriageModelHighLatency
  expr: histogram_quantile(0.95, inference_duration_seconds_bucket) > 4
  for: 5m
  annotations:
    summary: "AML triage model p95 latency above 4s"
- alert: TriageModelErrorRate
  expr: rate(inference_errors_total[5m]) / rate(inference_requests_total[5m]) > 0.02
  for: 5m

You own security patching for the serving stack, not just the model — the inference server, its dependencies, and the infrastructure it runs on all need the same patch cadence as anything else exposed to production traffic, even if that traffic is entirely internal.

The decision, stated plainly

Self-host when a hard constraint — data residency being the clearest example — rules out sending the data externally at all, or when sustained volume makes the amortized cost of owned hardware clearly beat per-token pricing, and you’re prepared to staff the update cycle, capacity planning, and on-call burden that comes with it. Use a hosted API everywhere else, including for most LLM features that don’t touch regulated data or run at sustained high volume — the operational cost of self-hosting isn’t worth paying just to avoid a per-token bill.

What this means for you

“Should we self-host our LLM” is really three separate questions — does data have to stay on our infrastructure, does the volume math favor owned hardware, and are we willing to run inference like a production service with its own on-call — and the answer to the first can override the other two entirely. Get that question right before evaluating hardware, because the operational commitment is the real cost, not the GPUs.