Observability for event-driven systems

Why event-driven systems need different dashboards

A request/response API has an obvious health signal: is the request succeeding, and how long is it taking. An event-driven system built around bounded modules and event handlers — a payments platform with 29 modules and 170-plus handlers reacting to events through a transactional outbox — has no single request to watch. Health is distributed across producers, consumers, queues, and workers that all have to be behaving for the system to be healthy, and a problem in any one of them can be invisible from any other. You need dashboards built around that shape, not around the request/response shape most default monitoring assumes.

Four dashboards cover the shape of what can actually go wrong in a system like this, and the reason to build them before an incident rather than during one is that during one, you don’t have time to figure out what to look at.

1. Event throughput

The first question in any event-driven system: are events flowing at the rate they should be. A drop in throughput for one specific event type, even while overall system throughput looks normal, is often the earliest signal of a stuck handler or a silently failing producer — the kind of thing that doesn’t trip a generic error-rate alert because nothing is technically erroring, it’s just not happening.

# rate of events published, by type, over 5-minute windows
rate(outbox_events_published_total[5m]) by (event_type)

# rate of events consumed, by handler
rate(events_consumed_total[5m]) by (handler_name)

Plotting published and consumed rates for the same event type on the same panel makes a growing gap between them visible immediately — that gap is backlog building somewhere, before it’s big enough to show up as latency anywhere else.

2. Success rate, per event type and per handler

An aggregate success rate across the whole system hides exactly the failures worth knowing about early: one handler with a real bug looks like noise in a system-wide number, but is obvious the moment it’s broken out by handler.

sum(rate(handler_processed_total{status="success"}[5m])) by (handler_name)
/
sum(rate(handler_processed_total[5m])) by (handler_name)

The panel worth building alongside this one is success rate specifically for money-moving event types — charge.succeeded, payout.initiated, refund.completed — kept visually distinct from the rest, because a dip there deserves faster attention than a dip in, say, a reporting handler.

3. p95 latency, end to end and per hop

Latency in an event-driven system has more than one meaningful measurement. There’s the latency of any single hop — how long a handler takes to process one event — and there’s end-to-end latency: how long from the originating action (a charge request arriving) to the last handler reacting to it finishing (a receipt sent, a ledger entry recorded). The second number is what a customer or a support ticket actually cares about, and it’s the sum of every hop plus every queue wait in between, so it can be degrading even while every individual handler’s own p95 looks fine.

histogram_quantile(0.95, sum(rate(handler_duration_seconds_bucket[5m])) by (le, handler_name))

histogram_quantile(0.95, sum(rate(event_chain_duration_seconds_bucket[5m])) by (le, event_type))

4. Worker health and HTTP error rates

The last dashboard is the infrastructure underneath the other three: are the workers that consume from the queue actually alive, keeping up, and not silently restarting in a crash loop, and is the HTTP surface — the public API, webhook delivery, any synchronous entry point — returning errors at a normal or abnormal rate.

# workers processing vs. configured worker count
worker_active_count / worker_configured_count

# worker restarts, which should be near zero in steady state
increase(worker_restarts_total[15m])

http_requests_total{status=~"5.."} / http_requests_total

A worker fleet that’s silently down to half capacity can still show “normal” throughput for a while if the queue is absorbing the backlog — this dashboard is what catches that before the queue stops absorbing it.

Why these four, and why before you need them

Each of these dashboards answers a different question — is work happening, is it succeeding, is it fast enough, is the infrastructure underneath it sound — and an incident in an event-driven system is almost always legible as an answer to exactly one of those four questions going wrong first. Building them ahead of time means the first thing anyone does during an incident is look at a dashboard that already exists and already has the right shape, rather than spending the first twenty minutes of the incident writing the query that should have been sitting there. In the Grafana and Loki setup behind the payments platform’s own dashboards, this is exactly the difference between a fifteen-minute diagnosis and an hour of guessing.

What this means for you

If your system is event-driven, don’t try to monitor it with the same handful of panels you’d use for a request/response API. Build throughput, success rate, latency, and worker/infrastructure health as four distinct dashboards, each broken out by event type or handler rather than aggregated system-wide, and build them before an incident forces you to improvise them.