Everyone building AI for infrastructure is having the same conversation: Kubernetes incidents, microservice failures, alert noise. Almost nobody in that conversation is talking about data governance. That's strange, because most of what actually breaks a business runs through it.
The layer nobody's modeling
A schema change nobody told the downstream team about. An Airflow DAG that fails silently because the topic it reads from got deleted. A consumer group drifting for three days while every dashboard stays green. PII moving through a pipeline nobody can trace when a regulator asks where it went.
Transactions, user events, inventory, financial decisions: most critical business outcomes travel through topics and pipelines before they reach anything a dashboard can query. When that layer is ungoverned, the blast radius isn't an engineering incident. It's wrong data powering a business decision, and compliance exposure.
Infrastructure tooling and data observability are both maturing, on their own. Neither connects the two causally. No single tool holds the full state of your data layer, ownership, schema versions, consumer health, lineage, in one place, right now.
That's the gap.
Modeling the runtime without the content is half a picture
Here's the structural problem. The data layer is where production state actually lives. The infrastructure is the runtime it moves through. Model the runtime without the content and you have half a picture, and failure doesn't respect that boundary: a schema change breaks a downstream service, a deleted topic kills a pipeline. If your causal model stops at the infrastructure boundary, you're always starting the investigation from the wrong place.
NOFire already tracks microservices, deployments, and cloud resources in the Production Context Graph: their state, their relationships, how they change over time. Kafka topics, schemas, and consumer groups are the same kind of entity. State is the operative word. That's what lets you answer hard questions at any moment, not just when the pager fires.
Kafka is in the graph
Kafka support puts your clusters, topics, schemas, and consumer groups into that same graph, read-only, from outside your applications, no code changes, no libraries to ship. NOFire reads the cluster from the broker itself and resolves every consumer group to the service, team, and deploy behind it.
Two failure modes drove this build, because they're the two that actually cost enterprise teams real engineering hours: a broken message a consumer can't process, and a consumer that's stopped or fallen behind. Both get told apart automatically, and both get tied to the service, team, and change behind them, not handed to you as a number with no owner.
A stopped consumer and a slow one look identical on a lag chart. NOFire reads the committed offset against the topic head, sampled over time, so a frozen offset under a growing backlog reads as a stall, not a scaling problem, before it becomes a page.
A worked example
Here's what this looks like end to end, on data taken directly from a real investigation.
Consumer group settlement-group froze with offsets stuck at 516 and 257, lag growing at 80 messages a minute on the busiest partition. Every broker dashboard stayed green. A stopped consumer doesn't move a broker metric.
NOFire traced the event sequence automatically:
- Schema Registry connection failures began at 10:45 PM, 12 occurrences over 34 seconds.
- The registry recovered two minutes later, serving normal responses again.
settlement-consumer's logs stopped at 10:58 PM. The last event it processed was logged at offset 515.- One minute earlier, the schema compatibility mode on
orders-events-valuehad changed from BACKWARD to NONE, at version 2. - By 11:02 PM, both partitions were frozen and lag was climbing.
Root cause: the compatibility change allowed an incompatible v2 schema onto the topic. settlement-consumer couldn't deserialize it, and it stalled silently, no error thrown, no alert fired, logs simply stopped. That's a poison pill: whoever pushed the schema, whichever team wrote the producer, the consumer that inherits the failure has no way to see it coming from a lag number alone.
NOFire ties the specific message to the specific change and writes the fix back as a guardrail: require consumer sign-off before a compatibility downgrade ships, alert on commit stalls, surface deserialization failures automatically. The next poison pill from a different cause, a malformed payload, a bad producer deploy, gets caught the same way, because the mechanism doesn't care which team or which cause produced the bad message.
What's next
Poison pills and lag are the two failure modes that cost teams the most, but they're not the only signals that matter once Kafka is in the graph. Two more are built and on their way: a security check that flags encryption-in-transit dropping from TLS to plaintext on a cluster, and a topic config-change audit that catches a retention cut, an ISR-floor drop, or a replication-factor change the moment it happens, tied to who made it, not to whoever notices the backup window already shrank.
Where to start
If your Kafka estate is a black box the business runs on and nobody quite governs it, see how Kafka fits into the graph, or get a demo and connect your first cluster.
