Apache Kafka Monitoring Guide 2026

Why Kafka Monitoring Is Uniquely Challenging

Apache Kafka exposes hundreds of JMX metrics across brokers, producers, consumers, and topics. The challenge isn't collecting metrics — it's knowing which ones matter and what thresholds to alert on. A misconfigured consumer quietly falls behind while all other metrics look normal. An under-replicated partition means you're one broker crash away from data loss.

Common Kafka incidents in production and their early warning signs:

Critical Kafka Metrics to Monitor

Kafka exposes hundreds of JMX metrics. These are the ones that actually matter for production reliability, organized by signal category:

Consumer Lag: The Most Important Kafka Metric

Consumer lag is the difference between the latest offset (log end offset) and the consumer's current position (committed offset) for each partition. A lag of 0 means the consumer is fully caught up. Growing lag means consumers can't keep pace with producers.

# List all consumer groups and their lag
kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --describe \
  --all-groups

# Check specific consumer group
kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --describe \
  --group my-consumer-group

# Output columns:
# GROUP | TOPIC | PARTITION | CURRENT-OFFSET | LOG-END-OFFSET | LAG

# Consumer lag per group/topic/partition
kafka_consumergroup_lag{group="my-group", topic="my-topic"}

# Total lag for a consumer group (sum across all partitions)
sum(kafka_consumergroup_lag{group="my-group"}) by (group, topic)

# Alert: lag has been growing for 5+ minutes
# (lag at time T > lag at time T-5m)
(
  sum(kafka_consumergroup_lag{group="my-group"}) by (group)
  >
  sum(kafka_consumergroup_lag{group="my-group"} offset 5m) by (group)
)

Kafka Monitoring Tools Comparison

The right tool depends on whether you're self-hosting Kafka or using a managed service, and your team's operational capacity:

Kafka Alerting Runbook

These are the alerts every Kafka cluster should have, ordered by severity:

Enabling JMX for Kafka Monitoring

Kafka exposes all metrics via JMX. You need JMX enabled on your brokers and a compatible exporter to get data into your monitoring stack:

# In kafka-server-start.sh or as environment variable
export JMX_PORT=9999

# For remote JMX access (production — add auth):
export KAFKA_JMX_OPTS="-Dcom.sun.jndi.rmiregistry.interfaces=127.0.0.1 \
  -Dcom.sun.management.jmxremote \
  -Dcom.sun.management.jmxremote.authenticate=false \
  -Dcom.sun.management.jmxremote.ssl=false \
  -Dcom.sun.management.jmxremote.port=9999 \
  -Dcom.sun.management.jmxremote.rmi.port=9999 \
  -Djava.rmi.server.hostname=<broker-hostname>"

lowercaseOutputName: true
rules:
  # Under-replicated partitions
  - pattern: 'kafka.server<type=ReplicaManager, name=UnderReplicatedPartitions><>Value'
    name: kafka_server_ReplicaManager_UnderReplicatedPartitions
    type: GAUGE
  # Active controller
  - pattern: 'kafka.controller<type=KafkaController, name=ActiveControllerCount><>Value'
    name: kafka_controller_KafkaController_ActiveControllerCount
    type: GAUGE
  # Offline partitions
  - pattern: 'kafka.controller<type=KafkaController, name=OfflinePartitionsCount><>Value'
    name: kafka_controller_KafkaController_OfflinePartitionsCount
    type: GAUGE
  # Request handler idle ratio
  - pattern: 'kafka.server<type=KafkaRequestHandlerPool, name=RequestHandlerAvgIdlePercent><>OneMinuteRate'
    name: kafka_server_KafkaRequestHandlerPool_RequestHandlerAvgIdlePercent
    type: GAUGE

Related Guides

Frequently Asked Questions