Kafka Metrics
Kafka uses Yammer metrics to record internal performance measurements. The metrics are exposed via Java Management Extensions (JMX) and can be read with a JMX console.
Continue reading:
Metrics Categories
There are metrics available in the various components of Kafka. In addition, there are some metrics specific to how Cloudera Manager and Kafka interact. This table has pointers to both the Apache Kafka metrics names and the Cloudera Manager metric names.
Category | Cloudera Manager Metrics Doc | Apache Kafka Metrics Doc |
---|---|---|
Cloudera Manager Kafka Service |
Base Metrics | |
Broker | Broker Metrics | Broker |
Common Producer/Consumer |
||
Producer | ||
Consumer | ||
Mirror Maker | Mirror Maker Metrics | Same as Producer or Consumer tables |
Viewing Metrics
Cloudera Manager records most of these metrics and makes them available via Chart Builder.
Because Cloudera Manager cannot track metrics on any clients (that is, producer or consumer), you may wish to use an alternative JMX console program to check metrics. There are several JMX console options:
Building Cloudera Manager Charts with Kafka Metrics
The Charts edit menu looks like a small pencil icon in the Charts page of the Cloudera Manager console. From there, choose Add from Chart Builder and enter a query for the appropriate metric.
SELECT metric WHERE filter
Some specific examples of queries for Cloudera Metrics are:
- Controllers across all brokers
- This chart shows the active controller across all brokers. It is useful for checking active controller status (should be one at any given time, transitions should be fast).
SELECT kafka_active_controller WHERE roleType=KAFKA_BROKER
- Network idle rate
- >Chart showing the network processor idle rate across all brokers. If idle time is always zero, then probably the num.network.threads property may
need to be increased.
SELECT kafka_network_processor_avg_idle_rate WHERE roleType=KAFKA_BROKER
- Partitions per broker
- Chart showing the number of partitions per broker. It is useful for detecting partition imbalances early.
SELECT kafka_partitions WHERE roleType=KAFKA_BROKER
- Partition activity
- Chart tracking partition activity on a single broker.
SELECT kafka_partitions, kafka_under_replicated_partitions WHERE hostname=host1.domain.com
- Mirror Maker activity
- Chart for tracking Mirror Maker behavior. Since Mirror Maker has one or more consumers and a single producer, most consumer or metrics should be usable with this query.
SELECT producer or consumer metric WHERE roleType=KAFKA_MIRROR_MAKER
<< Developing Kafka Clients | ©2016 Cloudera, Inc. All rights reserved | Kafka Administration >> |
Terms and Conditions Privacy Policy |