You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kafka.apache.org by ju...@apache.org on 2015/02/06 19:50:41 UTC
svn commit: r1657920 - /kafka/site/082/ops.html
Author: junrao
Date: Fri Feb 6 18:50:41 2015
New Revision: 1657920
URL: http://svn.apache.org/r1657920
Log:
add new producer metrics to 0.8.2 doc
Modified:
kafka/site/082/ops.html
Modified: kafka/site/082/ops.html
URL: http://svn.apache.org/viewvc/kafka/site/082/ops.html?rev=1657920&r1=1657919&r2=1657920&view=diff
==============================================================================
--- kafka/site/082/ops.html (original)
+++ kafka/site/082/ops.html Fri Feb 6 18:50:41 2015
@@ -586,6 +586,253 @@ We pay particular we do graphing and ale
</tr>
</tbody></table>
+<h4><a id="new_producer_monitoring">New producer monitoring</a></h4>
+
+The following metrics are available on new producer instances.
+
+<table class="data-table">
+<tbody><tr>
+ <th>Metric/Attribute name</th>
+ <th>Description</th>
+ <th>Mbean name</th>
+ </tr>
+ <tr>
+ <td>waiting-threads</td>
+ <td>The number of user threads blocked waiting for buffer memory to enqueue their records</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>buffer-total-bytes</td>
+ <td>The maximum amount of buffer memory the client can use (whether or not it is currently used).</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>buffer-available-bytes</td>
+ <td>The total amount of buffer memory that is not being used (either unallocated or in the free list).</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>bufferpool-wait-time</td>
+ <td>The fraction of time an appender waits for space allocation.</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>batch-size-avg</td>
+ <td>The average number of bytes sent per partition per-request.</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>batch-size-max</td>
+ <td>The max number of bytes sent per partition per-request.</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>compression-rate-avg</td>
+ <td>The average compression rate of record batches.</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>record-queue-time-avg</td>
+ <td>The average time in ms record batches spent in the record accumulator.</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>record-queue-time-max</td>
+ <td>The maximum time in ms record batches spent in the record accumulator</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>request-latency-avg</td>
+ <td>The average request latency in ms</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>request-latency-max</td>
+ <td>The maximum request latency in ms</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>record-send-rate</td>
+ <td>The average number of records sent per second.</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>records-per-request-avg</td>
+ <td>The average number of records per request.</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>record-retry-rate</td>
+ <td>The average per-second number of retried record sends</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>record-error-rate</td>
+ <td>The average per-second number of record sends that resulted in errors</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>record-size-max</td>
+ <td>The maximum record size</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>record-size-avg</td>
+ <td>The average record size</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>requests-in-flight</td>
+ <td>The current number of in-flight requests awaiting a response.</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>metadata-age</td>
+ <td>The age in seconds of the current producer metadata being used.</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>connection-close-rate</td>
+ <td>Connections closed per second in the window.</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>connection-creation-rate</td>
+ <td>New connections established per second in the window.</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>network-io-rate</td>
+ <td>The average number of network operations (reads or writes) on all connections per second.</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>outgoing-byte-rate</td>
+ <td>The average number of outgoing bytes sent per second to all servers.</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>request-rate</td>
+ <td>The average number of requests sent per second.</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>request-size-avg</td>
+ <td>The average size of all requests in the window.</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>request-size-max</td>
+ <td>The maximum size of any request sent in the window.</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>incoming-byte-rate</td>
+ <td>Bytes/second read off all sockets</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>response-rate</td>
+ <td>Responses received sent per second.</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>select-rate</td>
+ <td>Number of times the I/O layer checked for new I/O to perform per second</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>io-wait-time-ns-avg</td>
+ <td>The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds.</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>io-wait-ratio</td>
+ <td>The fraction of time the I/O thread spent waiting.</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>io-time-ns-avg</td>
+ <td>The average length of time for I/O per select call in nanoseconds.</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>io-ratio</td>
+ <td>The fraction of time the I/O thread spent doing I/O</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>connection-count</td>
+ <td>The current number of active connections.</td>
+ <td>kafka.producer:type=producer-metrics,client-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>outgoing-byte-rate</td>
+ <td>The average number of outgoing bytes sent per second for a node.</td>
+ <td>kafka.producer:type=producer-node-metrics,client-id=([-.\w]+),node-id=([0-9]+)</td>
+ </tr>
+ <tr>
+ <td>request-rate</td>
+ <td>The average number of requests sent per second for a node.</td>
+ <td>kafka.producer:type=producer-node-metrics,client-id=([-.\w]+),node-id=([0-9]+)</td>
+ </tr>
+ <tr>
+ <td>request-size-avg</td>
+ <td>The average size of all requests in the window for a node.</td>
+ <td>kafka.producer:type=producer-node-metrics,client-id=([-.\w]+),node-id=([0-9]+)</td>
+ </tr>
+ <tr>
+ <td>request-size-max</td>
+ <td>The maximum size of any request sent in the window for a node.</td>
+ <td>kafka.producer:type=producer-node-metrics,client-id=([-.\w]+),node-id=([0-9]+)</td>
+ </tr>
+ <tr>
+ <td>incoming-byte-rate</td>
+ <td>The average number of responses received per second for a node.</td>
+ <td>kafka.producer:type=producer-node-metrics,client-id=([-.\w]+),node-id=([0-9]+)</td>
+ </tr>
+ <tr>
+ <td>request-latency-avg</td>
+ <td>The average request latency in ms for a node.</td>
+ <td>kafka.producer:type=producer-node-metrics,client-id=([-.\w]+),node-id=([0-9]+)</td>
+ </tr>
+ <tr>
+ <td>request-latency-max</td>
+ <td>The maximum request latency in ms for a node.</td>
+ <td>kafka.producer:type=producer-node-metrics,client-id=([-.\w]+),node-id=([0-9]+)</td>
+ </tr>
+ <tr>
+ <td>response-rate</td>
+ <td>Responses received sent per second for a node.</td>
+ <td>kafka.producer:type=producer-node-metrics,client-id=([-.\w]+),node-id=([0-9]+)</td>
+ </tr>
+ <tr>
+ <td>record-send-rate</td>
+ <td>The average number of records sent per second for a topic.</td>
+ <td>kafka.producer:type=producer-topic-metrics,client-id=([-.\w]+),topic=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>byte-rate</td>
+ <td>The average number of bytes sent per second for a topic.</td>
+ <td>kafka.producer:type=producer-topic-metrics,client-id=([-.\w]+),topic=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>compression-rate</td>
+ <td>The average compression rate of record batches for a topic.</td>
+ <td>kafka.producer:type=producer-topic-metrics,client-id=([-.\w]+),topic=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>record-retry-rate</td>
+ <td>The average per-second number of retried record sends for a topic</td>
+ <td>kafka.producer:type=producer-topic-metrics,client-id=([-.\w]+),topic=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>record-error-rate</td>
+ <td>The average per-second number of record sends that resulted in errors for a topic.</td>
+ <td>kafka.producer:type=producer-topic-metrics,client-id=([-.\w]+),topic=([-.\w]+)</td>
+ </tr>
+</tbody></table>
+
We recommend monitor GC time and other stats and various server stats such as CPU utilization, I/O service time, etc.
On the client side, we recommend monitor the message/byte rate (global and per topic), request rate/size/time, and on the consumer side, max lag in messages among all partitions and min fetch request rate. For a consumer to keep up, max lag needs to be less than a threshold and min fetch rate needs to be larger than 0.