You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2021/01/15 21:28:09 UTC
[GitHub] [kafka] ableegoldman commented on a change in pull request #9895: KAFKA-9924: Add docs for RocksDB properties-based metrics
ableegoldman commented on a change in pull request #9895:
URL: https://github.com/apache/kafka/pull/9895#discussion_r558598465
##########
File path: docs/ops.html
##########
@@ -2032,17 +2032,26 @@ <h5 class="anchor-heading"><a id="kafka_streams_store_monitoring" class="anchor-
</table>
<h5 class="anchor-heading"><a id="kafka_streams_rocksdb_monitoring" class="anchor-link"></a><a href="#kafka_streams_rocksdb_monitoring">RocksDB Metrics</a></h5>
- All of the following metrics have a recording level of <code>debug</code>.
- The metrics are collected every minute from the RocksDB state stores.
- If a state store consists of multiple RocksDB instances as it is the case for aggregations over time and session windows,
- each metric reports an aggregation over the RocksDB instances of the state store.
+ RocksDB metrics are grouped into statistics-based metrics and properties-based metrics.
+ The former are recorded from statistics that a RocksDB state store collects whereas the latter are recorded from
+ properties that RocksDB exposes.
+ Statistics collected by RocksDB provide cumulative measurements over time, e.g. bytes written to the state store.
+ Properties exposed by RocksDB provide current measurements, e.g., the amount of memory currently used.
Note that the <code>store-scope</code> for built-in RocksDB state stores are currently the following:
<ul>
<li><code>rocksdb-state</code> (for RocksDB backed key-value store)</li>
<li><code>rocksdb-window-state</code> (for RocksDB backed window store)</li>
<li><code>rocksdb-session-state</code> (for RocksDB backed session store)</li>
</ul>
+ <strong>RocksDB Statistics-based Metrics:</strong>
+ All of the following statistics-based metrics have a recording level of <code>debug</code> because collecting
+ statistics in <a href="https://github.com/facebook/rocksdb/wiki/Statistics#stats-level-and-performance-costs">RocksDB
+ may have an impact on performance</a>.
+ Statistics-based metrics are collected every minute from the RocksDB state stores.
+ If a state store consists of multiple RocksDB instances, as is the case for aggregations over time and session windows,
Review comment:
```suggestion
If a state store consists of multiple RocksDB instances, as is the case for WindowStores and SessionStores,
```
I think we should be more specific here. Also users can have a Window/SessionStore that's unattached to an aggregation, eg in the PAPI, so we should avoid explicitly tying this to aggregations
##########
File path: docs/ops.html
##########
@@ -2032,17 +2032,25 @@ <h5 class="anchor-heading"><a id="kafka_streams_store_monitoring" class="anchor-
</table>
<h5 class="anchor-heading"><a id="kafka_streams_rocksdb_monitoring" class="anchor-link"></a><a href="#kafka_streams_rocksdb_monitoring">RocksDB Metrics</a></h5>
- All of the following metrics have a recording level of <code>debug</code>.
- The metrics are collected every minute from the RocksDB state stores.
- If a state store consists of multiple RocksDB instances as it is the case for aggregations over time and session windows,
- each metric reports an aggregation over the RocksDB instances of the state store.
+ RocksDB Metrics are grouped into statistics-based metrics and properties-based metrics.
+ The former are recorded from statistics that a RocksDB state store collects whereas the latter are recorded from
+ properties that RocksDB exposes.
Review comment:
Thanks for the explanation, I also didn't understand the difference at first
##########
File path: docs/ops.html
##########
@@ -2133,6 +2142,136 @@ <h5 class="anchor-heading"><a id="kafka_streams_rocksdb_monitoring" class="ancho
</tbody>
</table>
+ <strong>RocksDB Properties-based Metrics:</strong>
+ All of the following properties-based metrics have a recording level of <code>info</code> and are recorded when the
+ metrics are accessed.
+ If a state store consists of multiple RocksDB instances, as is the case for aggregations over time and session windows,
+ each metric reports the sum over all the RocksDB instances of the state store, except for the block cache metrics
+ <code>block-cache-*</code>. The block cache metrics report the sum over all RocksDB instances if each instance uses its
+ own block cache, and they report the recorded value from only one instance if a single block cache is shared
Review comment:
Technically I think it's possible for users to share a single block cache among only some stores, but not others. Or they could have two block caches shared across different state stores, etc. Would we detect this case and report the correct block cache for a given state store?
(Idk how common that pattern could possibly be, but this made me wonder)
##########
File path: docs/ops.html
##########
@@ -2133,6 +2142,136 @@ <h5 class="anchor-heading"><a id="kafka_streams_rocksdb_monitoring" class="ancho
</tbody>
</table>
+ <strong>RocksDB Properties-based Metrics:</strong>
+ All of the following properties-based metrics have a recording level of <code>info</code> and are recorded when the
+ metrics are accessed.
+ If a state store consists of multiple RocksDB instances, as is the case for aggregations over time and session windows,
Review comment:
```suggestion
If a state store consists of multiple RocksDB instances, as is the case for WindowStores and SessionStores,
```
##########
File path: docs/ops.html
##########
@@ -2133,6 +2142,136 @@ <h5 class="anchor-heading"><a id="kafka_streams_rocksdb_monitoring" class="ancho
</tbody>
</table>
+ <strong>RocksDB Properties-based Metrics:</strong>
+ All of the following properties-based metrics have a recording level of <code>info</code> and are recorded when the
+ metrics are accessed.
+ If a state store consists of multiple RocksDB instances, as is the case for aggregations over time and session windows,
+ each metric reports the sum over all the RocksDB instances of the state store, except for the block cache metrics
+ <code>block-cache-*</code>. The block cache metrics report the sum over all RocksDB instances if each instance uses its
+ own block cache, and they report the recorded value from only one instance if a single block cache is shared
+ among all instances.
+
+ <table class="data-table">
+ <tbody>
+ <tr>
+ <th>Metric/Attribute name</th>
+ <th>Description</th>
+ <th>Mbean name</th>
+ </tr>
+ <tr>
+ <td>num-immutable-mem-table</td>
+ <td>The number of immutable memtables that have not yet been flushed.</td>
+ <td>kafka.streams:type=stream-state-metrics,thread-id=([-.\w]+),task-id=([-.\w]+),[store-scope]-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>cur-size-active-mem-table</td>
+ <td>The approximate size of the active memtable in bytes.</td>
+ <td>kafka.streams:type=stream-state-metrics,thread-id=([-.\w]+),task-id=([-.\w]+),[store-scope]-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>cur-size-all-mem-tables</td>
+ <td>The approximate size of active and unflushed immutable memtables in bytes.</td>
+ <td>kafka.streams:type=stream-state-metrics,thread-id=([-.\w]+),task-id=([-.\w]+),[store-scope]-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>size-all-mem-tables</td>
+ <td>The approximate size of active, unflushed immutable, and pinned immutable memtables in bytes.</td>
+ <td>kafka.streams:type=stream-state-metrics,thread-id=([-.\w]+),task-id=([-.\w]+),[store-scope]-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>num-entries-active-mem-table</td>
+ <td>The number of entries in the active memtable.</td>
+ <td>kafka.streams:type=stream-state-metrics,thread-id=([-.\w]+),task-id=([-.\w]+),[store-scope]-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>num-entries-imm-mem-tables</td>
+ <td>The number of entries in the unflushed immutable memtables.</td>
+ <td>kafka.streams:type=stream-state-metrics,thread-id=([-.\w]+),task-id=([-.\w]+),[store-scope]-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>num-deletes-active-mem-table</td>
+ <td>The number of delete entries in the active memtable.</td>
+ <td>kafka.streams:type=stream-state-metrics,thread-id=([-.\w]+),task-id=([-.\w]+),[store-scope]-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>num-deletes-imm-mem-tables</td>
+ <td>The number of delete entries in the unflushed immutable memtables.</td>
+ <td>kafka.streams:type=stream-state-metrics,thread-id=([-.\w]+),task-id=([-.\w]+),[store-scope]-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>mem-table-flush-pending</td>
+ <td>This metric reports 1 if a memtable flush is pending, otherwise it reports 0.</td>
+ <td>kafka.streams:type=stream-state-metrics,thread-id=([-.\w]+),task-id=([-.\w]+),[store-scope]-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>num-running-flushes</td>
+ <td>The number of currently running flushes.</td>
+ <td>kafka.streams:type=stream-state-metrics,thread-id=([-.\w]+),task-id=([-.\w]+),[store-scope]-id=([-.\w]+)</td>
+ </tr>
+ <tr>
+ <td>compaction-pending</td>
+ <td>This metric reports 1 if at least one compaction is pending, otherwise it reports 0.</td>
Review comment:
Do you know if this applies only to a single rocksdb instance, or across all instances? By default rocksdb has a single shared Environment (basically a thread pool) for compactions so it seems like it could go either way. But it's ok if you don't know, I wouldn't waste hours trying to understand the rocksdb code trying to figure it out 😉
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org