[jira] [Created] (KAFKA-15541) RocksDB Iterator Metrics

["Nicholas Telford (Jira)" <ji...@apache.org>]

Nicholas Telford created KAFKA-15541:
----------------------------------------

             Summary: RocksDB Iterator Metrics
                 Key: KAFKA-15541
                 URL: https://issues.apache.org/jira/browse/KAFKA-15541
             Project: Kafka
          Issue Type: Improvement
          Components: streams
            Reporter: Nicholas Telford
            Assignee: Nicholas Telford


[KIP-989: RocksDB Iterator Metrics|https://cwiki.apache.org/confluence/display/KAFKA/KIP-989%3A+RocksDB+Iterator+Metrics]

RocksDB {{Iterators}} must be closed after use, to prevent memory leaks due to [blocks being "pinned" in-memory|https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB#blocks-pinned-by-iterators]. Pinned blocks can currently be tracked via the per-store {{block-cache-pinned-usage}} metric. However, it's common [(and even recommended)|https://docs.confluent.io/platform/current/streams/developer-guide/memory-mgmt.html#rocksdb] to share the Block Cache among all stores in an application, to enable users to globally bound native memory used by RocksDB. This results in the {{block-cache-pinned-usage}} reporting the same memory usage for every store in the application, irrespective of which store is actually pinning blocks in the block cache.

To aid users in finding leaked Iterators, as well as identifying the cause of a high number of pinned blocks, we introduce two new metrics.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)