You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by Muhammed Ashik <as...@gmail.com> on 2019/07/17 13:25:46 UTC

Print RocksDb Stats

Hi I'm trying to log the rocksdb stats with the below code, but not
observing any logs..
I'm enabling this as the off-heap memory grows indefinitely over a
period of time.
We were using inMemoryKeyValueStore only, I was not sure kafka-streams uses
rockdb as default in memory store.

Kafka Streams version - 2.0.0

class CustomRocksDBConfig extends RocksDBConfigSetter {
  override def setConfig(storeName: String, options: Options, configs:
util.Map[String, AnyRef]): Unit = {

    val stats = new Statistics
    stats.setStatsLevel(StatsLevel.ALL)
    options.setStatistics(stats)
      .setStatsDumpPeriodSec(600)
    options
      .setInfoLogLevel(InfoLogLevel.INFO_LEVEL)
    options.setDbLogDir("/tmp/dump")

  }
}

Re: Print RocksDb Stats

Posted by Bruno Cadonna <br...@confluent.io>.

Hi Muhammed,

RocksDB is not an in-memory store. If you use only
InMemoryKeyValueStore, you are not using any RocksDB.

Best,
Bruno

On Wed, Jul 17, 2019 at 3:26 PM Muhammed Ashik <as...@gmail.com> wrote:
>
> Hi I'm trying to log the rocksdb stats with the below code, but not
> observing any logs..
> I'm enabling this as the off-heap memory grows indefinitely over a
> period of time.
> We were using inMemoryKeyValueStore only, I was not sure kafka-streams uses
> rockdb as default in memory store.
>
> Kafka Streams version - 2.0.0
>
> class CustomRocksDBConfig extends RocksDBConfigSetter {
>   override def setConfig(storeName: String, options: Options, configs:
> util.Map[String, AnyRef]): Unit = {
>
>     val stats = new Statistics
>     stats.setStatsLevel(StatsLevel.ALL)
>     options.setStatistics(stats)
>       .setStatsDumpPeriodSec(600)
>     options
>       .setInfoLogLevel(InfoLogLevel.INFO_LEVEL)
>     options.setDbLogDir("/tmp/dump")
>
>   }
> }

Re: Print RocksDb Stats

Posted by Guozhang Wang <wa...@gmail.com>.

Hello Muhammed,

The community is working on KIP-444 that expose rocksDB metrics. There's an
on-going PR that you may find helpful for your own implementation:
https://github.com/apache/kafka/pull/6884


Guozhang


On Wed, Jul 17, 2019 at 6:26 AM Muhammed Ashik <as...@gmail.com> wrote:

> Hi I'm trying to log the rocksdb stats with the below code, but not
> observing any logs..
> I'm enabling this as the off-heap memory grows indefinitely over a
> period of time.
> We were using inMemoryKeyValueStore only, I was not sure kafka-streams uses
> rockdb as default in memory store.
>
> Kafka Streams version - 2.0.0
>
> class CustomRocksDBConfig extends RocksDBConfigSetter {
>   override def setConfig(storeName: String, options: Options, configs:
> util.Map[String, AnyRef]): Unit = {
>
>     val stats = new Statistics
>     stats.setStatsLevel(StatsLevel.ALL)
>     options.setStatistics(stats)
>       .setStatsDumpPeriodSec(600)
>     options
>       .setInfoLogLevel(InfoLogLevel.INFO_LEVEL)
>     options.setDbLogDir("/tmp/dump")
>
>   }
> }
>


-- 
-- Guozhang