RocksDB state store disk space estimation

[Chris Toomey <ct...@gmail.com>]

We're using RocksDB as a persistent Kafka state store for compacted topics
and need to be able to estimate the maximum disk space required.

We're using the default config. settings provided by Kafka, which include
Universal compaction, no compression, and 4k block size.

Given these settings and a topic w/ key size K, value size V, and number of
records R, I'd assume a rough disk space estimation would be of the form

max. disk space = (K+V)*R*M*2

where M is an unknown DB size -> disk size multiplier and *2 is to allow
for full compaction as per here
<https://github.com/facebook/rocksdb/wiki/Universal-Compaction>.

Does this look right, and can anyone provide a ballpark range for the
multiplier M and/or some guidelines for how to estimate it?

much thanks,
Chris

Re: RocksDB state store disk space estimation

[Chris Toomey <ct...@gmail.com>]

Thanks Bruno.

On Thu, Feb 18, 2021 at 9:19 AM Bruno Cadonna <br...@confluent.io> wrote:

> Hi Chris,
>
> your estimation looks correct to me.
>
> I do not know how big M might be. Maybe the following link can help you
> with the estimation:
>
> https://github.com/facebook/rocksdb/wiki/Rocksdb-BlockBasedTable-Format
>
> There are also some additional files that RocksDB keeps in its
> directory. I guess the best way to estimate the space is experimentally.
>
> Also take into account that, you will have one state store per partition.
>
> If you want to save disk space, you should try to use Leveled compaction
> (https://github.com/facebook/rocksdb/wiki/Leveled-Compaction) instead
> since it has a space amplification of 10% instead of 100% with Universal
> compaction. That is, you can replace the 2 in your formula with 1.1.
>
> Since AK 2.7, you can also monitor the sizes of your RocksDB state
> stores with the metric total-sst-files-size
> (https://kafka.apache.org/documentation/#kafka_streams_rocksdb_monitoring)
>
> Best,
> Bruno
>
> On 18.02.21 17:43, Chris Toomey wrote:
> > We're using RocksDB as a persistent Kafka state store for compacted
> topics
> > and need to be able to estimate the maximum disk space required.
> >
> > We're using the default config. settings provided by Kafka, which include
> > Universal compaction, no compression, and 4k block size.
> >
> > Given these settings and a topic w/ key size K, value size V, and number
> of
> > records R, I'd assume a rough disk space estimation would be of the form
> >
> > max. disk space = (K+V)*R*M*2
> >
> > where M is an unknown DB size -> disk size multiplier and *2 is to allow
> > for full compaction as per here
> > <https://github.com/facebook/rocksdb/wiki/Universal-Compaction>.
> >
> > Does this look right, and can anyone provide a ballpark range for the
> > multiplier M and/or some guidelines for how to estimate it?
> >
> > much thanks,
> > Chris
> >
>

Re: RocksDB state store disk space estimation

[Bruno Cadonna <br...@confluent.io>]

Hi Chris,

your estimation looks correct to me.

I do not know how big M might be. Maybe the following link can help you 
with the estimation:

https://github.com/facebook/rocksdb/wiki/Rocksdb-BlockBasedTable-Format

There are also some additional files that RocksDB keeps in its 
directory. I guess the best way to estimate the space is experimentally.

Also take into account that, you will have one state store per partition.

If you want to save disk space, you should try to use Leveled compaction 
(https://github.com/facebook/rocksdb/wiki/Leveled-Compaction) instead 
since it has a space amplification of 10% instead of 100% with Universal 
compaction. That is, you can replace the 2 in your formula with 1.1.

Since AK 2.7, you can also monitor the sizes of your RocksDB state 
stores with the metric total-sst-files-size 
(https://kafka.apache.org/documentation/#kafka_streams_rocksdb_monitoring)

Best,
Bruno

On 18.02.21 17:43, Chris Toomey wrote:
> We're using RocksDB as a persistent Kafka state store for compacted topics
> and need to be able to estimate the maximum disk space required.
> 
> We're using the default config. settings provided by Kafka, which include
> Universal compaction, no compression, and 4k block size.
> 
> Given these settings and a topic w/ key size K, value size V, and number of
> records R, I'd assume a rough disk space estimation would be of the form
> 
> max. disk space = (K+V)*R*M*2
> 
> where M is an unknown DB size -> disk size multiplier and *2 is to allow
> for full compaction as per here
> <https://github.com/facebook/rocksdb/wiki/Universal-Compaction>.
> 
> Does this look right, and can anyone provide a ballpark range for the
> multiplier M and/or some guidelines for how to estimate it?
> 
> much thanks,
> Chris
>