You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Aaron Levin <aa...@stripe.com> on 2020/04/08 17:02:36 UTC

ListState with millions of elements

Hello friendly Flink community!

I'm curious if anyone has operational experience with jobs that store
ListState where occasionally, due to skew, some small number of lists
stored in ListState (stored in RocksDB) will have millions of elements.
Here are the stats:

* millions of keys
* p95 size of list in ListState is ~2.
* some small number of keys (less than 100) may have lists whose size is on
the order of tens of thousands and up to millions.
* state is stored in RocksDB

Are there any known issues or limitations with storing or fetching that
much list state out of RocksDB? I realize fetching from RocksDB and
deserializing will be costly when hitting a key with a list of a million
elements, but is there anything else we should consider?

Thanks!

Best,

Aaron Levin

Re: ListState with millions of elements

Posted by Aljoscha Krettek <al...@apache.org>.
On 08.04.20 20:14, Seth Wiesman wrote:
> There is a limitation in RocksDB's JNI bridge that will cause applications
> to fail if list state exceeds 2GB. I am not aware of anyone working on this
> issue.

That is correct, and here's the Jira issue for it: 
https://issues.apache.org/jira/browse/FLINK-6761

There's also another issue about changing RocksDB list state to not 
store the whole ListState in a single RocksDB value but instead store 
the individual ListState entries in individual RocksDB values, under a 
specially formed "sequential" key. This is the issue: 
https://issues.apache.org/jira/browse/FLINK-8297

Best,
Aljoscha

Re: ListState with millions of elements

Posted by Seth Wiesman <sj...@gmail.com>.
There is a limitation in RocksDB's JNI bridge that will cause applications
to fail if list state exceeds 2GB. I am not aware of anyone working on this
issue.

Seth.

[1] https://github.com/facebook/rocksdb/issues/2383

On Wed, Apr 8, 2020 at 12:02 PM Aaron Levin <aa...@stripe.com> wrote:

> Hello friendly Flink community!
>
> I'm curious if anyone has operational experience with jobs that store
> ListState where occasionally, due to skew, some small number of lists
> stored in ListState (stored in RocksDB) will have millions of elements.
> Here are the stats:
>
> * millions of keys
> * p95 size of list in ListState is ~2.
> * some small number of keys (less than 100) may have lists whose size is
> on the order of tens of thousands and up to millions.
> * state is stored in RocksDB
>
> Are there any known issues or limitations with storing or fetching that
> much list state out of RocksDB? I realize fetching from RocksDB and
> deserializing will be costly when hitting a key with a list of a million
> elements, but is there anything else we should consider?
>
> Thanks!
>
> Best,
>
> Aaron Levin
>