Stream-Stream joins - restricting the size of the KV store

[Kishore N C <ki...@gmail.com>]

Hi all,

I just read through this post
<http://samza.apache.org/learn/documentation/0.7.0/container/state-management.html#stream-stream-join>
about implementing streaming joins in Samza using a KV store that Samza
provides. Can someone tell me how I can ensure that this KV storage does
not grow too large (even though I know that this is on disk)? Does Samza
provide a way to purge older keys, or is that expected to be done by the
application code?

Thanks.

KN.

Re: Stream-Stream joins - restricting the size of the KV store

[Chris Riccomini <cr...@apache.org>]

Hey Kishore,

Right now, that has to be done by the application code. Our plan is to
support a TTL in RocksDB once that feature is available to us in the Java
JNI bindings (should be soon). In the meantime, you'll have to implement
the WindowableTask.window() method, and call store.all() to periodically
clear out old keys.

Cheers,
Chris

On Tue, Mar 3, 2015 at 9:08 AM, Kishore N C <ki...@gmail.com> wrote:

> Hi all,
>
> I just read through this post
> <
> http://samza.apache.org/learn/documentation/0.7.0/container/state-management.html#stream-stream-join
> >
> about implementing streaming joins in Samza using a KV store that Samza
> provides. Can someone tell me how I can ensure that this KV storage does
> not grow too large (even though I know that this is on disk)? Does Samza
> provide a way to purge older keys, or is that expected to be done by the
> application code?
>
> Thanks.
>
> KN.
>