Kafka Streams table persistence

[Jan Ehrhardt <ja...@gmail.com>]

Hi,

I am curious about the relationship of KTables and state store in Kafka
Streams. When I create a table with the `table` method, it uses the
original topic as a changelog, thus persisting a table changelog does not
make much sense. When I create invoke `countByKey` on a KStream, the result
is a KTable and I find a changelog topic for it created. Thus Kafka Streams
can recreate the state store. When I join two tables, this will also result
in a new KTable, but this time, there is no changelog topic created.

This is a little bit confusing. Does the `table` method read the whole
original topic as an input again after a restart, or how does it load the
whole table again? Can someone explain the rules to persist or restore a
KTable to or from a changelog?

Best regards
Jan

Re: Kafka Streams table persistence

[Eno Thereska <en...@gmail.com>]

Hi Jan,

It's a good question. Your observations are correct. A KTable is sometimes materialised to a changelog topic lazily, depending on whether the result in the KTable is needed for subsequent operations or not.

We are working to improving the documentation of the materialisation rules and should have something shortly to share on the wiki https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams+Discussions.

Thanks
Eno

> On 16 Jun 2016, at 12:45, Jan Ehrhardt <ja...@gmail.com> wrote:
> 
> Hi,
> 
> I am curious about the relationship of KTables and state store in Kafka
> Streams. When I create a table with the `table` method, it uses the
> original topic as a changelog, thus persisting a table changelog does not
> make much sense. When I create invoke `countByKey` on a KStream, the result
> is a KTable and I find a changelog topic for it created. Thus Kafka Streams
> can recreate the state store. When I join two tables, this will also result
> in a new KTable, but this time, there is no changelog topic created.
> 
> This is a little bit confusing. Does the `table` method read the whole
> original topic as an input again after a restart, or how does it load the
> whole table again? Can someone explain the rules to persist or restore a
> KTable to or from a changelog?
> 
> Best regards
> Jan