Kafka Stream - Building KTable in Kafka 1.0.0

[Cedric BERTRAND <be...@gmail.com>]

Hello,

In the new api 1.0.0 for building KTable, it is written that No internal
changelod topic is created.

public <K,V> KTable
<https://kafka.apache.org/10/javadoc/org/apache/kafka/streams/kstream/KTable.html><K,V>
table(java.lang.String topic)

Create a KTable
<https://kafka.apache.org/10/javadoc/org/apache/kafka/streams/kstream/KTable.html>
for
the specified topic. The default "auto.offset.reset" strategy and default
key and value deserializers as specified in the config
<https://kafka.apache.org/10/javadoc/org/apache/kafka/streams/StreamsConfig.html>
are
used. Input records
<https://kafka.apache.org/10/javadoc/org/apache/kafka/streams/KeyValue.html>
 with null key will be dropped.

Note that the specified input topics must be partitioned by key. If this is
not the case the returned KTable
<https://kafka.apache.org/10/javadoc/org/apache/kafka/streams/kstream/KTable.html>
will
be corrupted.

The resulting KTable
<https://kafka.apache.org/10/javadoc/org/apache/kafka/streams/kstream/KTable.html>
will
be materialized in a local KeyValueStore
<https://kafka.apache.org/10/javadoc/org/apache/kafka/streams/state/KeyValueStore.html>
with
an internal store name. Note that that store name may not be queriable
through Interactive Queries. *No internal changelog topic is created since
the original input topic can be used for recovery (cf. methods
of KGroupedStream
<https://kafka.apache.org/10/javadoc/org/apache/kafka/streams/kstream/KGroupedStream.html>
and KGroupedTable
<https://kafka.apache.org/10/javadoc/org/apache/kafka/streams/kstream/KGroupedTable.html>
that
return a KTable
<https://kafka.apache.org/10/javadoc/org/apache/kafka/streams/kstream/KTable.html>).*
Parameters:topic - the topic name; cannot be nullReturns:a KTable
<https://kafka.apache.org/10/javadoc/org/apache/kafka/streams/kstream/KTable.html>
for
the specified topic

My code is as followed :KTable<K, V> table = builder.table("my_topic");

When I look at the created topics I can see an internal topic
"application_id-my_topicSTATE-STORE-0000000002-changelog".
Do I missed something ?
Thanks,
Cédric

Re: Kafka Stream - Building KTable in Kafka 1.0.0

[Guozhang Wang <wa...@gmail.com>]

Hello Cédric,

Your observation is correct, and I think we have some obsoleted docs that
we need to fix. In KIP-182 (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-182%3A+Reduce+Streams+DSL+overloads+and+allow+easier+use+of+custom+storage+engines)
we are effectively materializing all state stores with a changelog enabled
by default, but the javadocs have not be updated yet.

We are working on some topology optimization techniques in the near future
to re-enable such optimizations now:
https://issues.apache.org/jira/browse/KAFKA-6034.


Guozhang


On Thu, Mar 29, 2018 at 7:52 AM, Cedric BERTRAND <
bertrandcedric.cbe@gmail.com> wrote:

> Hello,
>
> In the new api 1.0.0 for building KTable, it is written that No internal
> changelod topic is created.
>
> public <K,V> KTable
> <https://kafka.apache.org/10/javadoc/org/apache/kafka/
> streams/kstream/KTable.html><K,V>
> table(java.lang.String topic)
>
> Create a KTable
> <https://kafka.apache.org/10/javadoc/org/apache/kafka/
> streams/kstream/KTable.html>
> for
> the specified topic. The default "auto.offset.reset" strategy and default
> key and value deserializers as specified in the config
> <https://kafka.apache.org/10/javadoc/org/apache/kafka/
> streams/StreamsConfig.html>
> are
> used. Input records
> <https://kafka.apache.org/10/javadoc/org/apache/kafka/
> streams/KeyValue.html>
>  with null key will be dropped.
>
> Note that the specified input topics must be partitioned by key. If this is
> not the case the returned KTable
> <https://kafka.apache.org/10/javadoc/org/apache/kafka/
> streams/kstream/KTable.html>
> will
> be corrupted.
>
> The resulting KTable
> <https://kafka.apache.org/10/javadoc/org/apache/kafka/
> streams/kstream/KTable.html>
> will
> be materialized in a local KeyValueStore
> <https://kafka.apache.org/10/javadoc/org/apache/kafka/
> streams/state/KeyValueStore.html>
> with
> an internal store name. Note that that store name may not be queriable
> through Interactive Queries. *No internal changelog topic is created since
> the original input topic can be used for recovery (cf. methods
> of KGroupedStream
> <https://kafka.apache.org/10/javadoc/org/apache/kafka/streams/kstream/
> KGroupedStream.html>
> and KGroupedTable
> <https://kafka.apache.org/10/javadoc/org/apache/kafka/
> streams/kstream/KGroupedTable.html>
> that
> return a KTable
> <https://kafka.apache.org/10/javadoc/org/apache/kafka/
> streams/kstream/KTable.html>).*
> Parameters:topic - the topic name; cannot be nullReturns:a KTable
> <https://kafka.apache.org/10/javadoc/org/apache/kafka/
> streams/kstream/KTable.html>
> for
> the specified topic
>
> My code is as followed :KTable<K, V> table = builder.table("my_topic");
>
> When I look at the created topics I can see an internal topic
> "application_id-my_topicSTATE-STORE-0000000002-changelog".
> Do I missed something ?
> Thanks,
> Cédric
>



-- 
-- Guozhang