[jira] [Commented] (KAFKA-7777) Decouple topic serdes from materialized serdes

["Guozhang Wang (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-7777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17169176#comment-17169176 ] 

Guozhang Wang commented on KAFKA-7777:
--------------------------------------

Another walkaround for now, is to use `KTable#mapValues()` in which we can project out the unwanted fields and then materialize. More specifically, you can write:

builder.table("topic", Consumed)           // do not use Materialized to enforce materializing the source table
           .mapValues(..., Materialized)       // project out those unwanted fields, and then Materialize with the new serde
           
As a result only one store would be created for the resulted KTable after the mapValues.

> Decouple topic serdes from materialized serdes
> ----------------------------------------------
>
>                 Key: KAFKA-7777
>                 URL: https://issues.apache.org/jira/browse/KAFKA-7777
>             Project: Kafka
>          Issue Type: Wish
>          Components: streams
>            Reporter: Maarten
>            Priority: Minor
>              Labels: needs-kip
>
> It would be valuable to us to have the the encoding format in a Kafka topic decoupled from the encoding format used to cache the data locally in a kafka streams app. 
> We would like to use the `range()` function in the interactive queries API to look up a series of results, but can't with our encoding scheme due to our keys being variable length.
> We use protobuf, but based on what I've read Avro, Flatbuffers and Cap'n proto have similar problems.
> Currently we use the following code to work around this problem:
> {code}
> builder
>     .stream("input-topic", Consumed.with(inputKeySerde, inputValueSerde))
>     .to("intermediate-topic", Produced.with(intermediateKeySerde, intermediateValueSerde)); 
> t1 = builder
>     .table("intermediate-topic", Consumed.with(intermediateKeySerde, intermediateValueSerde), t1Materialized);
> {code}
> With the encoding formats decoupled, the code above could be reduced to a single step, not requiring an intermediate topic.
> Based on feedback on my [SO question|https://stackoverflow.com/questions/53913571/is-there-a-way-to-separate-kafka-topic-serdes-from-materialized-serdes] a change that introduces this would impact state restoration when using an input topic for recovery.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)