[jira] [Created] (KAFKA-6036) Enable logical materialization to physical materialization

["Guozhang Wang (JIRA)" <ji...@apache.org>]

Guozhang Wang created KAFKA-6036:
------------------------------------

             Summary: Enable logical materialization to physical materialization
                 Key: KAFKA-6036
                 URL: https://issues.apache.org/jira/browse/KAFKA-6036
             Project: Kafka
          Issue Type: Sub-task
            Reporter: Guozhang Wang


Today whenever users specify a queryable store name for KTable, we would always add a physical state store in the translated processor topology.

For some scenarios, we should consider not physically materialize the KTable but only "logically" materialize it when you have some simple transformation operations or even join operations that generated new KTables, and which needs to be materialized with a state store, you can use the changelog topic of the previous KTable and applies the transformation logic upon restoration instead of creating a new changelog topic. For example:

{code}
table1 = builder.table("topic1");
table2 = table1.filter(..).join(table3); // table2 needs to be materialized for joining
{code}

We can actually set the {{getter}} function of table2's materialized store, say {{state2}} to be reading from {{topic1}} and then apply the filter operator, instead of creating a new {{state2-changelog}} topic in this case.

We can come up with a general internal impl optimizations to determine when to logically materialize, and whether we should actually allow users of DSL to "hint" whether to materialize or not (it then may need a KIP).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)