[jira] [Commented] (KAFKA-12453) Guidance on whether a topology is eligible for optimisation

["Matthias J. Sax (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-12453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17299935#comment-17299935 ] 

Matthias J. Sax commented on KAFKA-12453:
-----------------------------------------

Personally, I think it might be better to make the runtime smarter and allow it to have two types of repartition topics – regular repartition topics (as we have them today) and "table repartition topics" (for a lack of a good name) – for the new "table repartition topics" that might inserted via `toTable()` we should configure the topic with log compaction and skip the "purge data" calls – this way, enabling topology optimization becomes save.

> Guidance on whether a topology is eligible for optimisation
> -----------------------------------------------------------
>
>                 Key: KAFKA-12453
>                 URL: https://issues.apache.org/jira/browse/KAFKA-12453
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>            Reporter: Patrick O'Keeffe
>            Priority: Major
>
> Since the introduction of KStream.toTable() in Kafka 2.6.x, the decision about whether a topology is eligible for optimisation is no longer a simple one, and is related to whether toTable() operations are preceded by key changing operators.
> This decision requires expert level knowledge, and there are serious implications associated with getting it wrong in terms of fault tolerance
> Some ideas spring to mind around how to guide developers to make the correct decision:
>  # Topology.describe() could indicate whether this topology is eligible for optimisation
>  # Topologies could be automatically optimised - note this may have an impact at deployment time, in that an application reset may be required. The developer would need to made aware of this and adjust the deployment plan accordingly
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)