[jira] [Comment Edited] (KAFKA-18053) Clean up TopologyConfig and API for supplying configs needed by the topology

["Sebastien Viale (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17902246#comment-17902246 ] 

Sebastien Viale edited comment on KAFKA-18053 at 12/2/24 12:21 PM:
-------------------------------------------------------------------

[~ableegoldman]  [~mjsax] 

I added a comment in the discussion thread of the KIP-1111 https://lists.apache.org/thread/k05o50clxtjv0x748sw2wj5jj4389tc0]

Everything is clear to me.

I must say that I sometimes felt confused with the constructors, but now everything is much clearer. Based on my experience and discussions with my teammates, I believe this is a good opportunity to simplify and clarify the implementation.

The question is whether this should be part of a separate KIP or can it be part of the KIP-1111 ([https://cwiki.apache.org/confluence/display/KAFKA/KIP-1111%3A+Enforcing+Explicit+Naming+for+Kafka+Streams+Internal+Topics])
In both case, I would be happy to work on that !

 


was (Author: JIRAUSER304445):
[~ableegoldman]  [~mjsax] 

I added a comment in the discussion thread of the KIP-1111[ (https://lists.apache.org/thread/k05o50clxtjv0x748sw2wj5jj4389tc0):|https://lists.apache.org/thread/k05o50clxtjv0x748sw2wj5jj4389tc0]

Everything is clear to me.

I must say that I sometimes felt confused with the constructors, but now everything is much clearer. Based on my experience and discussions with my teammates, I believe this is a good opportunity to simplify and clarify the implementation.


The question is whether this should be part of a separate KIP or can it be part of the KIP-1111 (https://cwiki.apache.org/confluence/display/KAFKA/KIP-1111%3A+Enforcing+Explicit+Naming+for+Kafka+Streams+Internal+Topics)
In both case, I would be happy to work on that !

 

> Clean up TopologyConfig and API for supplying configs needed by the topology
> ----------------------------------------------------------------------------
>
>                 Key: KAFKA-18053
>                 URL: https://issues.apache.org/jira/browse/KAFKA-18053
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>            Reporter: A. Sophie Blee-Goldman
>            Priority: Major
>              Labels: needs-kip
>
> As mentioned briefly in KIP-1112, the current API around configs which need to be applied prior to building the topology can be cumbersome and easy to implement incorrectly. This leads to awkward code and confused users at best, and applications that silently drop some configs at worst.
> There are three main problems we want to solve:
>  # The most serious issue is that the KafkaStreams constructor requires the application configs whereas the topology can be built without them, so most people will only pass their configs into KafkaStreams. However certain configs have to be passed into the topology as well, which is easy for users to miss and leads to a silent misconfiguration that can easily go unnoticed
>  # Over time we've added several new APIs for configs that need to be applied at different/earlier stages of the topology building process. So now users are faced with multiple ways of passing these configs into the topology, and configs that may work with any of them or may need to be used with a specific API
>  # At some point we also made the TopologyConfig class public, which was originally intended only for internal use. Furthermore it's only purpose was to separate configs between different topologies, a feature that has now been deprecated and will soon be removed, at which point TopologyConfig serves no purpose and should be removed as well. Unfortunately, the newest APIs for passing configs into a topology are a Topology and StreamsBuilder constructor that have a TopologyConfig  parameter, rather than the StreamsConfig parameter used in the previous API (ie StreamsBuilder#build). This is unnecessary since StreamsConfig would work just as well, and in fact a TopologyConfig has to be constructed from a StreamsConfig so users are just adding a wrapper layer
> As of filing this ticket, the list of topology-specific configs is:
>  * topology.optimization
>  * processor.wrapper.class
> However there are new configs in this category being proposed in other KIPs currently being discussed



--
This message was sent by Atlassian Jira
(v8.20.10#820010)