[jira] [Commented] (KAFKA-20593) Log a diagnostic warning at startup when Kafka Streams detects internal repartition topics

["Jahnavi Patel (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-20593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18081325#comment-18081325 ] 

Jahnavi Patel commented on KAFKA-20593:
---------------------------------------

Can I assign this ticket to myself? I'd like to work on it.

> Log a diagnostic warning at startup when Kafka Streams detects internal repartition topics
> ------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-20593
>                 URL: https://issues.apache.org/jira/browse/KAFKA-20593
>             Project: Kafka
>          Issue Type: New Feature
>          Components: streams
>            Reporter: Ankur Sinha
>            Priority: Minor
>
> *The Problem*
> When a topology changes a record key (e.g., .selectKey(), .map()) and subsequently calls a stateful operation (e.g., .groupByKey(), .join()), Kafka Streams automatically provisions and manages internal repartition topics on the broker cluster.While this is core architectural behavior, these topics are created implicitly. Developers frequently introduce accidental, highly expensive network shuffles without realizing the operational and cloud infrastructure cost impact.Currently, discovering these requires manually printing Topology.describe() or digging through noisy, verbose consumer group rebalance logs long after the application has started. For example, a developer today has to parse runtime blocks like this just to find hidden shuffles:
> {code:java}
> 2026-05-11T16:18:33.861Z  INFO 1 --- [k-streams] [-StreamThread-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=k-streams-1-consumer, groupId=k-streams_app_id] Updating assignment with
>         Assigned partitions: [Topic1-3, k-streams_app_id-Topic2-repartition-1, k-streams_app_id-Topic2-repartition-5, k-streams_app_id-Topic3-repartition-3...]
> {code}
> This log makes it difficult to separate actual data storage from internal shuffle infrastructure, and it fails to explain which specific operator triggered the repartition.
> *The Solution*
> Add a configuration property that scans the compiled topology during initialization. If Kafka Streams detects that internal repartition topics will be generated on the cluster, it will log a clean, structured WARN block explicitly detailing them before the application begins processing data.New Configuration Property:
> {code:java}
> streams.warn.on.repartition (Boolean, default: true)
> {code}
> Proposed Warning Log Output Example:
> {code:java}
> WARN  org.apache.kafka.streams.KafkaStreams - [Topology Diagnostics] Internal repartition topics detected:
> 1. Topic: k-streams_app_id-Topic1-repartition
>    Trigger Operator: KSTREAM-AGGREGATE-0000000003 (groupByKey)
>    Upstream Cause: Key was flagged as changed by an upstream operator (selectKey)
> 2. Topic: k-streams_app_id-Topic2-repartition
>    Trigger Operator: KSTREAM-JOIN-0000000008 (join)
> {code}
> *Impact & CompatibilityBackward* 
> * Compatible: Yes. It only introduces diagnostic log messages. Existing applications will run completely unchanged.
> * Performance Impact: Negligible. The structural topology scan runs exactly once during the KafkaStreams startup sequence.
> * Opt-out: Users can disable this output by setting streams.warn.on.repartition=false.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)