[jira] [Comment Edited] (KAFKA-19935) Introduce TopologyValidator Utils for improving topology evolution comptiability.

["sanghyeok An (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-19935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18041421#comment-18041421 ] 

sanghyeok An edited comment on KAFKA-19935 at 11/29/25 2:26 AM:
----------------------------------------------------------------

[~mjsax] Hi!

Thank you for the thoughtful feedback. 
I completely agree with your point regarding the false positive alert. 
Attempting to mathematically guarantee 100% compatibility for all logical topology shifts is indeed extremely difficult and could be risky if users blindly trust it.

To address this and the implementation concerns, 
I propose refining the scope and approach.

 

*1. Implementation Strategy: Snapshot Testing*

You rightly asked how we would implement the detection since we cannot instantiate the previous topology object within the modified codebase.
I propose adopting a Snapshot Testing pattern (similar to Jest Snapshots in frontend or Approval Testing in Java).
 * Baseline{*}:{*} The user saves the stable {{TopologyDescription}} (via {{{}topology.describe().toString(){}}}) as a text file in the repository.

 * Validation{*}:{*} During unit tests or CI, the validator parses this baseline file and compares it against the _current_ {{Topology}} structure.

This approach solves the Time Paradox of needing the old object and allows for a reliable comparison of structural changes.

 

*2. Scope: Modular, Rule-Based Validator* 

Instead of a generic {{.diff()}} that tries to interpret intent, we can offer specific, opt-in rules. 
The primary use case is not to validate logic, but to catch {*}unintentional side effects{*}.
 * {{StateStoreNameCompatibility}} (Critical){*}:{*} This rule strictly compares the set of StateStore names. If {{STORE-0001}} disappears and {{STORE-0002}} appears, it flags a violation. 

 * 
 ** _Goal:_ Prevent accidental data loss due to ID shifting.

 * {{{}SourceTopicCompatibility{}}}{*}:{*} Ensures source topics remain consistent.

then we can throw an errors or print WARN log. Also, we can add an new Rule Interfaces for other rule.

 

*3. User Adoption*
Regarding user adoption, You are right that users unaware of Named Stores might miss this tool. 
However, if integrated into {{{}streams-test-utils{}}}, this tool acts as a Topology Linter in local unit tests.

When a developer runs a test, they would get immediate feedback. For instance:{*}{{*}}

*Error: StateStore names have changed. Did you mean to name your operators? See [link] for details.*

This effectively let the user know about the importance of naming stores, rather than just silently allowing the change.

 

*4. API Usage*
{code:java}
@Test
public void shouldKeepTopologyCompatible() {
    String previousDescription = Files.readString(Paths.get("src/test/resources/topology-baseline.txt"));
    String currentDescription = builder.build().describe().toString();

    TopologyValidator validator = new TopologyValidatorBuilder()
                                    .addRule(new StateStoreNameCompatibility())
                                    .addRule(new SourceTopicCompatibility())
                                    .build()
    
    // Option1. validator could throw errors such as StateStoreNameCompatibilityViolation()
    // Option2. validator print WARN log such as "StateStore names have changed. previous: [A, B], current: [A, C]"
    vaildator.check(previousDescription, currentDescription);
} {code}
This utility would be valuable for both local development (fast feedback) and CI pipelines (safety net).

 

In summary, by narrowing the scope to Snapshot-based Structural Validation, 
I believe we can provide a high-value utility that prevents common operational disasters without over-promising logical compatibility.
What do you think about this "Snapshot & Rule-based" direction?


was (Author: JIRAUSER303328):
[~mjsax] Hi!


Thank you for the thoughtful feedback. 
I completely agree with your point regarding the false positive alert. 
Attempting to mathematically guarantee 100% compatibility for all logical topology shifts is indeed extremely difficult and could be risky if users blindly trust it.

To address this and the implementation concerns, 
I propose refining the scope and approach in two key ways:


*1. Implementation Strategy: Snapshot Testing*

You rightly asked how we would implement the detection since we cannot instantiate the previous topology object within the modified codebase.
I propose adopting a Snapshot Testing pattern (similar to Jest Snapshots in frontend or Approval Testing in Java).
 * Baseline{*}:{*} The user saves the stable {{TopologyDescription}} (via {{{}topology.describe().toString(){}}}) as a text file in the repository.

 * Validation{*}:{*} During unit tests or CI, the validator parses this baseline file and compares it against the _current_ {{Topology}} structure.

This approach solves the Time Paradox of needing the old object and allows for a reliable comparison of structural changes.

 

*2. Scope: Modular, Rule-Based Validator* 

Instead of a generic {{.diff()}} that tries to interpret intent, we can offer specific, opt-in rules. 
The primary use case is not to validate logic, but to catch {*}unintentional side effects{*}.
 * {{StateStoreNameCompatibility}} (Critical){*}:{*} This rule strictly compares the set of StateStore names. If {{STORE-0001}} disappears and {{STORE-0002}} appears, it flags a violation. 

 ** _Goal:_ Prevent accidental data loss due to ID shifting.

 * {{{}SourceTopicCompatibility{}}}{*}:{*} Ensures source topics remain consistent.

then we can throw an errors or print WARN log. Also, we can add an new Rule Interfaces for other rule.

 

*3. User Adoption*
Regarding user adoption, You are right that users unaware of Named Stores might miss this tool. 
However, if integrated into {{{}streams-test-utils{}}}, this tool acts as a Topology Linter in local unit tests.


When a developer runs a test, they would get immediate feedback. For instance:{*}{*}

*Error: StateStore names have changed. Did you mean to name your operators? See [link] for details.*

This effectively let the user know about the importance of naming stores, rather than just silently allowing the change.

 

*4. API Usage*
{code:java}
@Test
public void shouldKeepTopologyCompatible() {
    String previousDescription = Files.readString(Paths.get("src/test/resources/topology-baseline.txt"));
    String currentDescription = builder.build().describe().toString();

    TopologyValidator validator = new TopologyValidatorBuilder()
                                    .addRule(new StateStoreNameCompatibility())
                                    .addRule(new SourceTopicCompatibility())
                                    .build()
    
    // Option1. validator could throw errors such as StateStoreNameCompatibilityViolation()
    // Option2. validator print WARN log such as "StateStore names have changed. previous: [A, B], current: [A, C]"
    vaildator.check(previousDescription, currentDescription);
} {code}
This utility would be valuable for both local development (fast feedback) and CI pipelines (safety net).

 

 

In summary, by narrowing the scope to Snapshot-based Structural Validation, 
I believe we can provide a high-value utility that prevents common operational disasters without over-promising logical compatibility.
What do you think about this "Snapshot & Rule-based" direction?

> Introduce TopologyValidator Utils for improving topology evolution comptiability.
> ---------------------------------------------------------------------------------
>
>                 Key: KAFKA-19935
>                 URL: https://issues.apache.org/jira/browse/KAFKA-19935
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams-test-utils
>            Reporter: sanghyeok An
>            Assignee: sanghyeok An
>            Priority: Minor
>              Labels: needs-kip
>
> If a developer does not explicitly specify a name for a {{{}StateStore{}}}, its name is generated using an incremental number. Consequently, the corresponding Changelog topic is created using this same generated name.
> Let's assume a scenario where the application evolves and a new Source Node is added. Since the new Source Node is typically built at the beginning of the topology, the incremental numbers for all subsequent nodes shift by 1. As a result, the names of the {{{}StateStore{}}}s change as well.
> For example, the change might look like this:
>  * *Previous:* {{MY-APP-KSTREAM-AGGREGATE-STATE-STORE-0000000001-changelog}}
>  * *New:* {{MY-APP-KSTREAM-AGGREGATE-STATE-STORE-0000000002-changelog}}
>  
> This shift can lead to significant operational issues. Simply replaying the Source Topic is often insufficient to make the new {{StateStore}} identical to the previous one.
>  
> There are several scenarios where this is impossible. For instance, if the application only processes messages after the last commit(most case) and the data includes keys that appear infrequently, the state will never match. (Consider a restaurant with very few orders compared to one with many; the "sparse" orders might have existed in the previous StateStore but will likely be missing in the new one after the shift).
> While this might be dismissed as a minor issue by some, it can be a critical problem for organizations where data consistency is paramount.
>  
> A clear solution to this problem is using {*}Named StateStores{*}. However, some users may not be aware of this feature or may not feel the need for it. Furthermore, they might not realize that the IDs of the StateStore and Changelog topic have incremented, leading to the unintentional creation of a new store and potential data loss.
> To address this, I propose introducing a {{TopologyValidator}} as a utility class. Ideally, the usage would look something like this:
> {code:java}
> TopologyValidator.of(prevTopology, newTopology).diff(); {code}
>  * When {{.diff()}} is called, the {{TopologyValidator}} would identify changes in the topology and issue a warning if {{StateStore}} IDs have shifted.
>  * The {{TopologyValidator}} could be included in the Kafka Streams application code or integrated into CI pipelines. Since it essentially requires only the code to build the topology, it does not need a connection to an actual broker. Therefore, we can verify the diff using just the previous and new topology definitions.
>  
> By introducing {{TopologyValidator}} under the {{streams/test-utils}} module and encouraging users to utilize it, I believe we can promote greater stability from the perspective of {*}Topology Evolution Compatibility{*}.
>  
> Please give your opinion. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)