You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "lucasbru (via GitHub)" <gi...@apache.org> on 2026/09/09 13:58:17 UTC
[PR] MINOR: Add cross-version compatibility tests for the KIP-1331 topology description plugin [kafka]
lucasbru opened a new pull request, #23413:
URL: https://github.com/apache/kafka/pull/23413
The heartbeat- and describe-based topology push protocol added by KIP-1331 relies on StreamsGroupHeartbeat and StreamsGroupDescribe negotiating down to earlier versions when talking to a broker or client that predates the feature, and on the admin client refusing to send a topology description request a broker cannot understand. This was previously only checked by unit tests exercising the version-gating logic directly; nothing ran an actual pre-4.4 Kafka broker or Streams client against the feature to prove the real wire negotiation degrades gracefully in both directions.
This adds three system tests covering the missing directions: a Streams client built from this branch (topology push enabled by default) talking to a genuine 4.3.1 broker, a genuine 4.3.1 Streams client talking to a broker on this branch with the plugin configured, and the admin client/CLI on this branch requesting a topology description from a 4.3.1 broker, which must fail client-side with UnsupportedVersionException rather than silently describing the group without it.
## Test plan
All three tests were run against real ducker containers (docker-based ducktape harness) to confirm they exercise the intended code paths rather than passing vacuously:
- Confirmed via broker/client logs that the pinned old-version processes actually ran Kafka 4.3.1, not the dev-branch build.
- Confirmed the admin-client test's failure text matches `UnsupportedVersionException: Attempted to write a non-default includeTopologyDescription at version 0` with exit code 1.
- Re-ran the full `streams_topology_description_plugin_test.py` suite (9 tests) plus the new smoke test together; all pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add cross-version compatibility tests for the KIP-1331 topology description plugin [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #23413:
URL: https://github.com/apache/kafka/pull/23413#discussion_r3970817505
##########
tests/kafkatest/tests/streams/streams_smoke_test.py:
##########
@@ -122,3 +123,45 @@ def test_streams(self, processing_guarantee, crash, metadata_quorum, group_proto
processor3.stop()
self.driver.node.account.ssh("grep SUCCESS %s" % self.driver.STDOUT_FILE, allow_fail=False)
+
+ @cluster(num_nodes=5)
+ @matrix(metadata_quorum=[quorum.combined_kraft])
+ def test_old_client_not_solicited_for_topology_push(self, metadata_quorum):
+ """
+ Test the situation when a pre-KIP-1331 Kafka Streams client uses the streams group
+ protocol against a broker on this branch with the topology description plugin configured
+ (the default broker setup from BaseStreamsTest). StreamsGroupHeartbeat negotiates down to
+ version 0 for this client, so the broker must never solicit a topology description push
+ for it.
+ """
+ processor = StreamsSmokeTestJobRunnerService(self.test_context, self.kafka, 'at_least_once', 'streams')
+ processor.set_version(str(LATEST_4_3))
+
+ broker_node = self.kafka.nodes[0]
Review Comment:
This test inherits the `BaseStreamsTest` setup, which runs 3 brokers, should we grep all of them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add cross-version compatibility tests for the KIP-1331 topology description plugin [kafka]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #23413:
URL: https://github.com/apache/kafka/pull/23413#discussion_r3971560629
##########
tests/kafkatest/tests/streams/streams_smoke_test.py:
##########
@@ -122,3 +123,45 @@ def test_streams(self, processing_guarantee, crash, metadata_quorum, group_proto
processor3.stop()
self.driver.node.account.ssh("grep SUCCESS %s" % self.driver.STDOUT_FILE, allow_fail=False)
+
+ @cluster(num_nodes=5)
+ @matrix(metadata_quorum=[quorum.combined_kraft])
+ def test_old_client_not_solicited_for_topology_push(self, metadata_quorum):
+ """
+ Test the situation when a pre-KIP-1331 Kafka Streams client uses the streams group
+ protocol against a broker on this branch with the topology description plugin configured
+ (the default broker setup from BaseStreamsTest). StreamsGroupHeartbeat negotiates down to
+ version 0 for this client, so the broker must never solicit a topology description push
+ for it.
+ """
+ processor = StreamsSmokeTestJobRunnerService(self.test_context, self.kafka, 'at_least_once', 'streams')
+ processor.set_version(str(LATEST_4_3))
+
+ broker_node = self.kafka.nodes[0]
Review Comment:
This only checks broker 0, but `BaseStreamsTest` starts three brokers (`base_streams_test.py:32,39-40`) and the heartbeat is processed by the broker leading this group's `__consumer_offsets` partition. If another broker is the coordinator, the assertion remains zero even when that broker incorrectly solicits a push, so this compatibility test can pass vacuously. Aggregate this log count across every node in `self.kafka.nodes`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add cross-version compatibility tests for the KIP-1331 topology description plugin [kafka]
Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru merged PR #23413:
URL: https://github.com/apache/kafka/pull/23413
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add cross-version compatibility tests for the KIP-1331 topology description plugin [kafka]
Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on code in PR #23413:
URL: https://github.com/apache/kafka/pull/23413#discussion_r3971571486
##########
tests/kafkatest/tests/streams/streams_smoke_test.py:
##########
@@ -122,3 +123,45 @@ def test_streams(self, processing_guarantee, crash, metadata_quorum, group_proto
processor3.stop()
self.driver.node.account.ssh("grep SUCCESS %s" % self.driver.STDOUT_FILE, allow_fail=False)
+
+ @cluster(num_nodes=5)
+ @matrix(metadata_quorum=[quorum.combined_kraft])
+ def test_old_client_not_solicited_for_topology_push(self, metadata_quorum):
+ """
+ Test the situation when a pre-KIP-1331 Kafka Streams client uses the streams group
+ protocol against a broker on this branch with the topology description plugin configured
+ (the default broker setup from BaseStreamsTest). StreamsGroupHeartbeat negotiates down to
+ version 0 for this client, so the broker must never solicit a topology description push
+ for it.
+ """
+ processor = StreamsSmokeTestJobRunnerService(self.test_context, self.kafka, 'at_least_once', 'streams')
+ processor.set_version(str(LATEST_4_3))
+
+ broker_node = self.kafka.nodes[0]
Review Comment:
Good catch, fixed — now checking all 3 broker logs and summing the count.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org