You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "FrankYang0529 (via GitHub)" <gi...@apache.org> on 2024/12/15 14:55:49 UTC
[PR] KAFKA-18201: testGroupMetadataMessageFormatter fails for new consumer protocol [kafka]
FrankYang0529 opened a new pull request, #18198:
URL: https://github.com/apache/kafka/pull/18198
We introduced new coordinator record for async kafka consumer like ConsumerGroupMetadataKey, ConsumerGroupPartitionMetadataKey, etc. The GroupMetadataMessageFormatter didn't support these message type. We should extend it to support new records.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18201: testGroupMetadataMessageFormatter fails for new consumer protocol [kafka]
Posted by "FrankYang0529 (via GitHub)" <gi...@apache.org>.
FrankYang0529 commented on code in PR #18198:
URL: https://github.com/apache/kafka/pull/18198#discussion_r4154438199
##########
tools/src/test/java/org/apache/kafka/tools/consumer/ConsoleConsumerTest.java:
##########
@@ -455,9 +490,10 @@ private Consumer<byte[], byte[]> createOffsetConsumer(ClusterInstance cluster) {
return new KafkaConsumer<>(props);
}
- private Consumer<byte[], byte[]> createGroupMetaDataConsumer(ClusterInstance cluster) {
+ private Consumer<byte[], byte[]> createGroupMetaDataConsumer(ClusterInstance cluster, GroupProtocol groupProtocol) {
Review Comment:
Thanks for the review. Updated it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18201: testGroupMetadataMessageFormatter fails for new consumer protocol [kafka]
Posted by "FrankYang0529 (via GitHub)" <gi...@apache.org>.
FrankYang0529 commented on code in PR #18198:
URL: https://github.com/apache/kafka/pull/18198#discussion_r3645160049
##########
tools/src/test/java/org/apache/kafka/tools/consumer/ConsoleConsumerTest.java:
##########
@@ -370,8 +374,11 @@ public void testOffsetsMessageFormatter(ClusterInstance cluster) throws Exceptio
}
}
- @ClusterTest(brokers = 3)
Review Comment:
The default `offsets.topic.replication.factor` is 3. We set it to 1, so we don't need 3 brokers here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18201: testGroupMetadataMessageFormatter fails for new consumer protocol [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #18198: KAFKA-18201: testGroupMetadataMessageFormatter fails for new consumer protocol
URL: https://github.com/apache/kafka/pull/18198
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18201: testGroupMetadataMessageFormatter fails for new consumer protocol [kafka]
Posted by "FrankYang0529 (via GitHub)" <gi...@apache.org>.
FrankYang0529 commented on PR #18198:
URL: https://github.com/apache/kafka/pull/18198#issuecomment-2595410633
Hi @lianetm, could you help me review this when you have time? Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18201: testGroupMetadataMessageFormatter fails for new consumer protocol [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18198:
URL: https://github.com/apache/kafka/pull/18198#issuecomment-4293501274
This PR is being marked as stale since it has not had any activity in 90 days. If you
would like to keep this PR alive, please leave a comment asking for a review. If the PR has
merge conflicts, update it with the latest from the base branch.
<p>
If you are having difficulty finding a reviewer, please reach out on the
[mailing list](https://kafka.apache.org/contact).
<p>
If this PR is no longer valid or desired, please feel free to close it. If no activity
occurs in the next 30 days, it will be automatically closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18201: testGroupMetadataMessageFormatter fails for new consumer protocol [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18198:
URL: https://github.com/apache/kafka/pull/18198#issuecomment-3573669217
This PR is being marked as stale since it has not had any activity in 90 days. If you
would like to keep this PR alive, please leave a comment asking for a review. If the PR has
merge conflicts, update it with the latest from the base branch.
<p>
If you are having difficulty finding a reviewer, please reach out on the
[mailing list](https://kafka.apache.org/contact).
<p>
If this PR is no longer valid or desired, please feel free to close it. If no activity
occurs in the next 30 days, it will be automatically closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18201: testGroupMetadataMessageFormatter fails for new consumer protocol [kafka]
Posted by "FrankYang0529 (via GitHub)" <gi...@apache.org>.
FrankYang0529 merged PR #18198:
URL: https://github.com/apache/kafka/pull/18198
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18201: testGroupMetadataMessageFormatter fails for new consumer protocol [kafka]
Posted by "gensericghiro (via GitHub)" <gi...@apache.org>.
gensericghiro commented on code in PR #18198:
URL: https://github.com/apache/kafka/pull/18198#discussion_r2684246938
##########
tools/src/test/java/org/apache/kafka/tools/consumer/ConsoleConsumerTest.java:
##########
@@ -370,8 +374,11 @@ public void testOffsetsMessageFormatter(ClusterInstance cluster) throws Exceptio
}
}
- @ClusterTest(brokers = 3)
Review Comment:
Was this needed before?
##########
tools/src/test/java/org/apache/kafka/tools/consumer/ConsoleConsumerTest.java:
##########
@@ -458,9 +493,10 @@ private Consumer<byte[], byte[]> createOffsetConsumer(ClusterInstance cluster) {
return new KafkaConsumer<>(props);
}
- private Consumer<byte[], byte[]> createGroupMetaDataConsumer(ClusterInstance cluster) {
+ private Consumer<byte[], byte[]> createGroupMetaDataConsumer(ClusterInstance cluster, GroupProtocol groupProtocol) {
Properties props = consumerProps(cluster);
props.put(AUTO_OFFSET_RESET_CONFIG, "earliest");
+ props.put(GROUP_PROTOCOL_CONFIG, groupProtocol.name);
Review Comment:
nit: could be good to use the getter for name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18201: testGroupMetadataMessageFormatter fails for new consumer protocol [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #18198:
URL: https://github.com/apache/kafka/pull/18198#discussion_r4172427792
##########
tools/src/test/java/org/apache/kafka/tools/consumer/ConsoleConsumerTest.java:
##########
@@ -411,6 +418,34 @@ public void testGroupMetadataMessageFormatter(ClusterInstance cluster) throws Ex
}
}
+ @ClusterTest(serverProperties = {
+ @ClusterConfigProperty(key = OFFSETS_TOPIC_PARTITIONS_CONFIG, value = "1"),
+ @ClusterConfigProperty(key = OFFSETS_TOPIC_REPLICATION_FACTOR_CONFIG, value = "1")
+ })
+ public void testGroupMetadataMessageFormatterWithConsumerGroupProtocol(ClusterInstance cluster) throws Exception {
+ cluster.createTopic(topic, 1, (short) 1);
+ produceMessages(cluster);
+
+ String[] groupMetadataMessageFormatter = createConsoleConsumerArgs(cluster,
+ Topic.GROUP_METADATA_TOPIC_NAME,
+ "org.apache.kafka.tools.consumer.GroupMetadataMessageFormatter");
+
+ ConsoleConsumerOptions options = new ConsoleConsumerOptions(groupMetadataMessageFormatter);
+ ConsoleConsumer.ConsumerWrapper consumerWrapper =
+ new ConsoleConsumer.ConsumerWrapper(options, createGroupMetadataConsumer(cluster, GroupProtocol.CONSUMER));
+
+ try (ByteArrayOutputStream out = new ByteArrayOutputStream();
+ PrintStream output = new PrintStream(out)) {
+ ConsoleConsumer.process(1, options.formatter(), consumerWrapper, output, true);
+
+ JsonNode jsonNode = objectMapper.reader().readTree(out.toByteArray());
+ // Consumer group protocol does not write GROUP_METADATA records, so the result is empty
Review Comment:
This is confusing at first glance. Please update the comment to explain that we don't have a formatter to read the records written by the CONSUMER protocol.
BTW, we have a ticket for the new formatter: https://issues.apache.org/jira/browse/KAFKA-19277
@dajac Do you have time to address KAFKA-19277? Or I could ask another contributor to take it over :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18201: testGroupMetadataMessageFormatter fails for new consumer protocol [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #18198:
URL: https://github.com/apache/kafka/pull/18198#issuecomment-3225517863
@FrankYang0529 could you please rebase code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18201: testGroupMetadataMessageFormatter fails for new consumer protocol [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18198:
URL: https://github.com/apache/kafka/pull/18198#issuecomment-5041906764
This PR is being marked as stale since it has not had any activity in 90 days. If you
would like to keep this PR alive, please leave a comment asking for a review. If the PR has
merge conflicts, update it with the latest from the base branch.
<p>
If you are having difficulty finding a reviewer, please reach out on the
[mailing list](https://kafka.apache.org/contact).
<p>
If this PR is no longer valid or desired, please feel free to close it. If no activity
occurs in the next 30 days, it will be automatically closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18201: testGroupMetadataMessageFormatter fails for new consumer protocol [kafka]
Posted by "FrankYang0529 (via GitHub)" <gi...@apache.org>.
FrankYang0529 commented on code in PR #18198:
URL: https://github.com/apache/kafka/pull/18198#discussion_r3645157256
##########
tools/src/test/java/org/apache/kafka/tools/consumer/ConsoleConsumerTest.java:
##########
@@ -458,9 +493,10 @@ private Consumer<byte[], byte[]> createOffsetConsumer(ClusterInstance cluster) {
return new KafkaConsumer<>(props);
}
- private Consumer<byte[], byte[]> createGroupMetaDataConsumer(ClusterInstance cluster) {
+ private Consumer<byte[], byte[]> createGroupMetaDataConsumer(ClusterInstance cluster, GroupProtocol groupProtocol) {
Properties props = consumerProps(cluster);
props.put(AUTO_OFFSET_RESET_CONFIG, "earliest");
+ props.put(GROUP_PROTOCOL_CONFIG, groupProtocol.name);
Review Comment:
Updated it. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18201: testGroupMetadataMessageFormatter fails for new consumer protocol [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18198:
URL: https://github.com/apache/kafka/pull/18198#issuecomment-2972220664
This PR is being marked as stale since it has not had any activity in 90 days. If you
would like to keep this PR alive, please leave a comment asking for a review. If the PR has
merge conflicts, update it with the latest from the base branch.
<p>
If you are having difficulty finding a reviewer, please reach out on the
[mailing list](https://kafka.apache.org/contact).
<p>
If this PR is no longer valid or desired, please feel free to close it. If no activity
occurs in the next 30 days, it will be automatically closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18201: testGroupMetadataMessageFormatter fails for new consumer protocol [kafka]
Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on code in PR #18198:
URL: https://github.com/apache/kafka/pull/18198#discussion_r4153573335
##########
tools/src/test/java/org/apache/kafka/tools/consumer/ConsoleConsumerTest.java:
##########
@@ -455,9 +490,10 @@ private Consumer<byte[], byte[]> createOffsetConsumer(ClusterInstance cluster) {
return new KafkaConsumer<>(props);
}
- private Consumer<byte[], byte[]> createGroupMetaDataConsumer(ClusterInstance cluster) {
+ private Consumer<byte[], byte[]> createGroupMetaDataConsumer(ClusterInstance cluster, GroupProtocol groupProtocol) {
Review Comment:
nit: `createGroupMetadataConsumer` would be better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18201: testGroupMetadataMessageFormatter fails for new consumer protocol [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18198:
URL: https://github.com/apache/kafka/pull/18198#issuecomment-5390479930
This PR has been closed since it has not had any activity in 120 days. If you feel like this
was a mistake, or you would like to continue working on it, please feel free to re-open the
PR and ask for a review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18201: testGroupMetadataMessageFormatter fails for new consumer protocol [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #18198: KAFKA-18201: testGroupMetadataMessageFormatter fails for new consumer protocol
URL: https://github.com/apache/kafka/pull/18198
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18201: testGroupMetadataMessageFormatter fails for new consumer protocol [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18198:
URL: https://github.com/apache/kafka/pull/18198#issuecomment-3067746729
This PR has been closed since it has not had any activity in 120 days. If you feel like this
was a mistake, or you would like to continue working on it, please feel free to re-open the
PR and ask for a review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18201: testGroupMetadataMessageFormatter fails for new consumer protocol [kafka]
Posted by "FrankYang0529 (via GitHub)" <gi...@apache.org>.
FrankYang0529 commented on PR #18198:
URL: https://github.com/apache/kafka/pull/18198#issuecomment-3226504178
@chia7712 I merge trunk and fix the code. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18201: testGroupMetadataMessageFormatter fails for new consumer protocol [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #18198:
URL: https://github.com/apache/kafka/pull/18198#discussion_r4172792736
##########
tools/src/test/java/org/apache/kafka/tools/consumer/ConsoleConsumerTest.java:
##########
@@ -411,6 +418,34 @@ public void testGroupMetadataMessageFormatter(ClusterInstance cluster) throws Ex
}
}
+ @ClusterTest(serverProperties = {
+ @ClusterConfigProperty(key = OFFSETS_TOPIC_PARTITIONS_CONFIG, value = "1"),
+ @ClusterConfigProperty(key = OFFSETS_TOPIC_REPLICATION_FACTOR_CONFIG, value = "1")
+ })
+ public void testGroupMetadataMessageFormatterWithConsumerGroupProtocol(ClusterInstance cluster) throws Exception {
+ cluster.createTopic(topic, 1, (short) 1);
+ produceMessages(cluster);
+
+ String[] groupMetadataMessageFormatter = createConsoleConsumerArgs(cluster,
+ Topic.GROUP_METADATA_TOPIC_NAME,
+ "org.apache.kafka.tools.consumer.GroupMetadataMessageFormatter");
+
+ ConsoleConsumerOptions options = new ConsoleConsumerOptions(groupMetadataMessageFormatter);
+ ConsoleConsumer.ConsumerWrapper consumerWrapper =
+ new ConsoleConsumer.ConsumerWrapper(options, createGroupMetadataConsumer(cluster, GroupProtocol.CONSUMER));
+
+ try (ByteArrayOutputStream out = new ByteArrayOutputStream();
+ PrintStream output = new PrintStream(out)) {
+ ConsoleConsumer.process(1, options.formatter(), consumerWrapper, output, true);
+
+ JsonNode jsonNode = objectMapper.reader().readTree(out.toByteArray());
+ // Consumer group protocol does not write GROUP_METADATA records, so the result is empty
Review Comment:
https://github.com/apache/kafka/pull/23684
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18201: testGroupMetadataMessageFormatter fails for new consumer protocol [kafka]
Posted by "FrankYang0529 (via GitHub)" <gi...@apache.org>.
FrankYang0529 commented on code in PR #18198:
URL: https://github.com/apache/kafka/pull/18198#discussion_r4172572410
##########
tools/src/test/java/org/apache/kafka/tools/consumer/ConsoleConsumerTest.java:
##########
@@ -411,6 +418,34 @@ public void testGroupMetadataMessageFormatter(ClusterInstance cluster) throws Ex
}
}
+ @ClusterTest(serverProperties = {
+ @ClusterConfigProperty(key = OFFSETS_TOPIC_PARTITIONS_CONFIG, value = "1"),
+ @ClusterConfigProperty(key = OFFSETS_TOPIC_REPLICATION_FACTOR_CONFIG, value = "1")
+ })
+ public void testGroupMetadataMessageFormatterWithConsumerGroupProtocol(ClusterInstance cluster) throws Exception {
+ cluster.createTopic(topic, 1, (short) 1);
+ produceMessages(cluster);
+
+ String[] groupMetadataMessageFormatter = createConsoleConsumerArgs(cluster,
+ Topic.GROUP_METADATA_TOPIC_NAME,
+ "org.apache.kafka.tools.consumer.GroupMetadataMessageFormatter");
+
+ ConsoleConsumerOptions options = new ConsoleConsumerOptions(groupMetadataMessageFormatter);
+ ConsoleConsumer.ConsumerWrapper consumerWrapper =
+ new ConsoleConsumer.ConsumerWrapper(options, createGroupMetadataConsumer(cluster, GroupProtocol.CONSUMER));
+
+ try (ByteArrayOutputStream out = new ByteArrayOutputStream();
+ PrintStream output = new PrintStream(out)) {
+ ConsoleConsumer.process(1, options.formatter(), consumerWrapper, output, true);
+
+ JsonNode jsonNode = objectMapper.reader().readTree(out.toByteArray());
+ // Consumer group protocol does not write GROUP_METADATA records, so the result is empty
Review Comment:
Create a PR for it. Thanks for the review. https://github.com/apache/kafka/pull/23683
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org