You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "dajac (via GitHub)" <gi...@apache.org> on 2023/09/01 14:35:13 UTC
[GitHub] [kafka] dajac opened a new pull request, #14321: KAFKA-14499: [6/N] Add MemberId and MemberEpoch to OffsetFetchRequest
dajac opened a new pull request, #14321:
URL: https://github.com/apache/kafka/pull/14321
This patch adds the MemberId and the MemberEpoch fields to the OffsetFetchRequest. Those fields will be populated when the new consumer group protocol is used to ensure that the member fetching the offset has the correct member id and epoch. If it does not, UNKNOWN_MEMBER_ID or STALE_MEMBER_EPOCH are returned to the client.
Our initial idea was to implement the same for the old protocol. The field is called GenerationIdOrMemberEpoch in KIP-848 to materialize this. As a second though, I think that we should only do it for the new protocol. The effort to implement it in the old protocol is not worth it in my opinion.
This patch adds unit test to test the new code. I will add integration tests in a separate PR.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dajac commented on pull request #14321: KAFKA-14499: [6/N] Add MemberId and MemberEpoch to OffsetFetchRequest
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on PR #14321:
URL: https://github.com/apache/kafka/pull/14321#issuecomment-1704917384
@rreddy-22 @CalvinConfluent @jolshan Thanks for your comments. I have addressed all of them. I have also changed a bit the schema and fixed a few other tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jolshan commented on a diff in pull request #14321: KAFKA-14499: [6/N] Add MemberId and MemberEpoch to OffsetFetchRequest
Posted by "jolshan (via GitHub)" <gi...@apache.org>.
jolshan commented on code in PR #14321:
URL: https://github.com/apache/kafka/pull/14321#discussion_r1313635548
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/consumer/ConsumerGroupTest.java:
##########
@@ -631,4 +631,33 @@ public void testValidateOffsetCommit() {
// This should succeed.
group.validateOffsetCommit("member-id", "", 0);
}
+
+ @Test
+ public void testValidateOffsetFetch() {
+ SnapshotRegistry snapshotRegistry = new SnapshotRegistry(new LogContext());
+ ConsumerGroup group = new ConsumerGroup(snapshotRegistry, "group-foo");
+
+ // Simulate a call from the admin client without member id and member epoch.
+ // This should pass only if the group is empty.
Review Comment:
Do we have an example where it fails because the group is not empty?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jolshan commented on a diff in pull request #14321: KAFKA-14499: [6/N] Add MemberId and MemberEpoch to OffsetFetchRequest
Posted by "jolshan (via GitHub)" <gi...@apache.org>.
jolshan commented on code in PR #14321:
URL: https://github.com/apache/kafka/pull/14321#discussion_r1316197710
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/OffsetMetadataManagerTest.java:
##########
@@ -181,10 +181,28 @@ public List<OffsetFetchResponseData.OffsetFetchResponseTopics> fetchOffsets(
String groupId,
List<OffsetFetchRequestData.OffsetFetchRequestTopics> topics,
long committedOffset
+ ) {
Review Comment:
It could be useful to add a small comment in case folks wanted to build upon these tests. But up to you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dajac commented on a diff in pull request #14321: KAFKA-14499: [6/N] Add MemberId and MemberEpoch to OffsetFetchRequest
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #14321:
URL: https://github.com/apache/kafka/pull/14321#discussion_r1314664510
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/OffsetMetadataManagerTest.java:
##########
@@ -181,10 +181,28 @@ public List<OffsetFetchResponseData.OffsetFetchResponseTopics> fetchOffsets(
String groupId,
List<OffsetFetchRequestData.OffsetFetchRequestTopics> topics,
long committedOffset
+ ) {
+ return fetchOffsets(
+ groupId,
+ "",
+ -1,
+ topics,
+ committedOffset
+ );
+ }
+
+ public List<OffsetFetchResponseData.OffsetFetchResponseTopics> fetchOffsets(
+ String groupId,
+ String memberId,
+ int memberEpoch,
+ List<OffsetFetchRequestData.OffsetFetchRequestTopics> topics,
+ long committedOffset
) {
OffsetFetchResponseData.OffsetFetchResponseGroup response = offsetMetadataManager.fetchOffsets(
new OffsetFetchRequestData.OffsetFetchRequestGroup()
.setGroupId(groupId)
+ .setMemberId(memberId)
Review Comment:
Correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jolshan commented on a diff in pull request #14321: KAFKA-14499: [6/N] Add MemberId and MemberEpoch to OffsetFetchRequest
Posted by "jolshan (via GitHub)" <gi...@apache.org>.
jolshan commented on code in PR #14321:
URL: https://github.com/apache/kafka/pull/14321#discussion_r1313582619
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/OffsetMetadataManager.java:
##########
@@ -225,17 +225,21 @@ private Group validateOffsetCommit(
}
/**
- * Validates an OffsetCommit request.
+ * Validates an OffsetFetch request.
Review Comment:
whoops -- was this a typo before 😅
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/OffsetMetadataManager.java:
##########
@@ -225,17 +225,21 @@ private Group validateOffsetCommit(
}
/**
- * Validates an OffsetCommit request.
+ * Validates an OffsetFetch request.
Review Comment:
whoops -- was this a typo before 😅
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jolshan commented on a diff in pull request #14321: KAFKA-14499: [6/N] Add MemberId and MemberEpoch to OffsetFetchRequest
Posted by "jolshan (via GitHub)" <gi...@apache.org>.
jolshan commented on code in PR #14321:
URL: https://github.com/apache/kafka/pull/14321#discussion_r1313613392
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/OffsetMetadataManagerTest.java:
##########
@@ -181,10 +181,28 @@ public List<OffsetFetchResponseData.OffsetFetchResponseTopics> fetchOffsets(
String groupId,
List<OffsetFetchRequestData.OffsetFetchRequestTopics> topics,
long committedOffset
+ ) {
+ return fetchOffsets(
+ groupId,
+ "",
+ -1,
+ topics,
+ committedOffset
+ );
+ }
+
+ public List<OffsetFetchResponseData.OffsetFetchResponseTopics> fetchOffsets(
+ String groupId,
+ String memberId,
+ int memberEpoch,
+ List<OffsetFetchRequestData.OffsetFetchRequestTopics> topics,
+ long committedOffset
) {
OffsetFetchResponseData.OffsetFetchResponseGroup response = offsetMetadataManager.fetchOffsets(
new OffsetFetchRequestData.OffsetFetchRequestGroup()
.setGroupId(groupId)
+ .setMemberId(memberId)
Review Comment:
Is this needed now because of validation (and wasn't needed before)
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/OffsetMetadataManagerTest.java:
##########
@@ -181,10 +181,28 @@ public List<OffsetFetchResponseData.OffsetFetchResponseTopics> fetchOffsets(
String groupId,
List<OffsetFetchRequestData.OffsetFetchRequestTopics> topics,
long committedOffset
+ ) {
Review Comment:
is this for generic groups only now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jolshan commented on a diff in pull request #14321: KAFKA-14499: [6/N] Add MemberId and MemberEpoch to OffsetFetchRequest
Posted by "jolshan (via GitHub)" <gi...@apache.org>.
jolshan commented on code in PR #14321:
URL: https://github.com/apache/kafka/pull/14321#discussion_r1313590994
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/generic/GenericGroup.java:
##########
@@ -823,9 +823,17 @@ public void validateOffsetCommit(
/**
* Validates the OffsetFetch request.
+ *
+ * @param memberId The member id. This is not provided for generic groups.
+ * @param memberEpoch The member epoch for consumer groups. This is not provided for generic groups.
+ * @param lastCommittedOffset The last committed offsets in the timeline.
*/
@Override
- public void validateOffsetFetch() throws GroupIdNotFoundException {
+ public void validateOffsetFetch(
+ String memberId,
Review Comment:
For my understanding, we pass these to be consistent with the interface, but we don't need it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dajac commented on a diff in pull request #14321: KAFKA-14499: [6/N] Add MemberId and MemberEpoch to OffsetFetchRequest
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #14321:
URL: https://github.com/apache/kafka/pull/14321#discussion_r1314671160
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/consumer/ConsumerGroupTest.java:
##########
@@ -631,4 +631,33 @@ public void testValidateOffsetCommit() {
// This should succeed.
group.validateOffsetCommit("member-id", "", 0);
}
+
+ @Test
+ public void testValidateOffsetFetch() {
+ SnapshotRegistry snapshotRegistry = new SnapshotRegistry(new LogContext());
+ ConsumerGroup group = new ConsumerGroup(snapshotRegistry, "group-foo");
+
+ // Simulate a call from the admin client without member id and member epoch.
+ // This should pass only if the group is empty.
+ group.validateOffsetFetch("", -1, Long.MAX_VALUE);
+
+ // The member does not exist.
+ assertThrows(UnknownMemberIdException.class, () ->
+ group.validateOffsetFetch("member-id", 0, Long.MAX_VALUE));
+
+ // Create a member.
+ snapshotRegistry.getOrCreateSnapshot(0);
+ group.getOrMaybeCreateMember("member-id", true);
+
+ // The member does not exist at epoch 0.
Review Comment:
This comment is confusing. The epoch was referring to the epoch of the snapshot here. I rephrased it to make it clearer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dajac commented on a diff in pull request #14321: KAFKA-14499: [6/N] Add MemberId and MemberEpoch to OffsetFetchRequest
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #14321:
URL: https://github.com/apache/kafka/pull/14321#discussion_r1314665557
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/OffsetMetadataManagerTest.java:
##########
@@ -181,10 +181,28 @@ public List<OffsetFetchResponseData.OffsetFetchResponseTopics> fetchOffsets(
String groupId,
List<OffsetFetchRequestData.OffsetFetchRequestTopics> topics,
long committedOffset
+ ) {
Review Comment:
This or to simulate the admin client case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jolshan commented on a diff in pull request #14321: KAFKA-14499: [6/N] Add MemberId and MemberEpoch to OffsetFetchRequest
Posted by "jolshan (via GitHub)" <gi...@apache.org>.
jolshan commented on code in PR #14321:
URL: https://github.com/apache/kafka/pull/14321#discussion_r1313631916
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/consumer/ConsumerGroup.java:
##########
@@ -538,17 +537,46 @@ public void validateOffsetCommit(
if (memberEpoch < 0 && members().isEmpty()) return;
final ConsumerGroupMember member = getOrMaybeCreateMember(memberId, false);
- if (memberEpoch != member.memberEpoch()) {
- throw Errors.STALE_MEMBER_EPOCH.exception();
- }
+ validateMemberEpoch(memberEpoch, member.memberEpoch());
}
/**
* Validates the OffsetFetch request.
+ *
+ * @param memberId The member id for consumer groups.
+ * @param memberEpoch The member epoch for consumer groups.
+ * @param lastCommittedOffset The last committed offsets in the timeline.
*/
@Override
- public void validateOffsetFetch() {
- // Nothing.
+ public void validateOffsetFetch(
+ String memberId,
+ int memberEpoch,
+ long lastCommittedOffset
+ ) throws UnknownMemberIdException, StaleMemberEpochException {
+ // When the member epoch is -1, the request comes from the admin client. In this case,
+ // the request can commit offsets if the group is empty.
Review Comment:
I made a comment on something similar in another PR. Might be a slightly different case, but similar wording about committing offsets is allowed and it being a bit unclear if fetching them is or isn't
https://github.com/apache/kafka/pull/14120/files/7e5f1f64d173d3e3b68fd6310fb76d6137daed2c#r1303310001
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jolshan commented on a diff in pull request #14321: KAFKA-14499: [6/N] Add MemberId and MemberEpoch to OffsetFetchRequest
Posted by "jolshan (via GitHub)" <gi...@apache.org>.
jolshan commented on code in PR #14321:
URL: https://github.com/apache/kafka/pull/14321#discussion_r1316207292
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/OffsetMetadataManagerTest.java:
##########
@@ -1491,10 +1526,15 @@ public void testConsumerGroupOffsetFetchWithUnknownMemberId() {
.setPartitionIndexes(Collections.singletonList(0))
);
+ // Fetch offsets cases.
+ assertThrows(UnknownMemberIdException.class,
+ () -> context.fetchOffsets("group", "", 0, topics, Long.MAX_VALUE));
Review Comment:
Is this testing that the empty member string will fail (ie if no member, it must be null?) I'm trying to understand how this test is different than the one below.
I also noticed that the test changed the default values from empty string to null. Was that an error before?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dajac commented on a diff in pull request #14321: KAFKA-14499: [6/N] Add MemberId and MemberEpoch to OffsetFetchRequest
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #14321:
URL: https://github.com/apache/kafka/pull/14321#discussion_r1316210535
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/OffsetMetadataManagerTest.java:
##########
@@ -1491,10 +1526,15 @@ public void testConsumerGroupOffsetFetchWithUnknownMemberId() {
.setPartitionIndexes(Collections.singletonList(0))
);
+ // Fetch offsets cases.
+ assertThrows(UnknownMemberIdException.class,
+ () -> context.fetchOffsets("group", "", 0, topics, Long.MAX_VALUE));
Review Comment:
Yeah, I changed the member id to be nullable and null by default. I think that it makes more sense. So now, an empty string raises an unknown member id error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jolshan commented on a diff in pull request #14321: KAFKA-14499: [6/N] Add MemberId and MemberEpoch to OffsetFetchRequest
Posted by "jolshan (via GitHub)" <gi...@apache.org>.
jolshan commented on code in PR #14321:
URL: https://github.com/apache/kafka/pull/14321#discussion_r1313618740
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/OffsetMetadataManagerTest.java:
##########
@@ -1409,6 +1444,82 @@ public void testFetchAllOffsetsAtDifferentCommittedOffset() {
), context.fetchAllOffsets("group", Long.MAX_VALUE));
}
+ @Test
+ public void testConsumerGroupOffsetFetchWithMemberIdAndEpoch() {
+ OffsetMetadataManagerTestContext context = new OffsetMetadataManagerTestContext.Builder().build();
+ // Create consumer group.
+ ConsumerGroup group = context.groupMetadataManager.getOrMaybeCreateConsumerGroup("group", true);
+ // Create member.
+ group.getOrMaybeCreateMember("member", true);
+ // Commit offset.
+ context.commitOffset("group", "foo", 0, 100L, 1);
+
+ // Fetch offsets case.
+ List<OffsetFetchRequestData.OffsetFetchRequestTopics> topics = Collections.singletonList(
+ new OffsetFetchRequestData.OffsetFetchRequestTopics()
+ .setName("foo")
+ .setPartitionIndexes(Collections.singletonList(0))
+ );
+
+ assertEquals(Collections.singletonList(
+ new OffsetFetchResponseData.OffsetFetchResponseTopics()
+ .setName("foo")
+ .setPartitions(Collections.singletonList(
+ mkOffsetPartitionResponse(0, 100L, 1, "metadata")
+ ))
+ ), context.fetchOffsets("group", "member", 0, topics, Long.MAX_VALUE));
+
+ // Fetch all offsets case.
+ assertEquals(Collections.singletonList(
+ new OffsetFetchResponseData.OffsetFetchResponseTopics()
+ .setName("foo")
+ .setPartitions(Collections.singletonList(
+ mkOffsetPartitionResponse(0, 100L, 1, "metadata")
+ ))
+ ), context.fetchAllOffsets("group", "member", 0, Long.MAX_VALUE));
+ }
+
+ @Test
+ public void testConsumerGroupOffsetFetchWithUnknownMemberId() {
+ OffsetMetadataManagerTestContext context = new OffsetMetadataManagerTestContext.Builder().build();
+ context.groupMetadataManager.getOrMaybeCreateConsumerGroup("group", true);
+
+ // Fetch offsets case.
+ List<OffsetFetchRequestData.OffsetFetchRequestTopics> topics = Collections.singletonList(
+ new OffsetFetchRequestData.OffsetFetchRequestTopics()
+ .setName("foo")
+ .setPartitionIndexes(Collections.singletonList(0))
+ );
+
+ assertThrows(UnknownMemberIdException.class,
+ () -> context.fetchOffsets("group", "member", 0, topics, Long.MAX_VALUE));
+
+ // Fetch all offsets case.
+ assertThrows(UnknownMemberIdException.class,
+ () -> context.fetchAllOffsets("group", "member", 0, Long.MAX_VALUE));
+ }
+
+ @Test
+ public void testConsumerGroupOffsetFetchWithStaleMemberEpoch() {
Review Comment:
This is the member epoch -1 right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jolshan commented on a diff in pull request #14321: KAFKA-14499: [6/N] Add MemberId and MemberEpoch to OffsetFetchRequest
Posted by "jolshan (via GitHub)" <gi...@apache.org>.
jolshan commented on code in PR #14321:
URL: https://github.com/apache/kafka/pull/14321#discussion_r1313636482
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/consumer/ConsumerGroupTest.java:
##########
@@ -631,4 +631,33 @@ public void testValidateOffsetCommit() {
// This should succeed.
group.validateOffsetCommit("member-id", "", 0);
}
+
+ @Test
+ public void testValidateOffsetFetch() {
+ SnapshotRegistry snapshotRegistry = new SnapshotRegistry(new LogContext());
+ ConsumerGroup group = new ConsumerGroup(snapshotRegistry, "group-foo");
+
+ // Simulate a call from the admin client without member id and member epoch.
+ // This should pass only if the group is empty.
+ group.validateOffsetFetch("", -1, Long.MAX_VALUE);
+
+ // The member does not exist.
+ assertThrows(UnknownMemberIdException.class, () ->
+ group.validateOffsetFetch("member-id", 0, Long.MAX_VALUE));
+
+ // Create a member.
+ snapshotRegistry.getOrCreateSnapshot(0);
+ group.getOrMaybeCreateMember("member-id", true);
+
+ // The member does not exist at epoch 0.
Review Comment:
Should this be offset 0? Epoch 0 seems to be correct in the last line of the test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rreddy-22 commented on a diff in pull request #14321: KAFKA-14499: [6/N] Add MemberId and MemberEpoch to OffsetFetchRequest
Posted by "rreddy-22 (via GitHub)" <gi...@apache.org>.
rreddy-22 commented on code in PR #14321:
URL: https://github.com/apache/kafka/pull/14321#discussion_r1313249102
##########
clients/src/main/resources/common/message/OffsetFetchRequest.json:
##########
@@ -47,8 +54,12 @@
]},
{ "name": "Groups", "type": "[]OffsetFetchRequestGroup", "versions": "8+",
"about": "Each group we would like to fetch offsets for", "fields": [
- { "name": "groupId", "type": "string", "versions": "8+", "entityType": "groupId",
+ { "name": "GroupId", "type": "string", "versions": "8+", "entityType": "groupId",
"about": "The group ID."},
+ { "name": "MemberId", "type": "string", "versions": "9+", "ignorable": true,
+ "about": "The member ID assigned by the group coordinator if using the consumer protocol." },
Review Comment:
should we specify new consumer group protocol?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jolshan commented on a diff in pull request #14321: KAFKA-14499: [6/N] Add MemberId and MemberEpoch to OffsetFetchRequest
Posted by "jolshan (via GitHub)" <gi...@apache.org>.
jolshan commented on code in PR #14321:
URL: https://github.com/apache/kafka/pull/14321#discussion_r1313582042
##########
clients/src/main/resources/common/message/OffsetFetchResponse.json:
##########
@@ -32,8 +32,21 @@
// Version 7 adds pending offset commit as new error response on partition level.
//
// Version 8 is adding support for fetching offsets for multiple groups
- "validVersions": "0-8",
+ //
+ // Version 9 is the first version that can be used with the new consumer group protocol (KIP-848). The response is
+ // the same as version 8 but can return STALE_MEMBER_EPOCH and UNKNOWN_MEMBER_ID errors when the new consumer group
+ // protocol is used.
+ "validVersions": "0-9",
"flexibleVersions": "6+",
+ // Supported errors:
+ // - GROUP_AUTHORIZATION_FAILED (version 0+)
+ // - NOT_COORDINATOR (version 0+)
+ // - COORDINATOR_NOT_AVAILABLE (version 0+)
+ // - COORDINATOR_LOAD_IN_PROGRESS (version 0+)
+ // - GROUP_ID_NOT_FOUND (version 0+)
+ // - UNSTABLE_OFFSET_COMMIT (version 7+)
Review Comment:
thanks for adding this block -- it's good to have such a mapping on the json for responses.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jolshan commented on a diff in pull request #14321: KAFKA-14499: [6/N] Add MemberId and MemberEpoch to OffsetFetchRequest
Posted by "jolshan (via GitHub)" <gi...@apache.org>.
jolshan commented on code in PR #14321:
URL: https://github.com/apache/kafka/pull/14321#discussion_r1316209008
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/OffsetMetadataManagerTest.java:
##########
@@ -1491,10 +1526,15 @@ public void testConsumerGroupOffsetFetchWithUnknownMemberId() {
.setPartitionIndexes(Collections.singletonList(0))
);
+ // Fetch offsets cases.
+ assertThrows(UnknownMemberIdException.class,
+ () -> context.fetchOffsets("group", "", 0, topics, Long.MAX_VALUE));
Review Comment:
Or is this part of the latest commit that made null the no member ID indicator?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dajac commented on a diff in pull request #14321: KAFKA-14499: [6/N] Add MemberId and MemberEpoch to OffsetFetchRequest
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #14321:
URL: https://github.com/apache/kafka/pull/14321#discussion_r1314663349
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/generic/GenericGroup.java:
##########
@@ -823,9 +823,17 @@ public void validateOffsetCommit(
/**
* Validates the OffsetFetch request.
+ *
+ * @param memberId The member id. This is not provided for generic groups.
+ * @param memberEpoch The member epoch for consumer groups. This is not provided for generic groups.
+ * @param lastCommittedOffset The last committed offsets in the timeline.
*/
@Override
- public void validateOffsetFetch() throws GroupIdNotFoundException {
+ public void validateOffsetFetch(
+ String memberId,
Review Comment:
Correct. I mentioned this in the javadoc of the method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dajac commented on a diff in pull request #14321: KAFKA-14499: [6/N] Add MemberId and MemberEpoch to OffsetFetchRequest
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #14321:
URL: https://github.com/apache/kafka/pull/14321#discussion_r1314590003
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/consumer/ConsumerGroup.java:
##########
@@ -538,17 +537,46 @@ public void validateOffsetCommit(
if (memberEpoch < 0 && members().isEmpty()) return;
final ConsumerGroupMember member = getOrMaybeCreateMember(memberId, false);
- if (memberEpoch != member.memberEpoch()) {
- throw Errors.STALE_MEMBER_EPOCH.exception();
- }
+ validateMemberEpoch(memberEpoch, member.memberEpoch());
}
/**
* Validates the OffsetFetch request.
+ *
+ * @param memberId The member id for consumer groups.
+ * @param memberEpoch The member epoch for consumer groups.
+ * @param lastCommittedOffset The last committed offsets in the timeline.
*/
@Override
- public void validateOffsetFetch() {
- // Nothing.
+ public void validateOffsetFetch(
+ String memberId,
+ int memberEpoch,
+ long lastCommittedOffset
+ ) throws UnknownMemberIdException, StaleMemberEpochException {
+ // When the member epoch is -1, the request comes from the admin client. In this case,
+ // the request can commit offsets if the group is empty.
Review Comment:
The comment is wrong. Let me fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jolshan commented on a diff in pull request #14321: KAFKA-14499: [6/N] Add MemberId and MemberEpoch to OffsetFetchRequest
Posted by "jolshan (via GitHub)" <gi...@apache.org>.
jolshan commented on code in PR #14321:
URL: https://github.com/apache/kafka/pull/14321#discussion_r1313587458
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/consumer/ConsumerGroup.java:
##########
@@ -538,17 +537,46 @@ public void validateOffsetCommit(
if (memberEpoch < 0 && members().isEmpty()) return;
final ConsumerGroupMember member = getOrMaybeCreateMember(memberId, false);
- if (memberEpoch != member.memberEpoch()) {
- throw Errors.STALE_MEMBER_EPOCH.exception();
- }
+ validateMemberEpoch(memberEpoch, member.memberEpoch());
}
/**
* Validates the OffsetFetch request.
+ *
+ * @param memberId The member id for consumer groups.
+ * @param memberEpoch The member epoch for consumer groups.
+ * @param lastCommittedOffset The last committed offsets in the timeline.
*/
@Override
- public void validateOffsetFetch() {
- // Nothing.
+ public void validateOffsetFetch(
+ String memberId,
+ int memberEpoch,
+ long lastCommittedOffset
+ ) throws UnknownMemberIdException, StaleMemberEpochException {
+ // When the member epoch is -1, the request comes from the admin client. In this case,
+ // the request can commit offsets if the group is empty.
Review Comment:
I think the "when the member epoch is -1" part applies here. Is the empty group part somewhere else?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] CalvinConfluent commented on a diff in pull request #14321: KAFKA-14499: [6/N] Add MemberId and MemberEpoch to OffsetFetchRequest
Posted by "CalvinConfluent (via GitHub)" <gi...@apache.org>.
CalvinConfluent commented on code in PR #14321:
URL: https://github.com/apache/kafka/pull/14321#discussion_r1313307656
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/consumer/ConsumerGroup.java:
##########
@@ -538,17 +537,46 @@ public void validateOffsetCommit(
if (memberEpoch < 0 && members().isEmpty()) return;
final ConsumerGroupMember member = getOrMaybeCreateMember(memberId, false);
- if (memberEpoch != member.memberEpoch()) {
- throw Errors.STALE_MEMBER_EPOCH.exception();
- }
+ validateMemberEpoch(memberEpoch, member.memberEpoch());
}
/**
* Validates the OffsetFetch request.
+ *
+ * @param memberId The member id for consumer groups.
+ * @param memberEpoch The member epoch for consumer groups.
+ * @param lastCommittedOffset The last committed offsets in the timeline.
*/
@Override
- public void validateOffsetFetch() {
- // Nothing.
+ public void validateOffsetFetch(
+ String memberId,
+ int memberEpoch,
+ long lastCommittedOffset
+ ) throws UnknownMemberIdException, StaleMemberEpochException {
+ // When the member epoch is -1, the request comes from the admin client. In this case,
+ // the request can commit offsets if the group is empty.
Review Comment:
Should we check the group is empty here? The code seems misaligned with the comment.
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/OffsetMetadataManagerTest.java:
##########
@@ -1409,6 +1444,82 @@ public void testFetchAllOffsetsAtDifferentCommittedOffset() {
), context.fetchAllOffsets("group", Long.MAX_VALUE));
}
+ @Test
+ public void testConsumerGroupOffsetFetchWithMemberIdAndEpoch() {
+ OffsetMetadataManagerTestContext context = new OffsetMetadataManagerTestContext.Builder().build();
+ // Create consumer group.
+ ConsumerGroup group = context.groupMetadataManager.getOrMaybeCreateConsumerGroup("group", true);
+ // Create member.
+ group.getOrMaybeCreateMember("member", true);
+ // Commit offset.
+ context.commitOffset("group", "foo", 0, 100L, 1);
+
+ // Fetch offsets case.
+ List<OffsetFetchRequestData.OffsetFetchRequestTopics> topics = Collections.singletonList(
+ new OffsetFetchRequestData.OffsetFetchRequestTopics()
+ .setName("foo")
+ .setPartitionIndexes(Collections.singletonList(0))
+ );
+
+ assertEquals(Collections.singletonList(
+ new OffsetFetchResponseData.OffsetFetchResponseTopics()
+ .setName("foo")
+ .setPartitions(Collections.singletonList(
+ mkOffsetPartitionResponse(0, 100L, 1, "metadata")
+ ))
+ ), context.fetchOffsets("group", "member", 0, topics, Long.MAX_VALUE));
+
+ // Fetch all offsets case.
+ assertEquals(Collections.singletonList(
+ new OffsetFetchResponseData.OffsetFetchResponseTopics()
+ .setName("foo")
+ .setPartitions(Collections.singletonList(
+ mkOffsetPartitionResponse(0, 100L, 1, "metadata")
+ ))
+ ), context.fetchAllOffsets("group", "member", 0, Long.MAX_VALUE));
+ }
+
+ @Test
+ public void testConsumerGroupOffsetFetchWithUnknownMemberId() {
+ OffsetMetadataManagerTestContext context = new OffsetMetadataManagerTestContext.Builder().build();
+ context.groupMetadataManager.getOrMaybeCreateConsumerGroup("group", true);
+
+ // Fetch offsets case.
+ List<OffsetFetchRequestData.OffsetFetchRequestTopics> topics = Collections.singletonList(
+ new OffsetFetchRequestData.OffsetFetchRequestTopics()
+ .setName("foo")
+ .setPartitionIndexes(Collections.singletonList(0))
+ );
+
+ assertThrows(UnknownMemberIdException.class,
+ () -> context.fetchOffsets("group", "member", 0, topics, Long.MAX_VALUE));
+
+ // Fetch all offsets case.
+ assertThrows(UnknownMemberIdException.class,
+ () -> context.fetchAllOffsets("group", "member", 0, Long.MAX_VALUE));
+ }
+
+ @Test
+ public void testConsumerGroupOffsetFetchWithStaleMemberEpoch() {
Review Comment:
Maybe add the case for admin client?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jolshan commented on a diff in pull request #14321: KAFKA-14499: [6/N] Add MemberId and MemberEpoch to OffsetFetchRequest
Posted by "jolshan (via GitHub)" <gi...@apache.org>.
jolshan commented on code in PR #14321:
URL: https://github.com/apache/kafka/pull/14321#discussion_r1313633100
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/OffsetMetadataManagerTest.java:
##########
@@ -1409,6 +1444,82 @@ public void testFetchAllOffsetsAtDifferentCommittedOffset() {
), context.fetchAllOffsets("group", Long.MAX_VALUE));
}
+ @Test
+ public void testConsumerGroupOffsetFetchWithMemberIdAndEpoch() {
+ OffsetMetadataManagerTestContext context = new OffsetMetadataManagerTestContext.Builder().build();
+ // Create consumer group.
+ ConsumerGroup group = context.groupMetadataManager.getOrMaybeCreateConsumerGroup("group", true);
+ // Create member.
+ group.getOrMaybeCreateMember("member", true);
+ // Commit offset.
+ context.commitOffset("group", "foo", 0, 100L, 1);
+
+ // Fetch offsets case.
+ List<OffsetFetchRequestData.OffsetFetchRequestTopics> topics = Collections.singletonList(
+ new OffsetFetchRequestData.OffsetFetchRequestTopics()
+ .setName("foo")
+ .setPartitionIndexes(Collections.singletonList(0))
+ );
+
+ assertEquals(Collections.singletonList(
+ new OffsetFetchResponseData.OffsetFetchResponseTopics()
+ .setName("foo")
+ .setPartitions(Collections.singletonList(
+ mkOffsetPartitionResponse(0, 100L, 1, "metadata")
+ ))
+ ), context.fetchOffsets("group", "member", 0, topics, Long.MAX_VALUE));
+
+ // Fetch all offsets case.
+ assertEquals(Collections.singletonList(
+ new OffsetFetchResponseData.OffsetFetchResponseTopics()
+ .setName("foo")
+ .setPartitions(Collections.singletonList(
+ mkOffsetPartitionResponse(0, 100L, 1, "metadata")
+ ))
+ ), context.fetchAllOffsets("group", "member", 0, Long.MAX_VALUE));
+ }
+
+ @Test
+ public void testConsumerGroupOffsetFetchWithUnknownMemberId() {
+ OffsetMetadataManagerTestContext context = new OffsetMetadataManagerTestContext.Builder().build();
+ context.groupMetadataManager.getOrMaybeCreateConsumerGroup("group", true);
+
+ // Fetch offsets case.
+ List<OffsetFetchRequestData.OffsetFetchRequestTopics> topics = Collections.singletonList(
+ new OffsetFetchRequestData.OffsetFetchRequestTopics()
+ .setName("foo")
+ .setPartitionIndexes(Collections.singletonList(0))
+ );
+
+ assertThrows(UnknownMemberIdException.class,
+ () -> context.fetchOffsets("group", "member", 0, topics, Long.MAX_VALUE));
+
+ // Fetch all offsets case.
+ assertThrows(UnknownMemberIdException.class,
+ () -> context.fetchAllOffsets("group", "member", 0, Long.MAX_VALUE));
+ }
+
+ @Test
+ public void testConsumerGroupOffsetFetchWithStaleMemberEpoch() {
Review Comment:
And to be clear -- this is the group has -1 and we are asking for 10.
the case Calvin is asking about seems to be passing -1 in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dajac commented on a diff in pull request #14321: KAFKA-14499: [6/N] Add MemberId and MemberEpoch to OffsetFetchRequest
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #14321:
URL: https://github.com/apache/kafka/pull/14321#discussion_r1314661887
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/OffsetMetadataManagerTest.java:
##########
@@ -1409,6 +1444,82 @@ public void testFetchAllOffsetsAtDifferentCommittedOffset() {
), context.fetchAllOffsets("group", Long.MAX_VALUE));
}
+ @Test
+ public void testConsumerGroupOffsetFetchWithMemberIdAndEpoch() {
+ OffsetMetadataManagerTestContext context = new OffsetMetadataManagerTestContext.Builder().build();
+ // Create consumer group.
+ ConsumerGroup group = context.groupMetadataManager.getOrMaybeCreateConsumerGroup("group", true);
+ // Create member.
+ group.getOrMaybeCreateMember("member", true);
+ // Commit offset.
+ context.commitOffset("group", "foo", 0, 100L, 1);
+
+ // Fetch offsets case.
+ List<OffsetFetchRequestData.OffsetFetchRequestTopics> topics = Collections.singletonList(
+ new OffsetFetchRequestData.OffsetFetchRequestTopics()
+ .setName("foo")
+ .setPartitionIndexes(Collections.singletonList(0))
+ );
+
+ assertEquals(Collections.singletonList(
+ new OffsetFetchResponseData.OffsetFetchResponseTopics()
+ .setName("foo")
+ .setPartitions(Collections.singletonList(
+ mkOffsetPartitionResponse(0, 100L, 1, "metadata")
+ ))
+ ), context.fetchOffsets("group", "member", 0, topics, Long.MAX_VALUE));
+
+ // Fetch all offsets case.
+ assertEquals(Collections.singletonList(
+ new OffsetFetchResponseData.OffsetFetchResponseTopics()
+ .setName("foo")
+ .setPartitions(Collections.singletonList(
+ mkOffsetPartitionResponse(0, 100L, 1, "metadata")
+ ))
+ ), context.fetchAllOffsets("group", "member", 0, Long.MAX_VALUE));
+ }
+
+ @Test
+ public void testConsumerGroupOffsetFetchWithUnknownMemberId() {
+ OffsetMetadataManagerTestContext context = new OffsetMetadataManagerTestContext.Builder().build();
+ context.groupMetadataManager.getOrMaybeCreateConsumerGroup("group", true);
+
+ // Fetch offsets case.
+ List<OffsetFetchRequestData.OffsetFetchRequestTopics> topics = Collections.singletonList(
+ new OffsetFetchRequestData.OffsetFetchRequestTopics()
+ .setName("foo")
+ .setPartitionIndexes(Collections.singletonList(0))
+ );
+
+ assertThrows(UnknownMemberIdException.class,
+ () -> context.fetchOffsets("group", "member", 0, topics, Long.MAX_VALUE));
+
+ // Fetch all offsets case.
+ assertThrows(UnknownMemberIdException.class,
+ () -> context.fetchAllOffsets("group", "member", 0, Long.MAX_VALUE));
+ }
+
+ @Test
+ public void testConsumerGroupOffsetFetchWithStaleMemberEpoch() {
Review Comment:
Sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dajac commented on a diff in pull request #14321: KAFKA-14499: [6/N] Add MemberId and MemberEpoch to OffsetFetchRequest
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #14321:
URL: https://github.com/apache/kafka/pull/14321#discussion_r1314666683
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/consumer/ConsumerGroupTest.java:
##########
@@ -631,4 +631,33 @@ public void testValidateOffsetCommit() {
// This should succeed.
group.validateOffsetCommit("member-id", "", 0);
}
+
+ @Test
+ public void testValidateOffsetFetch() {
+ SnapshotRegistry snapshotRegistry = new SnapshotRegistry(new LogContext());
+ ConsumerGroup group = new ConsumerGroup(snapshotRegistry, "group-foo");
+
+ // Simulate a call from the admin client without member id and member epoch.
+ // This should pass only if the group is empty.
Review Comment:
That part of the comment was wrong. I just removed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dajac merged pull request #14321: KAFKA-14499: [6/N] Add MemberId and MemberEpoch to OffsetFetchRequest
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac merged PR #14321:
URL: https://github.com/apache/kafka/pull/14321
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org