You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2021/03/01 13:56:39 UTC
[GitHub] [kafka] dajac opened a new pull request #10234: MINOR; Clean up LeaderAndIsrResponse construction in `ReplicaManager#becomeLeaderOrFollower`
dajac opened a new pull request #10234:
URL: https://github.com/apache/kafka/pull/10234
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dajac commented on a change in pull request #10234: MINOR; Clean up LeaderAndIsrResponse construction in `ReplicaManager#becomeLeaderOrFollower`
Posted by GitBox <gi...@apache.org>.
dajac commented on a change in pull request #10234:
URL: https://github.com/apache/kafka/pull/10234#discussion_r586310444
##########
File path: clients/src/main/java/org/apache/kafka/common/requests/LeaderAndIsrRequest.java
##########
@@ -145,24 +145,21 @@ public LeaderAndIsrResponse getErrorResponse(int throttleTimeMs, Throwable e) {
.setErrorCode(error.code()));
}
responseData.setPartitionErrors(partitions);
- return new LeaderAndIsrResponse(responseData, version());
- }
-
- List<LeaderAndIsrTopicError> topics = new ArrayList<>(data.topicStates().size());
- Map<String, Uuid> topicIds = topicIds();
- for (LeaderAndIsrTopicState topicState : data.topicStates()) {
- LeaderAndIsrTopicError topicError = new LeaderAndIsrTopicError();
- topicError.setTopicId(topicIds.get(topicState.topicName()));
- List<LeaderAndIsrPartitionError> partitions = new ArrayList<>(topicState.partitionStates().size());
- for (LeaderAndIsrPartitionState partition : topicState.partitionStates()) {
- partitions.add(new LeaderAndIsrPartitionError()
+ } else {
+ for (LeaderAndIsrTopicState topicState : data.topicStates()) {
+ List<LeaderAndIsrPartitionError> partitions = new ArrayList<>(
+ topicState.partitionStates().size());
+ for (LeaderAndIsrPartitionState partition : topicState.partitionStates()) {
+ partitions.add(new LeaderAndIsrPartitionError()
.setPartitionIndex(partition.partitionIndex())
Review comment:
Ack. So let's keep it as is.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dajac commented on a change in pull request #10234: MINOR; Clean up LeaderAndIsrResponse construction in `ReplicaManager#becomeLeaderOrFollower`
Posted by GitBox <gi...@apache.org>.
dajac commented on a change in pull request #10234:
URL: https://github.com/apache/kafka/pull/10234#discussion_r586188217
##########
File path: clients/src/main/java/org/apache/kafka/common/requests/LeaderAndIsrRequest.java
##########
@@ -145,24 +145,22 @@ public LeaderAndIsrResponse getErrorResponse(int throttleTimeMs, Throwable e) {
.setErrorCode(error.code()));
}
responseData.setPartitionErrors(partitions);
- return new LeaderAndIsrResponse(responseData, version());
- }
-
- List<LeaderAndIsrTopicError> topics = new ArrayList<>(data.topicStates().size());
- Map<String, Uuid> topicIds = topicIds();
- for (LeaderAndIsrTopicState topicState : data.topicStates()) {
- LeaderAndIsrTopicError topicError = new LeaderAndIsrTopicError();
- topicError.setTopicId(topicIds.get(topicState.topicName()));
- List<LeaderAndIsrPartitionError> partitions = new ArrayList<>(topicState.partitionStates().size());
- for (LeaderAndIsrPartitionState partition : topicState.partitionStates()) {
- partitions.add(new LeaderAndIsrPartitionError()
+ } else {
+ Map<String, Uuid> topicIds = topicIds();
+ for (LeaderAndIsrTopicState topicState : data.topicStates()) {
+ List<LeaderAndIsrPartitionError> partitions = new ArrayList<>(
+ topicState.partitionStates().size());
+ for (LeaderAndIsrPartitionState partition : topicState.partitionStates()) {
+ partitions.add(new LeaderAndIsrPartitionError()
.setPartitionIndex(partition.partitionIndex())
.setErrorCode(error.code()));
+ }
+ responseData.topics().add(new LeaderAndIsrTopicError()
+ .setTopicId(topicIds.get(topicState.topicName()))
Review comment:
Good catch. I missed this one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dajac merged pull request #10234: MINOR; Clean up LeaderAndIsrResponse construction in `ReplicaManager#becomeLeaderOrFollower`
Posted by GitBox <gi...@apache.org>.
dajac merged pull request #10234:
URL: https://github.com/apache/kafka/pull/10234
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dajac commented on pull request #10234: MINOR; Clean up LeaderAndIsrResponse construction in `ReplicaManager#becomeLeaderOrFollower`
Posted by GitBox <gi...@apache.org>.
dajac commented on pull request #10234:
URL: https://github.com/apache/kafka/pull/10234#issuecomment-790384482
@chia7712 I just pushed a commit to address your comment. Could you take a look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dajac commented on pull request #10234: MINOR; Clean up LeaderAndIsrResponse construction in `ReplicaManager#becomeLeaderOrFollower`
Posted by GitBox <gi...@apache.org>.
dajac commented on pull request #10234:
URL: https://github.com/apache/kafka/pull/10234#issuecomment-789522482
@chia7712 Thanks for the review. I just pushed a commit to address your comment. Could you take another look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] chia7712 commented on a change in pull request #10234: MINOR; Clean up LeaderAndIsrResponse construction in `ReplicaManager#becomeLeaderOrFollower`
Posted by GitBox <gi...@apache.org>.
chia7712 commented on a change in pull request #10234:
URL: https://github.com/apache/kafka/pull/10234#discussion_r586390524
##########
File path: clients/src/test/java/org/apache/kafka/common/requests/LeaderAndIsrRequestTest.java
##########
@@ -59,18 +61,41 @@ public void testUnsupportedVersion() {
@Test
public void testGetErrorResponse() {
- Uuid id = Uuid.randomUuid();
+ Uuid topicId = Uuid.randomUuid();
+ String topicName = "topic";
+ int partition = 0;
+
for (short version = LEADER_AND_ISR.oldestVersion(); version < LEADER_AND_ISR.latestVersion(); version++) {
Review comment:
It does not cover all versions. I had filed a PR for that (#10078) but I feel it can be fixed here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] chia7712 commented on a change in pull request #10234: MINOR; Clean up LeaderAndIsrResponse construction in `ReplicaManager#becomeLeaderOrFollower`
Posted by GitBox <gi...@apache.org>.
chia7712 commented on a change in pull request #10234:
URL: https://github.com/apache/kafka/pull/10234#discussion_r586289230
##########
File path: clients/src/main/java/org/apache/kafka/common/requests/LeaderAndIsrRequest.java
##########
@@ -145,24 +145,21 @@ public LeaderAndIsrResponse getErrorResponse(int throttleTimeMs, Throwable e) {
.setErrorCode(error.code()));
}
responseData.setPartitionErrors(partitions);
- return new LeaderAndIsrResponse(responseData, version());
- }
-
- List<LeaderAndIsrTopicError> topics = new ArrayList<>(data.topicStates().size());
- Map<String, Uuid> topicIds = topicIds();
- for (LeaderAndIsrTopicState topicState : data.topicStates()) {
- LeaderAndIsrTopicError topicError = new LeaderAndIsrTopicError();
- topicError.setTopicId(topicIds.get(topicState.topicName()));
- List<LeaderAndIsrPartitionError> partitions = new ArrayList<>(topicState.partitionStates().size());
- for (LeaderAndIsrPartitionState partition : topicState.partitionStates()) {
- partitions.add(new LeaderAndIsrPartitionError()
+ } else {
+ for (LeaderAndIsrTopicState topicState : data.topicStates()) {
+ List<LeaderAndIsrPartitionError> partitions = new ArrayList<>(
+ topicState.partitionStates().size());
+ for (LeaderAndIsrPartitionState partition : topicState.partitionStates()) {
+ partitions.add(new LeaderAndIsrPartitionError()
.setPartitionIndex(partition.partitionIndex())
Review comment:
> Do you really want to add a comment for this? That seems unnecessary to me.
I don't have strong reason for that. As you mentioned, we can read protocol file to understand the reasons of ignoring that filed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dajac commented on a change in pull request #10234: MINOR; Clean up LeaderAndIsrResponse construction in `ReplicaManager#becomeLeaderOrFollower`
Posted by GitBox <gi...@apache.org>.
dajac commented on a change in pull request #10234:
URL: https://github.com/apache/kafka/pull/10234#discussion_r586285455
##########
File path: clients/src/main/java/org/apache/kafka/common/requests/LeaderAndIsrRequest.java
##########
@@ -145,24 +145,21 @@ public LeaderAndIsrResponse getErrorResponse(int throttleTimeMs, Throwable e) {
.setErrorCode(error.code()));
}
responseData.setPartitionErrors(partitions);
- return new LeaderAndIsrResponse(responseData, version());
- }
-
- List<LeaderAndIsrTopicError> topics = new ArrayList<>(data.topicStates().size());
- Map<String, Uuid> topicIds = topicIds();
- for (LeaderAndIsrTopicState topicState : data.topicStates()) {
- LeaderAndIsrTopicError topicError = new LeaderAndIsrTopicError();
- topicError.setTopicId(topicIds.get(topicState.topicName()));
- List<LeaderAndIsrPartitionError> partitions = new ArrayList<>(topicState.partitionStates().size());
- for (LeaderAndIsrPartitionState partition : topicState.partitionStates()) {
- partitions.add(new LeaderAndIsrPartitionError()
+ } else {
+ for (LeaderAndIsrTopicState topicState : data.topicStates()) {
+ List<LeaderAndIsrPartitionError> partitions = new ArrayList<>(
+ topicState.partitionStates().size());
+ for (LeaderAndIsrPartitionState partition : topicState.partitionStates()) {
+ partitions.add(new LeaderAndIsrPartitionError()
.setPartitionIndex(partition.partitionIndex())
Review comment:
@chia7712 Starting from version 5, the topic name is no longer in the response: https://github.com/apache/kafka/blob/trunk/clients/src/main/resources/common/message/LeaderAndIsrResponse.json#L46 so we don't set it anymore. Do you really want to add a comment for this? That seems unnecessary to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] chia7712 commented on a change in pull request #10234: MINOR; Clean up LeaderAndIsrResponse construction in `ReplicaManager#becomeLeaderOrFollower`
Posted by GitBox <gi...@apache.org>.
chia7712 commented on a change in pull request #10234:
URL: https://github.com/apache/kafka/pull/10234#discussion_r585723099
##########
File path: clients/src/main/java/org/apache/kafka/common/requests/LeaderAndIsrRequest.java
##########
@@ -145,24 +145,22 @@ public LeaderAndIsrResponse getErrorResponse(int throttleTimeMs, Throwable e) {
.setErrorCode(error.code()));
}
responseData.setPartitionErrors(partitions);
- return new LeaderAndIsrResponse(responseData, version());
- }
-
- List<LeaderAndIsrTopicError> topics = new ArrayList<>(data.topicStates().size());
- Map<String, Uuid> topicIds = topicIds();
- for (LeaderAndIsrTopicState topicState : data.topicStates()) {
- LeaderAndIsrTopicError topicError = new LeaderAndIsrTopicError();
- topicError.setTopicId(topicIds.get(topicState.topicName()));
- List<LeaderAndIsrPartitionError> partitions = new ArrayList<>(topicState.partitionStates().size());
- for (LeaderAndIsrPartitionState partition : topicState.partitionStates()) {
- partitions.add(new LeaderAndIsrPartitionError()
+ } else {
+ Map<String, Uuid> topicIds = topicIds();
+ for (LeaderAndIsrTopicState topicState : data.topicStates()) {
+ List<LeaderAndIsrPartitionError> partitions = new ArrayList<>(
+ topicState.partitionStates().size());
+ for (LeaderAndIsrPartitionState partition : topicState.partitionStates()) {
+ partitions.add(new LeaderAndIsrPartitionError()
.setPartitionIndex(partition.partitionIndex())
.setErrorCode(error.code()));
+ }
+ responseData.topics().add(new LeaderAndIsrTopicError()
+ .setTopicId(topicIds.get(topicState.topicName()))
Review comment:
Could it be replaced by `setTopicId(topicState.topicId())`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dajac commented on pull request #10234: MINOR; Clean up LeaderAndIsrResponse construction in `ReplicaManager#becomeLeaderOrFollower`
Posted by GitBox <gi...@apache.org>.
dajac commented on pull request #10234:
URL: https://github.com/apache/kafka/pull/10234#issuecomment-789621268
@chia7712 I just pushed another small commit to fix/improve `LeaderAndIsrRequestTest#testGetErrorResponse`. It was not testing all the versions as claimed. I have also added assertions to verify the created response. Could you take a look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] chia7712 commented on a change in pull request #10234: MINOR; Clean up LeaderAndIsrResponse construction in `ReplicaManager#becomeLeaderOrFollower`
Posted by GitBox <gi...@apache.org>.
chia7712 commented on a change in pull request #10234:
URL: https://github.com/apache/kafka/pull/10234#discussion_r586270002
##########
File path: clients/src/main/java/org/apache/kafka/common/requests/LeaderAndIsrRequest.java
##########
@@ -145,24 +145,21 @@ public LeaderAndIsrResponse getErrorResponse(int throttleTimeMs, Throwable e) {
.setErrorCode(error.code()));
}
responseData.setPartitionErrors(partitions);
- return new LeaderAndIsrResponse(responseData, version());
- }
-
- List<LeaderAndIsrTopicError> topics = new ArrayList<>(data.topicStates().size());
- Map<String, Uuid> topicIds = topicIds();
- for (LeaderAndIsrTopicState topicState : data.topicStates()) {
- LeaderAndIsrTopicError topicError = new LeaderAndIsrTopicError();
- topicError.setTopicId(topicIds.get(topicState.topicName()));
- List<LeaderAndIsrPartitionError> partitions = new ArrayList<>(topicState.partitionStates().size());
- for (LeaderAndIsrPartitionState partition : topicState.partitionStates()) {
- partitions.add(new LeaderAndIsrPartitionError()
+ } else {
+ for (LeaderAndIsrTopicState topicState : data.topicStates()) {
+ List<LeaderAndIsrPartitionError> partitions = new ArrayList<>(
+ topicState.partitionStates().size());
+ for (LeaderAndIsrPartitionState partition : topicState.partitionStates()) {
+ partitions.add(new LeaderAndIsrPartitionError()
.setPartitionIndex(partition.partitionIndex())
Review comment:
Could you add comments to explain why we don't need to set topic name here? Otherwise, It looks like a bug to me :(
##########
File path: core/src/main/scala/kafka/server/ReplicaManager.scala
##########
@@ -1441,38 +1441,29 @@ class ReplicaManager(val config: KafkaConfig,
replicaFetcherManager.shutdownIdleFetcherThreads()
replicaAlterLogDirsManager.shutdownIdleFetcherThreads()
onLeadershipChange(partitionsBecomeLeader, partitionsBecomeFollower)
- if (leaderAndIsrRequest.version() < 5) {
- val responsePartitions = responseMap.iterator.map { case (tp, error) =>
- new LeaderAndIsrPartitionError()
+
+ val data = new LeaderAndIsrResponseData().setErrorCode(Errors.NONE.code)
+ if (leaderAndIsrRequest.version < 5) {
+ responseMap.forKeyValue { (tp, error) =>
+ data.partitionErrors.add(new LeaderAndIsrPartitionError()
.setTopicName(tp.topic)
.setPartitionIndex(tp.partition)
- .setErrorCode(error.code)
- }.toBuffer
- new LeaderAndIsrResponse(new LeaderAndIsrResponseData()
- .setErrorCode(Errors.NONE.code)
- .setPartitionErrors(responsePartitions.asJava), leaderAndIsrRequest.version())
+ .setErrorCode(error.code))
+ }
} else {
- val topics = new mutable.HashMap[String, List[LeaderAndIsrPartitionError]]
- responseMap.asJava.forEach { case (tp, error) =>
- if (!topics.contains(tp.topic)) {
- topics.put(tp.topic, List(new LeaderAndIsrPartitionError()
- .setPartitionIndex(tp.partition)
- .setErrorCode(error.code)))
- } else {
- topics.put(tp.topic, new LeaderAndIsrPartitionError()
- .setPartitionIndex(tp.partition)
- .setErrorCode(error.code)::topics(tp.topic))
+ responseMap.forKeyValue { (tp, error) =>
+ val topicId = topicIds.get(tp.topic)
+ var topic = data.topics.find(topicId)
+ if (topic == null) {
+ topic = new LeaderAndIsrTopicError().setTopicId(topicId)
+ data.topics.add(topic)
}
+ topic.partitionErrors.add(new LeaderAndIsrPartitionError()
+ .setPartitionIndex(tp.partition)
Review comment:
ditto
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dajac commented on a change in pull request #10234: MINOR; Clean up LeaderAndIsrResponse construction in `ReplicaManager#becomeLeaderOrFollower`
Posted by GitBox <gi...@apache.org>.
dajac commented on a change in pull request #10234:
URL: https://github.com/apache/kafka/pull/10234#discussion_r587207959
##########
File path: clients/src/test/java/org/apache/kafka/common/requests/LeaderAndIsrRequestTest.java
##########
@@ -59,18 +61,41 @@ public void testUnsupportedVersion() {
@Test
public void testGetErrorResponse() {
- Uuid id = Uuid.randomUuid();
+ Uuid topicId = Uuid.randomUuid();
+ String topicName = "topic";
+ int partition = 0;
+
for (short version = LEADER_AND_ISR.oldestVersion(); version < LEADER_AND_ISR.latestVersion(); version++) {
Review comment:
Good catch!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org