You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "ShivsundarR (via GitHub)" <gi...@apache.org> on 2024/12/05 15:50:16 UTC
[PR] KAFKA-18164: Clear existing acknowledgements on share session epoch reset. [kafka]
ShivsundarR opened a new pull request, #18063:
URL: https://github.com/apache/kafka/pull/18063
*What*
If there were acknowledgements to be sent, and we had a share session epoch of 0, we were still attempting to send these acknowledgements, as this map was not cleared at all.
This led to the consumer hanging, it assumed it had sent the acknowledgements, but it is actually just re-attempting forever, but the `ShareSessionHandler` would not allow a `ShareAcknowledge` to be sent on epoch = 0.
This meant both fetching and acknowledging were blocked in the consumers.
PR fixes this bug and the one mentioned in https://issues.apache.org/jira/browse/KAFKA-18165 where we were not updating `nodesWithPendingRequests` correctly in one case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18164: Clear existing acknowledgements on share session epoch reset. [kafka]
Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on code in PR #18063:
URL: https://github.com/apache/kafka/pull/18063#discussion_r1875029001
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -990,16 +990,18 @@ UnsentRequest buildRequest() {
}
ShareAcknowledgeRequest.Builder requestBuilder = sessionHandler.newShareAcknowledgeBuilder(groupId, fetchConfig);
Review Comment:
It seems to me that `ShareSessionHandler.addPartitionToFetch` saves partition and acknowledgement state into the handler to be built by the request builder. In the event that the request builder cannot be built, such as because a new share session needs to be created, the partition and acknowledgement state is still left in the share session. I worry that it is possible this could lead to an erroneous request in some situations when the session handler is used again when a new connection has been created. I think that it would be prudent to clear out `nextAcknowledgements` and `nextPartitions` in `ShareSessionHandler.newShareAcknowledgeBuilder` in the case where it returns null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18164: Clear existing acknowledgements on share session epoch reset. [kafka]
Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield merged PR #18063:
URL: https://github.com/apache/kafka/pull/18063
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18164: Clear existing acknowledgements on share session epoch reset. [kafka]
Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on code in PR #18063:
URL: https://github.com/apache/kafka/pull/18063#discussion_r1872975167
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -990,16 +990,17 @@ UnsentRequest buildRequest() {
}
ShareAcknowledgeRequest.Builder requestBuilder = sessionHandler.newShareAcknowledgeBuilder(groupId, fetchConfig);
- Node nodeToSend = metadata.fetch().nodeById(nodeId);
log.trace("Building acknowledgements to send : {}", finalAcknowledgementsToSend);
Review Comment:
This log line probably ought to move down into the else block too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18164: Clear existing acknowledgements on share session epoch reset. [kafka]
Posted by "ShivsundarR (via GitHub)" <gi...@apache.org>.
ShivsundarR commented on code in PR #18063:
URL: https://github.com/apache/kafka/pull/18063#discussion_r1874846243
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -990,16 +990,17 @@ UnsentRequest buildRequest() {
}
ShareAcknowledgeRequest.Builder requestBuilder = sessionHandler.newShareAcknowledgeBuilder(groupId, fetchConfig);
- Node nodeToSend = metadata.fetch().nodeById(nodeId);
log.trace("Building acknowledgements to send : {}", finalAcknowledgementsToSend);
Review Comment:
Yeah makes sense, thanks. I have moved it now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18164: Clear existing acknowledgements on share session epoch reset. [kafka]
Posted by "ShivsundarR (via GitHub)" <gi...@apache.org>.
ShivsundarR commented on code in PR #18063:
URL: https://github.com/apache/kafka/pull/18063#discussion_r1875353133
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -990,16 +990,18 @@ UnsentRequest buildRequest() {
}
ShareAcknowledgeRequest.Builder requestBuilder = sessionHandler.newShareAcknowledgeBuilder(groupId, fetchConfig);
Review Comment:
Yeah, right. The handler does not clear the acknowledgements which can be a problem when future requests are built. I have added code to clear the `nextPartitions` and `nextAcknowledgements` now and added a test for the same.
Thanks for catching this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org