[GitHub] [kafka] lizthegrey commented on pull request #10326: Avoid newly replicating brokers in RackAwareReplicaSelector

[GitBox <gi...@apache.org>]

lizthegrey commented on pull request #10326:
URL: https://github.com/apache/kafka/pull/10326#issuecomment-803178364


   > @lizthegrey Could you describe the case that you've hit with a bit more details? I suppose that only consumers which are lagging could hit this condition, right? We filter out replicas which do not contain the requested fetch offset.
   
   Correct, consumers which are lagging (or requesting older offsets in the hope of catching up later) would hit this condition, where they stay stuck on brokers that do not have the offsets they need. The behavior we observed was: we had a hiccup in our Kafka quorum and lost the necessary ISR, did an unclean election, started up a new broker to replace ISR, and only half of our consumers that were configured with rack awareness caught up (because they shared the same rack parameter as brokers that were still in ISR); the other half shared a rack with brokers that were new to the replica set, and thus hung waiting for that offset to become available on that broker before they could begin to catch up. Restarting the consumer process didn't help, they still attempted to fetch from the newly replicating brokers at the checkpointed offset and failed to make progress.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org