[GitHub] [beam] nbali commented on issue #25962: [Bug]: Java SDF-based Kafka ingestion can be stuck at certain offsets in certain conditions

["nbali (via GitHub)" <gi...@apache.org>]

nbali commented on issue #25962:
URL: https://github.com/apache/beam/issues/25962#issuecomment-1497956296

   Well, if we won't explicitly forbid using certain kafka versions with a failfast, that means we have to support even the currently "bugged" versions. That means "_as we can then remove it when there is a fix in kafka_" will never happen. So if the goal is to have an implementation that supports pre-3.2.0, the current bugged versions, the possibly fixed versions, and pretty much anything that might be changed by kafka implementation details then I would say none of the 5 proposal works perfectly.
   
   I would say the most generic solution would be to call the `consumer.poll(KAFKA_POLL_TIMEOUT)` in a loop as well, and break the loop only if we received a `rawRecords` that isn't empty OR the `KAFKA_POLL_TIMEOUT` is over. That essentially simulates pre-3.2.0 behaviour.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org