[GitHub] [beam] Abacn commented on issue #25962: [Bug]: Java SDF-based Kafka ingestion can be stuck at certain offsets in certain conditions

["Abacn (via GitHub)" <gi...@apache.org>]

Abacn commented on issue #25962:
URL: https://github.com/apache/beam/issues/25962#issuecomment-1489294702

   Thanks so much for the detailed investigation.
   
   > Essentially what was fixed with the watermark for streaming jobs with #24205 has to be also fixed for the tracker/restriction/offset.
   
   CC: @johnjcasey who did the fix for the watermark. A fix for the tracker would hopefully resolve this long standing issue.
   
   
   > the generated jar and every dependency tool output if I deploy it with 2.44 it still uses 'org.apache.kafka:kafka-clients:2.4.1'
   
   kafka client is a provided version for Beam: https://github.com/apache/beam/blob/9c614557c51ad55230211f864e70f48ad0914326/sdks/java/io/kafka/build.gradle#L68
   
   should be able to assign different versions. Not sure why it still picked the default 2.4.1
   
   Also, this reminds me that kafka version is also important for KafkaIO issue. We saw incoming issues that did not capture by our tests. We may want to update the default kafka version also.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org