[GitHub] [spark] gaborgsomogyi edited a comment on issue #25135: [SPARK-28367][SS] Use new KafkaConsumer.poll API in Kafka connector

[GitBox <gi...@apache.org>]

gaborgsomogyi edited a comment on issue #25135: [SPARK-28367][SS] Use new KafkaConsumer.poll API in Kafka connector
URL: https://github.com/apache/spark/pull/25135#issuecomment-512821931
 
 
   > It feels me like consumer.poll(jt.Duration.ZERO) just doesn't work.
   
   If that would be true several users would protest with blocker bugs on Kafka side but haven't seen such situation (polling without assignment is not possible). I've seen several heavy users who are doing the polling (and no other magic) with `consumer.poll(Duration.ZERO)`. The use-case what I've seen is something like this:
   ```
   wile (!interrupted) {
     val data = consumer.poll(Duration.ZERO)
     process(data)
   }
   ```
   
   Anyway, I'm going to have 2 weeks vacation but after that I'm going to sit together with the Kafka guys and having a deeper look at this (again, only couple of Spark tests are failing consistently and it may happen we're using the API not properly).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org