[GitHub] [spark] gaborgsomogyi edited a comment on pull request #29729: [SPARK-32032][SS] Avoid infinite wait in driver because of KafkaConsumer.poll(long) API

[GitBox <gi...@apache.org>]

gaborgsomogyi edited a comment on pull request #29729:
URL: https://github.com/apache/spark/pull/29729#issuecomment-733652656


   @zsxwing @viirya @HeartSaVioR @xuanyuanking I've added the asked change and I would like to you ask to have a look please.
   
   Since it's quite a heavy change I'm listing here the main changes:
   * Introduced the asked config flag `spark.sql.streaming.kafka.useDeprecatedOffsetFetching` with default `true`
   * `KafkaOffsetReader` moved to `KafkaOffsetReaderConsumer` as-is
   * Moved the already reviewed `Admin` based offset reader to `KafkaOffsetReaderAdmin` as-is
   * Made `KafkaOffsetReader` as a wrapper to choose between the 2 implementation
   * Re-added the original implementation of `createConsumer` to `ConsumerStrategy` classes
   * Re-added group ID prefix implementation for the old offset fetching way (the tests only added back in the later commit)
   * Added `KafkaMicroBatchV1SourceWithAdminSuite`, `KafkaMicroBatchV2SourceWithAdminSuite`, `KafkaRelationSuiteWithAdminV1 ` and `KafkaRelationSuiteWithAdminV2` to test batch and streaming with the `Admin` based approach
   * Changed `KafkaOffsetReaderSuite` to test both offset fetching way
   * Adapted the `ss-migration-guide.md` to reflect the new behavior
   
   When we agree that the way is good I'm intended to change the PR description.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org