You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ka...@apache.org on 2020/12/01 11:37:08 UTC
[spark] branch master updated (1034815 -> e5bb293)
This is an automated email from the ASF dual-hosted git repository.
kabhwan pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 1034815 [SPARK-33572][SQL] Datetime building should fail if the year, month, ..., second combination is invalid
add e5bb293 [SPARK-32032][SS] Avoid infinite wait in driver because of KafkaConsumer.poll(long) API
No new revisions were added by this update.
Summary of changes:
docs/ss-migration-guide.md | 5 +
docs/structured-streaming-kafka-integration.md | 20 +
.../spark/sql/kafka010/ConsumerStrategy.scala | 65 ++-
.../org/apache/spark/sql/kafka010/KafkaBatch.scala | 2 +-
.../spark/sql/kafka010/KafkaOffsetReader.scala | 601 ++-------------------
...etReader.scala => KafkaOffsetReaderAdmin.scala} | 284 +++++-----
...eader.scala => KafkaOffsetReaderConsumer.scala} | 39 +-
.../apache/spark/sql/kafka010/KafkaRelation.scala | 2 +-
.../spark/sql/kafka010/KafkaSourceProvider.scala | 6 +-
.../spark/sql/kafka010/ConsumerStrategySuite.scala | 147 +++++
.../sql/kafka010/KafkaMicroBatchSourceSuite.scala | 42 +-
.../sql/kafka010/KafkaOffsetReaderSuite.scala | 95 +++-
.../spark/sql/kafka010/KafkaRelationSuite.scala | 47 +-
.../org/apache/spark/sql/internal/SQLConf.scala | 13 +
14 files changed, 542 insertions(+), 826 deletions(-)
copy external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/{KafkaOffsetReader.scala => KafkaOffsetReaderAdmin.scala} (73%)
copy external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/{KafkaOffsetReader.scala => KafkaOffsetReaderConsumer.scala} (96%)
create mode 100644 external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/ConsumerStrategySuite.scala
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org