You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2019/05/08 23:42:19 UTC
[spark] branch master updated (57450ed -> 78a403f)
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 57450ed [MINOR][SS] Rename `secondLatestBatchId` to `secondLatestOffsets`
add 78a403f [SPARK-27627][SQL] Make option "pathGlobFilter" as a general option for all file sources
No new revisions were added by this update.
Summary of changes:
docs/sql-data-sources-binaryFile.md | 16 +----
docs/sql-data-sources-load-save-functions.md | 21 +++++++
.../examples/sql/JavaSQLDataSourceExample.java | 5 ++
examples/src/main/python/sql/datasource.py | 5 ++
examples/src/main/r/RSparkSQLExample.R | 4 ++
.../partitioned_users.orc/do_not_read_this.txt | 1 +
.../users.orc | Bin 0 -> 448 bytes
.../favorite_color=red/users.orc | Bin 0 -> 402 bytes
.../spark/examples/sql/SQLDataSourceExample.scala | 5 ++
.../org/apache/spark/sql/avro/AvroFileFormat.scala | 4 ++
.../org/apache/spark/sql/avro/AvroOptions.scala | 5 +-
python/pyspark/sql/readwriter.py | 6 ++
python/pyspark/sql/streaming.py | 6 ++
.../org/apache/spark/sql/DataFrameReader.scala | 9 +++
.../sql/execution/datasources/DataSource.scala | 3 +-
.../datasources/PartitioningAwareFileIndex.scala | 11 +++-
.../datasources/binaryfile/BinaryFileFormat.scala | 70 ++++++++-------------
.../sql/execution/streaming/FileStreamSource.scala | 4 +-
.../execution/streaming/MetadataLogFileIndex.scala | 3 +-
.../spark/sql/streaming/DataStreamReader.scala | 9 +++
.../spark/sql/FileBasedDataSourceSuite.scala | 32 ++++++++++
.../sql/streaming/FileStreamSourceSuite.scala | 19 ++++++
22 files changed, 173 insertions(+), 65 deletions(-)
create mode 100644 examples/src/main/resources/partitioned_users.orc/do_not_read_this.txt
create mode 100644 examples/src/main/resources/partitioned_users.orc/favorite_color=__HIVE_DEFAULT_PARTITION__/users.orc
create mode 100644 examples/src/main/resources/partitioned_users.orc/favorite_color=red/users.orc
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org