You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by karthikjay <as...@gmail.com> on 2018/06/14 13:59:40 UTC
[structured-streaming][parquet] readStream files order in Parquet
My parquet files are first partitioned by environment and then by date like:
env=testing/
date=2018-03-04/
part1.parquet
part2.parquet
part3.parquet
date=2018-03-05/
part1.parquet
part2.parquet
part3.parquet
date=2018-03-06/
part1.parquet
part2.parquet
part3.parquet
In our read stream, I do the following:
val tunerParquetDF = spark
.readStream
.schema(...)
.format("parquet")
.option("basePath", basePath)
.option("path", basePath+"/env*")
.option("maxFilesPerTrigger", 5)
.load()
The expected behavior is that readstream will read files in order of the
dates but the observed behavior is that files are shuffled in random order.
How do I force the date order of read in Parquet files ?
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: [structured-streaming][parquet] readStream files order in Parquet
Posted by Tathagata Das <ta...@gmail.com>.
The files are processed in the order the file last modified timestamp. The
path and partitioning scheme are not used for ordering.
On Thu, Jun 14, 2018 at 6:59 AM, karthikjay <as...@gmail.com> wrote:
> My parquet files are first partitioned by environment and then by date
> like:
>
> env=testing/
> date=2018-03-04/
> part1.parquet
> part2.parquet
> part3.parquet
> date=2018-03-05/
> part1.parquet
> part2.parquet
> part3.parquet
> date=2018-03-06/
> part1.parquet
> part2.parquet
> part3.parquet
> In our read stream, I do the following:
>
> val tunerParquetDF = spark
> .readStream
> .schema(...)
> .format("parquet")
> .option("basePath", basePath)
> .option("path", basePath+"/env*")
> .option("maxFilesPerTrigger", 5)
> .load()
>
> The expected behavior is that readstream will read files in order of the
> dates but the observed behavior is that files are shuffled in random order.
> How do I force the date order of read in Parquet files ?
>
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>