[jira] [Commented] (SPARK-20144) spark.read.parquet no long maintains ordering of the data

["sam (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-20144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203882#comment-16203882 ] 

sam commented on SPARK-20144:
-----------------------------

I think this is a regression.  We used to be able to easily control the number of output files / tasks based on num files and coalesce.  Now I have to use `repartition` to get the desired num files / partitions which is unnecessarily expensive.

I've tried playing with spark.sql.files.maxPartitionBytes and spark.sql.files.openCostInBytes to see if I can force the conventional behaviour.

{code}
val ss = SparkSession.builder().appName("uber-cp").master(conf.master())
             .config("spark.sql.files.maxPartitionBytes", 1)
             .config("spark.sql.files.openCostInBytes", Long.MaxValue)
{code}

This didn't work.  Spark just squashes all my parquet files into less partitions.

Suggest a simple `option` on DataFrameReader that can disable this (or enable it, default behaviour should be same as 1.6).

> spark.read.parquet no long maintains ordering of the data
> ---------------------------------------------------------
>
>                 Key: SPARK-20144
>                 URL: https://issues.apache.org/jira/browse/SPARK-20144
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.0.2
>            Reporter: Li Jin
>
> Hi, We are trying to upgrade Spark from 1.6.3 to 2.0.2. One issue we found is when we read parquet files in 2.0.2, the ordering of rows in the resulting dataframe is not the same as the ordering of rows in the dataframe that the parquet file was reproduced with. 
> This is because FileSourceStrategy.scala combines the parquet files into fewer partitions and also reordered them. This breaks our workflows because they assume the ordering of the data. 
> Is this considered a bug? Also FileSourceStrategy and FileSourceScanExec changed quite a bit from 2.0.2 to 2.1, so not sure if this is an issue with 2.1.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org