You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2015/07/11 00:14:04 UTC
[jira] [Assigned] (SPARK-8990) DataFrameReader.parquet() ignores
user specified data source options
[ https://issues.apache.org/jira/browse/SPARK-8990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-8990:
-----------------------------------
Assignee: Apache Spark (was: Cheng Lian)
> DataFrameReader.parquet() ignores user specified data source options
> --------------------------------------------------------------------
>
> Key: SPARK-8990
> URL: https://issues.apache.org/jira/browse/SPARK-8990
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.4.0
> Reporter: Cheng Lian
> Assignee: Apache Spark
>
> A bad consequence of this is that {{sqlContext.read.parquet(path)}} always do schema merging. For example:
> {code}
> import sqlContext._
> import sqlContext.implicits._
> val path = "s3n://my-bucket/parquet/tiny"
> range(0, 10).coalesce(1).write.mode("overwrite").parquet(path)
> // Explicitly disables schema merging
> read.option("mergeSchema", "false").format("parquet").load(path)
> {code}
> However, we still see all files are opened for schema discovery:
> {noformat}
> 15/07/10 14:46:52 INFO s3native.NativeS3FileSystem: Opening 's3n://databricks-lian/parquet/tiny/_metadata' for reading
> 15/07/10 14:46:52 INFO s3native.NativeS3FileSystem: Opening key 'parquet/tiny/_metadata' for reading at position '314'
> 15/07/10 14:46:52 INFO s3native.NativeS3FileSystem: Opening 's3n://databricks-lian/parquet/tiny/part-r-00000-da490c43-15e2-46b5-95ff-4863e6ab1cc4.gz.parquet' for reading
> 15/07/10 14:46:52 INFO s3native.NativeS3FileSystem: Opening 's3n://databricks-lian/parquet/tiny/_common_metadata' for reading
> 15/07/10 14:46:52 INFO s3native.NativeS3FileSystem: Opening key 'parquet/tiny/part-r-00000-da490c43-15e2-46b5-95ff-4863e6ab1cc4.gz.parquet' for reading at position '345'
> 15/07/10 14:46:52 INFO s3native.NativeS3FileSystem: Opening key 'parquet/tiny/_common_metadata' for reading at position '191'
> 15/07/10 14:46:52 INFO s3native.NativeS3FileSystem: Opening key 'parquet/tiny/_metadata' for reading at position '4'
> 15/07/10 14:46:52 INFO s3native.NativeS3FileSystem: Opening key 'parquet/tiny/part-r-00000-da490c43-15e2-46b5-95ff-4863e6ab1cc4.gz.parquet' for reading at position '97'
> 15/07/10 14:46:52 INFO s3native.NativeS3FileSystem: Opening key 'parquet/tiny/_common_metadata' for reading at position '4'
> {noformat}
> To workaround this issue, use the following instead:
> {noformat}
> sqlContext.read.option("mergeSchema", "false").format("parquet").load(path)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org