You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2016/03/14 10:59:33 UTC
[jira] [Updated] (SPARK-6504) Cannot read Parquet files generated
from different versions at once
[ https://issues.apache.org/jira/browse/SPARK-6504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen updated SPARK-6504:
-----------------------------
Assignee: Michael Armbrust
> Cannot read Parquet files generated from different versions at once
> -------------------------------------------------------------------
>
> Key: SPARK-6504
> URL: https://issues.apache.org/jira/browse/SPARK-6504
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.2.1
> Reporter: Marius Soutier
> Assignee: Michael Armbrust
> Fix For: 1.3.1
>
>
> When trying to read Parquet files generated by Spark 1.1.1 and 1.2.1 at the same time via `sqlContext.parquetFile("fileFrom1.1.parqut,fileFrom1.2.parquet")` an exception occurs:
> could not merge metadata: key org.apache.spark.sql.parquet.row.metadata has conflicting values: [{"type":"struct","fields":[{"name":"date","type":"string","nullable":true,"metadata":{}},{"name":"account","type":"string","nullable":true,"metadata":{}},{"name":"impressions","type":"long","nullable":false,"metadata":{}},{"name":"cost","type":"double","nullable":false,"metadata":{}},{"name":"clicks","type":"long","nullable":false,"metadata":{}},{"name":"conversions","type":"long","nullable":false,"metadata":{}},{"name":"orderValue","type":"double","nullable":false,"metadata":{}}]}, StructType(List(StructField(date,StringType,true), StructField(account,StringType,true), StructField(impressions,LongType,false), StructField(cost,DoubleType,false), StructField(clicks,LongType,false), StructField(conversions,LongType,false), StructField(orderValue,DoubleType,false)))]
> The Schema is exactly equal.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org