You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Michael Armbrust (JIRA)" <ji...@apache.org> on 2015/03/26 16:12:53 UTC

[jira] [Commented] (SPARK-6504) Cannot read Parquet files generated from different versions at once

    [ https://issues.apache.org/jira/browse/SPARK-6504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14382020#comment-14382020 ] 

Michael Armbrust commented on SPARK-6504:
-----------------------------------------

Have you tried this with Spark 1.3?

> Cannot read Parquet files generated from different versions at once
> -------------------------------------------------------------------
>
>                 Key: SPARK-6504
>                 URL: https://issues.apache.org/jira/browse/SPARK-6504
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.2.1
>            Reporter: Marius Soutier
>
> When trying to read Parquet files generated by Spark 1.1.1 and 1.2.1 at the same time via `sqlContext.parquetFile("fileFrom1.1.parqut,fileFrom1.2.parquet")` an exception occurs:
> could not merge metadata: key org.apache.spark.sql.parquet.row.metadata has conflicting values: [{"type":"struct","fields":[{"name":"date","type":"string","nullable":true,"metadata":{}},{"name":"account","type":"string","nullable":true,"metadata":{}},{"name":"impressions","type":"long","nullable":false,"metadata":{}},{"name":"cost","type":"double","nullable":false,"metadata":{}},{"name":"clicks","type":"long","nullable":false,"metadata":{}},{"name":"conversions","type":"long","nullable":false,"metadata":{}},{"name":"orderValue","type":"double","nullable":false,"metadata":{}}]}, StructType(List(StructField(date,StringType,true), StructField(account,StringType,true), StructField(impressions,LongType,false), StructField(cost,DoubleType,false), StructField(clicks,LongType,false), StructField(conversions,LongType,false), StructField(orderValue,DoubleType,false)))]
> The Schema is exactly equal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org