[jira] [Resolved] (SPARK-6774) Implement Parquet complex types backwards-compatiblity rules

["Cheng Lian (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-6774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Cheng Lian resolved SPARK-6774.
-------------------------------
    Resolution: Fixed

Finally, fixed all the Parquet compatibility issues after 6 months!

> Implement Parquet complex types backwards-compatiblity rules
> ------------------------------------------------------------
>
>                 Key: SPARK-6774
>                 URL: https://issues.apache.org/jira/browse/SPARK-6774
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.0.0, 1.1.1, 1.2.1, 1.3.0
>            Reporter: Cheng Lian
>            Priority: Critical
>
> [Parquet format PR #17|https://github.com/apache/incubator-parquet-format/pull/17] standardized representation of Parquet complex types and listed backwards-compatibility rules. Spark SQL should implement these compatibility rules to improve interoperatability.
> Before, Spark SQL is only compatible with parquet-avro, parquet-hive, and Impala. And it's done in an error prone ad-hoc way, because Parquet format spec didn't explicitly specify complex type structures at the time Spark SQL Parquet support was firstly authored. After fixing this issue, we are expected to be compatible with most (if not all) systems that generated Parquet data in a systematic way by conforming to Parquet format spec and implementing all backwards-compatibility rules.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org