[jira] [Resolved] (SPARK-6795) Avoid reading Parquet footers on driver side when an global arbitrative schema is available

["Cheng Lian (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-6795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Cheng Lian resolved SPARK-6795.
-------------------------------
    Resolution: Fixed

This has been fixed together with SPARK-8125.

> Avoid reading Parquet footers on driver side when an global arbitrative schema is available
> -------------------------------------------------------------------------------------------
>
>                 Key: SPARK-6795
>                 URL: https://issues.apache.org/jira/browse/SPARK-6795
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.0.2, 1.1.1, 1.2.1, 1.3.1
>            Reporter: Cheng Lian
>            Priority: Critical
>
> With the help of [Parquet MR PR #91|https://github.com/apache/incubator-parquet-mr/pull/91] which will be included in the official release of Parquet MR 1.6.0, now it's possible to avoid reading footers on the driver side completely when an global arbitrative schema is available.
> Currently, the global schema can be either Hive metastore schema or specified via data sources DDL. All tasks should verify Parquet data files and reconcile possible schema conflicts locally against this global schema.
> However, when no global schema is available and schema merging is enabled, we still need to read schemas from all data files to infer a valid global schema.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org