You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Michael Armbrust (JIRA)" <ji...@apache.org> on 2014/11/14 21:44:35 UTC

[jira] [Created] (SPARK-4413) Parquet support through datasource API

Michael Armbrust created SPARK-4413:
---------------------------------------

             Summary: Parquet support through datasource API
                 Key: SPARK-4413
                 URL: https://issues.apache.org/jira/browse/SPARK-4413
             Project: Spark
          Issue Type: Improvement
          Components: SQL
            Reporter: Michael Armbrust
            Assignee: Michael Armbrust
            Priority: Critical


Right now there are several issues with out parquet support.  Specifically, the only way to access parquet files though pure SQL is by including Hive, which has the following issues
 - fairly verbose syntax
 - requires you to explicitly add partitions
 - does not support decimal types.
 - querying tables with many partitions results in metadata operations dominating the query time (even worse when reading from S3).

It would be great to have better native support here though the new datasources API.  Ideally once that is in place we can deprecate the existing ParquetRelation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org