You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Samphel Norden (JIRA)" <ji...@apache.org> on 2015/07/28 12:36:04 UTC

[jira] [Reopened] (SPARK-9347) spark load of existing parquet files extremely slow if large number of files

     [ https://issues.apache.org/jira/browse/SPARK-9347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Samphel Norden reopened SPARK-9347:
-----------------------------------

I dont think spark-8125 is the issue to which this shd be linked... Spark 8838 is the relevant jira that Chi-Liang mentioned in the PR (earlier comments). 

The problem can be fixed by a distributed spark job but thats still inefficient. Instead there should be an option to skip reading the part files and just read the _common_metadata if schema is identical. Spark 8838 is referencing such an option. 

> spark load of existing parquet files extremely slow if large number of files
> ----------------------------------------------------------------------------
>
>                 Key: SPARK-9347
>                 URL: https://issues.apache.org/jira/browse/SPARK-9347
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 1.3.1
>            Reporter: Samphel Norden
>
> When spark sql shell is launched and we point it to a folder containing a large number of parquet files, the sqlContext.parquetFile() command takes a very long time to load the tables. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org