[jira] [Resolved] (SPARK-6330) newParquetRelation gets incorrect FileSystem

["Yin Huai (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-6330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yin Huai resolved SPARK-6330.
-----------------------------
    Resolution: Fixed

> newParquetRelation gets incorrect FileSystem
> --------------------------------------------
>
>                 Key: SPARK-6330
>                 URL: https://issues.apache.org/jira/browse/SPARK-6330
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.3.0
>            Reporter: Volodymyr Lyubinets
>            Assignee: Volodymyr Lyubinets
>            Priority: Blocker
>             Fix For: 1.3.1, 1.4.0
>
>
> Here's a snippet from newParquet.scala:
>     def refresh(): Unit = {
>       val fs = FileSystem.get(sparkContext.hadoopConfiguration)
>       // Support either reading a collection of raw Parquet part-files, or a collection of folders
>       // containing Parquet files (e.g. partitioned Parquet table).
>       val baseStatuses = paths.distinct.map { p =>
>         val qualified = fs.makeQualified(new Path(p))
>         if (!fs.exists(qualified) && maybeSchema.isDefined) {
>           fs.mkdirs(qualified)
>           prepareMetadata(qualified, maybeSchema.get, sparkContext.hadoopConfiguration)
>         }
>         fs.getFileStatus(qualified)
>       }.toArray
> If we are running this locally and path points to S3, fs would be incorrect. A fix is to construct fs for each file separately.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org