[jira] [Created] (SPARK-13207) _SUCCESS should not break partition discovery

["Yin Huai (JIRA)" <ji...@apache.org>]

Yin Huai created SPARK-13207:
--------------------------------

             Summary: _SUCCESS should not break partition discovery
                 Key: SPARK-13207
                 URL: https://issues.apache.org/jira/browse/SPARK-13207
             Project: Spark
          Issue Type: Bug
          Components: SQL
            Reporter: Yin Huai
            Assignee: Yin Huai


Partitioning discovery will fail with the following case
{code}
test("_SUCCESS should not break partitioning discovery") {
    withTempPath { dir =>
      val tablePath = new File(dir, "table")
      val df = (1 to 3).map(i => (i, i, i, i)).toDF("a", "b", "c", "d")

      df.write
        .format("parquet")
        .partitionBy("b", "c", "d")
        .save(tablePath.getCanonicalPath)

      Files.touch(new File(s"${tablePath.getCanonicalPath}/b=1", "_SUCCESS"))
      Files.touch(new File(s"${tablePath.getCanonicalPath}/b=1/c=1", "_SUCCESS"))
      Files.touch(new File(s"${tablePath.getCanonicalPath}/b=1/c=1/d=1", "_SUCCESS"))
      checkAnswer(sqlContext.read.format("parquet").load(tablePath.getCanonicalPath), df)
    }
  }
{code}

{{_SUCCESS}} is the cause of this problem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org