[jira] [Created] (SPARK-18659) Crash in overwrite table partitions due to hive metastore integration

["Eric Liang (JIRA)" <ji...@apache.org>]

Eric Liang created SPARK-18659:
----------------------------------

             Summary: Crash in overwrite table partitions due to hive metastore integration
                 Key: SPARK-18659
                 URL: https://issues.apache.org/jira/browse/SPARK-18659
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.1.0
            Reporter: Eric Liang
            Priority: Blocker


The following test cases fail due to a crash in hive client when dropping partitions that don't contain files. The last one crashes due to a partition case resolution failure.

{code}
  test("foo") {
    withTable("test") {
      spark.range(10)
        .selectExpr("id", "id as A", "'x' as B")
        .write.partitionBy("A", "B").mode("overwrite")
        .saveAsTable("test")
      spark.sql("insert overwrite table test select id, id, 'x' from range(1)")
      assert(spark.sql("select * from test").count() == 1)
    }
  }

  test("bar") {
    withTable("test") {
      spark.range(10)
        .selectExpr("id", "id as A", "'x' as B")
        .write.partitionBy("A", "B").mode("overwrite")
        .saveAsTable("test")
      spark.sql("insert overwrite table test partition (a, b) select id, id, 'x' from range(1)")
      assert(spark.sql("select * from test").count() == 1)
    }
  }

  test("baz") {
    withTable("test") {
      spark.range(10)
        .selectExpr("id", "id as A", "'x' as B")
        .write.partitionBy("A", "B").mode("overwrite")
        .saveAsTable("test")
      spark.sql("insert overwrite table test partition (A, B) select id, id, 'x' from range(1)")
      assert(spark.sql("select * from test").count() == 1)
    }
  }

  test("qux") {
    withTable("test") {
      spark.range(10)
        .selectExpr("id", "id as A", "'x' as B")
        .write.partitionBy("A", "B").mode("overwrite")
        .saveAsTable("test")
      spark.sql("insert overwrite table test partition (a=1, b) select id, 'x' from range(1)")
      assert(spark.sql("select * from test").count() == 10)
    }
  }
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org