[jira] [Created] (SPARK-24867) Add AnalysisBarrier to DataFrameWriter

["Xiao Li (JIRA)" <ji...@apache.org>]

Xiao Li created SPARK-24867:
-------------------------------

             Summary: Add AnalysisBarrier to DataFrameWriter 
                 Key: SPARK-24867
                 URL: https://issues.apache.org/jira/browse/SPARK-24867
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.3.1, 2.3.0
            Reporter: Xiao Li
            Assignee: Xiao Li



{code}
      val udf1 = udf({(x: Int, y: Int) => x + y})
      val df = spark.range(0, 3).toDF("a")
        .withColumn("b", udf1($"a", udf1($"a", lit(10))))
      df.cache()
      df.write.saveAsTable("t")
      df.write.saveAsTable("t1")
{code}

Cache is not being used because the plans do not match with the cached plan. This is a regression caused by the changes we made in AnalysisBarrier, since not all the Analyzer rules are idempotent. We need to fix it to Spark 2.3.2




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org