[jira] [Created] (SPARK-24696) ColumnPruning rule fails to remove extra Project

["Xiao Li (JIRA)" <ji...@apache.org>]

Xiao Li created SPARK-24696:
-------------------------------

             Summary: ColumnPruning rule fails to remove extra Project
                 Key: SPARK-24696
                 URL: https://issues.apache.org/jira/browse/SPARK-24696
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 2.3.1
            Reporter: Xiao Li


{code}

   test("optimization infinite loop") {
    withTempDir { dir =>
      val path = dir.getCanonicalPath
      import testImplicits._
      Seq((3, "a")).toDF("id", "value").write.format("parquet").mode("overwrite").save(path)
      import org.apache.spark.sql.functions.udf
      val filterIt = udf((value: Int) => value > 0).asNondeterministic
      spark.read.load(path)
        .where(filterIt('id))
        .where('id < 0)
        .select('id)
        .collect
    }
  }

{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org