[jira] [Created] (SPARK-19039) UDF ClosureCleaner bug when UDF, col applied in paste mode in REPL

["Joseph K. Bradley (JIRA)" <ji...@apache.org>]

Joseph K. Bradley created SPARK-19039:
-----------------------------------------

             Summary: UDF ClosureCleaner bug when UDF, col applied in paste mode in REPL
                 Key: SPARK-19039
                 URL: https://issues.apache.org/jira/browse/SPARK-19039
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.1.0, 2.0.2, 1.6.3
            Reporter: Joseph K. Bradley


When I try this:
* Define UDF
* Apply UDF to get Column
* Use Column in a DataFrame

I can find weird behavior in the spark-shell when using paste mode.

To reproduce this, paste this into the spark-shell:
{code}
import org.apache.spark.sql.functions._
val df = spark.createDataFrame(Seq(
  ("hi", 1),
  ("there", 2),
  ("the", 3),
  ("end", 4)
)).toDF("a", "b")

val myNumbers = Set(1,2,3)
val tmpUDF = udf { (n: Int) => myNumbers.contains(n) }

val rowHasMyNumber = tmpUDF($"b")
df.where(rowHasMyNumber).show()
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org