You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by maryannxue <gi...@git.apache.org> on 2018/10/15 18:44:28 UTC

[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

GitHub user maryannxue opened a pull request:

    https://github.com/apache/spark/pull/22732

    [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor signature

    ## What changes were proposed in this pull request?
    
    This is a follow-up PR for #22259. The extra field added in `ScalaUDF` with the original PR was declared optional, but should be indeed required, otherwise callers of `ScalaUDF`'s constructor could ignore this new field and cause the result to be incorrect.
    
    Meanwhile, now that we have this extra field indicating if a null-test should be applied on the corresponding input value, we can also make use of this flag to avoid the rule `HandleNullInputsForUDF` being applied infinitely.
    
    ## How was this patch tested?
    
    Passed affected existing UTs:
    AnalysisSuite
    UDFSuite


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/maryannxue/spark spark-25044-followup

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/22732.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #22732
    
----
commit 5cf3c898978388dc9700219aa6e82c24d5f52f33
Author: maryannxue <ma...@...>
Date:   2018-10-15T18:36:00Z

    Adjust ScalaUDF constructor signature

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    **[Test build #97409 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97409/testReport)** for PR 22732 at commit [`5cf3c89`](https://github.com/apache/spark/commit/5cf3c898978388dc9700219aa6e82c24d5f52f33).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225278847
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
    @@ -31,6 +31,7 @@ import org.apache.spark.sql.types.DataType
      *                  null. Use boxed type or [[Option]] if you wanna do the null-handling yourself.
      * @param dataType  Return type of function.
      * @param children  The input expressions of this UDF.
    + * @param handleNullForInputs Whether the inputs need null-value handling respectively.
    --- End diff --
    
    Maybe give a quick example to clarify? like "For example, primitive types can't take on the value null, and need special handling."


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225735714
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala ---
    @@ -81,11 +81,11 @@ case class UserDefinedFunction protected[sql] (
           f,
           dataType,
           exprs.map(_.expr),
    +      nullableTypes.map(_.map(!_)).getOrElse(exprs.map(_ => false)),
    --- End diff --
    
    Yes that's right. There are a number of UDFs in MLlib, etc that have inputs of type "Any", which isn't great, but I wanted to work around rather than change them for now.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by maryannxue <gi...@git.apache.org>.
Github user maryannxue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225583730
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
    @@ -2137,36 +2137,27 @@ class Analyzer(
     
           case p => p transformExpressionsUp {
     
    -        case udf@ScalaUDF(func, _, inputs, _, _, _, _, nullableTypes) =>
    -          if (nullableTypes.isEmpty) {
    -            // If no nullability info is available, do nothing. No fields will be specially
    -            // checked for null in the plan. If nullability info is incorrect, the results
    -            // of the UDF could be wrong.
    -            udf
    -          } else {
    -            // Otherwise, add special handling of null for fields that can't accept null.
    -            // The result of operations like this, when passed null, is generally to return null.
    -            assert(nullableTypes.length == inputs.length)
    -
    -            // TODO: skip null handling for not-nullable primitive inputs after we can completely
    -            // trust the `nullable` information.
    -            val needsNullCheck = (nullable: Boolean, expr: Expression) =>
    -              nullable && !expr.isInstanceOf[KnownNotNull]
    -            val inputsNullCheck = nullableTypes.zip(inputs)
    -              .filter { case (nullableType, expr) => needsNullCheck(!nullableType, expr) }
    -              .map { case (_, expr) => IsNull(expr) }
    -              .reduceLeftOption[Expression]((e1, e2) => Or(e1, e2))
    -            // Once we add an `If` check above the udf, it is safe to mark those checked inputs
    -            // as not nullable (i.e., wrap them with `KnownNotNull`), because the null-returning
    -            // branch of `If` will be called if any of these checked inputs is null. Thus we can
    -            // prevent this rule from being applied repeatedly.
    -            val newInputs = nullableTypes.zip(inputs).map { case (nullable, expr) =>
    -              if (nullable) expr else KnownNotNull(expr)
    -            }
    -            inputsNullCheck
    -              .map(If(_, Literal.create(null, udf.dataType), udf.copy(children = newInputs)))
    -              .getOrElse(udf)
    -          }
    +        case udf @ ScalaUDF(_, _, inputs, handleNullForInputs, _, _, _, _)
    +          if !handleNullForInputs.forall(!_) =>
    +          // Otherwise, add special handling of null for fields that can't accept null.
    +          // The result of operations like this, when passed null, is generally to return null.
    +          assert(handleNullForInputs.length == inputs.length)
    +
    +          // TODO: skip null handling for not-nullable primitive inputs after we can completely
    +          // trust the `nullable` information.
    +          val inputsNullCheck = handleNullForInputs.zip(inputs)
    --- End diff --
    
    This should answer/confirm a couple of other questions above:
    Since we already have this flag `handleNullForInputs` in `ScalaUDF`,  we can take advantage of it in this rule as well. Say, the first time a `ScalaUDF` hits this rule with `handleNullForInputs` as "false, true, false", then we add a null-handling (if clause) for the second input which is flagged with "true", and from this point on we are all good with all inputs, and we can flag the new `ScalaUDF`'s `handleNullForInputs` as all "false". So even if the new `ScalaUDF` hits this rule for a second time, nothing will be done.
    It should work the same way for the "TODO" above, if `handleNullForInputs` has a "true" flag and the corresponding expression is NOT nullable, we can skip the null handling while flagging it as "false" in the new `ScalaUDF` in the end.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4040/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225280838
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
    @@ -2137,36 +2137,27 @@ class Analyzer(
     
           case p => p transformExpressionsUp {
     
    -        case udf@ScalaUDF(func, _, inputs, _, _, _, _, nullableTypes) =>
    -          if (nullableTypes.isEmpty) {
    -            // If no nullability info is available, do nothing. No fields will be specially
    -            // checked for null in the plan. If nullability info is incorrect, the results
    -            // of the UDF could be wrong.
    -            udf
    -          } else {
    -            // Otherwise, add special handling of null for fields that can't accept null.
    -            // The result of operations like this, when passed null, is generally to return null.
    -            assert(nullableTypes.length == inputs.length)
    -
    -            // TODO: skip null handling for not-nullable primitive inputs after we can completely
    -            // trust the `nullable` information.
    -            val needsNullCheck = (nullable: Boolean, expr: Expression) =>
    -              nullable && !expr.isInstanceOf[KnownNotNull]
    -            val inputsNullCheck = nullableTypes.zip(inputs)
    -              .filter { case (nullableType, expr) => needsNullCheck(!nullableType, expr) }
    -              .map { case (_, expr) => IsNull(expr) }
    -              .reduceLeftOption[Expression]((e1, e2) => Or(e1, e2))
    -            // Once we add an `If` check above the udf, it is safe to mark those checked inputs
    -            // as not nullable (i.e., wrap them with `KnownNotNull`), because the null-returning
    -            // branch of `If` will be called if any of these checked inputs is null. Thus we can
    -            // prevent this rule from being applied repeatedly.
    -            val newInputs = nullableTypes.zip(inputs).map { case (nullable, expr) =>
    -              if (nullable) expr else KnownNotNull(expr)
    -            }
    -            inputsNullCheck
    -              .map(If(_, Literal.create(null, udf.dataType), udf.copy(children = newInputs)))
    -              .getOrElse(udf)
    -          }
    +        case udf @ ScalaUDF(_, _, inputs, handleNullForInputs, _, _, _, _)
    +          if !handleNullForInputs.forall(!_) =>
    --- End diff --
    
    how about `if handleNullForInputs.exists(_)`? if I understand the meaning right, that could be simpler. If anything is true you have to trigger the handling.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r226156536
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/UDFSuite.scala ---
    @@ -393,4 +393,30 @@ class UDFSuite extends QueryTest with SharedSQLContext {
           checkAnswer(df, Seq(Row("12"), Row("24"), Row("3null"), Row(null)))
         }
       }
    +
    +  test("SPARK-25044 Verify null input handling for primitive types - with udf()") {
    +    val udf1 = udf({(x: Long, y: Any) => x * 2 + (if (y == null) 1 else 0)})
    +    val df = spark.range(0, 3).toDF("a")
    +      .withColumn("b", udf1($"a", lit(null)))
    +      .withColumn("c", udf1(lit(null), $"a"))
    +
    +    checkAnswer(
    +      df,
    +      Seq(
    +        Row(0, 1, null),
    +        Row(1, 3, null),
    +        Row(2, 5, null)))
    +  }
    +
    +  test("SPARK-25044 Verify null input handling for primitive types - with udf.register") {
    +    withTable("t") {
    --- End diff --
    
    ah we shuold do `(null, new Integer(1), "x"), ("N", null, "y"), ("N", new Integer(3), null)`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225804995
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala ---
    @@ -81,11 +81,11 @@ case class UserDefinedFunction protected[sql] (
           f,
           dataType,
           exprs.map(_.expr),
    +      nullableTypes.map(_.map(!_)).getOrElse(exprs.map(_ => false)),
    --- End diff --
    
    If we can't get the type information at compile time(e.g. `(a: Any) => xxx`), I don't think `ScalaReflection.getParameterTypes` can get something useful at runtime. For this case we won't add null check anyway, so I think there is no behavior change.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    **[Test build #97573 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97573/testReport)** for PR 22732 at commit [`cf8c457`](https://github.com/apache/spark/commit/cf8c4573b0c69bf4ac5e83dd54c3ce129ff2b329).
     * This patch **fails Spark unit tests**.
     * This patch **does not merge cleanly**.
     * This patch adds the following public classes _(experimental)_:
      * `trait ScalaReflection extends Logging `
      * `    // TODO: make sure this class is only instantiated through `SparkUserDefinedFunction.create()``


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    **[Test build #97601 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97601/testReport)** for PR 22732 at commit [`cb7e97a`](https://github.com/apache/spark/commit/cb7e97a18c0a8d5d806c70771b2c85a01e5f0df5).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    retest this please


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225807388
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
    @@ -31,6 +31,9 @@ import org.apache.spark.sql.types.DataType
      *                  null. Use boxed type or [[Option]] if you wanna do the null-handling yourself.
      * @param dataType  Return type of function.
      * @param children  The input expressions of this UDF.
    + * @param handleNullForInputs Whether the inputs need null-value handling, which preserves the null
    --- End diff --
    
    `inputsNullSafe` SGTM, let's also rename `UserDefinedFunction.nullableTypes`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    For Scala 2.11, we should not introduce any behavior change and also keep binary and source compatibility. 
    
    ```
    scala> import org.apache.spark.sql.types.DataTypes
    import org.apache.spark.sql.types.DataTypes
    
    scala> val f2 = udf({(x: Int) => x}, DataTypes.IntegerType)
    f2: org.apache.spark.sql.expressions.UserDefinedFunction = UserDefinedFunction(<function1>,IntegerType,None)
    
    scala> spark.range(3).select(f2('id + null)).show()
    +----------------+
    |UDF((id + null))|
    +----------------+
    |            null|
    |            null|
    |            null|
    +----------------+
    ```
    
    For Scala 2.12, since we are unable to know the type nullability in a few APIs, we issue a warning message in these cases. Below is the example which will generate a different answer:
    
    ```
    scala> import org.apache.spark.sql.types.DataTypes
    import org.apache.spark.sql.types.DataTypes
    
    scala> val f2 = udf({(x: Int) => x}, DataTypes.IntegerType)
    f2: org.apache.spark.sql.expressions.UserDefinedFunction = UserDefinedFunction($Lambda$2801/26868055@5eb35a26,IntegerType,None)
    
    scala> spark.range(3).select(f2('id + null)).show()
    18/10/18 23:07:02 WARN ScalaReflection: Scala version 2.12.7 cannot get type nullability correctly via reflection, thus Spark cannot add proper input null check for UDF. To avoid this problem, use the typed UDF interfaces instead.
    +----------------+
    |UDF((id + null))|
    +----------------+
    |               0|
    |               0|
    |               0|
    +----------------+
    ```
    



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4069/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225693558
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala ---
    @@ -81,11 +81,11 @@ case class UserDefinedFunction protected[sql] (
           f,
           dataType,
           exprs.map(_.expr),
    +      nullableTypes.map(_.map(!_)).getOrElse(exprs.map(_ => false)),
    --- End diff --
    
    It's new in the sense that Scala 2.12 forced us to change where we get type information for UDFS, or specifically whether the type is allowed to be null. Before we could infer it directly from the closure, but not in 2.12. This change is supposed to just get the info from elsewhere -- from TypeTags when the UDFs are declared. The change ought to be invisible to users and if done correctly should not result in a behavior change. This follow-up change is more about being conservative and lessening the chance of a mistake (see the test case mary ann found) internally or of a mistake for a library that uses ScalaUDF directly anyway. 
    
    For that reason, yeah I agree with your comment here. Default to assuming null checks are needed. 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r226142651
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala ---
    @@ -73,19 +73,21 @@ case class UserDefinedFunction protected[sql] (
        */
       @scala.annotation.varargs
       def apply(exprs: Column*): Column = {
    -    if (inputTypes.isDefined && nullableTypes.isDefined) {
    -      require(inputTypes.get.length == nullableTypes.get.length)
    +    val numOfArgs = ScalaReflection.getParameterCount(f)
    --- End diff --
    
    since we are here, shall we also check `exprs.length == numOfArgs`?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    **[Test build #97587 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97587/testReport)** for PR 22732 at commit [`cb7e97a`](https://github.com/apache/spark/commit/cb7e97a18c0a8d5d806c70771b2c85a01e5f0df5).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r226517584
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
    @@ -39,29 +42,29 @@ import org.apache.spark.sql.types.DataType
      * @param nullable  True if the UDF can return null value.
      * @param udfDeterministic  True if the UDF is deterministic. Deterministic UDF returns same result
      *                          each time it is invoked with a particular input.
    - * @param nullableTypes which of the inputTypes are nullable (i.e. not primitive)
      */
     case class ScalaUDF(
         function: AnyRef,
         dataType: DataType,
         children: Seq[Expression],
    +    inputsNullSafe: Seq[Boolean],
         inputTypes: Seq[DataType] = Nil,
         udfName: Option[String] = None,
         nullable: Boolean = true,
    -    udfDeterministic: Boolean = true,
    -    nullableTypes: Seq[Boolean] = Nil)
    +    udfDeterministic: Boolean = true)
       extends Expression with ImplicitCastInputTypes with NonSQLExpression with UserDefinedExpression {
     
       // The constructor for SPARK 2.1 and 2.2
       def this(
           function: AnyRef,
           dataType: DataType,
           children: Seq[Expression],
    +      inputsNullSafe: Seq[Boolean],
    --- End diff --
    
    SGTM


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by maryannxue <gi...@git.apache.org>.
Github user maryannxue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r226155205
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/UDFSuite.scala ---
    @@ -393,4 +393,30 @@ class UDFSuite extends QueryTest with SharedSQLContext {
           checkAnswer(df, Seq(Row("12"), Row("24"), Row("3null"), Row(null)))
         }
       }
    +
    +  test("SPARK-25044 Verify null input handling for primitive types - with udf()") {
    +    val udf1 = udf({(x: Long, y: Any) => x * 2 + (if (y == null) 1 else 0)})
    +    val df = spark.range(0, 3).toDF("a")
    +      .withColumn("b", udf1($"a", lit(null)))
    +      .withColumn("c", udf1(lit(null), $"a"))
    +
    +    checkAnswer(
    +      df,
    +      Seq(
    +        Row(0, 1, null),
    +        Row(1, 3, null),
    +        Row(2, 5, null)))
    +  }
    +
    +  test("SPARK-25044 Verify null input handling for primitive types - with udf.register") {
    +    withTable("t") {
    --- End diff --
    
    I tried this first, but seems `toDF()` couldn't deduce the right type when nulls are present.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225614430
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
    @@ -31,6 +31,7 @@ import org.apache.spark.sql.types.DataType
      *                  null. Use boxed type or [[Option]] if you wanna do the null-handling yourself.
      * @param dataType  Return type of function.
      * @param children  The input expressions of this UDF.
    + * @param handleNullForInputs Whether the inputs need null-value handling respectively.
    --- End diff --
    
    how about `acceptNullInputs`?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    **[Test build #97581 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97581/testReport)** for PR 22732 at commit [`e848ec7`](https://github.com/apache/spark/commit/e848ec7a2d420c28764ac7319f801666c40684c2).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/22732


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225764876
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala ---
    @@ -81,11 +81,11 @@ case class UserDefinedFunction protected[sql] (
           f,
           dataType,
           exprs.map(_.expr),
    +      nullableTypes.map(_.map(!_)).getOrElse(exprs.map(_ => false)),
    --- End diff --
    
    Hm, but we can't use getParameterTypes anymore. It won't work in Scala 2.12. Where the nullability info is definitely not available, be conservative and assume it all needs null handling?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    **[Test build #97587 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97587/testReport)** for PR 22732 at commit [`cb7e97a`](https://github.com/apache/spark/commit/cb7e97a18c0a8d5d806c70771b2c85a01e5f0df5).
     * This patch **fails due to an unknown error code, -9**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225393919
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
    @@ -39,29 +40,29 @@ import org.apache.spark.sql.types.DataType
      * @param nullable  True if the UDF can return null value.
      * @param udfDeterministic  True if the UDF is deterministic. Deterministic UDF returns same result
      *                          each time it is invoked with a particular input.
    - * @param nullableTypes which of the inputTypes are nullable (i.e. not primitive)
      */
     case class ScalaUDF(
         function: AnyRef,
         dataType: DataType,
         children: Seq[Expression],
    +    handleNullForInputs: Seq[Boolean],
    --- End diff --
    
    It could be merged. I guess the preference is to minimize the change required, so we just have a new field.
    
    That said, UDFs now really require nullability information to work correctly in Scala 2.12. This is the reason the new field is required and not optional. It already caused a new test to fail, so I'm more persuaded that it's OK to make a less backwards-compatible change to make it clear to any future callers that new info is needed. It's an internal class, so reasonable to do so.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225280432
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/UDFRegistration.scala ---
    @@ -124,8 +124,10 @@ class UDFRegistration private[sql] (functionRegistry: FunctionRegistry) extends
             |  val ScalaReflection.Schema(dataType, nullable) = ScalaReflection.schemaFor[RT]
             |  val inputSchemas: Option[Seq[ScalaReflection.Schema]] = Try($inputSchemas).toOption
             |  def builder(e: Seq[Expression]) = if (e.length == $x) {
    -        |    ScalaUDF(func, dataType, e, inputSchemas.map(_.map(_.dataType)).getOrElse(Nil), Some(name), nullable,
    -        |      udfDeterministic = true, nullableTypes = inputSchemas.map(_.map(_.nullable)).getOrElse(Nil))
    +        |    ScalaUDF(func, dataType, e,
    +        |      inputSchemas.map(_.map(!_.nullable)).getOrElse(e.map(_ => false)),
    --- End diff --
    
    Passing !nullable instead of nullable, yep, looks right. The meaning is flipped here, correct.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97578/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by maryannxue <gi...@git.apache.org>.
Github user maryannxue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225585971
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
    @@ -2137,36 +2137,27 @@ class Analyzer(
     
           case p => p transformExpressionsUp {
     
    -        case udf@ScalaUDF(func, _, inputs, _, _, _, _, nullableTypes) =>
    -          if (nullableTypes.isEmpty) {
    -            // If no nullability info is available, do nothing. No fields will be specially
    -            // checked for null in the plan. If nullability info is incorrect, the results
    -            // of the UDF could be wrong.
    -            udf
    -          } else {
    -            // Otherwise, add special handling of null for fields that can't accept null.
    -            // The result of operations like this, when passed null, is generally to return null.
    -            assert(nullableTypes.length == inputs.length)
    -
    -            // TODO: skip null handling for not-nullable primitive inputs after we can completely
    -            // trust the `nullable` information.
    -            val needsNullCheck = (nullable: Boolean, expr: Expression) =>
    -              nullable && !expr.isInstanceOf[KnownNotNull]
    -            val inputsNullCheck = nullableTypes.zip(inputs)
    -              .filter { case (nullableType, expr) => needsNullCheck(!nullableType, expr) }
    -              .map { case (_, expr) => IsNull(expr) }
    -              .reduceLeftOption[Expression]((e1, e2) => Or(e1, e2))
    -            // Once we add an `If` check above the udf, it is safe to mark those checked inputs
    -            // as not nullable (i.e., wrap them with `KnownNotNull`), because the null-returning
    -            // branch of `If` will be called if any of these checked inputs is null. Thus we can
    -            // prevent this rule from being applied repeatedly.
    -            val newInputs = nullableTypes.zip(inputs).map { case (nullable, expr) =>
    -              if (nullable) expr else KnownNotNull(expr)
    -            }
    -            inputsNullCheck
    -              .map(If(_, Literal.create(null, udf.dataType), udf.copy(children = newInputs)))
    -              .getOrElse(udf)
    -          }
    +        case udf @ ScalaUDF(_, _, inputs, handleNullForInputs, _, _, _, _)
    +          if !handleNullForInputs.forall(!_) =>
    --- End diff --
    
    Good point.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97409/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by maryannxue <gi...@git.apache.org>.
Github user maryannxue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r226384713
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
    @@ -39,29 +42,29 @@ import org.apache.spark.sql.types.DataType
      * @param nullable  True if the UDF can return null value.
      * @param udfDeterministic  True if the UDF is deterministic. Deterministic UDF returns same result
      *                          each time it is invoked with a particular input.
    - * @param nullableTypes which of the inputTypes are nullable (i.e. not primitive)
      */
     case class ScalaUDF(
         function: AnyRef,
         dataType: DataType,
         children: Seq[Expression],
    +    inputsNullSafe: Seq[Boolean],
         inputTypes: Seq[DataType] = Nil,
         udfName: Option[String] = None,
         nullable: Boolean = true,
    -    udfDeterministic: Boolean = true,
    -    nullableTypes: Seq[Boolean] = Nil)
    +    udfDeterministic: Boolean = true)
       extends Expression with ImplicitCastInputTypes with NonSQLExpression with UserDefinedExpression {
     
       // The constructor for SPARK 2.1 and 2.2
       def this(
           function: AnyRef,
           dataType: DataType,
           children: Seq[Expression],
    +      inputsNullSafe: Seq[Boolean],
    --- End diff --
    
    We'll do the Scala 2.11 approach for such places where nullableTypes info is unavailable so to at least keep legacy usage of ScalaUDF working.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97581/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r226145051
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/UDFSuite.scala ---
    @@ -393,4 +393,30 @@ class UDFSuite extends QueryTest with SharedSQLContext {
           checkAnswer(df, Seq(Row("12"), Row("24"), Row("3null"), Row(null)))
         }
       }
    +
    +  test("SPARK-25044 Verify null input handling for primitive types - with udf()") {
    +    val udf1 = udf({(x: Long, y: Any) => x * 2 + (if (y == null) 1 else 0)})
    --- End diff --
    
    nit `udf((x: Long, y: Any) => x * 2 + (if (y == null) 1 else 0))`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by maryannxue <gi...@git.apache.org>.
Github user maryannxue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225724505
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala ---
    @@ -81,11 +81,11 @@ case class UserDefinedFunction protected[sql] (
           f,
           dataType,
           exprs.map(_.expr),
    +      nullableTypes.map(_.map(!_)).getOrElse(exprs.map(_ => false)),
    --- End diff --
    
    Looks like the only place where we'd get a not-specified `inputSchemas` is when `ScalaReflection.schemaFor` doesn't recognize a type and throws an exception (https://github.com/apache/spark/blob/1fd59c129a7aa16f9960b109128b166952992f32/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala#L786). The caller seems to be doing a bad job by calling it this way, for example:
    ```
        val inputSchemas = Try(ScalaReflection.schemaFor(typeTag[A1]) :: ScalaReflection.schemaFor(typeTag[A2]) :: Nil).toOption
        val udf = SparkUserDefinedFunction.create(f, dataType, inputSchemas)
    ```
    It would mean if the type of only one of the parameters is unrecognizable by `ScalaReflection`, we'd end up having the entire `Seq` as `None`. I think it's fine not to check null for user-defined types that we don't know, coz they can't be primitive types anyway, but I do think we should make the type inference of each parameter independent so we do handle the nulls that need to be taken care of.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by maryannxue <gi...@git.apache.org>.
Github user maryannxue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225580591
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
    @@ -2137,36 +2137,27 @@ class Analyzer(
     
           case p => p transformExpressionsUp {
     
    -        case udf@ScalaUDF(func, _, inputs, _, _, _, _, nullableTypes) =>
    -          if (nullableTypes.isEmpty) {
    -            // If no nullability info is available, do nothing. No fields will be specially
    -            // checked for null in the plan. If nullability info is incorrect, the results
    -            // of the UDF could be wrong.
    -            udf
    -          } else {
    -            // Otherwise, add special handling of null for fields that can't accept null.
    -            // The result of operations like this, when passed null, is generally to return null.
    -            assert(nullableTypes.length == inputs.length)
    -
    -            // TODO: skip null handling for not-nullable primitive inputs after we can completely
    -            // trust the `nullable` information.
    -            val needsNullCheck = (nullable: Boolean, expr: Expression) =>
    -              nullable && !expr.isInstanceOf[KnownNotNull]
    -            val inputsNullCheck = nullableTypes.zip(inputs)
    -              .filter { case (nullableType, expr) => needsNullCheck(!nullableType, expr) }
    -              .map { case (_, expr) => IsNull(expr) }
    -              .reduceLeftOption[Expression]((e1, e2) => Or(e1, e2))
    -            // Once we add an `If` check above the udf, it is safe to mark those checked inputs
    -            // as not nullable (i.e., wrap them with `KnownNotNull`), because the null-returning
    -            // branch of `If` will be called if any of these checked inputs is null. Thus we can
    -            // prevent this rule from being applied repeatedly.
    -            val newInputs = nullableTypes.zip(inputs).map { case (nullable, expr) =>
    -              if (nullable) expr else KnownNotNull(expr)
    -            }
    -            inputsNullCheck
    -              .map(If(_, Literal.create(null, udf.dataType), udf.copy(children = newInputs)))
    -              .getOrElse(udf)
    -          }
    +        case udf @ ScalaUDF(_, _, inputs, handleNullForInputs, _, _, _, _)
    +          if !handleNullForInputs.forall(!_) =>
    +          // Otherwise, add special handling of null for fields that can't accept null.
    --- End diff --
    
    I was wondering about that, too... Our scala style check is sometimes confusing. Let me double check then.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4001/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4110/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r226520350
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala ---
    @@ -932,6 +935,23 @@ trait ScalaReflection {
         tpe.dealias.erasure.typeSymbol.asClass.fullName
       }
     
    +  /**
    +   * Returns the nullability of the input parameter types of the scala function object.
    +   *
    +   * Note that this only works with Scala 2.11, and the information returned may be inaccurate if
    +   * used with a different Scala version.
    --- End diff --
    
    shall we explicitly return seq of true if it's not scala 2.11? Then the behavior is more predictable than `may be inaccurate`.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r226424308
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
    @@ -39,29 +42,29 @@ import org.apache.spark.sql.types.DataType
      * @param nullable  True if the UDF can return null value.
      * @param udfDeterministic  True if the UDF is deterministic. Deterministic UDF returns same result
      *                          each time it is invoked with a particular input.
    - * @param nullableTypes which of the inputTypes are nullable (i.e. not primitive)
      */
     case class ScalaUDF(
         function: AnyRef,
         dataType: DataType,
         children: Seq[Expression],
    +    inputsNullSafe: Seq[Boolean],
         inputTypes: Seq[DataType] = Nil,
         udfName: Option[String] = None,
         nullable: Boolean = true,
    -    udfDeterministic: Boolean = true,
    -    nullableTypes: Seq[Boolean] = Nil)
    +    udfDeterministic: Boolean = true)
       extends Expression with ImplicitCastInputTypes with NonSQLExpression with UserDefinedExpression {
     
       // The constructor for SPARK 2.1 and 2.2
       def this(
           function: AnyRef,
           dataType: DataType,
           children: Seq[Expression],
    +      inputsNullSafe: Seq[Boolean],
    --- End diff --
    
    I like the idea. We should not break our end users if they just use Scala 2.11. In Spark 3.0, we can be more aggressive and break the binary compatibility. 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r226143221
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/UDFSuite.scala ---
    @@ -393,4 +393,30 @@ class UDFSuite extends QueryTest with SharedSQLContext {
           checkAnswer(df, Seq(Row("12"), Row("24"), Row("3null"), Row(null)))
         }
       }
    +
    +  test("SPARK-25044 Verify null input handling for primitive types - with udf()") {
    +    val udf1 = udf({(x: Long, y: Any) => x * 2 + (if (y == null) 1 else 0)})
    +    val df = spark.range(0, 3).toDF("a")
    +      .withColumn("b", udf1($"a", lit(null)))
    +      .withColumn("c", udf1(lit(null), $"a"))
    +
    +    checkAnswer(
    +      df,
    +      Seq(
    +        Row(0, 1, null),
    +        Row(1, 3, null),
    +        Row(2, 5, null)))
    +  }
    +
    +  test("SPARK-25044 Verify null input handling for primitive types - with udf.register") {
    +    withTable("t") {
    --- End diff --
    
    we can use temp view
    ```
    withTempView("v") {
      Seq((null, 1, "x"), ("N", null, "y"), ("N", 3, null)).toDF("a", "b", "c").createTempView("v")
      ...
    }
    ```


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97601/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by maryannxue <gi...@git.apache.org>.
Github user maryannxue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r226527439
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala ---
    @@ -932,6 +935,23 @@ trait ScalaReflection {
         tpe.dealias.erasure.typeSymbol.asClass.fullName
       }
     
    +  /**
    +   * Returns the nullability of the input parameter types of the scala function object.
    +   *
    +   * Note that this only works with Scala 2.11, and the information returned may be inaccurate if
    +   * used with a different Scala version.
    --- End diff --
    
    The argument here is it's not necessarily wrong if using scala 2.12. if all inputs are of boxed types, then it can still be good. I think it's just enough to say "we don't support it. switch to the new interface otherwise we can't guarantee correctness."


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by maryannxue <gi...@git.apache.org>.
Github user maryannxue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r226155153
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/UDFSuite.scala ---
    @@ -393,4 +393,30 @@ class UDFSuite extends QueryTest with SharedSQLContext {
           checkAnswer(df, Seq(Row("12"), Row("24"), Row("3null"), Row(null)))
         }
       }
    +
    +  test("SPARK-25044 Verify null input handling for primitive types - with udf()") {
    +    val udf1 = udf({(x: Long, y: Any) => x * 2 + (if (y == null) 1 else 0)})
    +    val df = spark.range(0, 3).toDF("a")
    +      .withColumn("b", udf1($"a", lit(null)))
    +      .withColumn("c", udf1(lit(null), $"a"))
    +
    +    checkAnswer(
    +      df,
    +      Seq(
    +        Row(0, 1, null),
    +        Row(1, 3, null),
    +        Row(2, 5, null)))
    +  }
    +
    +  test("SPARK-25044 Verify null input handling for primitive types - with udf.register") {
    +    withTable("t") {
    --- End diff --
    
    I tried this first, but seems `toDF()` couldn't deduce the right type when nulls are present.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97573/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    **[Test build #97578 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97578/testReport)** for PR 22732 at commit [`84cb456`](https://github.com/apache/spark/commit/84cb456c18d02f8abb21934191508fca5e58e6e2).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225952267
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
    @@ -39,29 +40,29 @@ import org.apache.spark.sql.types.DataType
      * @param nullable  True if the UDF can return null value.
      * @param udfDeterministic  True if the UDF is deterministic. Deterministic UDF returns same result
      *                          each time it is invoked with a particular input.
    - * @param nullableTypes which of the inputTypes are nullable (i.e. not primitive)
      */
     case class ScalaUDF(
         function: AnyRef,
         dataType: DataType,
         children: Seq[Expression],
    +    handleNullForInputs: Seq[Boolean],
         inputTypes: Seq[DataType] = Nil,
         udfName: Option[String] = None,
         nullable: Boolean = true,
    -    udfDeterministic: Boolean = true,
    -    nullableTypes: Seq[Boolean] = Nil)
    +    udfDeterministic: Boolean = true)
       extends Expression with ImplicitCastInputTypes with NonSQLExpression with UserDefinedExpression {
     
       // The constructor for SPARK 2.1 and 2.2
       def this(
           function: AnyRef,
           dataType: DataType,
           children: Seq[Expression],
    +      handleNullForInputs: Seq[Boolean],
    --- End diff --
    
    I think we should just remove this constructor. It's weird to keep backward compatibility for a private class, and I don't think it can work anymore. It's not OK to omit the nullable info.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225281705
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
    @@ -2137,36 +2137,27 @@ class Analyzer(
     
           case p => p transformExpressionsUp {
     
    -        case udf@ScalaUDF(func, _, inputs, _, _, _, _, nullableTypes) =>
    -          if (nullableTypes.isEmpty) {
    -            // If no nullability info is available, do nothing. No fields will be specially
    -            // checked for null in the plan. If nullability info is incorrect, the results
    -            // of the UDF could be wrong.
    -            udf
    -          } else {
    -            // Otherwise, add special handling of null for fields that can't accept null.
    -            // The result of operations like this, when passed null, is generally to return null.
    -            assert(nullableTypes.length == inputs.length)
    -
    -            // TODO: skip null handling for not-nullable primitive inputs after we can completely
    -            // trust the `nullable` information.
    -            val needsNullCheck = (nullable: Boolean, expr: Expression) =>
    -              nullable && !expr.isInstanceOf[KnownNotNull]
    -            val inputsNullCheck = nullableTypes.zip(inputs)
    -              .filter { case (nullableType, expr) => needsNullCheck(!nullableType, expr) }
    -              .map { case (_, expr) => IsNull(expr) }
    -              .reduceLeftOption[Expression]((e1, e2) => Or(e1, e2))
    -            // Once we add an `If` check above the udf, it is safe to mark those checked inputs
    -            // as not nullable (i.e., wrap them with `KnownNotNull`), because the null-returning
    -            // branch of `If` will be called if any of these checked inputs is null. Thus we can
    -            // prevent this rule from being applied repeatedly.
    -            val newInputs = nullableTypes.zip(inputs).map { case (nullable, expr) =>
    -              if (nullable) expr else KnownNotNull(expr)
    -            }
    -            inputsNullCheck
    -              .map(If(_, Literal.create(null, udf.dataType), udf.copy(children = newInputs)))
    -              .getOrElse(udf)
    -          }
    +        case udf @ ScalaUDF(_, _, inputs, handleNullForInputs, _, _, _, _)
    +          if !handleNullForInputs.forall(!_) =>
    +          // Otherwise, add special handling of null for fields that can't accept null.
    +          // The result of operations like this, when passed null, is generally to return null.
    +          assert(handleNullForInputs.length == inputs.length)
    +
    +          // TODO: skip null handling for not-nullable primitive inputs after we can completely
    +          // trust the `nullable` information.
    +          val inputsNullCheck = handleNullForInputs.zip(inputs)
    --- End diff --
    
    The TODO above seems to apply to the line that was removed? I'm not totally clear. It's OK to not check `!expr.isInstanceOf[KnownNotNull]` now? you know this better than I. Hey if tests pass I think it proves it.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    **[Test build #97601 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97601/testReport)** for PR 22732 at commit [`cb7e97a`](https://github.com/apache/spark/commit/cb7e97a18c0a8d5d806c70771b2c85a01e5f0df5).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    At this point I think you both know more than I do about this, so go ahead. To the limits of my understanding it sounds reasonable. 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225278704
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
    @@ -39,29 +40,29 @@ import org.apache.spark.sql.types.DataType
      * @param nullable  True if the UDF can return null value.
      * @param udfDeterministic  True if the UDF is deterministic. Deterministic UDF returns same result
      *                          each time it is invoked with a particular input.
    - * @param nullableTypes which of the inputTypes are nullable (i.e. not primitive)
      */
     case class ScalaUDF(
         function: AnyRef,
         dataType: DataType,
         children: Seq[Expression],
    +    handleNullForInputs: Seq[Boolean],
         inputTypes: Seq[DataType] = Nil,
         udfName: Option[String] = None,
         nullable: Boolean = true,
    -    udfDeterministic: Boolean = true,
    -    nullableTypes: Seq[Boolean] = Nil)
    +    udfDeterministic: Boolean = true)
       extends Expression with ImplicitCastInputTypes with NonSQLExpression with UserDefinedExpression {
     
       // The constructor for SPARK 2.1 and 2.2
       def this(
           function: AnyRef,
           dataType: DataType,
           children: Seq[Expression],
    +      handleNullForInputs: Seq[Boolean],
    --- End diff --
    
    We don't want to change this constructor, I believe. It's preserved for backwards-compatibility. It should pass an empty Seq or Nil for the new param.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by maryannxue <gi...@git.apache.org>.
Github user maryannxue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225585740
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
    @@ -39,29 +40,29 @@ import org.apache.spark.sql.types.DataType
      * @param nullable  True if the UDF can return null value.
      * @param udfDeterministic  True if the UDF is deterministic. Deterministic UDF returns same result
      *                          each time it is invoked with a particular input.
    - * @param nullableTypes which of the inputTypes are nullable (i.e. not primitive)
      */
     case class ScalaUDF(
         function: AnyRef,
         dataType: DataType,
         children: Seq[Expression],
    +    handleNullForInputs: Seq[Boolean],
    --- End diff --
    
    Among all other reasons, one argument for not merging the flags with the types is https://github.com/apache/spark/pull/22732#discussion_r225583730.
    As to your second question, it would be for same reason as above, plus https://github.com/apache/spark/pull/22259#discussion_r224252642.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225654592
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala ---
    @@ -81,11 +81,11 @@ case class UserDefinedFunction protected[sql] (
           f,
           dataType,
           exprs.map(_.expr),
    +      nullableTypes.map(_.map(!_)).getOrElse(exprs.map(_ => false)),
    --- End diff --
    
    We need to be more conservative here. The default behavior needs to handle NULLs. We should fix it.
    
    ```
    // Need to handle NULL when the fields can't accept NULL. 
    val handleNullForInputs = nullableTypes.map(_.map(!_)).getOrElse(Seq.fill(exprs.size)(true))
    ```
    



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    **[Test build #97573 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97573/testReport)** for PR 22732 at commit [`cf8c457`](https://github.com/apache/spark/commit/cf8c4573b0c69bf4ac5e83dd54c3ce129ff2b329).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by maryannxue <gi...@git.apache.org>.
Github user maryannxue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r226156109
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala ---
    @@ -73,19 +73,21 @@ case class UserDefinedFunction protected[sql] (
        */
       @scala.annotation.varargs
       def apply(exprs: Column*): Column = {
    -    if (inputTypes.isDefined && nullableTypes.isDefined) {
    -      require(inputTypes.get.length == nullableTypes.get.length)
    +    val numOfArgs = ScalaReflection.getParameterCount(f)
    --- End diff --
    
    Actually I think the "assert" should better be in `ScalaUDF`, but then it would be slightly different from the current behavior whatsoever. I'm keeping other things as they are.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225280125
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala ---
    @@ -73,27 +73,27 @@ case class UserDefinedFunction protected[sql] (
        */
       @scala.annotation.varargs
       def apply(exprs: Column*): Column = {
    -    if (inputTypes.isDefined && nullableTypes.isDefined) {
    -      require(inputTypes.get.length == nullableTypes.get.length)
    +    if (inputTypes.isDefined && handleNullForInputs.isDefined) {
    +      require(inputTypes.get.length == handleNullForInputs.get.length)
         }
     
         Column(ScalaUDF(
           f,
           dataType,
           exprs.map(_.expr),
    +      handleNullForInputs.getOrElse(exprs.map(_ => false)),
    --- End diff --
    
    If not specified, assume nothing needs special null handling. OK, check.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4114/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97574/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    **[Test build #97581 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97581/testReport)** for PR 22732 at commit [`e848ec7`](https://github.com/apache/spark/commit/e848ec7a2d420c28764ac7319f801666c40684c2).
     * This patch **fails due to an unknown error code, -9**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    **[Test build #97578 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97578/testReport)** for PR 22732 at commit [`84cb456`](https://github.com/apache/spark/commit/84cb456c18d02f8abb21934191508fca5e58e6e2).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4109/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by maryannxue <gi...@git.apache.org>.
Github user maryannxue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225587391
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
    @@ -31,6 +31,7 @@ import org.apache.spark.sql.types.DataType
      *                  null. Use boxed type or [[Option]] if you wanna do the null-handling yourself.
      * @param dataType  Return type of function.
      * @param children  The input expressions of this UDF.
    + * @param handleNullForInputs Whether the inputs need null-value handling respectively.
    --- End diff --
    
    Yes, it needs a little explanation here regarding the primitive types. Since the types are not nullable, when the values are null, it usually ends up being represented as a zero value.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    LGTM pending Jenkins


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r226144670
  
    --- Diff: docs/sql-programming-guide.md ---
    @@ -1951,7 +1951,7 @@ working with timestamps in `pandas_udf`s to get the best performance, see
                 <b>AnalysisException is thrown since integer type can not be promoted to string type in a loss-less manner.</b>
               </th>
               <th>
    -            <b>Users can use explict cast</b>
    +            <b>Users can use explicit cast</b>
    --- End diff --
    
    There's the same word mistake right above `explict cast` btw.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    **[Test build #97508 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97508/testReport)** for PR 22732 at commit [`968ed26`](https://github.com/apache/spark/commit/968ed266c7aaf376d428d2ae4edf147036bee3e2).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225602236
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala ---
    @@ -73,27 +73,27 @@ case class UserDefinedFunction protected[sql] (
        */
       @scala.annotation.varargs
       def apply(exprs: Column*): Column = {
    -    if (inputTypes.isDefined && nullableTypes.isDefined) {
    -      require(inputTypes.get.length == nullableTypes.get.length)
    +    if (inputTypes.isDefined && handleNullForInputs.isDefined) {
    +      require(inputTypes.get.length == handleNullForInputs.get.length)
         }
     
         Column(ScalaUDF(
           f,
           dataType,
           exprs.map(_.expr),
    +      handleNullForInputs.getOrElse(exprs.map(_ => false)),
    --- End diff --
    
    I don't know of a case. I suppose that's fine default behavior, and matches what happened in the previous change.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r226141983
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
    @@ -39,29 +42,29 @@ import org.apache.spark.sql.types.DataType
      * @param nullable  True if the UDF can return null value.
      * @param udfDeterministic  True if the UDF is deterministic. Deterministic UDF returns same result
      *                          each time it is invoked with a particular input.
    - * @param nullableTypes which of the inputTypes are nullable (i.e. not primitive)
      */
     case class ScalaUDF(
         function: AnyRef,
         dataType: DataType,
         children: Seq[Expression],
    +    inputsNullSafe: Seq[Boolean],
         inputTypes: Seq[DataType] = Nil,
         udfName: Option[String] = None,
         nullable: Boolean = true,
    -    udfDeterministic: Boolean = true,
    -    nullableTypes: Seq[Boolean] = Nil)
    +    udfDeterministic: Boolean = true)
       extends Expression with ImplicitCastInputTypes with NonSQLExpression with UserDefinedExpression {
     
       // The constructor for SPARK 2.1 and 2.2
       def this(
           function: AnyRef,
           dataType: DataType,
           children: Seq[Expression],
    +      inputsNullSafe: Seq[Boolean],
    --- End diff --
    
    this constructor is here for backward compatibility. If we have to change it, I don't think we need to keep it anymore. cc @gatorsmile 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    After more thoughts, there is one case we don't handle well. For UDFs like `(a: Any, i: Int) => xxx`, previously we can still get the int type info at runtime, and add null check for it. Now we can't get type info at compile time, because we do something like `Try(ScalaReflection.schemaFor[A1] :: ScalaReflection.schemaFor[A2] :: Nil).toOption`. When we fail to get type info of `Any`, we just give up and don't try the next input.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    **[Test build #97574 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97574/testReport)** for PR 22732 at commit [`7a6b2e1`](https://github.com/apache/spark/commit/7a6b2e16a5f45da399da75ee15eb7d81fd8443cf).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225651483
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatDataWriter.scala ---
    @@ -179,7 +179,8 @@ class DynamicPartitionDataWriter(
           val partitionName = ScalaUDF(
             ExternalCatalogUtils.getPartitionPathString _,
             StringType,
    -        Seq(Literal(c.name), Cast(c, StringType, Option(description.timeZoneId))))
    +        Seq(Literal(c.name), Cast(c, StringType, Option(description.timeZoneId))),
    +        Seq(false, false))
    --- End diff --
    
    ```suggestion
            handleNullForInputs = Seq(false, false))
    ```


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225393220
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
    @@ -39,29 +40,29 @@ import org.apache.spark.sql.types.DataType
      * @param nullable  True if the UDF can return null value.
      * @param udfDeterministic  True if the UDF is deterministic. Deterministic UDF returns same result
      *                          each time it is invoked with a particular input.
    - * @param nullableTypes which of the inputTypes are nullable (i.e. not primitive)
      */
     case class ScalaUDF(
         function: AnyRef,
         dataType: DataType,
         children: Seq[Expression],
    +    handleNullForInputs: Seq[Boolean],
    --- End diff --
    
    Maybe I missed something but:
    
    1. Why don't we just merge `handleNullForInputs` and `inputTypes`?
    2. Why `handleNullForInputs` is required whereas `inputTypes`'s default is `Nil`?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    **[Test build #97409 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97409/testReport)** for PR 22732 at commit [`5cf3c89`](https://github.com/apache/spark/commit/5cf3c898978388dc9700219aa6e82c24d5f52f33).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    **[Test build #97508 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97508/testReport)** for PR 22732 at commit [`968ed26`](https://github.com/apache/spark/commit/968ed266c7aaf376d428d2ae4edf147036bee3e2).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by maryannxue <gi...@git.apache.org>.
Github user maryannxue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225588931
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala ---
    @@ -73,27 +73,27 @@ case class UserDefinedFunction protected[sql] (
        */
       @scala.annotation.varargs
       def apply(exprs: Column*): Column = {
    -    if (inputTypes.isDefined && nullableTypes.isDefined) {
    -      require(inputTypes.get.length == nullableTypes.get.length)
    +    if (inputTypes.isDefined && handleNullForInputs.isDefined) {
    +      require(inputTypes.get.length == handleNullForInputs.get.length)
         }
     
         Column(ScalaUDF(
           f,
           dataType,
           exprs.map(_.expr),
    +      handleNullForInputs.getOrElse(exprs.map(_ => false)),
    --- End diff --
    
    I am actually not so sure about this part, but this is just to be consistent with the behavior in your previous check-in. Can you give an example of such end-user cases where these flags are unavailable/not specified?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225280977
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
    @@ -2137,36 +2137,27 @@ class Analyzer(
     
           case p => p transformExpressionsUp {
     
    -        case udf@ScalaUDF(func, _, inputs, _, _, _, _, nullableTypes) =>
    -          if (nullableTypes.isEmpty) {
    -            // If no nullability info is available, do nothing. No fields will be specially
    -            // checked for null in the plan. If nullability info is incorrect, the results
    -            // of the UDF could be wrong.
    -            udf
    -          } else {
    -            // Otherwise, add special handling of null for fields that can't accept null.
    -            // The result of operations like this, when passed null, is generally to return null.
    -            assert(nullableTypes.length == inputs.length)
    -
    -            // TODO: skip null handling for not-nullable primitive inputs after we can completely
    -            // trust the `nullable` information.
    -            val needsNullCheck = (nullable: Boolean, expr: Expression) =>
    -              nullable && !expr.isInstanceOf[KnownNotNull]
    -            val inputsNullCheck = nullableTypes.zip(inputs)
    -              .filter { case (nullableType, expr) => needsNullCheck(!nullableType, expr) }
    -              .map { case (_, expr) => IsNull(expr) }
    -              .reduceLeftOption[Expression]((e1, e2) => Or(e1, e2))
    -            // Once we add an `If` check above the udf, it is safe to mark those checked inputs
    -            // as not nullable (i.e., wrap them with `KnownNotNull`), because the null-returning
    -            // branch of `If` will be called if any of these checked inputs is null. Thus we can
    -            // prevent this rule from being applied repeatedly.
    -            val newInputs = nullableTypes.zip(inputs).map { case (nullable, expr) =>
    -              if (nullable) expr else KnownNotNull(expr)
    -            }
    -            inputsNullCheck
    -              .map(If(_, Literal.create(null, udf.dataType), udf.copy(children = newInputs)))
    -              .getOrElse(udf)
    -          }
    +        case udf @ ScalaUDF(_, _, inputs, handleNullForInputs, _, _, _, _)
    +          if !handleNullForInputs.forall(!_) =>
    +          // Otherwise, add special handling of null for fields that can't accept null.
    --- End diff --
    
    Do we need to indent all of this code below one more unit? it's as deep as the 'if ... =>' above


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    That's a good point, but that was already an issue right? it isn't introduced by this change at least? 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    **[Test build #97464 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97464/testReport)** for PR 22732 at commit [`19ed5e8`](https://github.com/apache/spark/commit/19ed5e866c25dc1ad26e35d6478f57392eb6b671).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225279364
  
    --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala ---
    @@ -314,24 +314,24 @@ class AnalysisSuite extends AnalysisTest with Matchers {
         }
     
         // non-primitive parameters do not need special null handling
    -    val udf1 = ScalaUDF((s: String) => "x", StringType, string :: Nil)
    +    val udf1 = ScalaUDF((s: String) => "x", StringType, string :: Nil, false :: Nil)
         val expected1 = udf1
         checkUDF(udf1, expected1)
     
         // only primitive parameter needs special null handling
         val udf2 = ScalaUDF((s: String, d: Double) => "x", StringType, string :: double :: Nil,
    -      nullableTypes = true :: false :: Nil)
    +      false :: true :: Nil)
         val expected2 =
    -      If(IsNull(double), nullResult, udf2.copy(children = string :: KnownNotNull(double) :: Nil))
    +      If(IsNull(double), nullResult, udf2.copy(handleNullForInputs = false :: false :: Nil))
    --- End diff --
    
    Just checking my understanding -- string gets false because it's a reference type, and a double known to not be null also doesn't need null checking, so gets false. OK that sounds right.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    There is an argument about whether #22259 introduced behavior changes. Here is my analysis.
    
    Before #22259 , the type and null check was done as
    1. user registers UDFs, like `(a: Int) => xxx`, `(b: Any) => xxx`
    2. at compile time, get the type info, and set input types, so that the analyzer can add cast or fail the query if the real input data type doesn't match. Note that, UDFs like `(b: Any) => xxx` has no type info and we won't do type check.
    3. at runtime, use reflection to get type info again, and add null check if an input is primitive type.
    
    After #22259 , the type and null check is done as
    1. user registers UDFs, like `(a: Int) => xxx`, `(b: Any) => xxx`
    2. at compile time, get the type info, set input types and input nullable, so that the analyzer can add cast or fail the query if the real input data type doesn't match, and add null check if necessary. Note that, UDFs like `(b: Any) => xxx` has no type info and we won't do type and null check.
    
    So we may have a behavior change if users register UDFs in a weird way. e.g. they define `(a: Int) => xxx`, but cast it to `Any => xx` during registration. Then we can't get the real type info at compile time.
    
    I'd say this is an invalid use case, because the data type check is also lost. I think we don't need to treat it as a behavior change.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225393373
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
    @@ -39,29 +40,29 @@ import org.apache.spark.sql.types.DataType
      * @param nullable  True if the UDF can return null value.
      * @param udfDeterministic  True if the UDF is deterministic. Deterministic UDF returns same result
      *                          each time it is invoked with a particular input.
    - * @param nullableTypes which of the inputTypes are nullable (i.e. not primitive)
      */
     case class ScalaUDF(
         function: AnyRef,
         dataType: DataType,
         children: Seq[Expression],
    +    handleNullForInputs: Seq[Boolean],
    --- End diff --
    
    Adding `handleNullForInputs` doesn't look reducing confusion a lot to me. Since this PR targets only refactoring mainly to reduce confusion and the easy of use, this concern should be addressed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97508/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97587/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225655290
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala ---
    @@ -81,11 +81,11 @@ case class UserDefinedFunction protected[sql] (
           f,
           dataType,
           exprs.map(_.expr),
    +      nullableTypes.map(_.map(!_)).getOrElse(exprs.map(_ => false)),
    --- End diff --
    
    @srowen @cloud-fan This sounds a new behavior added in Spark 2.4 release by the PR https://github.com/apache/spark/pull/22259, right?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by maryannxue <gi...@git.apache.org>.
Github user maryannxue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225605581
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala ---
    @@ -73,27 +73,27 @@ case class UserDefinedFunction protected[sql] (
        */
       @scala.annotation.varargs
       def apply(exprs: Column*): Column = {
    -    if (inputTypes.isDefined && nullableTypes.isDefined) {
    -      require(inputTypes.get.length == nullableTypes.get.length)
    +    if (inputTypes.isDefined && handleNullForInputs.isDefined) {
    +      require(inputTypes.get.length == handleNullForInputs.get.length)
         }
     
         Column(ScalaUDF(
           f,
           dataType,
           exprs.map(_.expr),
    +      handleNullForInputs.getOrElse(exprs.map(_ => false)),
    --- End diff --
    
    I actually assume the default behavior should be the other way around. If we don't know, we just do the if-else null handling and it wouldn't do us any harm correctness-wise, right? Anyway I'm not gonna change that in this PR but hope we can get an idea if and when this default behavior will happen.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97464/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by maryannxue <gi...@git.apache.org>.
Github user maryannxue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225619242
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
    @@ -31,6 +31,7 @@ import org.apache.spark.sql.types.DataType
      *                  null. Use boxed type or [[Option]] if you wanna do the null-handling yourself.
      * @param dataType  Return type of function.
      * @param children  The input expressions of this UDF.
    + * @param handleNullForInputs Whether the inputs need null-value handling respectively.
    --- End diff --
    
    Makes sense. Since I'm also using this flag to replace `KnownNotNull`, I think `inputsNullSafe` would be a better name?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4112/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by maryannxue <gi...@git.apache.org>.
Github user maryannxue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225586820
  
    --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala ---
    @@ -314,24 +314,24 @@ class AnalysisSuite extends AnalysisTest with Matchers {
         }
     
         // non-primitive parameters do not need special null handling
    -    val udf1 = ScalaUDF((s: String) => "x", StringType, string :: Nil)
    +    val udf1 = ScalaUDF((s: String) => "x", StringType, string :: Nil, false :: Nil)
         val expected1 = udf1
         checkUDF(udf1, expected1)
     
         // only primitive parameter needs special null handling
         val udf2 = ScalaUDF((s: String, d: Double) => "x", StringType, string :: double :: Nil,
    -      nullableTypes = true :: false :: Nil)
    +      false :: true :: Nil)
         val expected2 =
    -      If(IsNull(double), nullResult, udf2.copy(children = string :: KnownNotNull(double) :: Nil))
    +      If(IsNull(double), nullResult, udf2.copy(handleNullForInputs = false :: false :: Nil))
    --- End diff --
    
    Yes, as explained in https://github.com/apache/spark/pull/22732#discussion_r225583730


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by maryannxue <gi...@git.apache.org>.
Github user maryannxue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225606907
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
    @@ -39,29 +40,29 @@ import org.apache.spark.sql.types.DataType
      * @param nullable  True if the UDF can return null value.
      * @param udfDeterministic  True if the UDF is deterministic. Deterministic UDF returns same result
      *                          each time it is invoked with a particular input.
    - * @param nullableTypes which of the inputTypes are nullable (i.e. not primitive)
      */
     case class ScalaUDF(
         function: AnyRef,
         dataType: DataType,
         children: Seq[Expression],
    +    handleNullForInputs: Seq[Boolean],
         inputTypes: Seq[DataType] = Nil,
         udfName: Option[String] = None,
         nullable: Boolean = true,
    -    udfDeterministic: Boolean = true,
    -    nullableTypes: Seq[Boolean] = Nil)
    +    udfDeterministic: Boolean = true)
       extends Expression with ImplicitCastInputTypes with NonSQLExpression with UserDefinedExpression {
     
       // The constructor for SPARK 2.1 and 2.2
       def this(
           function: AnyRef,
           dataType: DataType,
           children: Seq[Expression],
    +      handleNullForInputs: Seq[Boolean],
    --- End diff --
    
    Just realized this is different from you original PR already, @cloud-fan did a follow-up PR adding the "nullableTypes". I'll revert the change here.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r226145150
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/UDFSuite.scala ---
    @@ -393,4 +393,30 @@ class UDFSuite extends QueryTest with SharedSQLContext {
           checkAnswer(df, Seq(Row("12"), Row("24"), Row("3null"), Row(null)))
         }
       }
    +
    +  test("SPARK-25044 Verify null input handling for primitive types - with udf()") {
    +    val udf1 = udf({(x: Long, y: Any) => x * 2 + (if (y == null) 1 else 0)})
    +    val df = spark.range(0, 3).toDF("a")
    +      .withColumn("b", udf1($"a", lit(null)))
    +      .withColumn("c", udf1(lit(null), $"a"))
    +
    +    checkAnswer(
    +      df,
    +      Seq(
    +        Row(0, 1, null),
    +        Row(1, 3, null),
    +        Row(2, 5, null)))
    +  }
    +
    +  test("SPARK-25044 Verify null input handling for primitive types - with udf.register") {
    +    withTable("t") {
    +      sql("create table t(a varchar(10), b int, c varchar(10)) using parquet")
    --- End diff --
    
    nit: upper case for keywords


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by maryannxue <gi...@git.apache.org>.
Github user maryannxue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r226521257
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala ---
    @@ -932,6 +935,23 @@ trait ScalaReflection {
         tpe.dealias.erasure.typeSymbol.asClass.fullName
       }
     
    +  /**
    +   * Returns the nullability of the input parameter types of the scala function object.
    +   *
    +   * Note that this only works with Scala 2.11, and the information returned may be inaccurate if
    +   * used with a different Scala version.
    --- End diff --
    
    I think it simply returns if going through the below code path. I should probably make the java doc clearer.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r226145444
  
    --- Diff: docs/sql-programming-guide.md ---
    @@ -1978,6 +1978,7 @@ working with timestamps in `pandas_udf`s to get the best performance, see
       - Since Spark 2.4, empty strings are saved as quoted empty strings `""`. In version 2.3 and earlier, empty strings are equal to `null` values and do not reflect to any characters in saved CSV files. For example, the row of `"a", null, "", 1` was writted as `a,,,1`. Since Spark 2.4, the same row is saved as `a,,"",1`. To restore the previous behavior, set the CSV option `emptyValue` to empty (not quoted) string.  
       - Since Spark 2.4, The LOAD DATA command supports wildcard `?` and `*`, which match any one character, and zero or more characters, respectively. Example: `LOAD DATA INPATH '/tmp/folder*/'` or `LOAD DATA INPATH '/tmp/part-?'`. Special Characters like `space` also now work in paths. Example: `LOAD DATA INPATH '/tmp/folder name/'`.
       - In Spark version 2.3 and earlier, HAVING without GROUP BY is treated as WHERE. This means, `SELECT 1 FROM range(10) HAVING true` is executed as `SELECT 1 FROM range(10) WHERE true`  and returns 10 rows. This violates SQL standard, and has been fixed in Spark 2.4. Since Spark 2.4, HAVING without GROUP BY is treated as a global aggregate, which means `SELECT 1 FROM range(10) HAVING true` will return only one row. To restore the previous behavior, set `spark.sql.legacy.parser.havingWithoutGroupByAsWhere` to `true`.
    +  - Since Spark 2.4, use of the method `def udf(f: AnyRef, dataType: DataType): UserDefinedFunction` should not expect any automatic null handling of the input parameters, thus a null input of a Scala primitive type will be converted to the type's corresponding default value in the UDF. All other UDF declaration and registration methods remain the same behavior as before.
    --- End diff --
    
    Does this target to backport to 2.4?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r226156400
  
    --- Diff: docs/sql-programming-guide.md ---
    @@ -1978,6 +1978,7 @@ working with timestamps in `pandas_udf`s to get the best performance, see
       - Since Spark 2.4, empty strings are saved as quoted empty strings `""`. In version 2.3 and earlier, empty strings are equal to `null` values and do not reflect to any characters in saved CSV files. For example, the row of `"a", null, "", 1` was writted as `a,,,1`. Since Spark 2.4, the same row is saved as `a,,"",1`. To restore the previous behavior, set the CSV option `emptyValue` to empty (not quoted) string.  
       - Since Spark 2.4, The LOAD DATA command supports wildcard `?` and `*`, which match any one character, and zero or more characters, respectively. Example: `LOAD DATA INPATH '/tmp/folder*/'` or `LOAD DATA INPATH '/tmp/part-?'`. Special Characters like `space` also now work in paths. Example: `LOAD DATA INPATH '/tmp/folder name/'`.
       - In Spark version 2.3 and earlier, HAVING without GROUP BY is treated as WHERE. This means, `SELECT 1 FROM range(10) HAVING true` is executed as `SELECT 1 FROM range(10) WHERE true`  and returns 10 rows. This violates SQL standard, and has been fixed in Spark 2.4. Since Spark 2.4, HAVING without GROUP BY is treated as a global aggregate, which means `SELECT 1 FROM range(10) HAVING true` will return only one row. To restore the previous behavior, set `spark.sql.legacy.parser.havingWithoutGroupByAsWhere` to `true`.
    +  - Since Spark 2.4, use of the method `def udf(f: AnyRef, dataType: DataType): UserDefinedFunction` should not expect any automatic null handling of the input parameters, thus a null input of a Scala primitive type will be converted to the type's corresponding default value in the UDF. All other UDF declaration and registration methods remain the same behavior as before.
    --- End diff --
    
    yes, this fixes a bug introduced by a commit in 2.4


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    **[Test build #97464 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97464/testReport)** for PR 22732 at commit [`19ed5e8`](https://github.com/apache/spark/commit/19ed5e866c25dc1ad26e35d6478f57392eb6b671).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4115/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r226382186
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
    @@ -39,29 +42,29 @@ import org.apache.spark.sql.types.DataType
      * @param nullable  True if the UDF can return null value.
      * @param udfDeterministic  True if the UDF is deterministic. Deterministic UDF returns same result
      *                          each time it is invoked with a particular input.
    - * @param nullableTypes which of the inputTypes are nullable (i.e. not primitive)
      */
     case class ScalaUDF(
         function: AnyRef,
         dataType: DataType,
         children: Seq[Expression],
    +    inputsNullSafe: Seq[Boolean],
         inputTypes: Seq[DataType] = Nil,
         udfName: Option[String] = None,
         nullable: Boolean = true,
    -    udfDeterministic: Boolean = true,
    -    nullableTypes: Seq[Boolean] = Nil)
    +    udfDeterministic: Boolean = true)
       extends Expression with ImplicitCastInputTypes with NonSQLExpression with UserDefinedExpression {
     
       // The constructor for SPARK 2.1 and 2.2
       def this(
           function: AnyRef,
           dataType: DataType,
           children: Seq[Expression],
    +      inputsNullSafe: Seq[Boolean],
    --- End diff --
    
    We should not break it. 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4123/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    **[Test build #97574 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97574/testReport)** for PR 22732 at commit [`7a6b2e1`](https://github.com/apache/spark/commit/7a6b2e16a5f45da399da75ee15eb7d81fd8443cf).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by maryannxue <gi...@git.apache.org>.
Github user maryannxue commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    @srowen What @cloud-fan described is a change introduced in #22259. We can fix it by keeping each call to `ScalaReflection.schemaFor` in their own `Try` blocks.
    As to `UserDefinedFunction`, after offline discussions with @cloud-fan, we decided that there should be no occurrences of calling the constructor without setting the `nullableTypes`, so we'll just assert `nullableTypes.length == exprs.length`.
    I'll go ahead and fix these two items and update the PR.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by maryannxue <gi...@git.apache.org>.
Github user maryannxue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225986215
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatDataWriter.scala ---
    @@ -179,7 +179,8 @@ class DynamicPartitionDataWriter(
           val partitionName = ScalaUDF(
             ExternalCatalogUtils.getPartitionPathString _,
             StringType,
    -        Seq(Literal(c.name), Cast(c, StringType, Option(description.timeZoneId))))
    +        Seq(Literal(c.name), Cast(c, StringType, Option(description.timeZoneId))),
    +        Seq(false, false))
    --- End diff --
    
    Don't think we need to do that for required arguments, or we do?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF construc...

Posted by maryannxue <gi...@git.apache.org>.
Github user maryannxue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22732#discussion_r225762708
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala ---
    @@ -81,11 +81,11 @@ case class UserDefinedFunction protected[sql] (
           f,
           dataType,
           exprs.map(_.expr),
    +      nullableTypes.map(_.map(!_)).getOrElse(exprs.map(_ => false)),
    --- End diff --
    
    In addition to what I just pointed out, which is when we did try to get `inputSchemas` through `ScalaReflection.schemaFor` and got an exception for unrecognized types, there's another case where we could get an unspecified `nullableTypes`, and that is when `UserDefinedFunction` is instantiated calling the constructor but not the `create` method.
    Then I assume it's created by an earlier version, and we should use the old logic, i.e., `ScalaReflection.getParameterTypes` (https://github.com/apache/spark/pull/22259/files#diff-57b3d87be744b7d79a9beacf8e5e5eb2L2153) to get the correct information for `nullableTypes`. Is that right, @cloud-fan @srowen ?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #22732: [SPARK-25044][FOLLOW-UP] Change ScalaUDF constructor sig...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/22732
  
    thanks, merging to master/2.4!


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org