[GitHub] [spark] cloud-fan commented on a change in pull request #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default

[GitBox <gi...@apache.org>]

cloud-fan commented on a change in pull request #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#discussion_r376305764
 
 

 ##########
 File path: docs/sql-migration-guide.md
 ##########
 @@ -65,6 +65,8 @@ license: |
 
   - In Spark version 2.4 and earlier, if `org.apache.spark.sql.functions.udf(Any, DataType)` gets a Scala closure with primitive-type argument, the returned UDF will return null if the input values is null. Since Spark 3.0, the UDF will return the default value of the Java type if the input value is null. For example, `val f = udf((x: Int) => x, IntegerType)`, `f($"x")` will return null in Spark 2.4 and earlier if column `x` is null, and return 0 in Spark 3.0. This behavior change is introduced because Spark 3.0 is built with Scala 2.12 by default.
 
+  - Since Spark 3.0, using `org.apache.spark.sql.functions.udf(AnyRef, DataType)` is not allowed by default. Set `spark.sql.legacy.useUnTypedUdf.enabled` to true to keep use it.
 
 Review comment:
   can we merge the migration guide between this one and the one that changes the behavior?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org