You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Josh Rosen (Jira)" <ji...@apache.org> on 2024/12/19 23:24:00 UTC
[jira] [Updated] (SPARK-50629) UDFRegistration vs. functions.udf() differ in supported argument count (22 vs. 10)
[ https://issues.apache.org/jira/browse/SPARK-50629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Rosen updated SPARK-50629:
-------------------------------
Description:
`spark.udf.register` and `functions.udf()` differ in the supported argument count
[UDFRegistration|https://github.com/apache/spark/blob/09d93ece2cf78ff265c305fcf79c5a4f4b5bd5a4/sql/api/src/main/scala/org/apache/spark/sql/api/UDFRegistration.scala#L110-L140] supports up to 22 parameters, but [udf()|https://github.com/apache/spark/blob/09d93ece2cf78ff265c305fcf79c5a4f4b5bd5a4/sql/api/src/main/scala/org/apache/spark/sql/functions.scala#L8543-L8586] only supports 10.
The impact of this is that it's hard for users to register UDFs marked as non-deterministic because they can't use the `udf()` syntax.
I suspect that this is a simple accidental omission. I believe this is a straightforward, mechanical fix by re-running the quoted code-generation, so I'm filing this Jira in case someone with bandwidth wants to pick this up and fix it.
> UDFRegistration vs. functions.udf() differ in supported argument count (22 vs. 10)
> ----------------------------------------------------------------------------------
>
> Key: SPARK-50629
> URL: https://issues.apache.org/jira/browse/SPARK-50629
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.0.0
> Reporter: Josh Rosen
> Priority: Major
>
> `spark.udf.register` and `functions.udf()` differ in the supported argument count
> [UDFRegistration|https://github.com/apache/spark/blob/09d93ece2cf78ff265c305fcf79c5a4f4b5bd5a4/sql/api/src/main/scala/org/apache/spark/sql/api/UDFRegistration.scala#L110-L140] supports up to 22 parameters, but [udf()|https://github.com/apache/spark/blob/09d93ece2cf78ff265c305fcf79c5a4f4b5bd5a4/sql/api/src/main/scala/org/apache/spark/sql/functions.scala#L8543-L8586] only supports 10.
> The impact of this is that it's hard for users to register UDFs marked as non-deterministic because they can't use the `udf()` syntax.
> I suspect that this is a simple accidental omission. I believe this is a straightforward, mechanical fix by re-running the quoted code-generation, so I'm filing this Jira in case someone with bandwidth wants to pick this up and fix it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org