You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2018/01/05 00:00:08 UTC
[jira] [Commented] (SPARK-22965) Add deterministic parameter to
registerJavaFunction
[ https://issues.apache.org/jira/browse/SPARK-22965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16312232#comment-16312232 ]
Apache Spark commented on SPARK-22965:
--------------------------------------
User 'gatorsmile' has created a pull request for this issue:
https://github.com/apache/spark/pull/20162
> Add deterministic parameter to registerJavaFunction
> ---------------------------------------------------
>
> Key: SPARK-22965
> URL: https://issues.apache.org/jira/browse/SPARK-22965
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 2.3.0
> Reporter: Xiao Li
> Assignee: Xiao Li
>
> To register a JAVA UDF in PySpark, users are unable to specify the registered UDF is not deterministic. The proposal is to add the extra parameter deterministic at the end of the function registerJavaFunction
> Below is an example.
> {noformat}
> >>> from pyspark.sql.types import DoubleType
> >>> sqlContext.registerJavaFunction("javaRand",
> ... "test.org.apache.spark.sql.JavaRandUDF", DoubleType(), deterministic=False)
> >>> sqlContext.sql("SELECT javaRand(3)").collect() # doctest: +SKIP
> [Row(UDF:javaRand(3)=3.12345)]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org