You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2017/01/10 20:57:58 UTC
[jira] [Commented] (SPARK-19160) Decorator for UDF creation.
[ https://issues.apache.org/jira/browse/SPARK-19160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15816163#comment-15816163 ]
Apache Spark commented on SPARK-19160:
--------------------------------------
User 'zero323' has created a pull request for this issue:
https://github.com/apache/spark/pull/16533
> Decorator for UDF creation.
> ---------------------------
>
> Key: SPARK-19160
> URL: https://issues.apache.org/jira/browse/SPARK-19160
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark, SQL
> Affects Versions: 1.6.0, 2.0.0, 2.1.0, 2.2.0
> Reporter: Maciej Szymkiewicz
>
> Right now there are a few ways we can create UDF:
> - With standalone function:
> {code}
> def _add_one(x):
> """Adds one"""
> if x is not None:
> return x + 1
> add_one = udf(_add_one, IntegerType())
> {code}
> This allows for full control flow, including exception handling, but duplicates variables.
>
> - With `lambda` expression:
> {code}
> add_one = udf(lambda x: x + 1 if x is not None else None, IntegerType())
> {code}
> No variable duplication but only pure expressions.
> - Using nested functions with immediate call:
> {code}
> def add_one(c):
> def add_one_(x):
> if x is not None:
> return x + 1
> return udf(add_one_, IntegerType())(c)
> {code}
> Quite verbose but enables full control flow and clearly indicates expected number of arguments.
>
> - Using `udf` functions as a decorator:
> {code}
> @udf
> def add_one(x):
> """Adds one"""
> if x is not None:
> return x + 1
> {code}
> Possible but only with default `returnType` (or curried `@partial(udf, returnType=IntegerType())`).
>
> Proposed
> Add `udf` decorator which can be used as follows:
> {code}
> from pyspark.sql.decorators import udf
> @udf(IntegerType())
> def add_one(x):
> """Adds one"""
> if x is not None:
> return x + 1
> {code}
> or
> {code}
> @udf()
> def strip(x):
> """Strips String"""
> if x is not None:
> return x.strip()
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org