[jira] [Resolved] (SPARK-37752) Python UDF fails when it should not get evaluated

["Hyukjin Kwon (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-37752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon resolved SPARK-37752.
----------------------------------
    Resolution: Not A Bug

> Python UDF fails when it should not get evaluated
> -------------------------------------------------
>
>                 Key: SPARK-37752
>                 URL: https://issues.apache.org/jira/browse/SPARK-37752
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.3.4
>            Reporter: Ohad Raviv
>            Priority: Minor
>
> Haven't checked on newer versions yet.
> If i define in Python:
> {code:java}
> def udf1(col1):
>     print(col1[2])
>     return "blah"
> spark.udf.register("udf1", udf1) {code}
> and then use it in SQL:
> {code:java}
> select case when length(c)>2 then udf1(c) end
> from (
>     select explode(array("123","234","12")) as c
> ) {code}
> it fails on:
> {noformat}
>   File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/worker.py", line 253, in main
>     process()
>   File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/worker.py", line 248, in process
>     serializer.dump_stream(func(split_index, iterator), outfile)
>   File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/worker.py", line 155, in <lambda>
>     func = lambda _, it: map(mapper, it)
>   File "<string>", line 1, in <lambda>
>   File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/worker.py", line 76, in <lambda>
>     return lambda *a: f(*a)
>   File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/util.py", line 55, in wrapper
>     return f(*args, **kwargs)
>   File "<stdin>", line 3, in udf1
> IndexError: string index out of range{noformat}
> Although in the out-of-range row it should not get evaluated at all as the case-when filters for lengths of more than 2 letters.
> the same scenario works great when we define instead a Scala UDF.
> will check now if it happens also for newer versions.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org