[jira] [Commented] (SPARK-38981) Unexpected commutative property of udf/pandas_udf and filters

["Kshitij Singh (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-38981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17527208#comment-17527208 ] 

Kshitij Singh commented on SPARK-38981:
---------------------------------------

This is my first issue. Let me take a look at it [~msackel] and get back. 

> Unexpected commutative property of udf/pandas_udf and filters
> -------------------------------------------------------------
>
>                 Key: SPARK-38981
>                 URL: https://issues.apache.org/jira/browse/SPARK-38981
>             Project: Spark
>          Issue Type: Bug
>          Components: Optimizer
>    Affects Versions: 3.2.1
>            Reporter: Maximilian Sackel
>            Priority: Critical
>              Labels: beginner
>         Attachments: optimization_udf_filter.html, screenshot-1.png, screenshot-2.png
>
>
> Hello all,
> When running the attached minmal working example in the attachments, the order of the filter and the UDF is swapped by the optimizer. This can lead to errors, which are difficult to debug. In the documentation I have found no reference to such behavior. 
> Is this a bug or a functionality which is poorly documented?
> With kind regards,
> Max



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org