You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Maximilian Sackel (Jira)" <ji...@apache.org> on 2022/04/21 09:50:00 UTC

[jira] [Created] (SPARK-38981) Unexpected commutative property of udf/pandas_udf

Maximilian Sackel created SPARK-38981:
-----------------------------------------

             Summary: Unexpected commutative property of udf/pandas_udf
                 Key: SPARK-38981
                 URL: https://issues.apache.org/jira/browse/SPARK-38981
             Project: Spark
          Issue Type: Bug
          Components: Optimizer
    Affects Versions: 3.2.1
            Reporter: Maximilian Sackel
         Attachments: screenshot-1.png

Hello all,
When running the attached minmal working example, the order of the filter and the UDF is swapped by the optimizer. This can lead to errors, which are difficult to debug. In the documentation I have found no reference to such behavior. 

Is this a bug or a functionality which is poorly documented?

With kind regards,
Max

!image-2022-04-21-11-47-36-931.png!

!image-2022-04-21-11-48-17-966.png!



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org