[jira] [Resolved] (SPARK-35787) Does anyone has performance issue after upgrade from 3.0 to 3.1?

["Hyukjin Kwon (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-35787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon resolved SPARK-35787.
----------------------------------
    Resolution: Invalid

> Does anyone has performance issue after upgrade from 3.0 to 3.1?
> ----------------------------------------------------------------
>
>                 Key: SPARK-35787
>                 URL: https://issues.apache.org/jira/browse/SPARK-35787
>             Project: Spark
>          Issue Type: Question
>          Components: Spark Core
>    Affects Versions: 3.1.2
>            Reporter: Vidmantas Drasutis
>            Priority: Major
>         Attachments: Execution_plan_difference.png, spark_3.0_execution_plan_details_fast.txt, spark_3.1_execution_plan_details_slow.txt, spark_job_info_1.png, spark_job_info_2.png
>
>
> Hello.
>  
> We had using spark 3.0.2 and query was executed in ~100 seconds.
> After we upgraded Spark to 3.1.1 (tried also 3.1.2 - same, slow performance) - our query execution time started taking ~260 seconds it is huge increase 250-300 % of execution time increase.
>  
> We tried quite simple query.
> In query we using UDF (*org.apache.spark.sql.functions*)
> ) - which explodes data and do polygon hit test. Nothing changed in our code from query perspective.
>  It is 1 VM box cluster
>  
> Maybe anyone faced similar issue?
> Attached some details from spark dashboard.
>  
> *Looks like it is UDF related slowdown. As queries which does not use UDF`s performance is same and which uses UDFs - starting from 3.1 performance decreased.*
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org