You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "XiDuo You (Jira)" <ji...@apache.org> on 2021/10/18 12:13:00 UTC

[jira] [Updated] (SPARK-37043) Cancel all running job after AQE plan finished

     [ https://issues.apache.org/jira/browse/SPARK-37043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

XiDuo You updated SPARK-37043:
------------------------------
    Description: 
We see stage was still running after AQE plan finished. This is because the plan which contains a empty join has been converted to `LocalTableScanExec` during `AQEOptimizer`, but the other side of this join is still running (shuffle map stage).

 

It's no meaning to keep running the stage, It's better to cancel the running stage after AQE plan finished in case wasting the task resource.

  was:
We see stage was still running after AQE plan finished. This is because the plan which contains a empty join has been converted to `LocalTableScanExec` during `AQEOptimizer`, but the other side of this join is still running (shuffle map stage).

 

It's better to cancel the running stage after AQE plan finished in case wasting the task resource.


> Cancel all running job after AQE plan finished
> ----------------------------------------------
>
>                 Key: SPARK-37043
>                 URL: https://issues.apache.org/jira/browse/SPARK-37043
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 3.3.0
>            Reporter: XiDuo You
>            Priority: Major
>
> We see stage was still running after AQE plan finished. This is because the plan which contains a empty join has been converted to `LocalTableScanExec` during `AQEOptimizer`, but the other side of this join is still running (shuffle map stage).
>  
> It's no meaning to keep running the stage, It's better to cancel the running stage after AQE plan finished in case wasting the task resource.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org