[jira] [Created] (SPARK-43426) Introducing a new SparkListenerEvent for jobs running in AQE mode

["Faiz Halde (Jira)" <ji...@apache.org>]

Faiz Halde created SPARK-43426:
----------------------------------

             Summary: Introducing a new SparkListenerEvent for jobs running in AQE mode
                 Key: SPARK-43426
                 URL: https://issues.apache.org/jira/browse/SPARK-43426
             Project: Spark
          Issue Type: New Feature
          Components: Spark Core
    Affects Versions: 3.4.0
            Reporter: Faiz Halde


Prior to AQE, we used to rely on SparkListenerJobEnd to figure out when a job has been completed from Spark's scheduler point of view

Unfortunately with AQE a job is split into multiple jobs and there's no way for us to figure out when a job is completed

We have a listener that tracks some spark metrics. In order to establish the right relations among stages, jobs, and tasks we had to keep a mapping around. We used the SparkListenerJobEnd event to clear out these entries, however with AQE we cannot know when is the right time to remove these entries that we track

What suggestion would you have for this?

Would having a SparkListenerJobGroupEnd be a rationale addition?

Because our jobs have 1 single action, we probably can rely on stage metrics i.e if a stage completed that had some output metrics > 0, that's a useful indicator that the next Job End likely was the last job in that Job group



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org