You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Costas Zarifis (Jira)" <ji...@apache.org> on 2025/01/24 19:16:00 UTC

[jira] [Created] (SPARK-50977) Enhance availability of API performing aggregation of accumulator results

Costas Zarifis created SPARK-50977:
--------------------------------------

             Summary: Enhance availability of API performing aggregation of accumulator results
                 Key: SPARK-50977
                 URL: https://issues.apache.org/jira/browse/SPARK-50977
             Project: Spark
          Issue Type: Task
          Components: Spark Core
    Affects Versions: 4.0.0
            Reporter: Costas Zarifis


Currently the logic that performs aggregation of accumulator results among all tasks is present under the `org.apache.spark.sql.execution.metric` package. This package is not accessible from other modules such as the DAG Scheduler. This makes potential event queue and memory improvements that can be achieved by eagerly aggregating such metrics upstream (before they hit the taskEnd and stageCompletion event listeners) impossible.

 

This issue can be addressed by moving this logic to a common package that can be accessed both by the event listener classes and the DAG Scheduler.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org