[jira] [Updated] (SPARK-50977) Enhance availability of logic performing aggregation of accumulator results

["Costas Zarifis (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-50977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Costas Zarifis updated SPARK-50977:
-----------------------------------
    Summary: Enhance availability of logic performing aggregation of accumulator results  (was: Enhance availability of API performing aggregation of accumulator results)

> Enhance availability of logic performing aggregation of accumulator results
> ---------------------------------------------------------------------------
>
>                 Key: SPARK-50977
>                 URL: https://issues.apache.org/jira/browse/SPARK-50977
>             Project: Spark
>          Issue Type: Task
>          Components: Spark Core
>    Affects Versions: 4.0.0
>            Reporter: Costas Zarifis
>            Priority: Major
>
> Currently the logic that performs aggregation of accumulator results among all tasks is present under the `org.apache.spark.sql.execution.metric` package. This package is not accessible from other modules such as the DAG Scheduler. This makes potential event queue and memory improvements that can be achieved by eagerly aggregating such metrics upstream (before they hit the taskEnd and stageCompletion event listeners) impossible.
>  
> This issue can be addressed by moving this logic to a common package that can be accessed both by the event listener classes and the DAG Scheduler.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org