You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Costas Zarifis (Jira)" <ji...@apache.org> on 2025/01/24 19:16:00 UTC
[jira] [Created] (SPARK-50977) Enhance availability of API performing aggregation of accumulator results
Costas Zarifis created SPARK-50977:
--------------------------------------
Summary: Enhance availability of API performing aggregation of accumulator results
Key: SPARK-50977
URL: https://issues.apache.org/jira/browse/SPARK-50977
Project: Spark
Issue Type: Task
Components: Spark Core
Affects Versions: 4.0.0
Reporter: Costas Zarifis
Currently the logic that performs aggregation of accumulator results among all tasks is present under the `org.apache.spark.sql.execution.metric` package. This package is not accessible from other modules such as the DAG Scheduler. This makes potential event queue and memory improvements that can be achieved by eagerly aggregating such metrics upstream (before they hit the taskEnd and stageCompletion event listeners) impossible.
This issue can be addressed by moving this logic to a common package that can be accessed both by the event listener classes and the DAG Scheduler.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org