You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Josh Rosen (Jira)" <ji...@apache.org> on 2023/01/27 07:54:00 UTC

[jira] [Assigned] (SPARK-42206) Omit "Task Executor Metrics" field in JsonProtocol output if values are all zero

     [ https://issues.apache.org/jira/browse/SPARK-42206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Josh Rosen reassigned SPARK-42206:
----------------------------------

    Assignee: Josh Rosen

> Omit "Task Executor Metrics" field in JsonProtocol output if values are all zero
> --------------------------------------------------------------------------------
>
>                 Key: SPARK-42206
>                 URL: https://issues.apache.org/jira/browse/SPARK-42206
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 3.0.0
>            Reporter: Josh Rosen
>            Assignee: Josh Rosen
>            Priority: Major
>
> SPARK-26329 added "Task Executor Metrics" to JsonProtocol SparkListenerTaskEnd JSON. With the default `spark.executor.metrics.pollingInterval = 0` configuration these metric values are only updated when heartbeats occur. If a task launches and finishes between executor heartbeats then all of these metric values will be zero. For jobs with large numbers of short tasks, this contributes to significant event log bloat.
> JsonProtocol already knows how to handle the absence of the "Task Executor Metrics" field, so I think it's safe for us to omit this field when all values are zero.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org