You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Thejdeep Gudivada (Jira)" <ji...@apache.org> on 2023/01/26 21:31:00 UTC

[jira] [Commented] (SPARK-42203) JsonProtocol should skip logging of push-based shuffle read metrics when push-based shuffle is disabled

    [ https://issues.apache.org/jira/browse/SPARK-42203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17681126#comment-17681126 ] 

Thejdeep Gudivada commented on SPARK-42203:
-------------------------------------------

Thanks for filing this Josh. I will begin work on this soon.

> JsonProtocol should skip logging of push-based shuffle read metrics when push-based shuffle is disabled
> -------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-42203
>                 URL: https://issues.apache.org/jira/browse/SPARK-42203
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Shuffle
>    Affects Versions: 3.4.0
>            Reporter: Josh Rosen
>            Priority: Major
>
> This is a followup to SPARK-36620:
> When push-based shuffle is disabled (the default), I think that we should skip the logging of the new push-based shuffle read metrics. Because these metrics are logged for every task, they will add significant additional size to Spark event logs. It would be great to avoid this cost in cases where it's not necessary.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org