You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Deependra Patel (Jira)" <ji...@apache.org> on 2023/07/19 09:22:00 UTC
[jira] [Commented] (SPARK-44209) Expose amount of shuffle data available on the node
[ https://issues.apache.org/jira/browse/SPARK-44209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17744500#comment-17744500 ]
Deependra Patel commented on SPARK-44209:
-----------------------------------------
Created PR - [https://github.com/apache/spark/pull/42071]
> Expose amount of shuffle data available on the node
> ---------------------------------------------------
>
> Key: SPARK-44209
> URL: https://issues.apache.org/jira/browse/SPARK-44209
> Project: Spark
> Issue Type: New Feature
> Components: Shuffle
> Affects Versions: 3.4.1
> Reporter: Deependra Patel
> Priority: Trivial
>
> [ShuffleMetrics|https://github.com/apache/spark/blob/43f7a86a05ad8c7ec7060607e43d9ca4d0fe4166/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalBlockHandler.java#L318] doesn't have metrics like
> "totalShuffleDataBytes" and "numAppsWithShuffleData", these metrics are per node published by External Shuffle Service.
>
> Adding these metrics would help in -
> 1. Deciding if we can decommission the node if no shuffle data present
> 2. Better live monitoring of customer's workload to see if there is skewed shuffle data present on the node
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org