[jira] [Commented] (SPARK-21334) Fix metrics for external shuffle service

["Saisai Shao (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-21334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079389#comment-16079389 ] 

Saisai Shao commented on SPARK-21334:
-------------------------------------

Are you using external shuffle service with YARN? If so I think currently it cannot be supported, we could improve the code to make it work (either through Hadoop metrics system or Spark's own one).

> Fix metrics for external shuffle service
> ----------------------------------------
>
>                 Key: SPARK-21334
>                 URL: https://issues.apache.org/jira/browse/SPARK-21334
>             Project: Spark
>          Issue Type: Bug
>          Components: Shuffle
>    Affects Versions: 2.1.1
>            Reporter: Raajay Viswanathan
>              Labels: external-shuffle-service
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> SPARK-16405 introduced metrics for external shuffle service. However, as it is currently there are two issues.
> 1. The shuffle service metrics system does not report values ever.
> 2. The current method for determining "blockTransferRate" is incorrect. The entire block is assumed to be transferred once the OpenBlocks message if processed. The actual data fetch from the disk and the succeeding transfer over the wire happens much later when MessageEncoder invokes encode on ChunkFetchSuccess message. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org