You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Raajay Viswanathan (JIRA)" <ji...@apache.org> on 2017/07/07 03:36:00 UTC

[jira] [Created] (SPARK-21334) Fix metrics for external shuffle service

Raajay Viswanathan created SPARK-21334:
------------------------------------------

             Summary: Fix metrics for external shuffle service
                 Key: SPARK-21334
                 URL: https://issues.apache.org/jira/browse/SPARK-21334
             Project: Spark
          Issue Type: Bug
          Components: Shuffle
    Affects Versions: 2.1.1
            Reporter: Raajay Viswanathan


SPARK-16405 introduced metrics for external shuffle service. However, as it is currently there are two issues.

1. The shuffle service metrics system does not report values ever.
2. The current method for determining "blockTransferRate" is incorrect. The entire block is assumed to be transferred once the OpenBlocks message if processed. The actual data fetch from the disk and the succeeding transfer over the wire happens much later when MessageEncoder invokes encode on ChunkFetchSuccess message. 




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org