You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Kay Ousterhout (JIRA)" <ji...@apache.org> on 2014/07/18 04:32:06 UTC

[jira] [Updated] (SPARK-2571) Shuffle read bytes are reported incorrectly for stages with multiple shuffle dependencies

     [ https://issues.apache.org/jira/browse/SPARK-2571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kay Ousterhout updated SPARK-2571:
----------------------------------

    Description: 
In BlockStoreShuffleFetcher, we set the shuffle metrics for a task to include information about data fetched from one BlockFetcherIterator.  When tasks have multiple shuffle dependencies (e.g., a stage that joins two datasets together), the metrics will get set based on data fetched from the last BlockFetcherIterator to complete, rather than the sum of all data fetched from all BlockFetcherIterators.  This can lead to dramatically underreporting the shuffle read bytes.

Thanks [~andrewor14] and [~rxin] for helping to diagnose this issue.

  was:In BlockStoreShuffleFetcher, we set the shuffle metrics for a task to include information about data fetched from one BlockFetcherIterator.  When tasks have multiple shuffle dependencies (e.g., a stage that joins two datasets together), the metrics will get set based on data fetched from the last BlockFetcherIterator to complete, rather than the sum of all data fetched from all BlockFetcherIterators.  This can lead to dramatically underreporting the shuffle read bytes.


> Shuffle read bytes are reported incorrectly for stages with multiple shuffle dependencies
> -----------------------------------------------------------------------------------------
>
>                 Key: SPARK-2571
>                 URL: https://issues.apache.org/jira/browse/SPARK-2571
>             Project: Spark
>          Issue Type: Bug
>          Components: Web UI
>    Affects Versions: 1.0.1, 0.9.3
>            Reporter: Kay Ousterhout
>            Assignee: Kay Ousterhout
>
> In BlockStoreShuffleFetcher, we set the shuffle metrics for a task to include information about data fetched from one BlockFetcherIterator.  When tasks have multiple shuffle dependencies (e.g., a stage that joins two datasets together), the metrics will get set based on data fetched from the last BlockFetcherIterator to complete, rather than the sum of all data fetched from all BlockFetcherIterators.  This can lead to dramatically underreporting the shuffle read bytes.
> Thanks [~andrewor14] and [~rxin] for helping to diagnose this issue.



--
This message was sent by Atlassian JIRA
(v6.2#6252)