You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by kayousterhout <gi...@git.apache.org> on 2014/03/03 04:44:14 UTC

[GitHub] spark pull request: Remove the remoteFetchTime metric.

GitHub user kayousterhout opened a pull request:

    https://github.com/apache/spark/pull/62

    Remove the remoteFetchTime metric.

    This metric is confusing: it adds up all of the time to fetch
    shuffle inputs, but fetches often happen in parallel, so
    remoteFetchTime can be much longer than the task execution time.
    
    @squito it looks like you added this metric -- do you have a use case for it?
    
    cc @shivaram -- I know you've looked at the shuffle performance a lot so chime in here if this metric has turned out to be useful for you!

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/kayousterhout/spark-1 remove_fetch_variable

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/62.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #62
    
----
commit 43341ebed07971ccfa5ff0295cb4f6c5ab025403
Author: Kay Ousterhout <ka...@gmail.com>
Date:   2014-03-03T03:37:55Z

    Remote the remoteFetchTime metric.
    
    This metric is confusing: it adds up all of the time to fetch
    shuffle inputs, but fetches often happen in parallel, so
    remoteFetchTime can be much longer than the task execution time.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Remove the remoteFetchTime metric.

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/62


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Remove the remoteFetchTime metric.

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/62#issuecomment-36480712
  
    Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Remove the remoteFetchTime metric.

Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on the pull request:

    https://github.com/apache/spark/pull/62#issuecomment-36481963
  
    Hmm -- I have been confused by this before, but if I am reading the comment right, this could be useful for to get an estimate of the raw network bandwidth used for shuffle ? If not could we have an explicit metric for that ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Remove the remoteFetchTime metric.

Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on the pull request:

    https://github.com/apache/spark/pull/62#issuecomment-36483177
  
    Okay -- Thats seems like a separate conversation. This change looks good to me.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Remove the remoteFetchTime metric.

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/62#issuecomment-36482478
  
    All automated tests passed.
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12959/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Remove the remoteFetchTime metric.

Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:

    https://github.com/apache/spark/pull/62#issuecomment-36575370
  
    LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Remove the remoteFetchTime metric.

Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the pull request:

    https://github.com/apache/spark/pull/62#issuecomment-36577709
  
    I've merged this into master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Remove the remoteFetchTime metric.

Posted by squito <gi...@git.apache.org>.
Github user squito commented on the pull request:

    https://github.com/apache/spark/pull/62#issuecomment-36589314
  
    I think I just threw a bunch of different metrics together, I don't think this one was ever that useful.  It would be nice to have something to measure bandwidth, but I guess this isn't really useful.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Remove the remoteFetchTime metric.

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/62#issuecomment-36480711
  
     Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Remove the remoteFetchTime metric.

Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the pull request:

    https://github.com/apache/spark/pull/62#issuecomment-36483058
  
    Unfortunately this isn't very useful for getting network bandwidth...if you
    consider a simple case where two shuffle reads (for one task) occur
    simultaneously and both take t time, the metric will report a time of 2t.
     The metric would also report 2t if two shuffles happened one after
    another, so it's hard to extract raw network bandwidth from the metric.  I
    thought about adding a metric that just records the time spent doing
    network stuff (that doesn't double count when things are overlapping) to
    get bandwidth but @pwendell said this existed before and was also
    misleading because if many tasks are fetching shuffle data at the same
    time, they will each individually have low bandwidth, even if the aggregate
    bandwidth is good.
    
    
    On Sun, Mar 2, 2014 at 9:02 PM, Shivaram Venkataraman <
    notifications@github.com> wrote:
    
    > Hmm -- I have been confused by this before, but if I am reading the
    > comment right, this could be useful for to get an estimate of the raw
    > network bandwidth used for shuffle ? If not could we have an explicit
    > metric for that ?
    >
    > --
    > Reply to this email directly or view it on GitHub<https://github.com/apache/spark/pull/62#issuecomment-36481963>
    > .
    >


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: Remove the remoteFetchTime metric.

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/62#issuecomment-36482477
  
    Merged build finished.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---