You are viewing a plain text version of this content. The canonical link for it is here.

Posted to mapreduce-dev@hadoop.apache.org by "Liyin Liang (JIRA)" <ji...@apache.org> on 2013/12/19 08:17:06 UTC

[jira] [Created] (MAPREDUCE-5691) Throttle shuffle's bandwidth utilization

Liyin Liang created MAPREDUCE-5691:
--------------------------------------

             Summary: Throttle shuffle's bandwidth utilization
                 Key: MAPREDUCE-5691
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5691
             Project: Hadoop Map/Reduce
          Issue Type: Improvement
    Affects Versions: 2.2.0
            Reporter: Liyin Liang


In our hadoop cluster, a reducer of a big job can utilize all the bandwidth during shuffle phase. Then any task reading data from  the machine which running that reducer becomes very very slow.
It's better to move DataTransferThrottler from hadoop-hdfs to hadoop-common. And create a throttler for Shuffle to throttle each Fetcher.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)