You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Runping Qi (JIRA)" <ji...@apache.org> on 2008/06/27 19:15:44 UTC
[jira] Created: (HADOOP-3656) fetcher should re-use connection when
it needs to fetch multiple segments from the same task tracker
fetcher should re-use connection when it needs to fetch multiple segments from the same task tracker
----------------------------------------------------------------------------------------------------
Key: HADOOP-3656
URL: https://issues.apache.org/jira/browse/HADOOP-3656
Project: Hadoop Core
Issue Type: Bug
Reporter: Runping Qi
In the current implementation, the fetcher will fetch one segment a time from a task tracker.
In the case where a job has N mappers per tracker, each reducer will need N trips to each tracker.
That will generate a lot of network traffic when N and the number of reduces is large.
The problem will be improved if the fetcher can retrieve multiple segments from a tracker per connection, either through http keep alive or
through application level protocol.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-3656) fetcher should re-use connection
when it needs to fetch multiple segments from the same task tracker
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley resolved HADOOP-3656.
-----------------------------------
Resolution: Duplicate
This is a duplicate of HADOOP-1338.
> fetcher should re-use connection when it needs to fetch multiple segments from the same task tracker
> ----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3656
> URL: https://issues.apache.org/jira/browse/HADOOP-3656
> Project: Hadoop Core
> Issue Type: Bug
> Reporter: Runping Qi
>
> In the current implementation, the fetcher will fetch one segment a time from a task tracker.
> In the case where a job has N mappers per tracker, each reducer will need N trips to each tracker.
> That will generate a lot of network traffic when N and the number of reduces is large.
> The problem will be improved if the fetcher can retrieve multiple segments from a tracker per connection, either through http keep alive or
> through application level protocol.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.