[jira] Updated: (HADOOP-4888) Use Apache HttpClient for fetching map outputs

["Chris Douglas (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-4888:
----------------------------------

    Attachment: 4888-1.patch

@Zheng: You're right, I shouldn't have said "degraded."

@Steve: Thanks for the ivy settings; I hadn't started to consider that, yet. The goal of this is identical to HADOOP-1338, really. Reimplementing the connection pooling in Hadoop could offer some advantages (e.g. more granular progress reporting), but appropriating all the work done in HttpClient seems like a clear win until that work is completed.

I tried a similar, still preliminary patch, but with max connections per host set to 1 and on a job with different parameters, i.e. mapred.reduce.slowstart.completed.maps=1.0, 38272 maps, 448 reducers, 32MB (generated) per map on ~300 nodes. Times measured are from the start of the reduce (after all maps have finished, so the stragglers are not a factor) to end of the shuffle (avg / std.d):

|| Version || 1 || 2 || 3 || 4 || 5 || avg || avg job ||
| r732838 | 786.89 / 45.55 | 842.596 / 70.69 | 1458.75 / 83.88 | 1140.93 / 44.22 | 1294.67 / 58.87 | 1104.77 | 2479.8 |
| r732838 + patch | 803.261 / 73.36 | 783.243 / 93.34 | 792.106 / 78.94 | 917.153 / 52.91 | 776.756 / 113.56 | 814.50 | 1955.2 |

Many of the parameters need to be adjusted. In particular, the timeouts are worth revisiting, as are the number of connections and threads at the server and client. Whether the HEAD + GET imposes a measurable penalty may also merit consideration before this can be committed. However, the preceding demonstrates that a measurable improvement is possible, and that this part of the pipeline could be mined for performance improvements.

> Use Apache HttpClient for fetching map outputs
> ----------------------------------------------
>
>                 Key: HADOOP-4888
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4888
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Chris Douglas
>            Assignee: Chris Douglas
>         Attachments: 4888-0.patch, 4888-1.patch
>
>
> It's worth experimenting with the [HttpClient|http://hc.apache.org/httpclient-3.x/] library to speed up the shuffle.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.