You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Owen O'Malley (JIRA)" <ji...@apache.org> on 2007/05/24 08:12:16 UTC
[jira] Resolved: (HADOOP-165) long response times from task
trackers under load
[ https://issues.apache.org/jira/browse/HADOOP-165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley resolved HADOOP-165.
----------------------------------
Resolution: Duplicate
Assignee: (was: Owen O'Malley)
This has been fixed primarily by moving the pulling part of the shuffle from the task tracker to the reduce's jvm. Previously, we saw task trackers burning 100% cpu and now it is much better.
> long response times from task trackers under load
> -------------------------------------------------
>
> Key: HADOOP-165
> URL: https://issues.apache.org/jira/browse/HADOOP-165
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.2.0
> Environment: 190 redhat linux boxes, each with 2 hyperthreaded cpus, 2gb memory
> Reporter: Owen O'Malley
>
> we are seeing very slow response times from the task tracker. I put in some instrumentation to measure how long each call took for the RPC.Sever code to run the method (so it does not include serialization/deserialization time). The top of the list (in ms) looks like:
> 7581 progress node1192
> 7022 ping node1192
> 5393 ping node1162
> 4854 progress node1162
> 4749 progress node1194
> 4709 ping node1194
> 3813 ping node1100
> 3486 ping node1190
> 3266 progress node1190
> 3187 progress node1265
> 3078 ping node1203
> 2972 progress node1203
> 2947 progress node1240
> 2889 progress node1100
> 2875 ping node1116
> 2843 ping node1189
> 2772 ping node1183
> 2737 ping node1110
> 2727 progress node1183
> 2710 ping node1123
> 2563 ping node1304
> 2527 progress node1144
> 2479 ping node1137
> 2476 progress node1304
> 2430 ping node1240
> 2416 ping node1144
> 2377 progress node1176
> 2339 ping node1109
> 2321 progress node1114
> 2311 ping node1157
> 2185 ping node1265
> 2185 ping node1109
> 2172 ping node1114
> 2145 progress node1109
> 2127 ping node1176
> 2083 progress node1189
> 2076 ping node1229
> 2073 progress node1188
> 2072 progress node1123
> 2048 ping node1161
> 2003 progress node1110
> 1989 ping node1180
> 1963 ping node1114
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.