You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Owen O'Malley (JIRA)" <ji...@apache.org> on 2006/04/25 18:34:17 UTC

[jira] Created: (HADOOP-165) long response times from task trackers under load

long response times from task trackers under load
-------------------------------------------------

         Key: HADOOP-165
         URL: http://issues.apache.org/jira/browse/HADOOP-165
     Project: Hadoop
        Type: Bug

  Components: mapred  
    Versions: 0.2    
 Environment: 190 redhat linux boxes, each with 2 hyperthreaded cpus, 2gb memory
    Reporter: Owen O'Malley
 Assigned to: Owen O'Malley 


we are seeing very slow response times from the task tracker. I put in some instrumentation to measure how long each call took for the RPC.Sever code to run the method (so it does not include serialization/deserialization time). The top of the list (in ms) looks like:

7581 progress node1192
7022 ping node1192
5393 ping node1162
4854 progress node1162
4749 progress node1194
4709 ping node1194
3813 ping node1100
3486 ping node1190
3266 progress node1190
3187 progress node1265
3078 ping node1203
2972 progress node1203
2947 progress node1240
2889 progress node1100
2875 ping node1116
2843 ping node1189
2772 ping node1183
2737 ping node1110
2727 progress node1183
2710 ping node1123
2563 ping node1304
2527 progress node1144
2479 ping node1137
2476 progress node1304
2430 ping node1240
2416 ping node1144
2377 progress node1176
2339 ping node1109
2321 progress node1114
2311 ping node1157
2185 ping node1265
2185 ping node1109
2172 ping node1114
2145 progress node1109
2127 ping node1176
2083 progress node1189
2076 ping node1229
2073 progress node1188
2072 progress node1123
2048 ping node1161
2003 progress node1110
1989 ping node1180
1963 ping node1114


-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


[jira] Updated: (HADOOP-165) long response times from task trackers under load

Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/HADOOP-165?page=all ]

Sameer Paranjpye updated HADOOP-165:
------------------------------------

    Fix Version: 0.3

> long response times from task trackers under load
> -------------------------------------------------
>
>          Key: HADOOP-165
>          URL: http://issues.apache.org/jira/browse/HADOOP-165
>      Project: Hadoop
>         Type: Bug

>   Components: mapred
>     Versions: 0.2
>  Environment: 190 redhat linux boxes, each with 2 hyperthreaded cpus, 2gb memory
>     Reporter: Owen O'Malley
>     Assignee: Owen O'Malley
>      Fix For: 0.3

>
> we are seeing very slow response times from the task tracker. I put in some instrumentation to measure how long each call took for the RPC.Sever code to run the method (so it does not include serialization/deserialization time). The top of the list (in ms) looks like:
> 7581 progress node1192
> 7022 ping node1192
> 5393 ping node1162
> 4854 progress node1162
> 4749 progress node1194
> 4709 ping node1194
> 3813 ping node1100
> 3486 ping node1190
> 3266 progress node1190
> 3187 progress node1265
> 3078 ping node1203
> 2972 progress node1203
> 2947 progress node1240
> 2889 progress node1100
> 2875 ping node1116
> 2843 ping node1189
> 2772 ping node1183
> 2737 ping node1110
> 2727 progress node1183
> 2710 ping node1123
> 2563 ping node1304
> 2527 progress node1144
> 2479 ping node1137
> 2476 progress node1304
> 2430 ping node1240
> 2416 ping node1144
> 2377 progress node1176
> 2339 ping node1109
> 2321 progress node1114
> 2311 ping node1157
> 2185 ping node1265
> 2185 ping node1109
> 2172 ping node1114
> 2145 progress node1109
> 2127 ping node1176
> 2083 progress node1189
> 2076 ping node1229
> 2073 progress node1188
> 2072 progress node1123
> 2048 ping node1161
> 2003 progress node1110
> 1989 ping node1180
> 1963 ping node1114

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


[jira] Commented: (HADOOP-165) long response times from task trackers under load

Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/HADOOP-165?page=comments#action_12376291 ] 

Doug Cutting commented on HADOOP-165:
-------------------------------------

Can you please try placing 'Server.LOG.setLevel(Level.FINE);' in the tasktracker and 'Client.LOG.setLevel(Level.FINE);' in the child.  That should provide some details about where the delay(s) are.


> long response times from task trackers under load
> -------------------------------------------------
>
>          Key: HADOOP-165
>          URL: http://issues.apache.org/jira/browse/HADOOP-165
>      Project: Hadoop
>         Type: Bug

>   Components: mapred
>     Versions: 0.2
>  Environment: 190 redhat linux boxes, each with 2 hyperthreaded cpus, 2gb memory
>     Reporter: Owen O'Malley
>     Assignee: Owen O'Malley

>
> we are seeing very slow response times from the task tracker. I put in some instrumentation to measure how long each call took for the RPC.Sever code to run the method (so it does not include serialization/deserialization time). The top of the list (in ms) looks like:
> 7581 progress node1192
> 7022 ping node1192
> 5393 ping node1162
> 4854 progress node1162
> 4749 progress node1194
> 4709 ping node1194
> 3813 ping node1100
> 3486 ping node1190
> 3266 progress node1190
> 3187 progress node1265
> 3078 ping node1203
> 2972 progress node1203
> 2947 progress node1240
> 2889 progress node1100
> 2875 ping node1116
> 2843 ping node1189
> 2772 ping node1183
> 2737 ping node1110
> 2727 progress node1183
> 2710 ping node1123
> 2563 ping node1304
> 2527 progress node1144
> 2479 ping node1137
> 2476 progress node1304
> 2430 ping node1240
> 2416 ping node1144
> 2377 progress node1176
> 2339 ping node1109
> 2321 progress node1114
> 2311 ping node1157
> 2185 ping node1265
> 2185 ping node1109
> 2172 ping node1114
> 2145 progress node1109
> 2127 ping node1176
> 2083 progress node1189
> 2076 ping node1229
> 2073 progress node1188
> 2072 progress node1123
> 2048 ping node1161
> 2003 progress node1110
> 1989 ping node1180
> 1963 ping node1114

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


[jira] Commented: (HADOOP-165) long response times from task trackers under load

Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/HADOOP-165?page=comments#action_12376295 ] 

Owen O'Malley commented on HADOOP-165:
--------------------------------------

I guess I should have made it clear that most of the times aren't this bad. This is the head of a long list. For example, the tail of the ping times look like (count, time):

      2 134
      1 123
      1 116
      1 108
      1 101
      1 79
      1 62
      1 56
      1 55
      1 37
      1 30
      1 15
      1 12
      2 4
     11 3
     90 2
    312 1
   9836 0

so by far most of the pings take 0 or 1 ms to execute.

> long response times from task trackers under load
> -------------------------------------------------
>
>          Key: HADOOP-165
>          URL: http://issues.apache.org/jira/browse/HADOOP-165
>      Project: Hadoop
>         Type: Bug

>   Components: mapred
>     Versions: 0.2
>  Environment: 190 redhat linux boxes, each with 2 hyperthreaded cpus, 2gb memory
>     Reporter: Owen O'Malley
>     Assignee: Owen O'Malley

>
> we are seeing very slow response times from the task tracker. I put in some instrumentation to measure how long each call took for the RPC.Sever code to run the method (so it does not include serialization/deserialization time). The top of the list (in ms) looks like:
> 7581 progress node1192
> 7022 ping node1192
> 5393 ping node1162
> 4854 progress node1162
> 4749 progress node1194
> 4709 ping node1194
> 3813 ping node1100
> 3486 ping node1190
> 3266 progress node1190
> 3187 progress node1265
> 3078 ping node1203
> 2972 progress node1203
> 2947 progress node1240
> 2889 progress node1100
> 2875 ping node1116
> 2843 ping node1189
> 2772 ping node1183
> 2737 ping node1110
> 2727 progress node1183
> 2710 ping node1123
> 2563 ping node1304
> 2527 progress node1144
> 2479 ping node1137
> 2476 progress node1304
> 2430 ping node1240
> 2416 ping node1144
> 2377 progress node1176
> 2339 ping node1109
> 2321 progress node1114
> 2311 ping node1157
> 2185 ping node1265
> 2185 ping node1109
> 2172 ping node1114
> 2145 progress node1109
> 2127 ping node1176
> 2083 progress node1189
> 2076 ping node1229
> 2073 progress node1188
> 2072 progress node1123
> 2048 ping node1161
> 2003 progress node1110
> 1989 ping node1180
> 1963 ping node1114

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


[jira] Resolved: (HADOOP-165) long response times from task trackers under load

Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley resolved HADOOP-165.
----------------------------------

    Resolution: Duplicate
      Assignee:     (was: Owen O'Malley)

This has been fixed primarily by moving the pulling part of the shuffle from the task tracker to the reduce's jvm. Previously, we saw task trackers burning 100% cpu and now it is much better.

> long response times from task trackers under load
> -------------------------------------------------
>
>                 Key: HADOOP-165
>                 URL: https://issues.apache.org/jira/browse/HADOOP-165
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.2.0
>         Environment: 190 redhat linux boxes, each with 2 hyperthreaded cpus, 2gb memory
>            Reporter: Owen O'Malley
>
> we are seeing very slow response times from the task tracker. I put in some instrumentation to measure how long each call took for the RPC.Sever code to run the method (so it does not include serialization/deserialization time). The top of the list (in ms) looks like:
> 7581 progress node1192
> 7022 ping node1192
> 5393 ping node1162
> 4854 progress node1162
> 4749 progress node1194
> 4709 ping node1194
> 3813 ping node1100
> 3486 ping node1190
> 3266 progress node1190
> 3187 progress node1265
> 3078 ping node1203
> 2972 progress node1203
> 2947 progress node1240
> 2889 progress node1100
> 2875 ping node1116
> 2843 ping node1189
> 2772 ping node1183
> 2737 ping node1110
> 2727 progress node1183
> 2710 ping node1123
> 2563 ping node1304
> 2527 progress node1144
> 2479 ping node1137
> 2476 progress node1304
> 2430 ping node1240
> 2416 ping node1144
> 2377 progress node1176
> 2339 ping node1109
> 2321 progress node1114
> 2311 ping node1157
> 2185 ping node1265
> 2185 ping node1109
> 2172 ping node1114
> 2145 progress node1109
> 2127 ping node1176
> 2083 progress node1189
> 2076 ping node1229
> 2073 progress node1188
> 2072 progress node1123
> 2048 ping node1161
> 2003 progress node1110
> 1989 ping node1180
> 1963 ping node1114

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-165) long response times from task trackers under load

Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/HADOOP-165?page=all ]

Sameer Paranjpye updated HADOOP-165:
------------------------------------

    Fix Version: 0.4
                     (was: 0.3)

> long response times from task trackers under load
> -------------------------------------------------
>
>          Key: HADOOP-165
>          URL: http://issues.apache.org/jira/browse/HADOOP-165
>      Project: Hadoop
>         Type: Bug

>   Components: mapred
>     Versions: 0.2
>  Environment: 190 redhat linux boxes, each with 2 hyperthreaded cpus, 2gb memory
>     Reporter: Owen O'Malley
>     Assignee: Owen O'Malley
>      Fix For: 0.4

>
> we are seeing very slow response times from the task tracker. I put in some instrumentation to measure how long each call took for the RPC.Sever code to run the method (so it does not include serialization/deserialization time). The top of the list (in ms) looks like:
> 7581 progress node1192
> 7022 ping node1192
> 5393 ping node1162
> 4854 progress node1162
> 4749 progress node1194
> 4709 ping node1194
> 3813 ping node1100
> 3486 ping node1190
> 3266 progress node1190
> 3187 progress node1265
> 3078 ping node1203
> 2972 progress node1203
> 2947 progress node1240
> 2889 progress node1100
> 2875 ping node1116
> 2843 ping node1189
> 2772 ping node1183
> 2737 ping node1110
> 2727 progress node1183
> 2710 ping node1123
> 2563 ping node1304
> 2527 progress node1144
> 2479 ping node1137
> 2476 progress node1304
> 2430 ping node1240
> 2416 ping node1144
> 2377 progress node1176
> 2339 ping node1109
> 2321 progress node1114
> 2311 ping node1157
> 2185 ping node1265
> 2185 ping node1109
> 2172 ping node1114
> 2145 progress node1109
> 2127 ping node1176
> 2083 progress node1189
> 2076 ping node1229
> 2073 progress node1188
> 2072 progress node1123
> 2048 ping node1161
> 2003 progress node1110
> 1989 ping node1180
> 1963 ping node1114

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


[jira] Updated: (HADOOP-165) long response times from task trackers under load

Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/HADOOP-165?page=all ]

Doug Cutting updated HADOOP-165:
--------------------------------

    Fix Version: 0.5.0
                     (was: 0.4.0)

> long response times from task trackers under load
> -------------------------------------------------
>
>          Key: HADOOP-165
>          URL: http://issues.apache.org/jira/browse/HADOOP-165
>      Project: Hadoop
>         Type: Bug

>   Components: mapred
>     Versions: 0.2.0
>  Environment: 190 redhat linux boxes, each with 2 hyperthreaded cpus, 2gb memory
>     Reporter: Owen O'Malley
>     Assignee: Owen O'Malley
>      Fix For: 0.5.0

>
> we are seeing very slow response times from the task tracker. I put in some instrumentation to measure how long each call took for the RPC.Sever code to run the method (so it does not include serialization/deserialization time). The top of the list (in ms) looks like:
> 7581 progress node1192
> 7022 ping node1192
> 5393 ping node1162
> 4854 progress node1162
> 4749 progress node1194
> 4709 ping node1194
> 3813 ping node1100
> 3486 ping node1190
> 3266 progress node1190
> 3187 progress node1265
> 3078 ping node1203
> 2972 progress node1203
> 2947 progress node1240
> 2889 progress node1100
> 2875 ping node1116
> 2843 ping node1189
> 2772 ping node1183
> 2737 ping node1110
> 2727 progress node1183
> 2710 ping node1123
> 2563 ping node1304
> 2527 progress node1144
> 2479 ping node1137
> 2476 progress node1304
> 2430 ping node1240
> 2416 ping node1144
> 2377 progress node1176
> 2339 ping node1109
> 2321 progress node1114
> 2311 ping node1157
> 2185 ping node1265
> 2185 ping node1109
> 2172 ping node1114
> 2145 progress node1109
> 2127 ping node1176
> 2083 progress node1189
> 2076 ping node1229
> 2073 progress node1188
> 2072 progress node1123
> 2048 ping node1161
> 2003 progress node1110
> 1989 ping node1180
> 1963 ping node1114

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira