You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Andreas Kostyrka <an...@kostyrka.org> on 2008/05/27 16:27:10 UTC

slow hosts in reduc

Hi!

Any idea how to cope with the "slow hosts" message that I seem to get
everytime during the reduce phase?

E.g.:

2008-05-27 14:21:04,310 INFO org.apache.hadoop.mapred.ReduceTask: task_200805270514_0006_r_000000_0 Need 8 map output(s)
2008-05-27 14:21:04,311 INFO org.apache.hadoop.mapred.ReduceTask: task_200805270514_0006_r_000000_0: Got 0 new map-outputs & 0 obsolete map-outputs from tasktracker and 0 map-outputs from previous failures

Now, it gets ugly:

2008-05-27 12:48:50,461 WARN org.apache.hadoop.mapred.ReduceTask: task_200805270514_0006_r_000000_0 copy failed: task_200805270514_0006_m_000000_0 from ec2-67-202-58-97.compute-1.amazonaws.com
2008-05-27 12:48:50,461 WARN org.apache.hadoop.mapred.ReduceTask: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
        at java.net.Socket.connect(Socket.java:519)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:152)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
        at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
        at sun.net.www.http.HttpClient.New(HttpClient.java:306)
        at sun.net.www.http.HttpClient.New(HttpClient.java:323)
        at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:788)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:729)
        at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:654)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:977)
        at org.apache.hadoop.mapred.MapOutputLocation.getFile(MapOutputLocation.java:207)
        at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:744)
        at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:696)

2008-05-27 12:48:50,461 INFO org.apache.hadoop.mapred.ReduceTask: Task task_200805270514_0006_r_000000_0: Failed fetch #2 from task_200805270514_0006_m_000000_0

What makes me wonder what specific service/port it could not reach on
ec2-67-202-58-97.compute-1.amazonaws.com?

Thanks for any hints,

Andreas

Re: slow hosts in reduc

Posted by Andreas Kostyrka <an...@kostyrka.org>.
Ok, just for the next guy scratching his head when googling, I haven't
found a solution, but an upgrade from hadoop 0.16.3 to 0.17.0 seems to
have fixed the problems.

Andreas


Am Dienstag, den 27.05.2008, 16:35 +0200 schrieb Andreas Kostyrka:
> To make it more painful, that's what I get from an nmap:
> 
> 
> ~# nmap -v -sT -p9000-9050,56000-57000 ec2-67-202-58-97.compute-1.amazonaws.com
> 
> Starting Nmap 4.53 ( http://insecure.org ) at 2008-05-27 14:30 UTC
> Initiating Ping Scan at 14:30
> Scanning 10.251.75.165 [2 ports]
> Completed Ping Scan at 14:30, 0.02s elapsed (1 total hosts)
> Initiating Connect Scan at 14:30
> Scanning ip-10-251-75-165.ec2.internal. (10.251.75.165) [1052 ports]
> Discovered open port 9000/tcp on 10.251.75.165
> Discovered open port 9001/tcp on 10.251.75.165
> Completed Connect Scan at 14:30, 1.11s elapsed (1052 total ports)
> Host ip-10-251-75-165.ec2.internal. (10.251.75.165) appears to be up ... good.
> Interesting ports on ip-10-251-75-165.ec2.internal. (10.251.75.165):
> Not shown: 1050 closed ports
> PORT     STATE SERVICE
> 9000/tcp open  unknown
> 9001/tcp open  unknown
> 
> That maps very nicely to the expected ports:
> 
>   <name>fs.default.name</name>
>   <value>hdfs://ec2-67-202-58-97.compute-1.amazonaws.com:9000</value>
>    <name>mapred.job.tracker</name>
>   <value>ec2-67-202-58-97.compute-1.amazonaws.com:9001</value>
> 
> So it really really makes me wonder on what port should I expect
> services that did not allow connection?
> 
> Andreas
> 
> Am Dienstag, den 27.05.2008, 16:27 +0200 schrieb Andreas Kostyrka:
> > ec2-67-202-58-97.compute-1.amazonaws.com

Re: slow hosts in reduc

Posted by Andreas Kostyrka <an...@kostyrka.org>.
To make it more painful, that's what I get from an nmap:


~# nmap -v -sT -p9000-9050,56000-57000 ec2-67-202-58-97.compute-1.amazonaws.com

Starting Nmap 4.53 ( http://insecure.org ) at 2008-05-27 14:30 UTC
Initiating Ping Scan at 14:30
Scanning 10.251.75.165 [2 ports]
Completed Ping Scan at 14:30, 0.02s elapsed (1 total hosts)
Initiating Connect Scan at 14:30
Scanning ip-10-251-75-165.ec2.internal. (10.251.75.165) [1052 ports]
Discovered open port 9000/tcp on 10.251.75.165
Discovered open port 9001/tcp on 10.251.75.165
Completed Connect Scan at 14:30, 1.11s elapsed (1052 total ports)
Host ip-10-251-75-165.ec2.internal. (10.251.75.165) appears to be up ... good.
Interesting ports on ip-10-251-75-165.ec2.internal. (10.251.75.165):
Not shown: 1050 closed ports
PORT     STATE SERVICE
9000/tcp open  unknown
9001/tcp open  unknown

That maps very nicely to the expected ports:

  <name>fs.default.name</name>
  <value>hdfs://ec2-67-202-58-97.compute-1.amazonaws.com:9000</value>
   <name>mapred.job.tracker</name>
  <value>ec2-67-202-58-97.compute-1.amazonaws.com:9001</value>

So it really really makes me wonder on what port should I expect
services that did not allow connection?

Andreas

Am Dienstag, den 27.05.2008, 16:27 +0200 schrieb Andreas Kostyrka:
> ec2-67-202-58-97.compute-1.amazonaws.com