You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Li Li <fa...@gmail.com> on 2014/04/02 08:15:57 UTC
why copy from localhost.localdomain failed and throw UnknownHostException
the reducer hangs and the log is:
2014-04-02 14:03:57,235 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_201402271518_0260_r_000000_0 copy failed:
attempt_201402271518_0260_m_000001_1 from localhost.localdomain
2014-04-02 14:03:57,235 WARN org.apache.hadoop.mapred.ReduceTask:
java.net.UnknownHostException: localhost.localdomain
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:178)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
at java.net.Socket.connect(Socket.java:579)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:388)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:483)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:213)
at sun.net.www.http.HttpClient.New(HttpClient.java:300)
at sun.net.www.http.HttpClient.New(HttpClient.java:316)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:992)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:928)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:846)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStream(ReduceTask.java:1636)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.setupSecureConnection(ReduceTask.java:1593)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1493)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1401)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1333)
2014-04-02 14:03:57,236 INFO org.apache.hadoop.mapred.ReduceTask: Task
attempt_201402271518_0260_r_000000_0: Failed fetch #16 from
attempt_201402271518_0260_m_000001_1
2014-04-02 14:03:57,236 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_201402271518_0260_r_000000_0 adding host localhost.localdomain
to penalty box, next contact in 665 seconds
2014-04-02 14:03:57,236 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201402271518_0260_r_000000_0: Got 1 map-outputs from previous
failures
2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201402271518_0260_r_000000_0 Need another 1 map output(s)
where 0 is already in progress
2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201402271518_0260_r_000000_0 Scheduled 0 outputs (1 slow hosts
and0 dup hosts)
2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
Penalized(slow) Hosts:
2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
localhost.localdomain Will be considered after: 645 seconds.
2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201402271518_0260_r_000000_0 Need another 1 map output(s)
where 0 is already in progress
2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201402271518_0260_r_000000_0 Scheduled 0 outputs (1 slow hosts
and0 dup hosts)
2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
Penalized(slow) Hosts:
2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
localhost.localdomain Will be considered after: 585 seconds.
2014-04-02 14:06:17,294 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201402271518_0260_r_000000_0 Need another 1 map output(s)
where 0 is already in progress
2014-04-02 14:06:17,295 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201402271518_0260_r_000000_0 Scheduled 0 outputs (1 slow hosts
and0 dup hosts)
2014-04-02 14:06:17,295 INFO org.apache.hadoop.mapred.ReduceTask:
Penalized(slow) Hosts:
Re: why copy from localhost.localdomain failed and throw UnknownHostException
Posted by Li Li <fa...@gmail.com>.
/etc/hosts
#127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
should localhost.localdomain point to 127.0.0.1?
On Wed, Apr 2, 2014 at 3:55 PM, Gordon Wang <gw...@gopivotal.com> wrote:
> You may want to check the /etc/hosts. The configuration of localhost may not
> right.
> Make sure localhost.localdomain can be resolved.
>
>
> On Wed, Apr 2, 2014 at 2:15 PM, Li Li <fa...@gmail.com> wrote:
>>
>> the reducer hangs and the log is:
>>
>> 2014-04-02 14:03:57,235 WARN org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0 copy failed:
>> attempt_201402271518_0260_m_000001_1 from localhost.localdomain
>> 2014-04-02 14:03:57,235 WARN org.apache.hadoop.mapred.ReduceTask:
>> java.net.UnknownHostException: localhost.localdomain
>> at
>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:178)
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
>> at java.net.Socket.connect(Socket.java:579)
>> at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
>> at sun.net.www.http.HttpClient.openServer(HttpClient.java:388)
>> at sun.net.www.http.HttpClient.openServer(HttpClient.java:483)
>> at sun.net.www.http.HttpClient.<init>(HttpClient.java:213)
>> at sun.net.www.http.HttpClient.New(HttpClient.java:300)
>> at sun.net.www.http.HttpClient.New(HttpClient.java:316)
>> at
>> sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:992)
>> at
>> sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:928)
>> at
>> sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:846)
>> at
>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStream(ReduceTask.java:1636)
>> at
>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.setupSecureConnection(ReduceTask.java:1593)
>> at
>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1493)
>> at
>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1401)
>> at
>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1333)
>>
>> 2014-04-02 14:03:57,236 INFO org.apache.hadoop.mapred.ReduceTask: Task
>> attempt_201402271518_0260_r_000000_0: Failed fetch #16 from
>> attempt_201402271518_0260_m_000001_1
>> 2014-04-02 14:03:57,236 WARN org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0 adding host localhost.localdomain
>> to penalty box, next contact in 665 seconds
>> 2014-04-02 14:03:57,236 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0: Got 1 map-outputs from previous
>> failures
>> 2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0 Need another 1 map output(s)
>> where 0 is already in progress
>> 2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0 Scheduled 0 outputs (1 slow hosts
>> and0 dup hosts)
>> 2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
>> Penalized(slow) Hosts:
>> 2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
>> localhost.localdomain Will be considered after: 645 seconds.
>> 2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0 Need another 1 map output(s)
>> where 0 is already in progress
>> 2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0 Scheduled 0 outputs (1 slow hosts
>> and0 dup hosts)
>> 2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
>> Penalized(slow) Hosts:
>> 2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
>> localhost.localdomain Will be considered after: 585 seconds.
>> 2014-04-02 14:06:17,294 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0 Need another 1 map output(s)
>> where 0 is already in progress
>> 2014-04-02 14:06:17,295 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0 Scheduled 0 outputs (1 slow hosts
>> and0 dup hosts)
>> 2014-04-02 14:06:17,295 INFO org.apache.hadoop.mapred.ReduceTask:
>> Penalized(slow) Hosts:
>
>
>
>
> --
> Regards
> Gordon Wang
Re: why copy from localhost.localdomain failed and throw UnknownHostException
Posted by Li Li <fa...@gmail.com>.
/etc/hosts
#127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
should localhost.localdomain point to 127.0.0.1?
On Wed, Apr 2, 2014 at 3:55 PM, Gordon Wang <gw...@gopivotal.com> wrote:
> You may want to check the /etc/hosts. The configuration of localhost may not
> right.
> Make sure localhost.localdomain can be resolved.
>
>
> On Wed, Apr 2, 2014 at 2:15 PM, Li Li <fa...@gmail.com> wrote:
>>
>> the reducer hangs and the log is:
>>
>> 2014-04-02 14:03:57,235 WARN org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0 copy failed:
>> attempt_201402271518_0260_m_000001_1 from localhost.localdomain
>> 2014-04-02 14:03:57,235 WARN org.apache.hadoop.mapred.ReduceTask:
>> java.net.UnknownHostException: localhost.localdomain
>> at
>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:178)
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
>> at java.net.Socket.connect(Socket.java:579)
>> at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
>> at sun.net.www.http.HttpClient.openServer(HttpClient.java:388)
>> at sun.net.www.http.HttpClient.openServer(HttpClient.java:483)
>> at sun.net.www.http.HttpClient.<init>(HttpClient.java:213)
>> at sun.net.www.http.HttpClient.New(HttpClient.java:300)
>> at sun.net.www.http.HttpClient.New(HttpClient.java:316)
>> at
>> sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:992)
>> at
>> sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:928)
>> at
>> sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:846)
>> at
>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStream(ReduceTask.java:1636)
>> at
>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.setupSecureConnection(ReduceTask.java:1593)
>> at
>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1493)
>> at
>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1401)
>> at
>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1333)
>>
>> 2014-04-02 14:03:57,236 INFO org.apache.hadoop.mapred.ReduceTask: Task
>> attempt_201402271518_0260_r_000000_0: Failed fetch #16 from
>> attempt_201402271518_0260_m_000001_1
>> 2014-04-02 14:03:57,236 WARN org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0 adding host localhost.localdomain
>> to penalty box, next contact in 665 seconds
>> 2014-04-02 14:03:57,236 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0: Got 1 map-outputs from previous
>> failures
>> 2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0 Need another 1 map output(s)
>> where 0 is already in progress
>> 2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0 Scheduled 0 outputs (1 slow hosts
>> and0 dup hosts)
>> 2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
>> Penalized(slow) Hosts:
>> 2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
>> localhost.localdomain Will be considered after: 645 seconds.
>> 2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0 Need another 1 map output(s)
>> where 0 is already in progress
>> 2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0 Scheduled 0 outputs (1 slow hosts
>> and0 dup hosts)
>> 2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
>> Penalized(slow) Hosts:
>> 2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
>> localhost.localdomain Will be considered after: 585 seconds.
>> 2014-04-02 14:06:17,294 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0 Need another 1 map output(s)
>> where 0 is already in progress
>> 2014-04-02 14:06:17,295 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0 Scheduled 0 outputs (1 slow hosts
>> and0 dup hosts)
>> 2014-04-02 14:06:17,295 INFO org.apache.hadoop.mapred.ReduceTask:
>> Penalized(slow) Hosts:
>
>
>
>
> --
> Regards
> Gordon Wang
Re: why copy from localhost.localdomain failed and throw UnknownHostException
Posted by Li Li <fa...@gmail.com>.
/etc/hosts
#127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
should localhost.localdomain point to 127.0.0.1?
On Wed, Apr 2, 2014 at 3:55 PM, Gordon Wang <gw...@gopivotal.com> wrote:
> You may want to check the /etc/hosts. The configuration of localhost may not
> right.
> Make sure localhost.localdomain can be resolved.
>
>
> On Wed, Apr 2, 2014 at 2:15 PM, Li Li <fa...@gmail.com> wrote:
>>
>> the reducer hangs and the log is:
>>
>> 2014-04-02 14:03:57,235 WARN org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0 copy failed:
>> attempt_201402271518_0260_m_000001_1 from localhost.localdomain
>> 2014-04-02 14:03:57,235 WARN org.apache.hadoop.mapred.ReduceTask:
>> java.net.UnknownHostException: localhost.localdomain
>> at
>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:178)
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
>> at java.net.Socket.connect(Socket.java:579)
>> at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
>> at sun.net.www.http.HttpClient.openServer(HttpClient.java:388)
>> at sun.net.www.http.HttpClient.openServer(HttpClient.java:483)
>> at sun.net.www.http.HttpClient.<init>(HttpClient.java:213)
>> at sun.net.www.http.HttpClient.New(HttpClient.java:300)
>> at sun.net.www.http.HttpClient.New(HttpClient.java:316)
>> at
>> sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:992)
>> at
>> sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:928)
>> at
>> sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:846)
>> at
>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStream(ReduceTask.java:1636)
>> at
>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.setupSecureConnection(ReduceTask.java:1593)
>> at
>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1493)
>> at
>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1401)
>> at
>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1333)
>>
>> 2014-04-02 14:03:57,236 INFO org.apache.hadoop.mapred.ReduceTask: Task
>> attempt_201402271518_0260_r_000000_0: Failed fetch #16 from
>> attempt_201402271518_0260_m_000001_1
>> 2014-04-02 14:03:57,236 WARN org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0 adding host localhost.localdomain
>> to penalty box, next contact in 665 seconds
>> 2014-04-02 14:03:57,236 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0: Got 1 map-outputs from previous
>> failures
>> 2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0 Need another 1 map output(s)
>> where 0 is already in progress
>> 2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0 Scheduled 0 outputs (1 slow hosts
>> and0 dup hosts)
>> 2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
>> Penalized(slow) Hosts:
>> 2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
>> localhost.localdomain Will be considered after: 645 seconds.
>> 2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0 Need another 1 map output(s)
>> where 0 is already in progress
>> 2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0 Scheduled 0 outputs (1 slow hosts
>> and0 dup hosts)
>> 2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
>> Penalized(slow) Hosts:
>> 2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
>> localhost.localdomain Will be considered after: 585 seconds.
>> 2014-04-02 14:06:17,294 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0 Need another 1 map output(s)
>> where 0 is already in progress
>> 2014-04-02 14:06:17,295 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0 Scheduled 0 outputs (1 slow hosts
>> and0 dup hosts)
>> 2014-04-02 14:06:17,295 INFO org.apache.hadoop.mapred.ReduceTask:
>> Penalized(slow) Hosts:
>
>
>
>
> --
> Regards
> Gordon Wang
Re: why copy from localhost.localdomain failed and throw UnknownHostException
Posted by Li Li <fa...@gmail.com>.
/etc/hosts
#127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
should localhost.localdomain point to 127.0.0.1?
On Wed, Apr 2, 2014 at 3:55 PM, Gordon Wang <gw...@gopivotal.com> wrote:
> You may want to check the /etc/hosts. The configuration of localhost may not
> right.
> Make sure localhost.localdomain can be resolved.
>
>
> On Wed, Apr 2, 2014 at 2:15 PM, Li Li <fa...@gmail.com> wrote:
>>
>> the reducer hangs and the log is:
>>
>> 2014-04-02 14:03:57,235 WARN org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0 copy failed:
>> attempt_201402271518_0260_m_000001_1 from localhost.localdomain
>> 2014-04-02 14:03:57,235 WARN org.apache.hadoop.mapred.ReduceTask:
>> java.net.UnknownHostException: localhost.localdomain
>> at
>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:178)
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
>> at java.net.Socket.connect(Socket.java:579)
>> at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
>> at sun.net.www.http.HttpClient.openServer(HttpClient.java:388)
>> at sun.net.www.http.HttpClient.openServer(HttpClient.java:483)
>> at sun.net.www.http.HttpClient.<init>(HttpClient.java:213)
>> at sun.net.www.http.HttpClient.New(HttpClient.java:300)
>> at sun.net.www.http.HttpClient.New(HttpClient.java:316)
>> at
>> sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:992)
>> at
>> sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:928)
>> at
>> sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:846)
>> at
>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStream(ReduceTask.java:1636)
>> at
>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.setupSecureConnection(ReduceTask.java:1593)
>> at
>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1493)
>> at
>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1401)
>> at
>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1333)
>>
>> 2014-04-02 14:03:57,236 INFO org.apache.hadoop.mapred.ReduceTask: Task
>> attempt_201402271518_0260_r_000000_0: Failed fetch #16 from
>> attempt_201402271518_0260_m_000001_1
>> 2014-04-02 14:03:57,236 WARN org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0 adding host localhost.localdomain
>> to penalty box, next contact in 665 seconds
>> 2014-04-02 14:03:57,236 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0: Got 1 map-outputs from previous
>> failures
>> 2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0 Need another 1 map output(s)
>> where 0 is already in progress
>> 2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0 Scheduled 0 outputs (1 slow hosts
>> and0 dup hosts)
>> 2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
>> Penalized(slow) Hosts:
>> 2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
>> localhost.localdomain Will be considered after: 645 seconds.
>> 2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0 Need another 1 map output(s)
>> where 0 is already in progress
>> 2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0 Scheduled 0 outputs (1 slow hosts
>> and0 dup hosts)
>> 2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
>> Penalized(slow) Hosts:
>> 2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
>> localhost.localdomain Will be considered after: 585 seconds.
>> 2014-04-02 14:06:17,294 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0 Need another 1 map output(s)
>> where 0 is already in progress
>> 2014-04-02 14:06:17,295 INFO org.apache.hadoop.mapred.ReduceTask:
>> attempt_201402271518_0260_r_000000_0 Scheduled 0 outputs (1 slow hosts
>> and0 dup hosts)
>> 2014-04-02 14:06:17,295 INFO org.apache.hadoop.mapred.ReduceTask:
>> Penalized(slow) Hosts:
>
>
>
>
> --
> Regards
> Gordon Wang
Re: why copy from localhost.localdomain failed and throw UnknownHostException
Posted by Gordon Wang <gw...@gopivotal.com>.
You may want to check the /etc/hosts. The configuration of localhost may
not right.
Make sure localhost.localdomain can be resolved.
On Wed, Apr 2, 2014 at 2:15 PM, Li Li <fa...@gmail.com> wrote:
> the reducer hangs and the log is:
>
> 2014-04-02 14:03:57,235 WARN org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0 copy failed:
> attempt_201402271518_0260_m_000001_1 from localhost.localdomain
> 2014-04-02 14:03:57,235 WARN org.apache.hadoop.mapred.ReduceTask:
> java.net.UnknownHostException: localhost.localdomain
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:178)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
> at java.net.Socket.connect(Socket.java:579)
> at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:388)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:483)
> at sun.net.www.http.HttpClient.<init>(HttpClient.java:213)
> at sun.net.www.http.HttpClient.New(HttpClient.java:300)
> at sun.net.www.http.HttpClient.New(HttpClient.java:316)
> at
> sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:992)
> at
> sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:928)
> at
> sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:846)
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStream(ReduceTask.java:1636)
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.setupSecureConnection(ReduceTask.java:1593)
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1493)
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1401)
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1333)
>
> 2014-04-02 14:03:57,236 INFO org.apache.hadoop.mapred.ReduceTask: Task
> attempt_201402271518_0260_r_000000_0: Failed fetch #16 from
> attempt_201402271518_0260_m_000001_1
> 2014-04-02 14:03:57,236 WARN org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0 adding host localhost.localdomain
> to penalty box, next contact in 665 seconds
> 2014-04-02 14:03:57,236 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0: Got 1 map-outputs from previous
> failures
> 2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0 Need another 1 map output(s)
> where 0 is already in progress
> 2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0 Scheduled 0 outputs (1 slow hosts
> and0 dup hosts)
> 2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
> Penalized(slow) Hosts:
> 2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
> localhost.localdomain Will be considered after: 645 seconds.
> 2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0 Need another 1 map output(s)
> where 0 is already in progress
> 2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0 Scheduled 0 outputs (1 slow hosts
> and0 dup hosts)
> 2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
> Penalized(slow) Hosts:
> 2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
> localhost.localdomain Will be considered after: 585 seconds.
> 2014-04-02 14:06:17,294 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0 Need another 1 map output(s)
> where 0 is already in progress
> 2014-04-02 14:06:17,295 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0 Scheduled 0 outputs (1 slow hosts
> and0 dup hosts)
> 2014-04-02 14:06:17,295 INFO org.apache.hadoop.mapred.ReduceTask:
> Penalized(slow) Hosts:
>
--
Regards
Gordon Wang
Re: why copy from localhost.localdomain failed and throw UnknownHostException
Posted by Gordon Wang <gw...@gopivotal.com>.
You may want to check the /etc/hosts. The configuration of localhost may
not right.
Make sure localhost.localdomain can be resolved.
On Wed, Apr 2, 2014 at 2:15 PM, Li Li <fa...@gmail.com> wrote:
> the reducer hangs and the log is:
>
> 2014-04-02 14:03:57,235 WARN org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0 copy failed:
> attempt_201402271518_0260_m_000001_1 from localhost.localdomain
> 2014-04-02 14:03:57,235 WARN org.apache.hadoop.mapred.ReduceTask:
> java.net.UnknownHostException: localhost.localdomain
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:178)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
> at java.net.Socket.connect(Socket.java:579)
> at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:388)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:483)
> at sun.net.www.http.HttpClient.<init>(HttpClient.java:213)
> at sun.net.www.http.HttpClient.New(HttpClient.java:300)
> at sun.net.www.http.HttpClient.New(HttpClient.java:316)
> at
> sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:992)
> at
> sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:928)
> at
> sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:846)
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStream(ReduceTask.java:1636)
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.setupSecureConnection(ReduceTask.java:1593)
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1493)
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1401)
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1333)
>
> 2014-04-02 14:03:57,236 INFO org.apache.hadoop.mapred.ReduceTask: Task
> attempt_201402271518_0260_r_000000_0: Failed fetch #16 from
> attempt_201402271518_0260_m_000001_1
> 2014-04-02 14:03:57,236 WARN org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0 adding host localhost.localdomain
> to penalty box, next contact in 665 seconds
> 2014-04-02 14:03:57,236 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0: Got 1 map-outputs from previous
> failures
> 2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0 Need another 1 map output(s)
> where 0 is already in progress
> 2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0 Scheduled 0 outputs (1 slow hosts
> and0 dup hosts)
> 2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
> Penalized(slow) Hosts:
> 2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
> localhost.localdomain Will be considered after: 645 seconds.
> 2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0 Need another 1 map output(s)
> where 0 is already in progress
> 2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0 Scheduled 0 outputs (1 slow hosts
> and0 dup hosts)
> 2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
> Penalized(slow) Hosts:
> 2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
> localhost.localdomain Will be considered after: 585 seconds.
> 2014-04-02 14:06:17,294 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0 Need another 1 map output(s)
> where 0 is already in progress
> 2014-04-02 14:06:17,295 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0 Scheduled 0 outputs (1 slow hosts
> and0 dup hosts)
> 2014-04-02 14:06:17,295 INFO org.apache.hadoop.mapred.ReduceTask:
> Penalized(slow) Hosts:
>
--
Regards
Gordon Wang
Re: why copy from localhost.localdomain failed and throw UnknownHostException
Posted by Gordon Wang <gw...@gopivotal.com>.
You may want to check the /etc/hosts. The configuration of localhost may
not right.
Make sure localhost.localdomain can be resolved.
On Wed, Apr 2, 2014 at 2:15 PM, Li Li <fa...@gmail.com> wrote:
> the reducer hangs and the log is:
>
> 2014-04-02 14:03:57,235 WARN org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0 copy failed:
> attempt_201402271518_0260_m_000001_1 from localhost.localdomain
> 2014-04-02 14:03:57,235 WARN org.apache.hadoop.mapred.ReduceTask:
> java.net.UnknownHostException: localhost.localdomain
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:178)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
> at java.net.Socket.connect(Socket.java:579)
> at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:388)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:483)
> at sun.net.www.http.HttpClient.<init>(HttpClient.java:213)
> at sun.net.www.http.HttpClient.New(HttpClient.java:300)
> at sun.net.www.http.HttpClient.New(HttpClient.java:316)
> at
> sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:992)
> at
> sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:928)
> at
> sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:846)
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStream(ReduceTask.java:1636)
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.setupSecureConnection(ReduceTask.java:1593)
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1493)
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1401)
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1333)
>
> 2014-04-02 14:03:57,236 INFO org.apache.hadoop.mapred.ReduceTask: Task
> attempt_201402271518_0260_r_000000_0: Failed fetch #16 from
> attempt_201402271518_0260_m_000001_1
> 2014-04-02 14:03:57,236 WARN org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0 adding host localhost.localdomain
> to penalty box, next contact in 665 seconds
> 2014-04-02 14:03:57,236 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0: Got 1 map-outputs from previous
> failures
> 2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0 Need another 1 map output(s)
> where 0 is already in progress
> 2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0 Scheduled 0 outputs (1 slow hosts
> and0 dup hosts)
> 2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
> Penalized(slow) Hosts:
> 2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
> localhost.localdomain Will be considered after: 645 seconds.
> 2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0 Need another 1 map output(s)
> where 0 is already in progress
> 2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0 Scheduled 0 outputs (1 slow hosts
> and0 dup hosts)
> 2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
> Penalized(slow) Hosts:
> 2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
> localhost.localdomain Will be considered after: 585 seconds.
> 2014-04-02 14:06:17,294 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0 Need another 1 map output(s)
> where 0 is already in progress
> 2014-04-02 14:06:17,295 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0 Scheduled 0 outputs (1 slow hosts
> and0 dup hosts)
> 2014-04-02 14:06:17,295 INFO org.apache.hadoop.mapred.ReduceTask:
> Penalized(slow) Hosts:
>
--
Regards
Gordon Wang
Re: why copy from localhost.localdomain failed and throw UnknownHostException
Posted by Gordon Wang <gw...@gopivotal.com>.
You may want to check the /etc/hosts. The configuration of localhost may
not right.
Make sure localhost.localdomain can be resolved.
On Wed, Apr 2, 2014 at 2:15 PM, Li Li <fa...@gmail.com> wrote:
> the reducer hangs and the log is:
>
> 2014-04-02 14:03:57,235 WARN org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0 copy failed:
> attempt_201402271518_0260_m_000001_1 from localhost.localdomain
> 2014-04-02 14:03:57,235 WARN org.apache.hadoop.mapred.ReduceTask:
> java.net.UnknownHostException: localhost.localdomain
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:178)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
> at java.net.Socket.connect(Socket.java:579)
> at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:388)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:483)
> at sun.net.www.http.HttpClient.<init>(HttpClient.java:213)
> at sun.net.www.http.HttpClient.New(HttpClient.java:300)
> at sun.net.www.http.HttpClient.New(HttpClient.java:316)
> at
> sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:992)
> at
> sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:928)
> at
> sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:846)
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStream(ReduceTask.java:1636)
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.setupSecureConnection(ReduceTask.java:1593)
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1493)
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1401)
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1333)
>
> 2014-04-02 14:03:57,236 INFO org.apache.hadoop.mapred.ReduceTask: Task
> attempt_201402271518_0260_r_000000_0: Failed fetch #16 from
> attempt_201402271518_0260_m_000001_1
> 2014-04-02 14:03:57,236 WARN org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0 adding host localhost.localdomain
> to penalty box, next contact in 665 seconds
> 2014-04-02 14:03:57,236 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0: Got 1 map-outputs from previous
> failures
> 2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0 Need another 1 map output(s)
> where 0 is already in progress
> 2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0 Scheduled 0 outputs (1 slow hosts
> and0 dup hosts)
> 2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
> Penalized(slow) Hosts:
> 2014-04-02 14:04:17,244 INFO org.apache.hadoop.mapred.ReduceTask:
> localhost.localdomain Will be considered after: 645 seconds.
> 2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0 Need another 1 map output(s)
> where 0 is already in progress
> 2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0 Scheduled 0 outputs (1 slow hosts
> and0 dup hosts)
> 2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
> Penalized(slow) Hosts:
> 2014-04-02 14:05:17,269 INFO org.apache.hadoop.mapred.ReduceTask:
> localhost.localdomain Will be considered after: 585 seconds.
> 2014-04-02 14:06:17,294 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0 Need another 1 map output(s)
> where 0 is already in progress
> 2014-04-02 14:06:17,295 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201402271518_0260_r_000000_0 Scheduled 0 outputs (1 slow hosts
> and0 dup hosts)
> 2014-04-02 14:06:17,295 INFO org.apache.hadoop.mapred.ReduceTask:
> Penalized(slow) Hosts:
>
--
Regards
Gordon Wang