You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by kumar y <yk...@gmail.com> on 2013/11/15 02:45:10 UTC

org.apache.hadoop.mapred.TaskTracker: Caught exception: java.net.UnknownHostException: Invalid host name:

Hi,

we changed the jobtracker name recently in our production and populated the
same value in conf files across all our worker nodes.

couple of tasktrackers ( atleast from what we saw ) are started filling  up
with the below log messages. It works when  i manual do nslookup ,dig for
the jobtracker name.

My question is

1) if it resolved the jobtracker dns name to ip , will it cache that info
in jvm or will it retry everytime ?

2) I have read something on hadoop doing reverse lookup and might run it
these kind of issues if something is not right, any suggestions on what i
can look for ?

Any help is much appreciated. Thanks in advance !!!

 ERROR org.apache.hadoop.mapred.TaskTracker: Caught exception:
java.net.UnknownHostException: Invalid host name: local host is: (unknown);
destination host is: "jobtracker.xxx.xxxx":8021;
java.net.UnknownHostException; For more details see:
http://wiki.apache.org/hadoop/UnknownHost
    at sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown Source)
    at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:782)
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:741)
    at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:286)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1280)
    at org.apache.hadoop.ipc.Client.call(Client.java:1208)
    at
org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:225)
    at org.apache.hadoop.mapred.$Proxy9.getBuildVersion(Unknown Source)
    at
org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1956)
    at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:2872)
    at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:4119)
Caused by: java.net.UnknownHostException
    ... 8 more

Re: org.apache.hadoop.mapred.TaskTracker: Caught exception: java.net.UnknownHostException: Invalid host name:

Posted by Adam Kawa <ka...@gmail.com>.

As far as I remember (we might run into such a issue ~6 months ago), the
TaskTracker can cache the hostname of JobTracker. Try to restart a
TaskTrackers, to check if it connects correctly. Please let me know, if
restart of TT helped.


2013/11/15 kumar y <yk...@gmail.com>

>
> Hi,
>
> we changed the jobtracker name recently in our production and populated
> the same value in conf files across all our worker nodes.
>
> couple of tasktrackers ( atleast from what we saw ) are started filling
> up with the below log messages. It works when  i manual do nslookup ,dig
> for the jobtracker name.
>
> My question is
>
> 1) if it resolved the jobtracker dns name to ip , will it cache that info
> in jvm or will it retry everytime ?
>
> 2) I have read something on hadoop doing reverse lookup and might run it
> these kind of issues if something is not right, any suggestions on what i
> can look for ?
>
> Any help is much appreciated. Thanks in advance !!!
>
>  ERROR org.apache.hadoop.mapred.TaskTracker: Caught exception:
> java.net.UnknownHostException: Invalid host name: local host is: (unknown);
> destination host is: "jobtracker.xxx.xxxx":8021;
> java.net.UnknownHostException; For more details see:
> http://wiki.apache.org/hadoop/UnknownHost
>     at sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown
> Source)
>     at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>     at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:782)
>     at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:741)
>     at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:286)
>     at org.apache.hadoop.ipc.Client.getConnection(Client.java:1280)
>     at org.apache.hadoop.ipc.Client.call(Client.java:1208)
>     at
> org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:225)
>     at org.apache.hadoop.mapred.$Proxy9.getBuildVersion(Unknown Source)
>     at
> org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1956)
>     at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:2872)
>     at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:4119)
> Caused by: java.net.UnknownHostException
>     ... 8 more
>
>
>

Re: org.apache.hadoop.mapred.TaskTracker: Caught exception: java.net.UnknownHostException: Invalid host name:

Posted by Adam Kawa <ka...@gmail.com>.

As far as I remember (we might run into such a issue ~6 months ago), the
TaskTracker can cache the hostname of JobTracker. Try to restart a
TaskTrackers, to check if it connects correctly. Please let me know, if
restart of TT helped.


2013/11/15 kumar y <yk...@gmail.com>

>
> Hi,
>
> we changed the jobtracker name recently in our production and populated
> the same value in conf files across all our worker nodes.
>
> couple of tasktrackers ( atleast from what we saw ) are started filling
> up with the below log messages. It works when  i manual do nslookup ,dig
> for the jobtracker name.
>
> My question is
>
> 1) if it resolved the jobtracker dns name to ip , will it cache that info
> in jvm or will it retry everytime ?
>
> 2) I have read something on hadoop doing reverse lookup and might run it
> these kind of issues if something is not right, any suggestions on what i
> can look for ?
>
> Any help is much appreciated. Thanks in advance !!!
>
>  ERROR org.apache.hadoop.mapred.TaskTracker: Caught exception:
> java.net.UnknownHostException: Invalid host name: local host is: (unknown);
> destination host is: "jobtracker.xxx.xxxx":8021;
> java.net.UnknownHostException; For more details see:
> http://wiki.apache.org/hadoop/UnknownHost
>     at sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown
> Source)
>     at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>     at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:782)
>     at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:741)
>     at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:286)
>     at org.apache.hadoop.ipc.Client.getConnection(Client.java:1280)
>     at org.apache.hadoop.ipc.Client.call(Client.java:1208)
>     at
> org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:225)
>     at org.apache.hadoop.mapred.$Proxy9.getBuildVersion(Unknown Source)
>     at
> org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1956)
>     at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:2872)
>     at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:4119)
> Caused by: java.net.UnknownHostException
>     ... 8 more
>
>
>

Re: org.apache.hadoop.mapred.TaskTracker: Caught exception: java.net.UnknownHostException: Invalid host name:

Posted by Adam Kawa <ka...@gmail.com>.

As far as I remember (we might run into such a issue ~6 months ago), the
TaskTracker can cache the hostname of JobTracker. Try to restart a
TaskTrackers, to check if it connects correctly. Please let me know, if
restart of TT helped.


2013/11/15 kumar y <yk...@gmail.com>

>
> Hi,
>
> we changed the jobtracker name recently in our production and populated
> the same value in conf files across all our worker nodes.
>
> couple of tasktrackers ( atleast from what we saw ) are started filling
> up with the below log messages. It works when  i manual do nslookup ,dig
> for the jobtracker name.
>
> My question is
>
> 1) if it resolved the jobtracker dns name to ip , will it cache that info
> in jvm or will it retry everytime ?
>
> 2) I have read something on hadoop doing reverse lookup and might run it
> these kind of issues if something is not right, any suggestions on what i
> can look for ?
>
> Any help is much appreciated. Thanks in advance !!!
>
>  ERROR org.apache.hadoop.mapred.TaskTracker: Caught exception:
> java.net.UnknownHostException: Invalid host name: local host is: (unknown);
> destination host is: "jobtracker.xxx.xxxx":8021;
> java.net.UnknownHostException; For more details see:
> http://wiki.apache.org/hadoop/UnknownHost
>     at sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown
> Source)
>     at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>     at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:782)
>     at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:741)
>     at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:286)
>     at org.apache.hadoop.ipc.Client.getConnection(Client.java:1280)
>     at org.apache.hadoop.ipc.Client.call(Client.java:1208)
>     at
> org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:225)
>     at org.apache.hadoop.mapred.$Proxy9.getBuildVersion(Unknown Source)
>     at
> org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1956)
>     at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:2872)
>     at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:4119)
> Caused by: java.net.UnknownHostException
>     ... 8 more
>
>
>

Re: org.apache.hadoop.mapred.TaskTracker: Caught exception: java.net.UnknownHostException: Invalid host name:

Posted by Adam Kawa <ka...@gmail.com>.

As far as I remember (we might run into such a issue ~6 months ago), the
TaskTracker can cache the hostname of JobTracker. Try to restart a
TaskTrackers, to check if it connects correctly. Please let me know, if
restart of TT helped.


2013/11/15 kumar y <yk...@gmail.com>

>
> Hi,
>
> we changed the jobtracker name recently in our production and populated
> the same value in conf files across all our worker nodes.
>
> couple of tasktrackers ( atleast from what we saw ) are started filling
> up with the below log messages. It works when  i manual do nslookup ,dig
> for the jobtracker name.
>
> My question is
>
> 1) if it resolved the jobtracker dns name to ip , will it cache that info
> in jvm or will it retry everytime ?
>
> 2) I have read something on hadoop doing reverse lookup and might run it
> these kind of issues if something is not right, any suggestions on what i
> can look for ?
>
> Any help is much appreciated. Thanks in advance !!!
>
>  ERROR org.apache.hadoop.mapred.TaskTracker: Caught exception:
> java.net.UnknownHostException: Invalid host name: local host is: (unknown);
> destination host is: "jobtracker.xxx.xxxx":8021;
> java.net.UnknownHostException; For more details see:
> http://wiki.apache.org/hadoop/UnknownHost
>     at sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown
> Source)
>     at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>     at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:782)
>     at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:741)
>     at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:286)
>     at org.apache.hadoop.ipc.Client.getConnection(Client.java:1280)
>     at org.apache.hadoop.ipc.Client.call(Client.java:1208)
>     at
> org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:225)
>     at org.apache.hadoop.mapred.$Proxy9.getBuildVersion(Unknown Source)
>     at
> org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1956)
>     at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:2872)
>     at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:4119)
> Caused by: java.net.UnknownHostException
>     ... 8 more
>
>
>