You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by kumar y <yk...@gmail.com> on 2013/11/15 02:45:10 UTC
org.apache.hadoop.mapred.TaskTracker: Caught exception:
java.net.UnknownHostException: Invalid host name:
Hi,
we changed the jobtracker name recently in our production and populated the
same value in conf files across all our worker nodes.
couple of tasktrackers ( atleast from what we saw ) are started filling up
with the below log messages. It works when i manual do nslookup ,dig for
the jobtracker name.
My question is
1) if it resolved the jobtracker dns name to ip , will it cache that info
in jvm or will it retry everytime ?
2) I have read something on hadoop doing reverse lookup and might run it
these kind of issues if something is not right, any suggestions on what i
can look for ?
Any help is much appreciated. Thanks in advance !!!
ERROR org.apache.hadoop.mapred.TaskTracker: Caught exception:
java.net.UnknownHostException: Invalid host name: local host is: (unknown);
destination host is: "jobtracker.xxx.xxxx":8021;
java.net.UnknownHostException; For more details see:
http://wiki.apache.org/hadoop/UnknownHost
at sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:782)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:741)
at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:286)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1280)
at org.apache.hadoop.ipc.Client.call(Client.java:1208)
at
org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:225)
at org.apache.hadoop.mapred.$Proxy9.getBuildVersion(Unknown Source)
at
org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1956)
at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:2872)
at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:4119)
Caused by: java.net.UnknownHostException
... 8 more
Re: org.apache.hadoop.mapred.TaskTracker: Caught exception:
java.net.UnknownHostException: Invalid host name:
Posted by Adam Kawa <ka...@gmail.com>.
As far as I remember (we might run into such a issue ~6 months ago), the
TaskTracker can cache the hostname of JobTracker. Try to restart a
TaskTrackers, to check if it connects correctly. Please let me know, if
restart of TT helped.
2013/11/15 kumar y <yk...@gmail.com>
>
> Hi,
>
> we changed the jobtracker name recently in our production and populated
> the same value in conf files across all our worker nodes.
>
> couple of tasktrackers ( atleast from what we saw ) are started filling
> up with the below log messages. It works when i manual do nslookup ,dig
> for the jobtracker name.
>
> My question is
>
> 1) if it resolved the jobtracker dns name to ip , will it cache that info
> in jvm or will it retry everytime ?
>
> 2) I have read something on hadoop doing reverse lookup and might run it
> these kind of issues if something is not right, any suggestions on what i
> can look for ?
>
> Any help is much appreciated. Thanks in advance !!!
>
> ERROR org.apache.hadoop.mapred.TaskTracker: Caught exception:
> java.net.UnknownHostException: Invalid host name: local host is: (unknown);
> destination host is: "jobtracker.xxx.xxxx":8021;
> java.net.UnknownHostException; For more details see:
> http://wiki.apache.org/hadoop/UnknownHost
> at sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:782)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:741)
> at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:286)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1280)
> at org.apache.hadoop.ipc.Client.call(Client.java:1208)
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:225)
> at org.apache.hadoop.mapred.$Proxy9.getBuildVersion(Unknown Source)
> at
> org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1956)
> at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:2872)
> at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:4119)
> Caused by: java.net.UnknownHostException
> ... 8 more
>
>
>
Re: org.apache.hadoop.mapred.TaskTracker: Caught exception:
java.net.UnknownHostException: Invalid host name:
Posted by Adam Kawa <ka...@gmail.com>.
As far as I remember (we might run into such a issue ~6 months ago), the
TaskTracker can cache the hostname of JobTracker. Try to restart a
TaskTrackers, to check if it connects correctly. Please let me know, if
restart of TT helped.
2013/11/15 kumar y <yk...@gmail.com>
>
> Hi,
>
> we changed the jobtracker name recently in our production and populated
> the same value in conf files across all our worker nodes.
>
> couple of tasktrackers ( atleast from what we saw ) are started filling
> up with the below log messages. It works when i manual do nslookup ,dig
> for the jobtracker name.
>
> My question is
>
> 1) if it resolved the jobtracker dns name to ip , will it cache that info
> in jvm or will it retry everytime ?
>
> 2) I have read something on hadoop doing reverse lookup and might run it
> these kind of issues if something is not right, any suggestions on what i
> can look for ?
>
> Any help is much appreciated. Thanks in advance !!!
>
> ERROR org.apache.hadoop.mapred.TaskTracker: Caught exception:
> java.net.UnknownHostException: Invalid host name: local host is: (unknown);
> destination host is: "jobtracker.xxx.xxxx":8021;
> java.net.UnknownHostException; For more details see:
> http://wiki.apache.org/hadoop/UnknownHost
> at sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:782)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:741)
> at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:286)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1280)
> at org.apache.hadoop.ipc.Client.call(Client.java:1208)
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:225)
> at org.apache.hadoop.mapred.$Proxy9.getBuildVersion(Unknown Source)
> at
> org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1956)
> at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:2872)
> at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:4119)
> Caused by: java.net.UnknownHostException
> ... 8 more
>
>
>
Re: org.apache.hadoop.mapred.TaskTracker: Caught exception:
java.net.UnknownHostException: Invalid host name:
Posted by Adam Kawa <ka...@gmail.com>.
As far as I remember (we might run into such a issue ~6 months ago), the
TaskTracker can cache the hostname of JobTracker. Try to restart a
TaskTrackers, to check if it connects correctly. Please let me know, if
restart of TT helped.
2013/11/15 kumar y <yk...@gmail.com>
>
> Hi,
>
> we changed the jobtracker name recently in our production and populated
> the same value in conf files across all our worker nodes.
>
> couple of tasktrackers ( atleast from what we saw ) are started filling
> up with the below log messages. It works when i manual do nslookup ,dig
> for the jobtracker name.
>
> My question is
>
> 1) if it resolved the jobtracker dns name to ip , will it cache that info
> in jvm or will it retry everytime ?
>
> 2) I have read something on hadoop doing reverse lookup and might run it
> these kind of issues if something is not right, any suggestions on what i
> can look for ?
>
> Any help is much appreciated. Thanks in advance !!!
>
> ERROR org.apache.hadoop.mapred.TaskTracker: Caught exception:
> java.net.UnknownHostException: Invalid host name: local host is: (unknown);
> destination host is: "jobtracker.xxx.xxxx":8021;
> java.net.UnknownHostException; For more details see:
> http://wiki.apache.org/hadoop/UnknownHost
> at sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:782)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:741)
> at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:286)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1280)
> at org.apache.hadoop.ipc.Client.call(Client.java:1208)
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:225)
> at org.apache.hadoop.mapred.$Proxy9.getBuildVersion(Unknown Source)
> at
> org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1956)
> at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:2872)
> at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:4119)
> Caused by: java.net.UnknownHostException
> ... 8 more
>
>
>
Re: org.apache.hadoop.mapred.TaskTracker: Caught exception:
java.net.UnknownHostException: Invalid host name:
Posted by Adam Kawa <ka...@gmail.com>.
As far as I remember (we might run into such a issue ~6 months ago), the
TaskTracker can cache the hostname of JobTracker. Try to restart a
TaskTrackers, to check if it connects correctly. Please let me know, if
restart of TT helped.
2013/11/15 kumar y <yk...@gmail.com>
>
> Hi,
>
> we changed the jobtracker name recently in our production and populated
> the same value in conf files across all our worker nodes.
>
> couple of tasktrackers ( atleast from what we saw ) are started filling
> up with the below log messages. It works when i manual do nslookup ,dig
> for the jobtracker name.
>
> My question is
>
> 1) if it resolved the jobtracker dns name to ip , will it cache that info
> in jvm or will it retry everytime ?
>
> 2) I have read something on hadoop doing reverse lookup and might run it
> these kind of issues if something is not right, any suggestions on what i
> can look for ?
>
> Any help is much appreciated. Thanks in advance !!!
>
> ERROR org.apache.hadoop.mapred.TaskTracker: Caught exception:
> java.net.UnknownHostException: Invalid host name: local host is: (unknown);
> destination host is: "jobtracker.xxx.xxxx":8021;
> java.net.UnknownHostException; For more details see:
> http://wiki.apache.org/hadoop/UnknownHost
> at sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:782)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:741)
> at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:286)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1280)
> at org.apache.hadoop.ipc.Client.call(Client.java:1208)
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:225)
> at org.apache.hadoop.mapred.$Proxy9.getBuildVersion(Unknown Source)
> at
> org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1956)
> at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:2872)
> at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:4119)
> Caused by: java.net.UnknownHostException
> ... 8 more
>
>
>