You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Li Li <fa...@gmail.com> on 2014/06/11 05:12:32 UTC

one node can't start tasktracker

I have a namenode/jobtracker, 5 datanode/tasktracker. hadoop version is 1.2.1
I use stop-mapred.sh and start-mapred.sh. But there is one node always fails.
the error log is:

2014-06-11 10:46:57,475 INFO org.apache.hadoop.http.HttpServer: Port
returned by webServer.getConnectors()[0].getLocalPort() before open()
is -1. Opening the listener on 50060
2014-06-11 10:46:57,477 ERROR org.apache.hadoop.mapred.TaskTracker:
Can not start task tracker because java.net.BindException: Address
already in use
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:444)
        at sun.nio.ch.Net.bind(Net.java:436)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
        at org.apache.hadoop.http.HttpServer.start(HttpServer.java:602)
        at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1616)
        at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3937)

2014-06-11 10:46:57,479 INFO org.apache.hadoop.mapred.TaskTracker: SHUTDOWN_MSG:

the other correct nodes' log:
2014-06-11 10:44:32,767 INFO org.apache.hadoop.http.HttpServer: Port
returned by webServer.getConnectors()[0].getLocalPort() before open()
is -1. Opening the listener on 50060
2014-06-11 10:44:32,768 INFO org.apache.hadoop.http.HttpServer:
listener.getLocalPort() returned 50060
webServer.getConnectors()[0].getLocalPort() returned 50060
2014-06-11 10:44:32,768 INFO org.apache.hadoop.http.HttpServer: Jetty
bound to port 50060
2014-06-11 10:44:32,768 INFO org.mortbay.log: jetty-6.1.26

it seems the reason is that port 50060 is used by other process.
But I use sudo netstat and also lsof. no process is using 50060. What's wrong?

Re: one node can't start tasktracker

Posted by Raj K Singh <ra...@gmail.com>.
​the problem seems with the jetty server port binding, as per the posted
log​

2014-06-11 10:46:57,475 INFO org.apache.hadoop.http.HttpServer: Port
returned by webServer.getConnectors()[0].getLocalPort() before open()
is -1. Opening the listener on 50060

before opening a connection the port which was retrieved -1
(webServer.getConnectors()[0].getLocalPort()) and oriPort (The original
requested port) is 50060

you should look into the log containing the string "listener.getLocalPort()
returned " which will give you idea about the port no which is bound with
some process on the machine.



::::::::::::::::::::::::::::::::::::::::
Raj K Singh
http://in.linkedin.com/in/rajkrrsingh
http://www.rajkrrsingh.blogspot.com
Mobile  Tel: +91 (0)9899821370


On Wed, Jun 11, 2014 at 8:42 AM, Li Li <fa...@gmail.com> wrote:

> I have a namenode/jobtracker, 5 datanode/tasktracker. hadoop version is
> 1.2.1
> I use stop-mapred.sh and start-mapred.sh. But there is one node always
> fails.
> the error log is:
>
> 2014-06-11 10:46:57,475 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open()
> is -1. Opening the listener on 50060
> 2014-06-11 10:46:57,477 ERROR org.apache.hadoop.mapred.TaskTracker:
> Can not start task tracker because java.net.BindException: Address
> already in use
>         at sun.nio.ch.Net.bind0(Native Method)
>         at sun.nio.ch.Net.bind(Net.java:444)
>         at sun.nio.ch.Net.bind(Net.java:436)
>         at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
>         at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
>         at
> org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
>         at org.apache.hadoop.http.HttpServer.start(HttpServer.java:602)
>         at
> org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1616)
>         at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3937)
>
> 2014-06-11 10:46:57,479 INFO org.apache.hadoop.mapred.TaskTracker:
> SHUTDOWN_MSG:
>
> the other correct nodes' log:
> 2014-06-11 10:44:32,767 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open()
> is -1. Opening the listener on 50060
> 2014-06-11 10:44:32,768 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50060
> webServer.getConnectors()[0].getLocalPort() returned 50060
> 2014-06-11 10:44:32,768 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 50060
> 2014-06-11 10:44:32,768 INFO org.mortbay.log: jetty-6.1.26
>
> it seems the reason is that port 50060 is used by other process.
> But I use sudo netstat and also lsof. no process is using 50060. What's
> wrong?
>

Re: one node can't start tasktracker

Posted by Raj K Singh <ra...@gmail.com>.
​the problem seems with the jetty server port binding, as per the posted
log​

2014-06-11 10:46:57,475 INFO org.apache.hadoop.http.HttpServer: Port
returned by webServer.getConnectors()[0].getLocalPort() before open()
is -1. Opening the listener on 50060

before opening a connection the port which was retrieved -1
(webServer.getConnectors()[0].getLocalPort()) and oriPort (The original
requested port) is 50060

you should look into the log containing the string "listener.getLocalPort()
returned " which will give you idea about the port no which is bound with
some process on the machine.



::::::::::::::::::::::::::::::::::::::::
Raj K Singh
http://in.linkedin.com/in/rajkrrsingh
http://www.rajkrrsingh.blogspot.com
Mobile  Tel: +91 (0)9899821370


On Wed, Jun 11, 2014 at 8:42 AM, Li Li <fa...@gmail.com> wrote:

> I have a namenode/jobtracker, 5 datanode/tasktracker. hadoop version is
> 1.2.1
> I use stop-mapred.sh and start-mapred.sh. But there is one node always
> fails.
> the error log is:
>
> 2014-06-11 10:46:57,475 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open()
> is -1. Opening the listener on 50060
> 2014-06-11 10:46:57,477 ERROR org.apache.hadoop.mapred.TaskTracker:
> Can not start task tracker because java.net.BindException: Address
> already in use
>         at sun.nio.ch.Net.bind0(Native Method)
>         at sun.nio.ch.Net.bind(Net.java:444)
>         at sun.nio.ch.Net.bind(Net.java:436)
>         at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
>         at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
>         at
> org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
>         at org.apache.hadoop.http.HttpServer.start(HttpServer.java:602)
>         at
> org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1616)
>         at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3937)
>
> 2014-06-11 10:46:57,479 INFO org.apache.hadoop.mapred.TaskTracker:
> SHUTDOWN_MSG:
>
> the other correct nodes' log:
> 2014-06-11 10:44:32,767 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open()
> is -1. Opening the listener on 50060
> 2014-06-11 10:44:32,768 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50060
> webServer.getConnectors()[0].getLocalPort() returned 50060
> 2014-06-11 10:44:32,768 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 50060
> 2014-06-11 10:44:32,768 INFO org.mortbay.log: jetty-6.1.26
>
> it seems the reason is that port 50060 is used by other process.
> But I use sudo netstat and also lsof. no process is using 50060. What's
> wrong?
>

Re: one node can't start tasktracker

Posted by Raj K Singh <ra...@gmail.com>.
​the problem seems with the jetty server port binding, as per the posted
log​

2014-06-11 10:46:57,475 INFO org.apache.hadoop.http.HttpServer: Port
returned by webServer.getConnectors()[0].getLocalPort() before open()
is -1. Opening the listener on 50060

before opening a connection the port which was retrieved -1
(webServer.getConnectors()[0].getLocalPort()) and oriPort (The original
requested port) is 50060

you should look into the log containing the string "listener.getLocalPort()
returned " which will give you idea about the port no which is bound with
some process on the machine.



::::::::::::::::::::::::::::::::::::::::
Raj K Singh
http://in.linkedin.com/in/rajkrrsingh
http://www.rajkrrsingh.blogspot.com
Mobile  Tel: +91 (0)9899821370


On Wed, Jun 11, 2014 at 8:42 AM, Li Li <fa...@gmail.com> wrote:

> I have a namenode/jobtracker, 5 datanode/tasktracker. hadoop version is
> 1.2.1
> I use stop-mapred.sh and start-mapred.sh. But there is one node always
> fails.
> the error log is:
>
> 2014-06-11 10:46:57,475 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open()
> is -1. Opening the listener on 50060
> 2014-06-11 10:46:57,477 ERROR org.apache.hadoop.mapred.TaskTracker:
> Can not start task tracker because java.net.BindException: Address
> already in use
>         at sun.nio.ch.Net.bind0(Native Method)
>         at sun.nio.ch.Net.bind(Net.java:444)
>         at sun.nio.ch.Net.bind(Net.java:436)
>         at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
>         at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
>         at
> org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
>         at org.apache.hadoop.http.HttpServer.start(HttpServer.java:602)
>         at
> org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1616)
>         at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3937)
>
> 2014-06-11 10:46:57,479 INFO org.apache.hadoop.mapred.TaskTracker:
> SHUTDOWN_MSG:
>
> the other correct nodes' log:
> 2014-06-11 10:44:32,767 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open()
> is -1. Opening the listener on 50060
> 2014-06-11 10:44:32,768 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50060
> webServer.getConnectors()[0].getLocalPort() returned 50060
> 2014-06-11 10:44:32,768 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 50060
> 2014-06-11 10:44:32,768 INFO org.mortbay.log: jetty-6.1.26
>
> it seems the reason is that port 50060 is used by other process.
> But I use sudo netstat and also lsof. no process is using 50060. What's
> wrong?
>

Re: one node can't start tasktracker

Posted by Raj K Singh <ra...@gmail.com>.
​the problem seems with the jetty server port binding, as per the posted
log​

2014-06-11 10:46:57,475 INFO org.apache.hadoop.http.HttpServer: Port
returned by webServer.getConnectors()[0].getLocalPort() before open()
is -1. Opening the listener on 50060

before opening a connection the port which was retrieved -1
(webServer.getConnectors()[0].getLocalPort()) and oriPort (The original
requested port) is 50060

you should look into the log containing the string "listener.getLocalPort()
returned " which will give you idea about the port no which is bound with
some process on the machine.



::::::::::::::::::::::::::::::::::::::::
Raj K Singh
http://in.linkedin.com/in/rajkrrsingh
http://www.rajkrrsingh.blogspot.com
Mobile  Tel: +91 (0)9899821370


On Wed, Jun 11, 2014 at 8:42 AM, Li Li <fa...@gmail.com> wrote:

> I have a namenode/jobtracker, 5 datanode/tasktracker. hadoop version is
> 1.2.1
> I use stop-mapred.sh and start-mapred.sh. But there is one node always
> fails.
> the error log is:
>
> 2014-06-11 10:46:57,475 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open()
> is -1. Opening the listener on 50060
> 2014-06-11 10:46:57,477 ERROR org.apache.hadoop.mapred.TaskTracker:
> Can not start task tracker because java.net.BindException: Address
> already in use
>         at sun.nio.ch.Net.bind0(Native Method)
>         at sun.nio.ch.Net.bind(Net.java:444)
>         at sun.nio.ch.Net.bind(Net.java:436)
>         at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
>         at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
>         at
> org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
>         at org.apache.hadoop.http.HttpServer.start(HttpServer.java:602)
>         at
> org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1616)
>         at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3937)
>
> 2014-06-11 10:46:57,479 INFO org.apache.hadoop.mapred.TaskTracker:
> SHUTDOWN_MSG:
>
> the other correct nodes' log:
> 2014-06-11 10:44:32,767 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open()
> is -1. Opening the listener on 50060
> 2014-06-11 10:44:32,768 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50060
> webServer.getConnectors()[0].getLocalPort() returned 50060
> 2014-06-11 10:44:32,768 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 50060
> 2014-06-11 10:44:32,768 INFO org.mortbay.log: jetty-6.1.26
>
> it seems the reason is that port 50060 is used by other process.
> But I use sudo netstat and also lsof. no process is using 50060. What's
> wrong?
>