You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Bhupesh Bansal <bb...@linkedin.com> on 2008/11/17 20:31:20 UTC
tasktracker startup Time
Hey folks,
I re-started my cluster after some node failures and saw couple of
tasktrackers not being up (they finally did after abt 20 Mins)
In the logs below check the blue timestamp to Red timestamp.
I was just curious what do we do while starting tasktracker that could
should take so much time ???
Best
Bhupesh
2008-11-17 10:43:04,094 INFO org.apache.hadoop.mapred.TaskTracker:
STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting TaskTracker
STARTUP_MSG: host = ....
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.18.1
STARTUP_MSG: build = ...
************************************************************/
2008-11-17 10:43:04,292 INFO org.mortbay.util.Credential: Checking Resource
aliases
2008-11-17 10:43:04,400 INFO org.mortbay.http.HttpServer: Version
Jetty/5.1.4
2008-11-17 10:43:04,401 INFO org.mortbay.util.Container: Started
HttpContext[/static,/static]
2008-11-17 10:43:04,401 INFO org.mortbay.util.Container: Started
HttpContext[/logs,/logs]
2008-11-17 10:43:04,713 INFO org.mortbay.util.Container: Started
org.mortbay.jetty.servlet.WebApplicationHandler@73da669c
2008-11-17 10:43:04,753 INFO org.mortbay.util.Container: Started
WebApplicationContext[/,/]
2008-11-17 10:43:04,757 INFO org.mortbay.http.SocketListener: Started
SocketListener on 0.0.0.0:50060
2008-11-17 10:43:04,757 INFO org.mortbay.util.Container: Started
org.mortbay.jetty.Server@20b9b538
2008-11-17 11:12:38,373 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=TaskTracker, sessionId=
2008-11-17 11:12:38,410 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=TaskTracker, port=47601
2008-11-17 11:12:38,487 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2008-11-17 11:12:38,488 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 47601: starting
2008-11-17 11:12:38,490 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 1 on 47601: starting
2008-11-17 11:12:38,490 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 2 on 47601: starting
2008-11-17 11:12:38,490 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 0 on 47601: starting
2008-11-17 11:12:38,490 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 3 on 47601: starting
2008-11-17 11:12:38,491 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 4 on 47601: starting
2008-11-17 11:12:38,491 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 5 on 47601: starting
2008-11-17 11:12:38,491 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 6 on 47601: starting
Re: tasktracker startup Time
Posted by Bhupesh Bansal <bb...@linkedin.com>.
Thanks Steve,
I will try kill -QUIT and report back.
Best
Bhupesh
On 11/18/08 5:45 AM, "Steve Loughran" <st...@apache.org> wrote:
> Bhupesh Bansal wrote:
>> Hey folks,
>>
>> I re-started my cluster after some node failures and saw couple of
>> tasktrackers not being up (they finally did after abt 20 Mins)
>> In the logs below check the blue timestamp to Red timestamp.
>>
>> I was just curious what do we do while starting tasktracker that could
>> should take so much time ???
>>
>>
>
>> 2008-11-17 10:43:04,757 INFO org.mortbay.util.Container: Started
>> org.mortbay.jetty.Server@20b9b538
>> 2008-11-17 11:12:38,373 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
>> Initializing JVM Metrics with processName=TaskTracker, sessionId=
>> 2008-11-17 11:12:38,410 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
>> Initializing RPC Metrics with hostName=TaskTracker, port=47601
>
>
> Off the top of my head
> -DNS lookups can introduce delays if your network's DNS is wrong, but
> that shouldn't take so long
> -The task tracker depends on the job tracker and the filesystem being
> up. If the filesystem is recovering: no task trackers
>
> Next time, get the process ID (via a jps -v call), then do kill -QUIT on
> the process. This will print out to the process's console the stack
> trace of all its threads; this could track down where it is hanging
Re: tasktracker startup Time
Posted by Steve Loughran <st...@apache.org>.
Bhupesh Bansal wrote:
> Hey folks,
>
> I re-started my cluster after some node failures and saw couple of
> tasktrackers not being up (they finally did after abt 20 Mins)
> In the logs below check the blue timestamp to Red timestamp.
>
> I was just curious what do we do while starting tasktracker that could
> should take so much time ???
>
>
> 2008-11-17 10:43:04,757 INFO org.mortbay.util.Container: Started
> org.mortbay.jetty.Server@20b9b538
> 2008-11-17 11:12:38,373 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=TaskTracker, sessionId=
> 2008-11-17 11:12:38,410 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> Initializing RPC Metrics with hostName=TaskTracker, port=47601
Off the top of my head
-DNS lookups can introduce delays if your network's DNS is wrong, but
that shouldn't take so long
-The task tracker depends on the job tracker and the filesystem being
up. If the filesystem is recovering: no task trackers
Next time, get the process ID (via a jps -v call), then do kill -QUIT on
the process. This will print out to the process's console the stack
trace of all its threads; this could track down where it is hanging