You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Manish Bhoge <ma...@rocketmail.com> on 2012/03/23 08:50:57 UTC
Job tracker service start issue.
I have Hadoop running on Standalone box. When I am starting deamon for
namenode, secondarynamenode, job tracker, task tracker and data node, it is starting gracefully. But soon after it start job tracker it doesn't
show up job tracker service. when i run 'jps' it is showing me all the
services including task tracker except Job Tracker.
Is there any time limit that need to set up or is it going into the safe
mode. Because when i saw job tracker log this what it is showing, looks
like it is starting the namenode but soon after it shutdown:
2012-03-22 23:26:04,061 INFO org.apache.hadoop.mapred.JobTracker: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting JobTracker
STARTUP_MSG: host = manish/10.131.18.119
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.2-cdh3u3
STARTUP_MSG: build = file:///data/1/tmp/nightly_2012-02-16_09-46-24_3/hadoop-0.20-0.20.2+923.195-1~maverick -r 217a3767c48ad11d4632e19a22897677268c40c4; compiled by 'root' on Thu Feb 16 10:22:53 PST 2012
************************************************************/
2012-03-22 23:26:04,140 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2012-03-22 23:26:04,141 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
2012-03-22 23:26:04,141 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2012-03-22 23:26:04,142 INFO org.apache.hadoop.mapred.JobTracker: Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT, limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
2012-03-22 23:26:04,143 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2012-03-22 23:26:04,186 INFO org.apache.hadoop.mapred.JobTracker: Starting jobtracker with owner as mapred
2012-03-22 23:26:04,201 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 54311
2012-03-22 23:26:04,203 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=JobTracker, port=54311
2012-03-22 23:26:04,206 INFO org.apache.hadoop.ipc.metrics.RpcDetailedMetrics: Initializing RPC Metrics with hostName=JobTracker, port=54311
2012-03-22 23:26:09,250 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2012-03-22 23:26:09,298 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2012-03-22 23:26:09,318 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50030
2012-03-22 23:26:09,318 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50030 webServer.getConnectors()[0].getLocalPort() returned 50030
2012-03-22 23:26:09,318 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50030
2012-03-22 23:26:09,319 INFO org.mortbay.log: jetty-6.1.26.cloudera.1
2012-03-22 23:26:09,517 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50030
2012-03-22 23:26:09,519 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
2012-03-22 23:26:09,519 INFO org.apache.hadoop.mapred.JobTracker: JobTracker up at: 54311
2012-03-22 23:26:09,519 INFO org.apache.hadoop.mapred.JobTracker: JobTracker webserver: 50030
2012-03-22 23:26:09,648 WARN org.apache.hadoop.mapred.JobTracker: Failed to operate on mapred.system.dir (hdfs://localhost:54310/app/hadoop/tmp/mapred/system) because of permissions.
2012-03-22 23:26:09,648 WARN org.apache.hadoop.mapred.JobTracker: This directory should be owned by the user 'mapred (auth:SIMPLE)'
2012-03-22 23:26:09,650 WARN org.apache.hadoop.mapred.JobTracker: Bailing out ...
org.apache.hadoop.security.AccessControlException: The systemdir hdfs://localhost:54310/app/hadoop/tmp/mapred/system is not owned by mapred at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2241) at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2050) at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:296) at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:288) at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4792)
2012-03-22 23:26:09,652 FATAL org.apache.hadoop.mapred.JobTracker: org.apache.hadoop.security.AccessControlException: The systemdir hdfs://localhost:54310/app/hadoop/tmp/mapred/system is not owned by mapred at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2241) at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2050) at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:296) at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:288) at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4792) 2012-03-22 23:26:09,654 INFO org.apache.hadoop.mapred.JobTracker: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down JobTracker at manish/10.131.18.119
************************************************************/
Re: Job tracker service start issue.
Posted by Marcos Ortiz <ml...@uci.cu>.
On 03/23/2012 06:57 AM, kasi subrahmanyam wrote:
> Hi Oliver,
>
> I am not sure my suggestion might solve your problem or it might be already
> solved on your side.
> It seems the task tracker is having a problem accessing the tmp directory.
> Try going to the core and mapred site xml and change the tmp directory to a
> new one.
> If this is not yet working then manually change the permissions of theat
> directory using :
> chmod -R 777 tmp
Please, don´t do chmod -R 777 in tmp directory. It´s not recommendable
for production servers.
The first option is more wise:
1- change the tmp directory in the core and mapreduce files
2- chown this new directory to group hadoop, where are the mapred and
hdfs users
>
> On Fri, Mar 23, 2012 at 3:33 PM, Olivier Sallou<ol...@irisa.fr>wrote:
>
>>
>> Le 3/23/12 8:50 AM, Manish Bhoge a écrit :
>>> I have Hadoop running on Standalone box. When I am starting deamon for
>>> namenode, secondarynamenode, job tracker, task tracker and data node, it
>> is starting gracefully. But soon after it start job tracker it doesn't
>>> show up job tracker service. when i run 'jps' it is showing me all the
>>> services including task tracker except Job Tracker.
>>>
>>> Is there any time limit that need to set up or is it going into the safe
>>> mode. Because when i saw job tracker log this what it is showing, looks
>>> like it is starting the namenode but soon after it shutdown:
>>>
>>> 2012-03-22 23:26:04,061 INFO org.apache.hadoop.mapred.JobTracker:
>> STARTUP_MSG:
>>> /************************************************************
>>> STARTUP_MSG: Starting JobTracker
>>> STARTUP_MSG: host = manish/10.131.18.119
>>> STARTUP_MSG: args = []
>>> STARTUP_MSG: version = 0.20.2-cdh3u3
>>> STARTUP_MSG: build =
>> file:///data/1/tmp/nightly_2012-02-16_09-46-24_3/hadoop-0.20-0.20.2+923.195-1~maverick
>> -r 217a3767c48ad11d4632e19a22897677268c40c4; compiled by 'root' on Thu Feb
>> 16 10:22:53 PST 2012
>>> ************************************************************/
>>> 2012-03-22 23:26:04,140 INFO
>> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
>> Updating the current master key for generating delegation tokens
>>> 2012-03-22 23:26:04,141 INFO
>> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
>> Starting expired delegation token remover thread,
>> tokenRemoverScanInterval=60 min(s)
>>> 2012-03-22 23:26:04,141 INFO
>> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
>> Updating the current master key for generating delegation tokens
>>> 2012-03-22 23:26:04,142 INFO org.apache.hadoop.mapred.JobTracker:
>> Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
>> limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
>>> 2012-03-22 23:26:04,143 INFO org.apache.hadoop.util.HostsFileReader:
>> Refreshing hosts (include/exclude) list
>>> 2012-03-22 23:26:04,186 INFO org.apache.hadoop.mapred.JobTracker:
>> Starting jobtracker with owner as mapred
>>> 2012-03-22 23:26:04,201 INFO org.apache.hadoop.ipc.Server: Starting
>> Socket Reader #1 for port 54311
>>> 2012-03-22 23:26:04,203 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
>> Initializing RPC Metrics with hostName=JobTracker, port=54311
>>> 2012-03-22 23:26:04,206 INFO
>> org.apache.hadoop.ipc.metrics.RpcDetailedMetrics: Initializing RPC Metrics
>> with hostName=JobTracker, port=54311
>>> 2012-03-22 23:26:09,250 INFO org.mortbay.log: Logging to
>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>> org.mortbay.log.Slf4jLog
>>> 2012-03-22 23:26:09,298 INFO org.apache.hadoop.http.HttpServer: Added
>> global filtersafety
>> (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>>> 2012-03-22 23:26:09,318 INFO org.apache.hadoop.http.HttpServer: Port
>> returned by webServer.getConnectors()[0].getLocalPort() before open() is
>> -1. Opening the listener on 50030
>>> 2012-03-22 23:26:09,318 INFO org.apache.hadoop.http.HttpServer:
>> listener.getLocalPort() returned 50030
>> webServer.getConnectors()[0].getLocalPort() returned 50030
>>> 2012-03-22 23:26:09,318 INFO org.apache.hadoop.http.HttpServer: Jetty
>> bound to port 50030
>>> 2012-03-22 23:26:09,319 INFO org.mortbay.log: jetty-6.1.26.cloudera.1
>>> 2012-03-22 23:26:09,517 INFO org.mortbay.log: Started
>> SelectChannelConnector@0.0.0.0:50030
>>> 2012-03-22 23:26:09,519 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
>> Initializing JVM Metrics with processName=JobTracker, sessionId=
>>> 2012-03-22 23:26:09,519 INFO org.apache.hadoop.mapred.JobTracker:
>> JobTracker up at: 54311
>>> 2012-03-22 23:26:09,519 INFO org.apache.hadoop.mapred.JobTracker:
>> JobTracker webserver: 50030
>>> 2012-03-22 23:26:09,648 WARN org.apache.hadoop.mapred.JobTracker: Failed
>> to operate on mapred.system.dir
>> (hdfs://localhost:54310/app/hadoop/tmp/mapred/system) because of
>> permissions.
>>> 2012-03-22 23:26:09,648 WARN org.apache.hadoop.mapred.JobTracker: This
>> directory should be owned by the user 'mapred (auth:SIMPLE)'
>>> 2012-03-22 23:26:09,650 WARN org.apache.hadoop.mapred.JobTracker:
>> Bailing out ...
>>> org.apache.hadoop.security.AccessControlException: The systemdir
>> hdfs://localhost:54310/app/hadoop/tmp/mapred/system is not owned by mapred
>> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2241) at
>> org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2050) at
>> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:296) at
>> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:288) at
>> org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4792)
>>> 2012-03-22 23:26:09,652 FATAL org.apache.hadoop.mapred.JobTracker:
>> org.apache.hadoop.security.AccessControlException: The systemdir
>> hdfs://localhost:54310/app/hadoop/tmp/mapred/system is not owned by mapred
>> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2241) at
>> org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2050) at
>> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:296) at
>> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:288) at
>> org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4792) 2012-03-22
>> 23:26:09,654 INFO org.apache.hadoop.mapred.JobTracker: SHUTDOWN_MSG:
>>> /************************************************************
>> you seem to have dfs access right issues.
>> Debian/Ubuntu packages create 2 users mapred and hdfs, you certainly
>> have the same.
>> A hdfs chmod in you hdfs dir for mapred should solve your issue : chown
>> to mapred or add write access to hadoop group as mapred and hdfs user
>> are both in hadoop group
>>
>> Olivier
>>> SHUTDOWN_MSG: Shutting down JobTracker at manish/10.131.18.119
>>> ************************************************************/
>> --
>>
>>
>> gpg key id: 4096R/326D8438 (keyring.debian.org)
>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
>>
>>
>>
>
> 10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
> CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
>
> http://www.uci.cu
> http://www.facebook.com/universidad.uci
> http://www.flickr.com/photos/universidad_uci
>
>
--
Marcos Luis Ortíz Valmaseda (@marcosluis2186)
Data Engineer at UCI
http://marcosluis2186.posterous.com
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
Re: Job tracker service start issue.
Posted by kasi subrahmanyam <ka...@gmail.com>.
Hi Oliver,
I am not sure my suggestion might solve your problem or it might be already
solved on your side.
It seems the task tracker is having a problem accessing the tmp directory.
Try going to the core and mapred site xml and change the tmp directory to a
new one.
If this is not yet working then manually change the permissions of theat
directory using :
chmod -R 777 tmp
On Fri, Mar 23, 2012 at 3:33 PM, Olivier Sallou <ol...@irisa.fr>wrote:
>
>
> Le 3/23/12 8:50 AM, Manish Bhoge a écrit :
> > I have Hadoop running on Standalone box. When I am starting deamon for
> > namenode, secondarynamenode, job tracker, task tracker and data node, it
> is starting gracefully. But soon after it start job tracker it doesn't
> > show up job tracker service. when i run 'jps' it is showing me all the
> > services including task tracker except Job Tracker.
> >
> > Is there any time limit that need to set up or is it going into the safe
> > mode. Because when i saw job tracker log this what it is showing, looks
> > like it is starting the namenode but soon after it shutdown:
> >
> > 2012-03-22 23:26:04,061 INFO org.apache.hadoop.mapred.JobTracker:
> STARTUP_MSG:
> > /************************************************************
> > STARTUP_MSG: Starting JobTracker
> > STARTUP_MSG: host = manish/10.131.18.119
> > STARTUP_MSG: args = []
> > STARTUP_MSG: version = 0.20.2-cdh3u3
> > STARTUP_MSG: build =
> file:///data/1/tmp/nightly_2012-02-16_09-46-24_3/hadoop-0.20-0.20.2+923.195-1~maverick
> -r 217a3767c48ad11d4632e19a22897677268c40c4; compiled by 'root' on Thu Feb
> 16 10:22:53 PST 2012
> > ************************************************************/
> > 2012-03-22 23:26:04,140 INFO
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> Updating the current master key for generating delegation tokens
> > 2012-03-22 23:26:04,141 INFO
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> Starting expired delegation token remover thread,
> tokenRemoverScanInterval=60 min(s)
> > 2012-03-22 23:26:04,141 INFO
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> Updating the current master key for generating delegation tokens
> > 2012-03-22 23:26:04,142 INFO org.apache.hadoop.mapred.JobTracker:
> Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
> limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
> > 2012-03-22 23:26:04,143 INFO org.apache.hadoop.util.HostsFileReader:
> Refreshing hosts (include/exclude) list
> > 2012-03-22 23:26:04,186 INFO org.apache.hadoop.mapred.JobTracker:
> Starting jobtracker with owner as mapred
> > 2012-03-22 23:26:04,201 INFO org.apache.hadoop.ipc.Server: Starting
> Socket Reader #1 for port 54311
> > 2012-03-22 23:26:04,203 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> Initializing RPC Metrics with hostName=JobTracker, port=54311
> > 2012-03-22 23:26:04,206 INFO
> org.apache.hadoop.ipc.metrics.RpcDetailedMetrics: Initializing RPC Metrics
> with hostName=JobTracker, port=54311
> > 2012-03-22 23:26:09,250 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> > 2012-03-22 23:26:09,298 INFO org.apache.hadoop.http.HttpServer: Added
> global filtersafety
> (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
> > 2012-03-22 23:26:09,318 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open() is
> -1. Opening the listener on 50030
> > 2012-03-22 23:26:09,318 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50030
> webServer.getConnectors()[0].getLocalPort() returned 50030
> > 2012-03-22 23:26:09,318 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 50030
> > 2012-03-22 23:26:09,319 INFO org.mortbay.log: jetty-6.1.26.cloudera.1
> > 2012-03-22 23:26:09,517 INFO org.mortbay.log: Started
> SelectChannelConnector@0.0.0.0:50030
> > 2012-03-22 23:26:09,519 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=JobTracker, sessionId=
> > 2012-03-22 23:26:09,519 INFO org.apache.hadoop.mapred.JobTracker:
> JobTracker up at: 54311
> > 2012-03-22 23:26:09,519 INFO org.apache.hadoop.mapred.JobTracker:
> JobTracker webserver: 50030
> > 2012-03-22 23:26:09,648 WARN org.apache.hadoop.mapred.JobTracker: Failed
> to operate on mapred.system.dir
> (hdfs://localhost:54310/app/hadoop/tmp/mapred/system) because of
> permissions.
> > 2012-03-22 23:26:09,648 WARN org.apache.hadoop.mapred.JobTracker: This
> directory should be owned by the user 'mapred (auth:SIMPLE)'
> > 2012-03-22 23:26:09,650 WARN org.apache.hadoop.mapred.JobTracker:
> Bailing out ...
> > org.apache.hadoop.security.AccessControlException: The systemdir
> hdfs://localhost:54310/app/hadoop/tmp/mapred/system is not owned by mapred
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2241) at
> org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2050) at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:296) at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:288) at
> org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4792)
> > 2012-03-22 23:26:09,652 FATAL org.apache.hadoop.mapred.JobTracker:
> org.apache.hadoop.security.AccessControlException: The systemdir
> hdfs://localhost:54310/app/hadoop/tmp/mapred/system is not owned by mapred
> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2241) at
> org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2050) at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:296) at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:288) at
> org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4792) 2012-03-22
> 23:26:09,654 INFO org.apache.hadoop.mapred.JobTracker: SHUTDOWN_MSG:
> > /************************************************************
> you seem to have dfs access right issues.
> Debian/Ubuntu packages create 2 users mapred and hdfs, you certainly
> have the same.
> A hdfs chmod in you hdfs dir for mapred should solve your issue : chown
> to mapred or add write access to hadoop group as mapred and hdfs user
> are both in hadoop group
>
> Olivier
> > SHUTDOWN_MSG: Shutting down JobTracker at manish/10.131.18.119
> > ************************************************************/
>
> --
>
>
> gpg key id: 4096R/326D8438 (keyring.debian.org)
> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
>
>
>
Re: Job tracker service start issue.
Posted by Olivier Sallou <ol...@irisa.fr>.
Le 3/23/12 8:50 AM, Manish Bhoge a écrit :
> I have Hadoop running on Standalone box. When I am starting deamon for
> namenode, secondarynamenode, job tracker, task tracker and data node, it is starting gracefully. But soon after it start job tracker it doesn't
> show up job tracker service. when i run 'jps' it is showing me all the
> services including task tracker except Job Tracker.
>
> Is there any time limit that need to set up or is it going into the safe
> mode. Because when i saw job tracker log this what it is showing, looks
> like it is starting the namenode but soon after it shutdown:
>
> 2012-03-22 23:26:04,061 INFO org.apache.hadoop.mapred.JobTracker: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting JobTracker
> STARTUP_MSG: host = manish/10.131.18.119
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.20.2-cdh3u3
> STARTUP_MSG: build = file:///data/1/tmp/nightly_2012-02-16_09-46-24_3/hadoop-0.20-0.20.2+923.195-1~maverick -r 217a3767c48ad11d4632e19a22897677268c40c4; compiled by 'root' on Thu Feb 16 10:22:53 PST 2012
> ************************************************************/
> 2012-03-22 23:26:04,140 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
> 2012-03-22 23:26:04,141 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
> 2012-03-22 23:26:04,141 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
> 2012-03-22 23:26:04,142 INFO org.apache.hadoop.mapred.JobTracker: Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT, limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
> 2012-03-22 23:26:04,143 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
> 2012-03-22 23:26:04,186 INFO org.apache.hadoop.mapred.JobTracker: Starting jobtracker with owner as mapred
> 2012-03-22 23:26:04,201 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 54311
> 2012-03-22 23:26:04,203 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=JobTracker, port=54311
> 2012-03-22 23:26:04,206 INFO org.apache.hadoop.ipc.metrics.RpcDetailedMetrics: Initializing RPC Metrics with hostName=JobTracker, port=54311
> 2012-03-22 23:26:09,250 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
> 2012-03-22 23:26:09,298 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
> 2012-03-22 23:26:09,318 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50030
> 2012-03-22 23:26:09,318 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50030 webServer.getConnectors()[0].getLocalPort() returned 50030
> 2012-03-22 23:26:09,318 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50030
> 2012-03-22 23:26:09,319 INFO org.mortbay.log: jetty-6.1.26.cloudera.1
> 2012-03-22 23:26:09,517 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50030
> 2012-03-22 23:26:09,519 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
> 2012-03-22 23:26:09,519 INFO org.apache.hadoop.mapred.JobTracker: JobTracker up at: 54311
> 2012-03-22 23:26:09,519 INFO org.apache.hadoop.mapred.JobTracker: JobTracker webserver: 50030
> 2012-03-22 23:26:09,648 WARN org.apache.hadoop.mapred.JobTracker: Failed to operate on mapred.system.dir (hdfs://localhost:54310/app/hadoop/tmp/mapred/system) because of permissions.
> 2012-03-22 23:26:09,648 WARN org.apache.hadoop.mapred.JobTracker: This directory should be owned by the user 'mapred (auth:SIMPLE)'
> 2012-03-22 23:26:09,650 WARN org.apache.hadoop.mapred.JobTracker: Bailing out ...
> org.apache.hadoop.security.AccessControlException: The systemdir hdfs://localhost:54310/app/hadoop/tmp/mapred/system is not owned by mapred at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2241) at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2050) at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:296) at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:288) at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4792)
> 2012-03-22 23:26:09,652 FATAL org.apache.hadoop.mapred.JobTracker: org.apache.hadoop.security.AccessControlException: The systemdir hdfs://localhost:54310/app/hadoop/tmp/mapred/system is not owned by mapred at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2241) at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2050) at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:296) at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:288) at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4792) 2012-03-22 23:26:09,654 INFO org.apache.hadoop.mapred.JobTracker: SHUTDOWN_MSG:
> /************************************************************
you seem to have dfs access right issues.
Debian/Ubuntu packages create 2 users mapred and hdfs, you certainly
have the same.
A hdfs chmod in you hdfs dir for mapred should solve your issue : chown
to mapred or add write access to hadoop group as mapred and hdfs user
are both in hadoop group
Olivier
> SHUTDOWN_MSG: Shutting down JobTracker at manish/10.131.18.119
> ************************************************************/
--
gpg key id: 4096R/326D8438 (keyring.debian.org)
Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
Re: Job tracker service start issue.
Posted by Harsh J <ha...@cloudera.com>.
Hi Manish,
You seem to be using CDH3u3, so I have moved your question to the
CDH-users community (cdh-user@cloudera.org, via Google Groups:
https://groups.google.com/a/cloudera.org/forum/#!forum/cdh-user). I
have bcc'd common-user@.
For fixing your issue, make sure you follow the 'Create and configure
mapred.system.dir on HDFS' part on the Installation Guide:
https://ccp.cloudera.com/display/CDHDOC/CDH3+Deployment+on+a+Cluster#CDH3DeploymentonaCluster-Step3%3ACreateandConfigurethe%7B%7Bmapred.system.dir%7D%7DDirectoryinHDFS
Let us know if that fixes it up for you.
(Tip: Following that whole document will get you a perfect Hadoop deployment.)
On Fri, Mar 23, 2012 at 1:20 PM, Manish Bhoge
<ma...@rocketmail.com> wrote:
> I have Hadoop running on Standalone box. When I am starting deamon for
> namenode, secondarynamenode, job tracker, task tracker and data node, it is starting gracefully. But soon after it start job tracker it doesn't
> show up job tracker service. when i run 'jps' it is showing me all the
> services including task tracker except Job Tracker.
>
> Is there any time limit that need to set up or is it going into the safe
> mode. Because when i saw job tracker log this what it is showing, looks
> like it is starting the namenode but soon after it shutdown:
>
> 2012-03-22 23:26:04,061 INFO org.apache.hadoop.mapred.JobTracker: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting JobTracker
> STARTUP_MSG: host = manish/10.131.18.119
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.20.2-cdh3u3
> STARTUP_MSG: build = file:///data/1/tmp/nightly_2012-02-16_09-46-24_3/hadoop-0.20-0.20.2+923.195-1~maverick -r 217a3767c48ad11d4632e19a22897677268c40c4; compiled by 'root' on Thu Feb 16 10:22:53 PST 2012
> ************************************************************/
> 2012-03-22 23:26:04,140 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
> 2012-03-22 23:26:04,141 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
> 2012-03-22 23:26:04,141 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
> 2012-03-22 23:26:04,142 INFO org.apache.hadoop.mapred.JobTracker: Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT, limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
> 2012-03-22 23:26:04,143 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
> 2012-03-22 23:26:04,186 INFO org.apache.hadoop.mapred.JobTracker: Starting jobtracker with owner as mapred
> 2012-03-22 23:26:04,201 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 54311
> 2012-03-22 23:26:04,203 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=JobTracker, port=54311
> 2012-03-22 23:26:04,206 INFO org.apache.hadoop.ipc.metrics.RpcDetailedMetrics: Initializing RPC Metrics with hostName=JobTracker, port=54311
> 2012-03-22 23:26:09,250 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
> 2012-03-22 23:26:09,298 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
> 2012-03-22 23:26:09,318 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50030
> 2012-03-22 23:26:09,318 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50030 webServer.getConnectors()[0].getLocalPort() returned 50030
> 2012-03-22 23:26:09,318 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50030
> 2012-03-22 23:26:09,319 INFO org.mortbay.log: jetty-6.1.26.cloudera.1
> 2012-03-22 23:26:09,517 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50030
> 2012-03-22 23:26:09,519 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
> 2012-03-22 23:26:09,519 INFO org.apache.hadoop.mapred.JobTracker: JobTracker up at: 54311
> 2012-03-22 23:26:09,519 INFO org.apache.hadoop.mapred.JobTracker: JobTracker webserver: 50030
> 2012-03-22 23:26:09,648 WARN org.apache.hadoop.mapred.JobTracker: Failed to operate on mapred.system.dir (hdfs://localhost:54310/app/hadoop/tmp/mapred/system) because of permissions.
> 2012-03-22 23:26:09,648 WARN org.apache.hadoop.mapred.JobTracker: This directory should be owned by the user 'mapred (auth:SIMPLE)'
> 2012-03-22 23:26:09,650 WARN org.apache.hadoop.mapred.JobTracker: Bailing out ...
> org.apache.hadoop.security.AccessControlException: The systemdir hdfs://localhost:54310/app/hadoop/tmp/mapred/system is not owned by mapred at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2241) at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2050) at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:296) at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:288) at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4792)
> 2012-03-22 23:26:09,652 FATAL org.apache.hadoop.mapred.JobTracker: org.apache.hadoop.security.AccessControlException: The systemdir hdfs://localhost:54310/app/hadoop/tmp/mapred/system is not owned by mapred at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2241) at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2050) at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:296) at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:288) at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4792) 2012-03-22 23:26:09,654 INFO org.apache.hadoop.mapred.JobTracker: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down JobTracker at manish/10.131.18.119
> ************************************************************/
--
Harsh J