You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Harsh J <ha...@cloudera.com> on 2013/04/06 22:24:08 UTC
Re: Oozie workflow error - renewing token issue
Hey Daryn and Oozie folks as well,
I've reported this issue as
https://issues.apache.org/jira/browse/HADOOP-9461, posting as I
figured you may be interested in it (Oozie gets affected on the
outcome).
On Thu, Jan 31, 2013 at 2:36 AM, Daryn Sharp <da...@yahoo-inc.com> wrote:
> The token renewer needs to be the job tracker principal. I think oozie had
> "mr token" hardcoded at one point, but later changed it to use a conf
> setting.
>
> The rest of the log looks very odd - ie. it looks like security is off, but
> it can't be. It's trying to renew hdfs tokens issued for the "hdfs"
> superuser. Are you running your job as hdfs? With security enabled, the
> container executor won't run jobs for hdfs… It's also trying to renew as
> "mapred (auth:SIMPLE)". SIMPLE means that security is disabled, which
> doesn't make sense since tokens require kerberos.
>
> The cannot find renewer class for MAPREDUCE_DELEGATION_TOKEN is also odd
> because it did subsequently try to renew the token.
>
> Daryn
>
>
>
> On Jan 30, 2013, at 2:14 PM, Alejandro Abdelnur wrote:
>
> Cobert,
>
> [Moving thread to user@oozie.a.o, BCCing common-user@hadoop.a.o]
>
> * What version of Oozie are you using?
> * Is the cluster a secure setup (Kerberos enabled)?
> * Would you mind posting the complete launcher logs?
>
> Thx
>
>
>
> On Wed, Jan 30, 2013 at 6:14 AM, Corbett Martin <co...@nhin.com> wrote:
>>
>> Thanks for the tip.
>>
>> The sqoop command listed in the stdout log file is:
>> sqoop
>> import
>> --driver
>> org.apache.derby.jdbc.ClientDriver
>> --connect
>> jdbc:derby://test-server:1527/mondb
>> --username
>> monuser
>> --password
>> xxxxxxxxx
>> --table
>> MONITOR
>> --split-by
>> request_id
>> --target-dir
>> /mon/import
>> --append
>> --incremental
>> append
>> --check-column
>> request_id
>> --last-value
>> 2000000
>>
>> The following information is from the sterr and stdout log files.
>>
>>
>> /var/log/hadoop-0.20-mapreduce/userlogs/job_201301231648_0029/attempt_201301231648_0029_m_000000_0
>> # more stderr
>> No such sqoop tool: sqoop. See 'sqoop help'.
>> Intercepting System.exit(1)
>> Failing Oozie Launcher, Main class
>> [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]
>>
>>
>>
>> /var/log/hadoop-0.20-mapreduce/userlogs/job_201301231648_0029/attempt_201301231648_0029_m_000000_0
>> # more stdout
>> ...
>> ...
>> ...
>> >>> Invoking Sqoop command line now >>>
>>
>> 1598 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has
>> not been set in the environment. Cannot check for additional configuration.
>> Intercepting System.exit(1)
>>
>> <<< Invocation of Main class completed <<<
>>
>> Failing Oozie Launcher, Main class
>> [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]
>>
>> Oozie Launcher failed, finishing Hadoop job gracefully
>>
>> Oozie Launcher ends
>>
>>
>> ~Corbett Martin
>> Software Architect
>> AbsoluteAR Accounts Receivable Services - An NHIN Solution
>>
>> -----Original Message-----
>> From: Harsh J [mailto:harsh@cloudera.com]
>> Sent: Tuesday, January 29, 2013 10:11 PM
>> To: <us...@hadoop.apache.org>
>> Subject: Re: Oozie workflow error - renewing token issue
>>
>> The job that Oozie launches for your action, which you are observing is
>> failing, does its own logs (task logs) show any errors?
>>
>> On Wed, Jan 30, 2013 at 4:59 AM, Corbett Martin <co...@nhin.com> wrote:
>> > Oozie question
>> >
>> >
>> >
>> > I'm trying to run an Oozie workflow (sqoop action) from the Hue
>> > console and it fails every time. No exception in the oozie log but I
>> > see this in the Job Tracker log file.
>> >
>> >
>> >
>> > Two primary issues seem to be
>> >
>> > 1. Client mapred tries to renew a token with renewer specified as mr
>> > token
>> >
>> >
>> >
>> > And
>> >
>> >
>> >
>> > 2. Cannot find class for token kind MAPREDUCE_DELEGATION_TOKEN
>> >
>> >
>> >
>> > Any ideas how to get past this?
>> >
>> >
>> >
>> > Full Stacktrace:
>> >
>> >
>> >
>> > 2013-01-29 17:11:28,860 INFO
>> >
>> > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
>> > Creating password for identifier: owner=hdfs, renewer=mr token,
>> > realUser=oozie, issueDate=1359501088860, maxDate=1360105888860,
>> > sequenceNumber=75, masterKeyId=8
>> >
>> > 2013-01-29 17:11:28,871 INFO
>> >
>> > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
>> > Creating password for identifier: owner=hdfs, renewer=mr token,
>> > realUser=oozie, issueDate=1359501088871, maxDate=1360105888871,
>> > sequenceNumber=76, masterKeyId=8
>> >
>> > 2013-01-29 17:11:29,202 INFO
>> > org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal:
>> > registering token for renewal for service =10.204.12.62:8021 and jobID
>> > =
>> > job_201301231648_0029
>> >
>> > 2013-01-29 17:11:29,211 INFO
>> >
>> > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
>> > Token renewal requested for identifier: owner=hdfs, renewer=mr token,
>> > realUser=oozie, issueDate=1359501088871, maxDate=1360105888871,
>> > sequenceNumber=76, masterKeyId=8
>> >
>> > 2013-01-29 17:11:29,211 ERROR
>> > org.apache.hadoop.security.UserGroupInformation:
>> > PriviledgedActionException as:mapred (auth:SIMPLE)
>> > cause:org.apache.hadoop.security.AccessControlException: Client mapred
>> > tries to renew a token with renewer specified as mr token
>> >
>> > 2013-01-29 17:11:29,211 WARN org.apache.hadoop.security.token.Token:
>> > Cannot find class for token kind MAPREDUCE_DELEGATION_TOKEN
>> >
>> > 2013-01-29 17:11:29,211 INFO org.apache.hadoop.ipc.Server: IPC Server
>> > handler 9 on 8021, call renewDelegationToken(Kind:
>> > MAPREDUCE_DELEGATION_TOKEN, Service: 10.204.12.62:8021, Ident: 00 04
>> > 68 64
>> > 66 73 08 6d 72 20 74 6f 6b 65 6e 05 6f 6f 7a 69 65 8a 01 3c 88 94 58
>> > 67 8a
>> > 01 3c ac a0 dc 67 4c 08), rpc version=2, client version=28,
>> > methodsFingerPrint=1830206421 from 10.204.12.62:9706: error:
>> > org.apache.hadoop.security.AccessControlException: Client mapred tries
>> > to renew a token with renewer specified as mr token
>> >
>> > org.apache.hadoop.security.AccessControlException: Client mapred tries
>> > to renew a token with renewer specified as mr token
>> >
>> > at
>> > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSec
>> > retManager.renewToken(AbstractDelegationTokenSecretManager.java:274)
>> >
>> > at
>> > org.apache.hadoop.mapred.JobTracker.renewDelegationToken(JobTracker.ja
>> > va:3738)
>> >
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> > Method)
>> >
>> > at
>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
>> > ava:39)
>> >
>> > at
>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
>> > orImpl.java:25)
>> >
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >
>> > at
>> > org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call
>> > (WritableRpcEngine.java:474)
>> >
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
>> >
>> > at
>> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
>> >
>> > at
>> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
>> >
>> > at java.security.AccessController.doPrivileged(Native
>> > Method)
>> >
>> > at javax.security.auth.Subject.doAs(Subject.java:396)
>> >
>> > at
>> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat
>> > ion.java:1332)
>> >
>> > at
>> > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
>> >
>> > 2013-01-29 17:11:29,212 ERROR
>> > org.apache.hadoop.security.UserGroupInformation:
>> > PriviledgedActionException as:mapred (auth:SIMPLE)
>> >
>> > cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException):
>> > Client mapred tries to renew a token with renewer specified as mr
>> > token
>> >
>> > at
>> > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSec
>> > retManager.renewToken(AbstractDelegationTokenSecretManager.java:274)
>> >
>> > at
>> > org.apache.hadoop.mapred.JobTracker.renewDelegationToken(JobTracker.ja
>> > va:3738)
>> >
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> > Method)
>> >
>> > at
>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
>> > ava:39)
>> >
>> > at
>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
>> > orImpl.java:25)
>> >
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >
>> > at
>> > org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call
>> > (WritableRpcEngine.java:474)
>> >
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
>> >
>> > at
>> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
>> >
>> > at
>> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
>> >
>> > at java.security.AccessController.doPrivileged(Native
>> > Method)
>> >
>> > at javax.security.auth.Subject.doAs(Subject.java:396)
>> >
>> > at
>> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat
>> > ion.java:1332)
>> >
>> > at
>> > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
>> >
>> >
>> >
>> > 2013-01-29 17:11:29,212 WARN org.apache.hadoop.security.token.Token:
>> > Cannot find class for token kind MAPREDUCE_DELEGATION_TOKEN
>> >
>> > 2013-01-29 17:11:29,212 ERROR
>> > org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal:
>> > Exception renewing tokenKind: MAPREDUCE_DELEGATION_TOKEN, Service:
>> > 10.204.12.62:8021,
>> > Ident: 00 04 68 64 66 73 08 6d 72 20 74 6f 6b 65 6e 05 6f 6f 7a 69 65
>> > 8a 01 3c 88 94 58 67 8a 01 3c ac a0 dc 67 4c 08. Not rescheduled
>> >
>> >
>> > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException):
>> > Client mapred tries to renew a token with renewer specified as mr
>> > token
>> >
>> > at
>> > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSec
>> > retManager.renewToken(AbstractDelegationTokenSecretManager.java:274)
>> >
>> > at
>> > org.apache.hadoop.mapred.JobTracker.renewDelegationToken(JobTracker.ja
>> > va:3738)
>> >
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> > Method)
>> >
>> > at
>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
>> > ava:39)
>> >
>> > at
>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
>> > orImpl.java:25)
>> >
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >
>> > at
>> > org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call
>> > (WritableRpcEngine.java:474)
>> >
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
>> >
>> > at
>> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
>> >
>> > at
>> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
>> >
>> > at java.security.AccessController.doPrivileged(Native
>> > Method)
>> >
>> > at javax.security.auth.Subject.doAs(Subject.java:396)
>> >
>> > at
>> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat
>> > ion.java:1332)
>> >
>> > at
>> > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
>> >
>> >
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:1160)
>> >
>> > at
>> > org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngi
>> > ne.java:225)
>> >
>> > at
>> > org.apache.hadoop.mapred.$Proxy12.renewDelegationToken(Unknown
>> > Source)
>> >
>> > at
>> > org.apache.hadoop.mapred.JobClient$Renewer.renew(JobClient.java:541)
>> >
>> > at
>> > org.apache.hadoop.security.token.Token.renew(Token.java:372)
>> >
>> > at
>> > org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$Rene
>> > walTimerTask$1.run(DelegationTokenRenewal.java:218)
>> >
>> > at
>> > org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$Rene
>> > walTimerTask$1.run(DelegationTokenRenewal.java:214)
>> >
>> > at java.security.AccessController.doPrivileged(Native
>> > Method)
>> >
>> > at javax.security.auth.Subject.doAs(Subject.java:396)
>> >
>> > at
>> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat
>> > ion.java:1332)
>> >
>> > at
>> > org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$Rene
>> > walTimerTask.run(DelegationTokenRenewal.java:213)
>> >
>> > at java.util.TimerThread.mainLoop(Timer.java:512)
>> >
>> > at java.util.TimerThread.run(Timer.java:462)
>> >
>> >
>> >
>> > ~Corbett Martin
>> >
>> > Software Architect
>> >
>> > AbsoluteAR Accounts Receivable Services - An NHIN Solution
>> >
>> >
>> >
>> >
>> > ________________________________
>> > This message and its contents (to include attachments) are the
>> > property of National Health Systems, Inc. and may contain confidential
>> > and proprietary information. This email and any files transmitted with
>> > it are intended solely for the use of the individual or entity to whom
>> > they are addressed.
>> > You are hereby notified that any unauthorized disclosure, copying, or
>> > distribution of this message, or the taking of any unauthorized action
>> > based on information contained herein is strictly prohibited.
>> > Unauthorized use of information contained herein may subject you to
>> > civil and criminal prosecution and penalties. If you are not the
>> > intended recipient, you should delete this message immediately and
>> > notify the sender immediately by telephone or by replying to this
>> > transmission.
>>
>>
>>
>> --
>> Harsh J
>>
>> This message and its contents (to include attachments) are the property of
>> National Health Systems, Inc. and may contain confidential and proprietary
>> information. This email and any files transmitted with it are intended
>> solely for the use of the individual or entity to whom they are addressed.
>> You are hereby notified that any unauthorized disclosure, copying, or
>> distribution of this message, or the taking of any unauthorized action based
>> on information contained herein is strictly prohibited. Unauthorized use of
>> information contained herein may subject you to civil and criminal
>> prosecution and penalties. If you are not the intended recipient, you should
>> delete this message immediately and notify the sender immediately by
>> telephone or by replying to this transmission.
>
>
>
>
> --
> Alejandro
>
>
--
Harsh J