You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Robert Kanter (JIRA)" <ji...@apache.org> on 2013/10/01 00:44:24 UTC
[jira] [Commented] (MAPREDUCE-5088) MR Client gets an renewer token
exception while Oozie is submitting a job
[ https://issues.apache.org/jira/browse/MAPREDUCE-5088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782364#comment-13782364 ]
Robert Kanter commented on MAPREDUCE-5088:
------------------------------------------
We're seeing an issue where Oozie can't submit to a secure cluster when the JHS is down because it can't get a delegation token from the Job History Server. However, if I submit a job without Oozie (i.e. hadoop jar ....) it works fine. When investigating, it looks like this JIRA may be the cause, or is at least related. Any ideas on why this is happening?
{noformat}
2013-09-30 13:14:48,544 WARN ActionStartXCommand:542 - USER[testuser] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000055-130929201011702-oozie-oozi-W] ACTION[0000055-130929201011702-oozie-oozi-W@mr-node] Er
ror starting action [mr-node]. ErrorType [TRANSIENT], ErrorCode [ JA006], Message [ JA006: Connection refused]
org.apache.oozie.action.ActionExecutorException: JA006: Connection refused
at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:412)
at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:392)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:794)
at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:948)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:215)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:60)
at org.apache.oozie.command.XCommand.call(XCommand.java:280)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:326)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:255)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:547)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:642)
at org.apache.hadoop.ipc.Client$Connection.access$2600(Client.java:314)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1399)
at org.apache.hadoop.ipc.Client.call(Client.java:1318)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy31.getDelegationToken(Unknown Source)
at org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getDelegationToken(MRClientProtocolPBClientImpl.java:211)
at org.apache.hadoop.mapred.YARNRunner.getDelegationTokenFromHS(YARNRunner.java:208)
at org.apache.hadoop.mapred.YARNRunner.addHistoryToken(YARNRunner.java:195)
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:280)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:430)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:552)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:552)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:543)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:779)
... 10 more
{noformat}
> MR Client gets an renewer token exception while Oozie is submitting a job
> -------------------------------------------------------------------------
>
> Key: MAPREDUCE-5088
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5088
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 2.0.3-alpha
> Reporter: Roman Shaposhnik
> Assignee: Daryn Sharp
> Priority: Blocker
> Fix For: 2.0.4-alpha
>
> Attachments: HADOOP-9409.patch, HADOOP-9409.patch, MAPREDUCE-5088.patch, MAPREDUCE-5088.patch, MAPREDUCE-5088.txt
>
>
> After the fix for HADOOP-9299 I'm now getting the following bizzare exception in Oozie while trying to submit a job. This also seems to be KRB related:
> {noformat}
> 2013-03-15 13:34:16,555 WARN ActionStartXCommand:542 - USER[hue] GROUP[-] TOKEN[] APP[MapReduce] JOB[0000001-130315123130987-oozie-oozi-W] ACTION[0000001-130315123130987-oozie-oozi-W@Sleep] Error starting action [Sleep]. ErrorType [ERROR], ErrorCode [UninitializedMessageException], Message [UninitializedMessageException: Message missing required fields: renewer]
> org.apache.oozie.action.ActionExecutorException: UninitializedMessageException: Message missing required fields: renewer
> at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:401)
> at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:738)
> at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:889)
> at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:211)
> at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:59)
> at org.apache.oozie.command.XCommand.call(XCommand.java:277)
> at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:326)
> at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:255)
> at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: com.google.protobuf.UninitializedMessageException: Message missing required fields: renewer
> at com.google.protobuf.AbstractMessage$Builder.newUninitializedMessageException(AbstractMessage.java:605)
> at org.apache.hadoop.security.proto.SecurityProtos$GetDelegationTokenRequestProto$Builder.build(SecurityProtos.java:973)
> at org.apache.hadoop.mapreduce.v2.api.protocolrecords.impl.pb.GetDelegationTokenRequestPBImpl.mergeLocalToProto(GetDelegationTokenRequestPBImpl.java:84)
> at org.apache.hadoop.mapreduce.v2.api.protocolrecords.impl.pb.GetDelegationTokenRequestPBImpl.getProto(GetDelegationTokenRequestPBImpl.java:67)
> at org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getDelegationToken(MRClientProtocolPBClientImpl.java:200)
> at org.apache.hadoop.mapred.YARNRunner.getDelegationTokenFromHS(YARNRunner.java:194)
> at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:273)
> at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:392)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1439)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215)
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:581)
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:576)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1439)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:576)
> at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:723)
> ... 10 more
> 2013-03-15 13:34:16,555 WARN ActionStartXCommand:542 - USER[hue] GROUP[-] TOKEN[] APP[MapReduce] JOB[0000001-13031512313
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)