You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by qq <98...@qq.com> on 2020/07/17 03:41:11 UTC
Oozie failed to run shell task under kerberos authentication environment
Hello:
The following error occurred while running the shell task using Oozie:
W] ACTION[0000001-200717100610211-oozie-root-W@shell-node] Adding default credentials for action: hdfs, yarn and jhs
2020-07-17 10:12:58,520 INFO ShellActionExecutor:520 - SERVER[master] USER[oozie] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-200717100610211-oozie-root-W] ACTION[0000001-200717100610211-oozie-root-W@shell-node] Adding default credentials for action: hdfs, yarn and jhs
2020-07-17 10:12:58,522 INFO deprecation:1395 - SERVER[master] fs.default.name is deprecated. Instead, use fs.defaultFS
2020-07-17 10:12:58,522 INFO deprecation:1395 - SERVER[master] fs.default.name is deprecated. Instead, use fs.defaultFS
2020-07-17 10:12:58,523 WARN HadoopTokenHelper:523 - SERVER[master] USER[oozie] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-200717100610211-oozie-root-W] ACTION[0000001-200717100610211-oozie-root-W@shell-node] An error happened while trying to get server principal. Getting it from service principal anyway.
2020-07-17 10:12:58,523 WARN HadoopTokenHelper:523 - SERVER[master] USER[oozie] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-200717100610211-oozie-root-W] ACTION[0000001-200717100610211-oozie-root-W@shell-node] An error happened while trying to get server principal. Getting it from service principal anyway.
2020-07-17 10:12:58,523 INFO HadoopTokenHelper:520 - SERVER[master] USER[oozie] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-200717100610211-oozie-root-W] ACTION[0000001-200717100610211-oozie-root-W@shell-node] Delegation Token Renewer for yarncluster is yarn
2020-07-17 10:12:58,523 INFO HadoopTokenHelper:520 - SERVER[master] USER[oozie] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-200717100610211-oozie-root-W] ACTION[0000001-200717100610211-oozie-root-W@shell-node] Delegation Token Renewer for yarncluster is yarn
2020-07-17 10:12:58,525 WARN Client:765 - SERVER[master] Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
2020-07-17 10:12:58,525 WARN Client:765 - SERVER[master] Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
2020-07-17 10:12:58,525 WARN ActionStartXCommand:523 - SERVER[master] USER[oozie] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-200717100610211-oozie-root-W] ACTION[0000001-200717100610211-oozie-root-W@shell-node] Error starting action [shell-node]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: DestHost:destPort master:10020 , LocalHost:localPort master/28.11.1.14:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]]
org.apache.oozie.action.ActionExecutorException: JA009: DestHost:destPort master:10020 , LocalHost:localPort master/28.11.1.14:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:463)
at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:443)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1134)
at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1644)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:243)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:68)
at org.apache.oozie.command.XCommand.call(XCommand.java:290)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:363)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:210)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: DestHost:destPort master:10020 , LocalHost:localPort master/28.11.1.14:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:833)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:808)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1549)
at org.apache.hadoop.ipc.Client.call(Client.java:1491)
at org.apache.hadoop.ipc.Client.call(Client.java:1388)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
at com.sun.proxy.$Proxy42.getDelegationToken(Unknown Source)
at org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getDelegationToken(MRClientProtocolPBClientImpl.java:211)
at org.apache.oozie.action.hadoop.JHSCredentials.getDelegationTokenFromJHS(JHSCredentials.java:88)
at org.apache.oozie.action.hadoop.JHSCredentials.updateCredentials(JHSCredentials.java:65)
at org.apache.oozie.action.hadoop.JavaActionExecutor.setCredentialTokens(JavaActionExecutor.java:1546)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1082)
... 11 more
Caused by: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:770)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:733)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:827)
at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:421)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1606)
at org.apache.hadoop.ipc.Client.call(Client.java:1435)
... 20 more
Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:173)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:627)
at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:421)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:814)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:810)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:810)
... 23 more
2020-07-17 10:12:58,525 WARN ActionStartXCommand:523 - SERVER[master] USER[oozie] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-200717100610211-oozie-root-W] ACTION[0000001-200717100610211-oozie-root-W@shell-node] Error starting action [shell-node]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: DestHost:destPort master:10020 , LocalHost:localPort master/28.11.1.14:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]]
org.apache.oozie.action.ActionExecutorException: JA009: DestHost:destPort master:10020 , LocalHost:localPort master/28.11.1.14:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:463)
at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:443)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1134)
at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1644)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:243)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:68)
at org.apache.oozie.command.XCommand.call(XCommand.java:290)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:363)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:210)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: DestHost:destPort master:10020 , LocalHost:localPort master/28.11.1.14:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:833)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:808)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1549)
at org.apache.hadoop.ipc.Client.call(Client.java:1491)
at org.apache.hadoop.ipc.Client.call(Client.java:1388)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
at com.sun.proxy.$Proxy42.getDelegationToken(Unknown Source)
at org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getDelegationToken(MRClientProtocolPBClientImpl.java:211)
at org.apache.oozie.action.hadoop.JHSCredentials.getDelegationTokenFromJHS(JHSCredentials.java:88)
at org.apache.oozie.action.hadoop.JHSCredentials.updateCredentials(JHSCredentials.java:65)
at org.apache.oozie.action.hadoop.JavaActionExecutor.setCredentialTokens(JavaActionExecutor.java:1546)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1082)
... 11 more
Caused by: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:770)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:733)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:827)
at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:421)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1606)
at org.apache.hadoop.ipc.Client.call(Client.java:1435)
... 20 more
Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:173)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:627)
at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:421)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:814)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:810)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:810)
... 23 more
2020-07-17 10:12:58,527 INFO ActionStartXCommand:520 - SERVER[master] USER[oozie] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-200717100610211-oozie-root-W] ACTION[0000001-200717100610211-oozie-root-W@shell-node] Next Retry, Attempt Number [1] in [10,000] milliseconds
2020-07-17 10:12:58,527 INFO ActionStartXCommand:520 - SERVER[master] USER[oozie] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000001-200717100610211-oozie-root-W] ACTION[0000001-200717100610211-oozie-root-W@shell-node] Next Retry, Attempt Number [1] in [10,000] milliseconds
The cluster environment information is as follows:
oozie version is 5.2.0
hadoop version is 3.2.1
the authentication mode is kerberos.
This problem also occurs when running other types of tasks, such as MapReduce.
Can anyone tell me other ways to solve this problem?
thinks.
I am looking forward to your reply！