You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/04/26 05:39:04 UTC
[jira] [Commented] (FLINK-6376) when deploy flink cluster on the
yarn, it is lack of hdfs delegation token.
[ https://issues.apache.org/jira/browse/FLINK-6376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15984204#comment-15984204 ]
ASF GitHub Bot commented on FLINK-6376:
---------------------------------------
GitHub user Rucongzhang opened a pull request:
https://github.com/apache/flink/pull/3776
[FLINK-6376]when deploy flink cluster on the yarn, it is lack of hdfs delegation.
Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful description of your changes.
- [ ] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the JIRA id)
- [ ] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Rucongzhang/flink flink-6376
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3776.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3776
----
commit 256f519cda73571c73914c98b3c9ff4381520907
Author: z00376786 <zh...@huawei.com>
Date: 2017-04-26T03:36:43Z
when deploy flink cluster on the yarn, it is lack of hdfs delegation token
----
> when deploy flink cluster on the yarn, it is lack of hdfs delegation token.
> ---------------------------------------------------------------------------
>
> Key: FLINK-6376
> URL: https://issues.apache.org/jira/browse/FLINK-6376
> Project: Flink
> Issue Type: Bug
> Components: Security, YARN
> Reporter: zhangrucong1982
> Assignee: zhangrucong1982
>
> 1、I use the flink of version 1.2.0. And I deploy the flink cluster on the yarn. The hadoop version is 2.7.2.
> 2、I use flink in security model with the keytab and principal. And the key configuration is :security.kerberos.login.keytab: /home/ketab/test.keytab 、security.kerberos.login.principal: test.
> 3、The yarn configuration is default and enable the yarn log aggregation configuration" yarn.log-aggregation-enable : true";
> 4、 Deploying the flink cluster on the yarn, the yarn Node manager occur the following failure when aggregation the log in HDFS. The basic reason is lack of HDFS delegation token.
> java.io.IOException: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "SZV1000258954/10.162.181.24"; destination host is: "SZV1000258954":25000;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:796)
> at org.apache.hadoop.ipc.Client.call(Client.java:1515)
> at org.apache.hadoop.ipc.Client.call(Client.java:1447)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
> at com.sun.proxy.$Proxy26.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:802)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:201)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at com.sun.proxy.$Proxy27.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1919)
> at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1500)
> at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1496)
> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1496)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.checkExists(LogAggregationService.java:271)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.access$100(LogAggregationService.java:68)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService$1.run(LogAggregationService.java:299)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1769)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.createAppDir(LogAggregationService.java:284)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.initAppAggregator(LogAggregationService.java:390)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.initApp(LogAggregationService.java:342)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:470)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:68)
> at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:194)
> at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:120)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:722)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1769)
> at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:685)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:772)
> at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:394)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1564)
> at org.apache.hadoop.ipc.Client.call(Client.java:1486)
> ... 29 more
> Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
> at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:177)
> at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:404)
> at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:581)
> at org.apache.hadoop.ipc.Client$Connection.access$1900(Client.java:394)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:764)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:760)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1769)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:759)
> ... 32 more
> 5、the hadoop fix the hadoop issue 14116(https://issues.apache.org/jira/browse/HADOOP-14116), if there is no HDFS delegation token, it will try 20 times after sleeping 1 second. So it will cause the flink cluster deploy on yarn is very slowly, it will spent about 5 minutes to deploy the cluster with 2 taskmanagers.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)