You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Till Rohrmann (Jira)" <ji...@apache.org> on 2021/01/08 08:49:00 UTC
[jira] [Comment Edited] (FLINK-20714) Hive delegation token is not
obtained when using `kinit` to submit Yarn per-job
[ https://issues.apache.org/jira/browse/FLINK-20714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17261132#comment-17261132 ]
Till Rohrmann edited comment on FLINK-20714 at 1/8/21, 8:48 AM:
----------------------------------------------------------------
Thanks for reporting this issue [~jackwangcs]. It looks indeed as if this is a bug. [~lirui] did we ever try the Hive connector with Kerberos enabled? [~jackwangcs] do you want to try to solve this problem?
was (Author: till.rohrmann):
Thanks for reporting this issue [~jackwangcs]. It looks indeed as if this is a bug. [~lirui] did we ever try the Hive connector with Kerberos enabled? [~xintongsong] could someone from your team help with this issue?
> Hive delegation token is not obtained when using `kinit` to submit Yarn per-job
> --------------------------------------------------------------------------------
>
> Key: FLINK-20714
> URL: https://issues.apache.org/jira/browse/FLINK-20714
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
> Affects Versions: 1.12.0, 1.11.2, 1.11.3
> Environment: Flink 1.11.2 on Yarn
> Reporter: jackwangcs
> Priority: Critical
> Labels: keberos
> Fix For: 1.13.0, 1.11.4, 1.12.1
>
>
> Hive delegation token is not obtained when using `kinit` to submit Yarn per-job.
> In YarnClusterDescriptor, it calls org.apache.flink.yarn.Utils#setTokensFor to obtain tokens for the job. But setTokensFor only obtains HDFS and HBase tokens currently, since the Hive integration is supported, the Hive delegation should be obtained also.
> Otherwise, it will throw the following error when it tries to connect to Hive metastore:
> {code:java}
> Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: GSS initiate failedCaused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: GSS initiate failed at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316) at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:464) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:244) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:187) at org.apache.flink.table.catalog.hive.client.HiveShimV100.getHiveMetastoreClient(HiveShimV100.java:97) at org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.createMetastoreClient(HiveMetastoreClientWrapper.java:240) at org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.<init>(HiveMetastoreClientWrapper.java:71) at org.apache.flink.table.catalog.hive.client.HiveMetastoreClientFactory.create(HiveMetastoreClientFactory.java:35) at org.apache.flink.connectors.hive.HiveTableMetaStoreFactory$HiveTableMetaStore.<init>(HiveTableMetaStoreFactory.java:74) at org.apache.flink.connectors.hive.HiveTableMetaStoreFactory$HiveTableMetaStore.<init>(HiveTableMetaStoreFactory.java:68) at org.apache.flink.connectors.hive.HiveTableMetaStoreFactory.createTableMetaStore(HiveTableMetaStoreFactory.java:65) at org.apache.flink.connectors.hive.HiveTableMetaStoreFactory.createTableMetaStore(HiveTableMetaStoreFactory.java:43) at org.apache.flink.table.filesystem.PartitionLoader.<init>(PartitionLoader.java:61) at org.apache.flink.table.filesystem.FileSystemCommitter.commitUpToCheckpoint(FileSystemCommitter.java:97) at org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:95) {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)