You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "koodin9 (via GitHub)" <gi...@apache.org> on 2025/04/19 07:14:05 UTC
[PR] When calling the reconnect() method in HiveMetaStoreClient, if HADOOP_PROXY_USER is set, a delegation token must be issued for the proxy user. [hive]
koodin9 opened a new pull request, #5775:
URL: https://github.com/apache/hive/pull/5775
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Added logic within the `reconnect()` method to obtain a Delegation Token for the proxy user if the `HADOOP_PROXY_USER` environment variable is set.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
When the Hive Metastore is restarted, the `reconnect()` method of the `HiveMetaStoreClient` included in the Iceberg API fails (when `HADOOP_PROXY_USER` is configured without this change).
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducable example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
If no, write 'No'.
-->
Yes, this affects users who specifically set the `HADOOP_PROXY_USER` environment variable. (The behavior upon HMS restart will change from failure to success for them).
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Testing was conducted using Kafka Connect with the Iceberg Sink Connector to save data to an Iceberg table.
During the test, while write operations were in progress, the connected Hive Metastore was restarted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28913: When calling the reconnect() method in HiveMetaStoreClient, if HADOOP_PROXY_USER is set, a delegation token must be issued for the proxy user. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5775:
URL: https://github.com/apache/hive/pull/5775#discussion_r2058036404
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
##########
@@ -502,6 +506,9 @@ public void reconnect() throws MetaException {
// connection has died and the default connection is likely to be the first array element.
promoteRandomMetaStoreURI();
}
+
+ generateProxyUserDelegationToken();
Review Comment:
No idea, that would require some debugging. However, if `UserGroupInformation.getCurrentUser().getTokens()`. is empty, creating a new token seems to be the only option
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28913: When calling the reconnect() method in HiveMetaStoreClient, if HADOOP_PROXY_USER is set, a delegation token must be issued for the proxy user. [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5775:
URL: https://github.com/apache/hive/pull/5775#issuecomment-2826986269
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5775) **Quality Gate passed**
Issues
 [3 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5775&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5775&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5775&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5775&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5775&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5775)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28913: When calling the reconnect() method in HiveMetaStoreClient, if HADOOP_PROXY_USER is set, a delegation token must be issued for the proxy user. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5775:
URL: https://github.com/apache/hive/pull/5775#discussion_r2058038643
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
##########
@@ -502,6 +510,24 @@ public void reconnect() throws MetaException {
// connection has died and the default connection is likely to be the first array element.
promoteRandomMetaStoreURI();
}
+
+ String tokenSig = MetastoreConf.getVar(conf, ConfVars.TOKEN_SIGNATURE);
+ try {
+ tokenStrForm = SecurityUtils.getTokenStrForm(tokenSig);
Review Comment:
This seems redundant based on the findings from https://github.com/apache/hive/pull/5775#discussion_r2057615303 `getTokenStrForm` uses `UserGroupInformation.getCurrentUser().getTokens()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28913: When calling the reconnect() method in HiveMetaStoreClient, if HADOOP_PROXY_USER is set, a delegation token must be issued for the proxy user. [hive]
Posted by "koodin9 (via GitHub)" <gi...@apache.org>.
koodin9 commented on code in PR #5775:
URL: https://github.com/apache/hive/pull/5775#discussion_r2053576969
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
##########
@@ -502,6 +506,9 @@ public void reconnect() throws MetaException {
// connection has died and the default connection is likely to be the first array element.
promoteRandomMetaStoreURI();
}
+
+ generateProxyUserDelegationToken();
Review Comment:
@deniskuzZ
If the HiveMetaStore is restarted while a service is running with the HADOOP_PROXY_USER environment variable configured, the following error occurs:
```
[2025-04-19 15:45:48,446] INFO [45337-limtan-ib-g19-3|task-0] Trying to connect to metastore with URI thrift://koodin-test-metastore-1.com:9083 (hive.metastore:410)
[2025-04-19 15:45:48,452] ERROR [45337-limtan-ib-g19-3|task-0] SASL negotiation failure (org.apache.thrift.transport.TSaslTransport:278)
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at jdk.security.jgss/com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(Unknown Source)
at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:96)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:236)
at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:39)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at java.base/javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:494)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.reconnect(HiveMetaStoreClient.java:341)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:163)
at jdk.proxy4/jdk.proxy4.$Proxy188.getTable(Unknown Source)
at org.apache.iceberg.hive.HiveTableOperations.lambda$doRefresh$0(HiveTableOperations.java:146)
at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:58)
at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:51)
at org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:122)
at org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:146)
at org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:97)
at org.apache.iceberg.BaseTable.refresh(BaseTable.java:73)
at io.tabular.iceberg.connect.channel.Worker.write(Worker.java:133)
at io.tabular.iceberg.connect.channel.TaskImpl.put(TaskImpl.java:51)
at io.tabular.iceberg.connect.IcebergSinkTask.lambda$put$4(IcebergSinkTask.java:181)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at java.base/javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
at io.tabular.iceberg.connect.IcebergSinkTask.put(IcebergSinkTask.java:179)
at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:605)
at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:344)
at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:246)
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:215)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:225)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:280)
at org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:237)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at java.security.jgss/sun.security.jgss.krb5.Krb5InitCredential.getInstance(Unknown Source)
at java.security.jgss/sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Unknown Source)
at java.security.jgss/sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Unknown Source)
at java.security.jgss/sun.security.jgss.GSSManagerImpl.getMechanismContext(Unknown Source)
at java.security.jgss/sun.security.jgss.GSSContextImpl.initSecContext(Unknown Source)
at java.security.jgss/sun.security.jgss.GSSContextImpl.initSecContext(Unknown Source)
... 40 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28913: When calling the reconnect() method in HiveMetaStoreClient, if HADOOP_PROXY_USER is set, a delegation token must be issued for the proxy user. [hive]
Posted by "koodin9 (via GitHub)" <gi...@apache.org>.
koodin9 commented on PR #5775:
URL: https://github.com/apache/hive/pull/5775#issuecomment-2820150366
@prasanthj
Would you mind taking a look at this pull request? thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28913: When calling the reconnect() method in HiveMetaStoreClient, if HADOOP_PROXY_USER is set, a delegation token must be issued for the proxy user. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5775:
URL: https://github.com/apache/hive/pull/5775#discussion_r2056278593
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
##########
@@ -502,6 +506,9 @@ public void reconnect() throws MetaException {
// connection has died and the default connection is likely to be the first array element.
promoteRandomMetaStoreURI();
}
+
+ generateProxyUserDelegationToken();
Review Comment:
@koodin9, you can get a token like
````
String tokenSig = MetastoreConf.getVar(conf, ConfVars.TOKEN_SIGNATURE);
tokenStrForm = SecurityUtils.getTokenStrForm(tokenSig);
````
do you know if renew helps here?
````
if (tokenStrForm != null) {
renewDelegationToken(tokenStrForm);
}
````
I could only imagine that the issue is caused by the expired TGT and not the HMS service restart per se
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28913: When calling the reconnect() method in HiveMetaStoreClient, if HADOOP_PROXY_USER is set, a delegation token must be issued for the proxy user. [hive]
Posted by "koodin9 (via GitHub)" <gi...@apache.org>.
koodin9 commented on code in PR #5775:
URL: https://github.com/apache/hive/pull/5775#discussion_r2055467422
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
##########
@@ -502,6 +506,9 @@ public void reconnect() throws MetaException {
// connection has died and the default connection is likely to be the first array element.
promoteRandomMetaStoreURI();
}
+
+ generateProxyUserDelegationToken();
Review Comment:
[1c4b095](https://github.com/apache/hive/pull/5775/commits/1c4b095105cfcedc1d615adef7da7669d7f9417c)
@deniskuzZ
I modified the code to retrieve the tokens from the proxy user's UserGroupInformation (via getCurrentUser()) and return if a HIVE_DELEGATION_TOKEN exists.
However, I'm not sure how best to implement the part that fetches the HIVE_DELEGATION_TOKEN and checks if it's valid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28913: When calling the reconnect() method in HiveMetaStoreClient, if HADOOP_PROXY_USER is set, a delegation token must be issued for the proxy user. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5775:
URL: https://github.com/apache/hive/pull/5775#discussion_r2058038643
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
##########
@@ -502,6 +510,24 @@ public void reconnect() throws MetaException {
// connection has died and the default connection is likely to be the first array element.
promoteRandomMetaStoreURI();
}
+
+ String tokenSig = MetastoreConf.getVar(conf, ConfVars.TOKEN_SIGNATURE);
+ try {
+ tokenStrForm = SecurityUtils.getTokenStrForm(tokenSig);
Review Comment:
This seems redundant based on the findings from https://github.com/apache/hive/pull/5775#discussion_r2057615303
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28913: When calling the reconnect() method in HiveMetaStoreClient, if HADOOP_PROXY_USER is set, a delegation token must be issued for the proxy user. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5775:
URL: https://github.com/apache/hive/pull/5775#discussion_r2058381508
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
##########
@@ -502,6 +510,24 @@ public void reconnect() throws MetaException {
// connection has died and the default connection is likely to be the first array element.
promoteRandomMetaStoreURI();
}
+
+ String tokenSig = MetastoreConf.getVar(conf, ConfVars.TOKEN_SIGNATURE);
+ try {
+ tokenStrForm = SecurityUtils.getTokenStrForm(tokenSig);
Review Comment:
based on your own findings, the issue seems not with the expired token, but with the fact that token is gone, so there is nothing to renew.
It would be great if you could debug it further to understand why UGI service token is lost, but [1d9d9c2](https://github.com/apache/hive/pull/5775/commits/1d9d9c2a6a55a950893624e1a6fb5ed93cce8a34) should do it for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28913: When calling the reconnect() method in HiveMetaStoreClient, if HADOOP_PROXY_USER is set, a delegation token must be issued for the proxy user. [hive]
Posted by "koodin9 (via GitHub)" <gi...@apache.org>.
koodin9 commented on code in PR #5775:
URL: https://github.com/apache/hive/pull/5775#discussion_r2057615303
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
##########
@@ -502,6 +506,9 @@ public void reconnect() throws MetaException {
// connection has died and the default connection is likely to be the first array element.
promoteRandomMetaStoreURI();
}
+
+ generateProxyUserDelegationToken();
Review Comment:
@deniskuzZ
It seems the process of generating a new token is definitely necessary.
When the HiveMetaStoreClient constructor created the Delegation Token for the proxy user, the result of calling UserGroupInformation.getCurrentUser().getTokens() correctly contained the token, as expected.
```
[2025-04-24 14:01:33,413] INFO [45337-limtan-ib-g19-3|task-0] finish creating DT. current User(koodin-9 (auth:PROXY) via superduper@KERBEROS.HADOOP (auth:KERBEROS)) getTokens: [Kind: HIVE_DELEGATION_TOKEN, Service: DelegationTokenForHiveMetaStoreServer, Ident: ...] (hive.metastore:334)
```
However, when checking by calling UserGroupInformation.getCurrentUser().getTokens() at the point reconnect() was invoked, it always showed an empty value.
In the 3f5a258, logging was added to the SecurityUtils.getTokenStrForm function, and when it was called, the results were as follows.
```
public static String getTokenStrForm(String tokenSignature) throws IOException {
UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
TokenSelector<? extends TokenIdentifier> tokenSelector = new DelegationTokenSelector();
LOG.info("ok, ugi getTokens? : {}", ugi.getTokens());
LOG.info("ok, ugi realUser getTokens? : {}", ugi.getRealUser().getTokens());
Token<? extends TokenIdentifier> token = tokenSelector.selectToken(
tokenSignature == null ? new Text() : new Text(tokenSignature), ugi.getTokens());
return token != null ? token.encodeToUrlString() : null;
}
```
```
[2025-04-24 14:32:51,577] INFO [45337-limtan-ib-g19-3|task-2] ok, ugi getTokens? : [] (org.apache.hadoop.hive.metastore.utils.SecurityUtils:159)
[2025-04-24 14:32:51,578] INFO [45337-limtan-ib-g19-3|task-2] ok, ugi realUser getTokens? : [] (org.apache.hadoop.hive.metastore.utils.SecurityUtils:160)
```
I am not sure why the token information in UserGroupInformation.getCurrentUser().getTokens() disappears after the Hive Metastore restarts. Would you happen to know the reason for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28913: When calling the reconnect() method in HiveMetaStoreClient, if HADOOP_PROXY_USER is set, a delegation token must be issued for the proxy user. [hive]
Posted by "koodin9 (via GitHub)" <gi...@apache.org>.
koodin9 commented on code in PR #5775:
URL: https://github.com/apache/hive/pull/5775#discussion_r2058176831
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
##########
@@ -502,6 +510,24 @@ public void reconnect() throws MetaException {
// connection has died and the default connection is likely to be the first array element.
promoteRandomMetaStoreURI();
}
+
+ String tokenSig = MetastoreConf.getVar(conf, ConfVars.TOKEN_SIGNATURE);
+ try {
+ tokenStrForm = SecurityUtils.getTokenStrForm(tokenSig);
Review Comment:
Do you mean I should reset the branch to commit [1d9d9c2](https://github.com/apache/hive/pull/5775/commits/1d9d9c2a6a55a950893624e1a6fb5ed93cce8a34)?
As @dengzhhu653 mentioned, if the cost of issuing a new delegation token is similar to the cost of checking if the token is valid, then resetting to [1d9d9c2](https://github.com/apache/hive/pull/5775/commits/1d9d9c2a6a55a950893624e1a6fb5ed93cce8a34) seems like a good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28913: When calling the reconnect() method in HiveMetaStoreClient, if HADOOP_PROXY_USER is set, a delegation token must be issued for the proxy user. [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5775:
URL: https://github.com/apache/hive/pull/5775#issuecomment-2828817108
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5775) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5775&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5775&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5775&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5775&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5775&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5775)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28913: When calling the reconnect() method in HiveMetaStoreClient, if HADOOP_PROXY_USER is set, a delegation token must be issued for the proxy user. [hive]
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 merged PR #5775:
URL: https://github.com/apache/hive/pull/5775
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28913: When calling the reconnect() method in HiveMetaStoreClient, if HADOOP_PROXY_USER is set, a delegation token must be issued for the proxy user. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5775:
URL: https://github.com/apache/hive/pull/5775#discussion_r2053558283
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
##########
@@ -502,6 +506,9 @@ public void reconnect() throws MetaException {
// connection has died and the default connection is likely to be the first array element.
promoteRandomMetaStoreURI();
}
+
+ generateProxyUserDelegationToken();
Review Comment:
This way we'll be requesting a new delegation token on every `reconnect()` attempt. ATM delegation token is created in `HiveMetaStoreClient` constructor.
What exception is being thrown? Do we need a new delegation token?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] When calling the reconnect() method in HiveMetaStoreClient, if HADOOP_PROXY_USER is set, a delegation token must be issued for the proxy user. [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5775:
URL: https://github.com/apache/hive/pull/5775#issuecomment-2817607990
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5775) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5775&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5775&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5775&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5775&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5775&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5775)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28913: When calling the reconnect() method in HiveMetaStoreClient, if HADOOP_PROXY_USER is set, a delegation token must be issued for the proxy user. [hive]
Posted by "koodin9 (via GitHub)" <gi...@apache.org>.
koodin9 commented on code in PR #5775:
URL: https://github.com/apache/hive/pull/5775#discussion_r2058648309
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
##########
@@ -502,6 +510,24 @@ public void reconnect() throws MetaException {
// connection has died and the default connection is likely to be the first array element.
promoteRandomMetaStoreURI();
}
+
+ String tokenSig = MetastoreConf.getVar(conf, ConfVars.TOKEN_SIGNATURE);
+ try {
+ tokenStrForm = SecurityUtils.getTokenStrForm(tokenSig);
Review Comment:
[8cd5b36](https://github.com/apache/hive/pull/5775/commits/8cd5b36f19e66a73cb6d5a6b0e070779e1e420b3)
I reverted the code to the initial commit state. Additionally, I moved the close statement to a finally block.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28913: When calling the reconnect() method in HiveMetaStoreClient, if HADOOP_PROXY_USER is set, a delegation token must be issued for the proxy user. [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5775:
URL: https://github.com/apache/hive/pull/5775#issuecomment-2823616843
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5775) **Quality Gate passed**
Issues
 [2 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5775&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5775&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5775&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5775&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5775&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5775)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28913: When calling the reconnect() method in HiveMetaStoreClient, if HADOOP_PROXY_USER is set, a delegation token must be issued for the proxy user. [hive]
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #5775:
URL: https://github.com/apache/hive/pull/5775#discussion_r2057423009
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
##########
@@ -389,6 +359,52 @@ private void resolveUris() throws MetaException {
}
}
+ private void generateProxyUserDelegationToken() throws MetaException {
+ //If HADOOP_PROXY_USER is set in env or property,
+ //then need to create metastore client that proxies as that user.
+ String HADOOP_PROXY_USER = "HADOOP_PROXY_USER";
+ String proxyUser = System.getenv(HADOOP_PROXY_USER);
+ if (proxyUser == null) {
+ proxyUser = System.getProperty(HADOOP_PROXY_USER);
+ }
+ //if HADOOP_PROXY_USER is set, create DelegationToken using real user
+ if (proxyUser != null) {
+ LOG.info(HADOOP_PROXY_USER + " is set. Using delegation "
+ + "token for HiveMetaStore connection.");
+ try {
+ UserGroupInformation.getRealUserOrSelf(UserGroupInformation.getLoginUser()).doAs(
+ new PrivilegedExceptionAction<Void>() {
+ @Override
+ public Void run() throws Exception {
+ open();
+ return null;
+ }
+ });
+ Collection<Token<? extends TokenIdentifier>> tokens = UserGroupInformation.getCurrentUser().getTokens();
+ for (Token<? extends TokenIdentifier> token : tokens) {
+ if (token.getKind().equals(DelegationTokenIdentifier.HIVE_DELEGATION_KIND)) {
+ // We have a delegation token for current user, so we can use it to connect to the metastore
+ LOG.info("Found delegation token for " + proxyUser);
+ return;
+ }
+ }
+
+ String delegationTokenPropString = "DelegationTokenForHiveMetaStoreServer";
+ String delegationTokenStr = getDelegationToken(proxyUser, proxyUser);
+ SecurityUtils.setTokenStr(UserGroupInformation.getCurrentUser(), delegationTokenStr,
+ delegationTokenPropString);
+ MetastoreConf.setVar(this.conf, ConfVars.TOKEN_SIGNATURE, delegationTokenPropString);
+ close();
Review Comment:
move this `close` to `finally`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28913: When calling the reconnect() method in HiveMetaStoreClient, if HADOOP_PROXY_USER is set, a delegation token must be issued for the proxy user. [hive]
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #5775:
URL: https://github.com/apache/hive/pull/5775#discussion_r2057421276
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
##########
@@ -502,6 +506,9 @@ public void reconnect() throws MetaException {
// connection has died and the default connection is likely to be the first array element.
promoteRandomMetaStoreURI();
}
+
+ generateProxyUserDelegationToken();
Review Comment:
For me it's reasonable to get a new token every time we reconnect, as this is the same cost as connecting to HMS and check that token(and a call to get a new token if expired then).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] When calling the reconnect() method in HiveMetaStoreClient, if HADOOP_PROXY_USER is set, a delegation token must be issued for the proxy user. [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5775:
URL: https://github.com/apache/hive/pull/5775#issuecomment-2816599786
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5775) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5775&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5775&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5775&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5775&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5775&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5775)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28913: When calling the reconnect() method in HiveMetaStoreClient, if HADOOP_PROXY_USER is set, a delegation token must be issued for the proxy user. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5775:
URL: https://github.com/apache/hive/pull/5775#discussion_r2058022815
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
##########
@@ -502,6 +506,9 @@ public void reconnect() throws MetaException {
// connection has died and the default connection is likely to be the first array element.
promoteRandomMetaStoreURI();
}
+
+ generateProxyUserDelegationToken();
Review Comment:
@koodin9, could you please check if only the UGI is missing the token, but it's still present in conf?
````
String tokenSig = MetastoreConf.getVar(conf, ConfVars.TOKEN_SIGNATURE);
tokenStrForm = SecurityUtils.getTokenStrForm(tokenSig);
if (tokenStrForm != null) {
SecurityUtils.setTokenStr(UserGroupInformation.getCurrentUser(), delegationTokenStr,
delegationTokenPropString);
MetastoreConf.setVar(this.conf, ConfVars.TOKEN_SIGNATURE, delegationTokenPropString);
}
````
PS: I don't think it's reasonable to create a new delegation token on reconnect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28913: When calling the reconnect() method in HiveMetaStoreClient, if HADOOP_PROXY_USER is set, a delegation token must be issued for the proxy user. [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5775:
URL: https://github.com/apache/hive/pull/5775#issuecomment-2826392201
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5775) **Quality Gate passed**
Issues
 [2 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5775&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5775&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5775&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5775&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5775&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5775)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28913: When calling the reconnect() method in HiveMetaStoreClient, if HADOOP_PROXY_USER is set, a delegation token must be issued for the proxy user. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5775:
URL: https://github.com/apache/hive/pull/5775#discussion_r2053558283
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
##########
@@ -502,6 +506,9 @@ public void reconnect() throws MetaException {
// connection has died and the default connection is likely to be the first array element.
promoteRandomMetaStoreURI();
}
+
+ generateProxyUserDelegationToken();
Review Comment:
This way we'll be requesting a new delegation token on every `reconnect()` attempt. ATM delegation token is created in `HiveMetaStoreClient` constructor.
What exception is being thrown?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28913: When calling the reconnect() method in HiveMetaStoreClient, if HADOOP_PROXY_USER is set, a delegation token must be issued for the proxy user. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5775:
URL: https://github.com/apache/hive/pull/5775#discussion_r2056278593
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
##########
@@ -502,6 +506,9 @@ public void reconnect() throws MetaException {
// connection has died and the default connection is likely to be the first array element.
promoteRandomMetaStoreURI();
}
+
+ generateProxyUserDelegationToken();
Review Comment:
@koodin9, you can get a token like
````
String tokenSig = MetastoreConf.getVar(conf, ConfVars.TOKEN_SIGNATURE);
tokenStrForm = SecurityUtils.getTokenStrForm(tokenSig);
````
However, what I was trying to say is that a new token should be requested only in case of GSSException. After the HMS restart client token is no longer valid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28913: When calling the reconnect() method in HiveMetaStoreClient, if HADOOP_PROXY_USER is set, a delegation token must be issued for the proxy user. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5775:
URL: https://github.com/apache/hive/pull/5775#discussion_r2056278593
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
##########
@@ -502,6 +506,9 @@ public void reconnect() throws MetaException {
// connection has died and the default connection is likely to be the first array element.
promoteRandomMetaStoreURI();
}
+
+ generateProxyUserDelegationToken();
Review Comment:
@koodin9, you can get a token like
````
String tokenSig = MetastoreConf.getVar(conf, ConfVars.TOKEN_SIGNATURE);
tokenStrForm = SecurityUtils.getTokenStrForm(tokenSig);
````
do you know if renew helps here?
````
if (tokenStrForm != null) {
renewDelegationToken(tokenStrForm);
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28913: When calling the reconnect() method in HiveMetaStoreClient, if HADOOP_PROXY_USER is set, a delegation token must be issued for the proxy user. [hive]
Posted by "koodin9 (via GitHub)" <gi...@apache.org>.
koodin9 commented on code in PR #5775:
URL: https://github.com/apache/hive/pull/5775#discussion_r2053576969
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
##########
@@ -502,6 +506,9 @@ public void reconnect() throws MetaException {
// connection has died and the default connection is likely to be the first array element.
promoteRandomMetaStoreURI();
}
+
+ generateProxyUserDelegationToken();
Review Comment:
@deniskuzZ
The error stack trace below was generated from Hive version 2.3.x. However, it appears the same issue likely exists in the master branch as well, so I've created this Pull Request.
If the HiveMetaStore is restarted while a service is running with the HADOOP_PROXY_USER environment variable configured, the following error occurs.
```
[2025-04-19 15:45:48,446] INFO [45337-limtan-ib-g19-3|task-0] Trying to connect to metastore with URI thrift://koodin-test-metastore-1.com:9083 (hive.metastore:410)
[2025-04-19 15:45:48,452] ERROR [45337-limtan-ib-g19-3|task-0] SASL negotiation failure (org.apache.thrift.transport.TSaslTransport:278)
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at jdk.security.jgss/com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(Unknown Source)
at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:96)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:236)
at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:39)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at java.base/javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:494)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.reconnect(HiveMetaStoreClient.java:341)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:163)
at jdk.proxy4/jdk.proxy4.$Proxy188.getTable(Unknown Source)
at org.apache.iceberg.hive.HiveTableOperations.lambda$doRefresh$0(HiveTableOperations.java:146)
at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:58)
at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:51)
at org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:122)
at org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:146)
at org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:97)
at org.apache.iceberg.BaseTable.refresh(BaseTable.java:73)
at io.tabular.iceberg.connect.channel.Worker.write(Worker.java:133)
at io.tabular.iceberg.connect.channel.TaskImpl.put(TaskImpl.java:51)
at io.tabular.iceberg.connect.IcebergSinkTask.lambda$put$4(IcebergSinkTask.java:181)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at java.base/javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
at io.tabular.iceberg.connect.IcebergSinkTask.put(IcebergSinkTask.java:179)
at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:605)
at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:344)
at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:246)
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:215)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:225)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:280)
at org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:237)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at java.security.jgss/sun.security.jgss.krb5.Krb5InitCredential.getInstance(Unknown Source)
at java.security.jgss/sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Unknown Source)
at java.security.jgss/sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Unknown Source)
at java.security.jgss/sun.security.jgss.GSSManagerImpl.getMechanismContext(Unknown Source)
at java.security.jgss/sun.security.jgss.GSSContextImpl.initSecContext(Unknown Source)
at java.security.jgss/sun.security.jgss.GSSContextImpl.initSecContext(Unknown Source)
... 40 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28913: When calling the reconnect() method in HiveMetaStoreClient, if HADOOP_PROXY_USER is set, a delegation token must be issued for the proxy user. [hive]
Posted by "koodin9 (via GitHub)" <gi...@apache.org>.
koodin9 commented on code in PR #5775:
URL: https://github.com/apache/hive/pull/5775#discussion_r2057446098
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
##########
@@ -389,6 +359,52 @@ private void resolveUris() throws MetaException {
}
}
+ private void generateProxyUserDelegationToken() throws MetaException {
+ //If HADOOP_PROXY_USER is set in env or property,
+ //then need to create metastore client that proxies as that user.
+ String HADOOP_PROXY_USER = "HADOOP_PROXY_USER";
+ String proxyUser = System.getenv(HADOOP_PROXY_USER);
+ if (proxyUser == null) {
+ proxyUser = System.getProperty(HADOOP_PROXY_USER);
+ }
+ //if HADOOP_PROXY_USER is set, create DelegationToken using real user
+ if (proxyUser != null) {
+ LOG.info(HADOOP_PROXY_USER + " is set. Using delegation "
+ + "token for HiveMetaStore connection.");
+ try {
+ UserGroupInformation.getRealUserOrSelf(UserGroupInformation.getLoginUser()).doAs(
+ new PrivilegedExceptionAction<Void>() {
+ @Override
+ public Void run() throws Exception {
+ open();
+ return null;
+ }
+ });
+ Collection<Token<? extends TokenIdentifier>> tokens = UserGroupInformation.getCurrentUser().getTokens();
+ for (Token<? extends TokenIdentifier> token : tokens) {
+ if (token.getKind().equals(DelegationTokenIdentifier.HIVE_DELEGATION_KIND)) {
+ // We have a delegation token for current user, so we can use it to connect to the metastore
+ LOG.info("Found delegation token for " + proxyUser);
+ return;
+ }
+ }
+
+ String delegationTokenPropString = "DelegationTokenForHiveMetaStoreServer";
+ String delegationTokenStr = getDelegationToken(proxyUser, proxyUser);
+ SecurityUtils.setTokenStr(UserGroupInformation.getCurrentUser(), delegationTokenStr,
+ delegationTokenPropString);
+ MetastoreConf.setVar(this.conf, ConfVars.TOKEN_SIGNATURE, delegationTokenPropString);
+ close();
Review Comment:
[c1d5ce3](https://github.com/apache/hive/pull/5775/commits/c1d5ce37f3a8fabd1d2e499141ba6e7294164629)
Thank you for reviewing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28913: When calling the reconnect() method in HiveMetaStoreClient, if HADOOP_PROXY_USER is set, a delegation token must be issued for the proxy user. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5775:
URL: https://github.com/apache/hive/pull/5775#discussion_r2058022815
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
##########
@@ -502,6 +506,9 @@ public void reconnect() throws MetaException {
// connection has died and the default connection is likely to be the first array element.
promoteRandomMetaStoreURI();
}
+
+ generateProxyUserDelegationToken();
Review Comment:
@koodin9, could you please check if only the UGI is missing the token, but it's still present in conf?
````
String tokenSig = MetastoreConf.getVar(conf, ConfVars.TOKEN_SIGNATURE);
tokenStrForm = SecurityUtils.getTokenStrForm(tokenSig);
if (tokenStrForm != null) {
SecurityUtils.setTokenStr(UserGroupInformation.getCurrentUser(), delegationTokenStr,
delegationTokenPropString);
MetastoreConf.setVar(this.conf, ConfVars.TOKEN_SIGNATURE, delegationTokenPropString);
}
````
PS: I don't think it's reasonable to create a new delegation token on reconnect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28913: When calling the reconnect() method in HiveMetaStoreClient, if HADOOP_PROXY_USER is set, a delegation token must be issued for the proxy user. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5775:
URL: https://github.com/apache/hive/pull/5775#discussion_r2053823396
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
##########
@@ -502,6 +506,9 @@ public void reconnect() throws MetaException {
// connection has died and the default connection is likely to be the first array element.
promoteRandomMetaStoreURI();
}
+
+ generateProxyUserDelegationToken();
Review Comment:
Could we request a new token only when the current one is invalid?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28913: When calling the reconnect() method in HiveMetaStoreClient, if HADOOP_PROXY_USER is set, a delegation token must be issued for the proxy user. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5775:
URL: https://github.com/apache/hive/pull/5775#discussion_r2056278593
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
##########
@@ -502,6 +506,9 @@ public void reconnect() throws MetaException {
// connection has died and the default connection is likely to be the first array element.
promoteRandomMetaStoreURI();
}
+
+ generateProxyUserDelegationToken();
Review Comment:
@koodin9, you can get a token like
````
String tokenSig = MetastoreConf.getVar(conf, ConfVars.TOKEN_SIGNATURE);
tokenStrForm = SecurityUtils.getTokenStrForm(tokenSig);
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28913: When calling the reconnect() method in HiveMetaStoreClient, if HADOOP_PROXY_USER is set, a delegation token must be issued for the proxy user. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5775:
URL: https://github.com/apache/hive/pull/5775#discussion_r2058381508
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
##########
@@ -502,6 +510,24 @@ public void reconnect() throws MetaException {
// connection has died and the default connection is likely to be the first array element.
promoteRandomMetaStoreURI();
}
+
+ String tokenSig = MetastoreConf.getVar(conf, ConfVars.TOKEN_SIGNATURE);
+ try {
+ tokenStrForm = SecurityUtils.getTokenStrForm(tokenSig);
Review Comment:
based on your own findings, the issue seems to be not with the expired token, but with the fact that the is no service token - it's gone, so there is nothing to renew.
It would be great if you could debug it further to understand why UGI service token is lost, but [1d9d9c2](https://github.com/apache/hive/pull/5775/commits/1d9d9c2a6a55a950893624e1a6fb5ed93cce8a34) should do it for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org