You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "Sunwoo-Shin (via GitHub)" <gi...@apache.org> on 2026/06/18 13:31:02 UTC
[PR] [KYUUBI #7433][SPARK] Preserve Hive delegation tokens with non-empty service [kyuubi]
Sunwoo-Shin opened a new pull request, #7518:
URL: https://github.com/apache/kyuubi/pull/7518
### Why are the changes needed?
Closes #7433.
`SparkTBinaryFrontendService#addHiveToken` keeps only the Hive delegation token whose `service` field is empty (the one `HiveMetaStoreClient` selects by default) and silently drops every Hive token whose `service` is non-empty.
A Hive delegation token gets a non-empty `service` when it is bound to a specific metastore via `hive.metastore.token.signature` (the signature is stored in the token service). When an engine talks to multiple Hive metastores that use different Kerberos principals — e.g. two Iceberg catalogs, each backed by its own HMS — each metastore produces its own signature-bound token. Because these tokens are dropped before reaching the engine UGI, the engine fails to authenticate against the non-default metastore with `DIGEST-MD5: IO error acquiring password`.
This is the engine-side counterpart of #1091 (renewing delegation tokens for multiple Hive metastore clusters): even when the server pushes per-metastore tokens, the engine drops them.
Affected versions: 1.11.1.
This change partitions the incoming Hive tokens by their `service` field:
- Tokens with a non-empty `service` are added to the engine credentials keyed by their alias, reusing the same issue-date downgrade guard the default path already applies.
- The existing single-metastore URI matching now runs only over the default (empty-`service`) tokens, so behavior for the common single-HMS case is unchanged.
- The `No matching Hive token found ...` warning is emitted only when a default-`service` token was actually expected, to avoid noise in metastore deployments that rely solely on signature-bound tokens.
### How was this patch tested?
Added unit tests in `SparkTBinaryFrontendServiceSuite` (the token-merging logic was extracted into `mergeHiveTokens` so it can be exercised without a `SparkContext`):
- signature-bound tokens for multiple metastores are preserved, keyed by alias;
- a signature-bound token with an earlier issue date is ignored, and a later one replaces the existing token;
- the existing single-metastore matching for default-`service` tokens is unchanged;
- signature-bound tokens are added without disturbing the default-`service` path.
```
build/mvn test -pl externals/kyuubi-spark-sql-engine -am \
-Dtest=none \
-DwildcardSuites=org.apache.kyuubi.engine.spark.SparkTBinaryFrontendServiceSuite
```
### Was this patch authored or co-authored using generative AI tooling?
Assisted-by: Claude:claude-opus-4-8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7433][SPARK] Preserve Hive delegation tokens with non-empty service [kyuubi]
Posted by "Sunwoo-Shin (via GitHub)" <gi...@apache.org>.
Sunwoo-Shin commented on PR #7518:
URL: https://github.com/apache/kyuubi/pull/7518#issuecomment-4828427522
Thanks for the review @pan3793!
Yes — besides the UTs, this has been running on a production Kerberized cluster for a while now.
For context on our setup: the built-in [HiveDelegationTokenProvider](https://github.com/apache/kyuubi/blob/master/kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HiveDelegationTokenProvider.scala) only supports a single Hive metastore, so we additionally implemented a custom provider through the DelegationTokenProvider SPI that fetches delegation tokens from multiple Hive metastores. Combined with this patch — which preserves the tokens carrying a non-empty service field — the per-metastore (signature-bound) tokens are now propagated correctly to the executors, while the single-metastore path keeps working as before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7433][SPARK] Preserve Hive delegation tokens with non-empty service [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #7518:
URL: https://github.com/apache/kyuubi/pull/7518#issuecomment-4799601206
Code LGTM, but as you know, the Kerberos/UGI/DT part is always tricky stuff, have you tested it in a real cluster in addition to the UT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7433][SPARK] Preserve Hive delegation tokens with non-empty service [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed pull request #7518: [KYUUBI #7433][SPARK] Preserve Hive delegation tokens with non-empty service
URL: https://github.com/apache/kyuubi/pull/7518
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7433][SPARK] Preserve Hive delegation tokens with non-empty service [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #7518:
URL: https://github.com/apache/kyuubi/pull/7518#issuecomment-4829323026
thanks, merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7433][SPARK] Preserve Hive delegation tokens with non-empty service [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #7518:
URL: https://github.com/apache/kyuubi/pull/7518#issuecomment-4799609621
also cc @zhouyifan279 and @cxzl25, who have better knowledge of this part
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org