HIVE-27260 discussion

[Zsolt Miskolczi <zs...@gmail.com>]

Hi community,

I'm planning to pick up HIVE-27260
<https://issues.apache.org/jira/browse/HIVE-27260> and please let me bring
up some possible ways to solve the problem.

So, the issue that we faced at multiple customers is when the expired token
cleanup and renewal process (
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/security/TokenStoreDelegationTokenSecretManager.java#L263)
runs getToken for each token id previously collected in
tokenStore.getAllDelegationTokenIdentifiers().

In a really bad case scenario, assuming the customer has ~200k tokens and
only half of them pass to that point, it is ~100k calls to the metastore
database. And it is worse if they have multiple HMS instances and the clean
up fires in an overlapping time window.

I see multiple possible ways to handle the issue, let me bring some of them
up. As I'm mostly familiar with the DbTokenStore I'm especially curious if
the possible solutions can potentially bring up doubts at
ZooKeeperTokenStore.

So, the possible solutions came into my mind:

1.: Modify the data structure: Add renew date as a new column. So that,
getAllDelegationTokenIdentifier can be modified to contain the renew date
and we can make a decision about removing or renewing with a single
database call. Please note, with this, there will still be large reads (a
full table scan, no filtering or batching). But at least there will be only
one call, not 100k.

2.: Combine possible solution 1 with batching. Multiple small calls, like
5k items in a batch. Honestly, I would do it in case solution 1 still has
performance problems due to the large amount of data transferred in a
single call.

3.: Instead of using getAllDelegationTokenIdentifiers() introduce a new
method that queries both the ids and the tokens together (basically the
whole table). The drawback is the traffic will be doubled (both of the
columns are varchar(767)). And also, I'm not completely sure if it can lead
to security issues (querying both the token ids and the tokens theirself
together).

4.: Instead of using a single getToken() call for each record, do it in
batches, like 1000 tokens in a batch. The problem with this solution is the
size of the token id (varchar(767)). Using this solution can lead to really
huge select texts (where token_ident in (....)).

I personally think solution 1 can have the best performance but as it also
means schema change, it is the solution that will be hard to modify later.
Solution 4 can look like an easy solution but due to the large size of the
id, I believe we should choose a really small batch size and after all,
there would be no significant performance gain.

Now I want to start an open discussion about the possible solutions. In
case there will be no second thoughts against that, I'm willing to start
developing solution 1.


So, my question is: what is your opinion?

Thank you,
Zsolt

Re: HIVE-27260 discussion

[Ayush Saxena <ay...@gmail.com>]

Hi folks,
Thanx for the discussion so far, I concur with Sai: combining Option 1
(renew date column so cleanup can decide renew vs remove without
per-token getToken calls) with Option 2 (batched reads) looks like the
best path forward for HIVE-27260.

If doing both in a single PR feels too heavy, we can land it in
stages—for example schema + bulk fetch with renew date first, then
batching/streaming and any cross-instance rate limiting as follow-ups.

-Ayush

On Wed, 21 Oct 2026 at 23:42, Sai Hemanth Gantasala
<gs...@apache.org> wrote:
>
> Hi Zsolt,
>
> Thanks for bringing this up! The high volume of database calls during token cleanup across multiple HMS instances is a significant pain point for large clusters.
>
> Here is my feedback on the proposed options:
>
>   - Option 1 & 2: Combining a renewal date column (Option 1) with batching (Option 2) is the most robust approach, efficiently handling this on the DB side without pounding the metastore with 100k individual calls.
>   - Option 3: Fetching both token IDs and full tokens in a single bulk query raises security concerns about loading sensitive token material into memory at once and doubles transferred data.
>   - Option 4: Passing large collections of IDs in WHERE token_ident IN (...) clauses will quickly hit SQL statement length limits or degrade query planner performance given the varchar(767) size.
>
> A few additional ideas to consider alongside Options 1 and 2:
>
>   - Cursor-based pagination / Streaming: Avoids loading massive result sets into memory and keeps HMS memory usage flat.
>   - Asynchronous / Rate-limited worker: Running cleanup via a background worker thread with rate limiting and random jitter across HMS instances would prevent thundering herd problems from concurrent metastores.
>
> My opinon: Combining Options 1 and 2 with batching seems like a solid direction forward.
>
> Thanks,
> Sai
>
>
> On Mon, Oct 5, 2026 at 6:16 AM Zsolt Miskolczi <zs...@gmail.com> wrote:
>>
>> Hi community,
>>
>> I'm planning to pick up HIVE-27260 and please let me bring up some possible ways to solve the problem.
>>
>> So, the issue that we faced at multiple customers is when the expired token cleanup and renewal process (https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/security/TokenStoreDelegationTokenSecretManager.java#L263) runs getToken for each token id previously collected in tokenStore.getAllDelegationTokenIdentifiers().
>>
>> In a really bad case scenario, assuming the customer has ~200k tokens and only half of them pass to that point, it is ~100k calls to the metastore database. And it is worse if they have multiple HMS instances and the clean up fires in an overlapping time window.
>>
>> I see multiple possible ways to handle the issue, let me bring some of them up. As I'm mostly familiar with the DbTokenStore I'm especially curious if the possible solutions can potentially bring up doubts at ZooKeeperTokenStore.
>>
>> So, the possible solutions came into my mind:
>>
>> 1.: Modify the data structure: Add renew date as a new column. So that, getAllDelegationTokenIdentifier can be modified to contain the renew date and we can make a decision about removing or renewing with a single database call. Please note, with this, there will still be large reads (a full table scan, no filtering or batching). But at least there will be only one call, not 100k.
>>
>> 2.: Combine possible solution 1 with batching. Multiple small calls, like 5k items in a batch. Honestly, I would do it in case solution 1 still has performance problems due to the large amount of data transferred in a single call.
>>
>> 3.: Instead of using getAllDelegationTokenIdentifiers() introduce a new method that queries both the ids and the tokens together (basically the whole table). The drawback is the traffic will be doubled (both of the columns are varchar(767)). And also, I'm not completely sure if it can lead to security issues (querying both the token ids and the tokens theirself together).
>>
>> 4.: Instead of using a single getToken() call for each record, do it in batches, like 1000 tokens in a batch. The problem with this solution is the size of the token id (varchar(767)). Using this solution can lead to really huge select texts (where token_ident in (....)).
>>
>> I personally think solution 1 can have the best performance but as it also means schema change, it is the solution that will be hard to modify later. Solution 4 can look like an easy solution but due to the large size of the id, I believe we should choose a really small batch size and after all, there would be no significant performance gain.
>>
>> Now I want to start an open discussion about the possible solutions. In case there will be no second thoughts against that, I'm willing to start developing solution 1.
>>
>>
>> So, my question is: what is your opinion?
>>
>> Thank you,
>> Zsolt

Re: HIVE-27260 discussion

[Sai Hemanth Gantasala <gs...@apache.org>]

Hi Zsolt,

Thanks for bringing this up! The high volume of database calls during token
cleanup across multiple HMS instances is a significant pain point for large
clusters.

Here is my feedback on the proposed options:

  - Option 1 & 2: Combining a renewal date column (Option 1) with batching
(Option 2) is the most robust approach, efficiently handling this on the DB
side without pounding the metastore with 100k individual calls.
  - Option 3: Fetching both token IDs and full tokens in a single bulk
query raises security concerns about loading sensitive token material into
memory at once and doubles transferred data.
  - Option 4: Passing large collections of IDs in WHERE token_ident IN
(...) clauses will quickly hit SQL statement length limits or degrade query
planner performance given the varchar(767) size.

A few additional ideas to consider alongside Options 1 and 2:

  - Cursor-based pagination / Streaming: Avoids loading massive result sets
into memory and keeps HMS memory usage flat.
  - Asynchronous / Rate-limited worker: Running cleanup via a background
worker thread with rate limiting and random jitter across HMS instances
would prevent thundering herd problems from concurrent metastores.

My opinon: Combining Options 1 and 2 with batching seems like a solid
direction forward.

Thanks,
Sai


On Mon, Oct 5, 2026 at 6:16 AM Zsolt Miskolczi <zs...@gmail.com>
wrote:

> Hi community,
>
> I'm planning to pick up HIVE-27260
> <https://issues.apache.org/jira/browse/HIVE-27260> and please let me
> bring up some possible ways to solve the problem.
>
> So, the issue that we faced at multiple customers is when the expired
> token cleanup and renewal process (
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/security/TokenStoreDelegationTokenSecretManager.java#L263)
> runs getToken for each token id previously collected in
> tokenStore.getAllDelegationTokenIdentifiers().
>
> In a really bad case scenario, assuming the customer has ~200k tokens and
> only half of them pass to that point, it is ~100k calls to the metastore
> database. And it is worse if they have multiple HMS instances and the clean
> up fires in an overlapping time window.
>
> I see multiple possible ways to handle the issue, let me bring some of
> them up. As I'm mostly familiar with the DbTokenStore I'm
> especially curious if the possible solutions can potentially bring up
> doubts at ZooKeeperTokenStore.
>
> So, the possible solutions came into my mind:
>
> 1.: Modify the data structure: Add renew date as a new column. So that,
> getAllDelegationTokenIdentifier can be modified to contain the renew date
> and we can make a decision about removing or renewing with a single
> database call. Please note, with this, there will still be large reads (a
> full table scan, no filtering or batching). But at least there will be only
> one call, not 100k.
>
> 2.: Combine possible solution 1 with batching. Multiple small calls, like
> 5k items in a batch. Honestly, I would do it in case solution 1 still has
> performance problems due to the large amount of data transferred in a
> single call.
>
> 3.: Instead of using getAllDelegationTokenIdentifiers() introduce a new
> method that queries both the ids and the tokens together (basically the
> whole table). The drawback is the traffic will be doubled (both of the
> columns are varchar(767)). And also, I'm not completely sure if it can lead
> to security issues (querying both the token ids and the tokens theirself
> together).
>
> 4.: Instead of using a single getToken() call for each record, do it in
> batches, like 1000 tokens in a batch. The problem with this solution is the
> size of the token id (varchar(767)). Using this solution can lead to really
> huge select texts (where token_ident in (....)).
>
> I personally think solution 1 can have the best performance but as it also
> means schema change, it is the solution that will be hard to modify later.
> Solution 4 can look like an easy solution but due to the large size of the
> id, I believe we should choose a really small batch size and after all,
> there would be no significant performance gain.
>
> Now I want to start an open discussion about the possible solutions. In
> case there will be no second thoughts against that, I'm willing to start
> developing solution 1.
>
>
> So, my question is: what is your opinion?
>
> Thank you,
> Zsolt
>