You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "arunk-kumar (via GitHub)" <gi...@apache.org> on 2026/07/21 12:45:52 UTC
[PR] HIVE-29720: Abort the service if Metastore cannot participate in the election [hive]
arunk-kumar opened a new pull request, #6621:
URL: https://github.com/apache/hive/pull/6621
### What changes were proposed in this pull request?
Install an `UncaughtExceptionHandler` on each election daemon in `LeaderElectionContext.start()`. On retry exhaustion it logs the failing election at `ERROR` and calls an injectable `abortAction` (default `System.exit(1)`), which triggers the existing HMS shutdown-hook chain (election `close()`, ZooKeeper deregistration, metrics, servlet server). An `AtomicBoolean` guards against a second abort when both electors fail concurrently. Also adds `@Category(MetastoreUnitTest.class)` to `TestLeaderElection` — the class was previously excluded by the module's `test.groups` filter.
### Why are the changes needed?
Retry exhaustion in `LeaseLeaderElection` previously terminated the daemon thread with no `UncaughtExceptionHandler` — no log, no audit, no shutdown. HMS kept running with no leader for `HOUSEKEEPING` / `ALWAYS_TASKS`, silently stalling compaction and stats updates until users noticed via query slowdowns.
### Does this PR introduce _any_ user-facing change?
Yes. When leader election exhausts metastore.lock.numretries retries, HMS now shuts down cleanly instead of remaining up without a leader for its housekeeping / always-tasks work. The existing retry knobs remain the tolerance dial.
### How was this patch tested?
Added two unit tests to `TestLeaderElection` that inject a failing `LeaderElection` and a fake `abortAction`, asserting the handler fires and that concurrent failures still trigger the abort exactly once. Full class runs in ~4.5s: `Tests run: 4, Failures: 0, Errors: 0, Skipped: 0`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29720: Abort the service if Metastore cannot participate in the election [hive]
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #6621:
URL: https://github.com/apache/hive/pull/6621#discussion_r3629030495
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/LeaderElectionContext.java:
##########
@@ -66,24 +68,26 @@ public String getName() {
private final Configuration conf;
private final String servHost;
- // Whether the context should be started as a daemon
- private final boolean startAsDaemon;
// Audit the event of election
private AuditLeaderListener auditLeaderListener;
// State change listeners group by type
private final Map<TTYPE, List<LeadershipStateListener>> listeners;
// Collection of leader candidates
private final List<LeaderElection<?>> leaderElections = new ArrayList<>();
- // Property for testing, a single leader will be created
+ // Ensures the abort action runs at most once even if both electors fail concurrently.
+ private final AtomicBoolean isAborting = new AtomicBoolean(false);
+ // Action to run when a leader election exhausts retries. Defaults to triggering an
+ // orderly JVM shutdown via System.exit, which fires the HMS shutdown-hook chain
+ // (election close, ZK deregistration, metrics, servlet server). Overridable for tests.
+ private Runnable abortAction = () -> System.exit(1);
Review Comment:
Can you use `ExitUtil.terminate` instead, refer to `https://github.com/apache/hive/pull/5914`?
And before exiting the system, try best to close the opened electors in `leaderElections`, note this `leaderElections` might be modified during the close, might be better to copy it to a new list, then travel the new list to close them.
Rest looks good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29720: Abort the service if Metastore cannot participate in the election [hive]
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 merged PR #6621:
URL: https://github.com/apache/hive/pull/6621
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29720: Abort the service if Metastore cannot participate in the election [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6621:
URL: https://github.com/apache/hive/pull/6621#issuecomment-5034947402
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6621) **Quality Gate passed**
Issues
 [6 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6621&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6621&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6621&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6621&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6621&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6621)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29720: Abort the service if Metastore cannot participate in the election [hive]
Posted by "arunk-kumar (via GitHub)" <gi...@apache.org>.
arunk-kumar commented on PR #6621:
URL: https://github.com/apache/hive/pull/6621#issuecomment-5044706178
Thanks @dengzhhu653 — both addressed in the latest commit:
- Switched the default `abortAction` from `System.exit(1)` to `ExitUtil.terminate(1)`, following the HIVE-29056 pattern.
- Abort handler now closes the electors before invoking `abortAction`, so the mutex is released promptly. `close()` iterates a snapshot copy of `leaderElections` to avoid mid-iteration mutation as you suggested.
`TestLeaderElection` still passes: `Tests run: 4, Failures: 0, Errors: 0, Skipped: 0`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29720: Abort the service if Metastore cannot participate in the election [hive]
Posted by "arunk-kumar (via GitHub)" <gi...@apache.org>.
arunk-kumar commented on PR #6621:
URL: https://github.com/apache/hive/pull/6621#issuecomment-5069536410
Thanks for the stack trace — that pinpointed it. The switch from `daemon.run()` (synchronous) to `daemon.start()` (async) in the earlier commit broke a pre-existing invariant: `LeaderElectionContext.start()` used to not return until listener callbacks (housekeeping thread creation etc.) had fired synchronously, and `TestMetastoreHousekeepingLeaderEmptyConfig` scans for those threads immediately after HMS starts.
Fix: join each daemon after starting all of them, so elections still run in parallel but `start()` waits until every initial `tryBeLeader()` completes. Since `LeaseLeaderElection.tryBeLeader` dispatches lease renewal to a separate `Heartbeater`/`NonLeaderWatcher` thread and then returns (rather than looping on renewal itself), `join()` terminates in bounded time on success, on wait-state, and on retry exhaustion.
`TestLeaderElection` still passes locally: `Tests run: 4, Failures: 0, Errors: 0, Skipped: 0`. Couldn't reproduce `TestMetastoreHousekeepingLeaderEmptyConfig` locally — upstream `itests/hive-unit` build is transiently broken on my end (Iceberg module compile issues on master). Relying on Jenkins to verify the fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29720: Abort the service if Metastore cannot participate in the election [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6621:
URL: https://github.com/apache/hive/pull/6621#issuecomment-5038674800
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6621) **Quality Gate passed**
Issues
 [3 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6621&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6621&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6621&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6621&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6621&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6621)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29720: Abort the service if Metastore cannot participate in the election [hive]
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #6621:
URL: https://github.com/apache/hive/pull/6621#discussion_r3629030495
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/LeaderElectionContext.java:
##########
@@ -66,24 +68,26 @@ public String getName() {
private final Configuration conf;
private final String servHost;
- // Whether the context should be started as a daemon
- private final boolean startAsDaemon;
// Audit the event of election
private AuditLeaderListener auditLeaderListener;
// State change listeners group by type
private final Map<TTYPE, List<LeadershipStateListener>> listeners;
// Collection of leader candidates
private final List<LeaderElection<?>> leaderElections = new ArrayList<>();
- // Property for testing, a single leader will be created
+ // Ensures the abort action runs at most once even if both electors fail concurrently.
+ private final AtomicBoolean isAborting = new AtomicBoolean(false);
+ // Action to run when a leader election exhausts retries. Defaults to triggering an
+ // orderly JVM shutdown via System.exit, which fires the HMS shutdown-hook chain
+ // (election close, ZK deregistration, metrics, servlet server). Overridable for tests.
+ private Runnable abortAction = () -> System.exit(1);
Review Comment:
Can you use `ExitUtil.terminate` instead, refer to `https://github.com/apache/hive/pull/5914`?
And before exiting the system, try best to close the opened electors in `leaderElections`, note this `leaderElections` might be modified during the close, might be better to copy it to a new list, then travel the new list to close them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29720: Abort the service if Metastore cannot participate in the election [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6621:
URL: https://github.com/apache/hive/pull/6621#issuecomment-5046687180
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6621) **Quality Gate passed**
Issues
 [3 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6621&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6621&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6621&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6621&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6621&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6621)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29720: Abort the service if Metastore cannot participate in the election [hive]
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on PR #6621:
URL: https://github.com/apache/hive/pull/6621#issuecomment-5066099179
Sounds like the CI failures are related, e.g, TestMetastoreHousekeepingLeaderEmptyConfig, can you please take a look? Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29720: Abort the service if Metastore cannot participate in the election [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6621:
URL: https://github.com/apache/hive/pull/6621#issuecomment-5043653416
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6621) **Quality Gate passed**
Issues
 [3 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6621&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6621&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6621&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6621&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6621&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6621)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29720: Abort the service if Metastore cannot participate in the election [hive]
Posted by "arunk-kumar (via GitHub)" <gi...@apache.org>.
arunk-kumar commented on PR #6621:
URL: https://github.com/apache/hive/pull/6621#issuecomment-5043022157
Thanks for the review! Pushed a commit addressing both points:
- `start()` now unconditionally starts each election candidate in its own daemon thread, so the `UncaughtExceptionHandler` fires regardless of how the context is constructed. The `startAsDaemon` builder flag and its backing field are removed since no production caller sets it to `true` and the synchronous `run()` path bypassed the abort handler.
- Updated the error message in `HiveMetaStore` to distinguish election setup failures from runtime election failures.
`TestLeaderElection` still passes locally: `Tests run: 4, Failures: 0, Errors: 0, Skipped: 0`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29720: Abort the service if Metastore cannot participate in the election [hive]
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #6621:
URL: https://github.com/apache/hive/pull/6621#discussion_r3626234590
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/LeaderElectionContext.java:
##########
@@ -117,6 +125,7 @@ public void start() throws Exception {
throw new RuntimeException("Error claiming to be leader: " + leaderElection.getName(), e);
}
});
+ daemon.setUncaughtExceptionHandler(newAbortOnElectionFailureHandler(leaderElection));
Review Comment:
Thank you for the PR!
If `startAsDaemon` is false, this hander might not be able to react to the election failure, an idea is to remove `daemon.run` in else branch, make every candidate run the election in their own thread.
Also please take care of the error message in `HiveMetaStore`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29720: Abort the service if Metastore cannot participate in the election [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6621:
URL: https://github.com/apache/hive/pull/6621#issuecomment-5070265406
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6621) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6621&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6621&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6621&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6621&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6621&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6621)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org