You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "dengzhhu653 (via GitHub)" <gi...@apache.org> on 2025/10/18 12:03:34 UTC

[PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

dengzhhu653 opened a new pull request, #6142:
URL: https://github.com/apache/hive/pull/6142

   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
     2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
     3. Ensure you have added or run the appropriate tests for your PR: 
     4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX:  Your PR title ...'.
     5. Be sure to keep the PR description updated to reflect all changes.
     6. Please write your PR title to summarize what this PR proposes.
     7. If possible, provide a concise example to reproduce the issue for a faster review.
   
   -->
   
   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   
   
   ### Does this PR introduce _any_ user-facing change?
   No 
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducable example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   
   ### How was this patch tested?
   https://ci.hive.apache.org/job/hive-flaky-check/915
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2444582270


##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/MetastoreHousekeepingLeaderTestBase.java:
##########
@@ -54,19 +56,24 @@ class MetastoreHousekeepingLeaderTestBase {
   static Map<String, Boolean> threadNames = new HashMap<>();
   static Map<Class<? extends Thread>, Boolean> threadClasses = new HashMap<>();
 
-  void internalSetup(final String leaderHostName, boolean configuredLeader) throws Exception {
+  void internalSetup(final String leaderHostName, Configuration configuration) throws Exception {
+    this.conf = configuration;
     MetaStoreTestUtils.setConfForStandloneMode(conf);
     MetastoreConf.setVar(conf, ConfVars.THRIFT_BIND_HOST, "localhost");
-    MetastoreConf.setVar(conf, ConfVars.METASTORE_HOUSEKEEPING_LEADER_HOSTNAME, leaderHostName);
     MetastoreConf.setVar(conf, ConfVars.METASTORE_HOUSEKEEPING_LEADER_ELECTION,
-        configuredLeader ? "host" : "lock");
+        leaderHostName != null ? "host" : "lock");
+    if (leaderHostName != null) {
+      MetastoreConf.setVar(conf, ConfVars.METASTORE_HOUSEKEEPING_LEADER_HOSTNAME, leaderHostName);
+    }
     MetastoreConf.setBoolVar(conf, MetastoreConf.ConfVars.COMPACTOR_INITIATOR_ON, true);
     MetastoreConf.setBoolVar(conf, MetastoreConf.ConfVars.COMPACTOR_CLEANER_ON, true);
 
     addHouseKeepingThreadConfigs();
 
     warehouse = new Warehouse(conf);
 
+    conf.set("metastore.leader.test.listener", TestLeaderNotification.class.getName());

Review Comment:
   This property is for test only, it's shouldn't be exposed to user or else where in the unit test



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2444616976


##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetastoreHousekeepingLeader.java:
##########
@@ -34,7 +35,7 @@ public class TestMetastoreHousekeepingLeader extends MetastoreHousekeepingLeader
 
   @Before
   public void setUp() throws Exception {
-    internalSetup("localhost", true);
+    internalSetup("localhost", MetastoreConf.newMetastoreConf());

Review Comment:
   It's used here: https://github.com/apache/hive/pull/6142/files#diff-bd73307ca500a5ceb6b862f1819eb919a4f03f8db1737fad425a1b684353c713R52, mainly for sharing the same connection url for different leader voter



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2462920404


##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/MetastoreHousekeepingLeaderTestBase.java:
##########
@@ -54,12 +63,15 @@ class MetastoreHousekeepingLeaderTestBase {
   static Map<String, Boolean> threadNames = new HashMap<>();
   static Map<Class<? extends Thread>, Boolean> threadClasses = new HashMap<>();
 
-  void internalSetup(final String leaderHostName, boolean configuredLeader) throws Exception {
+  void setup(final String leaderHostName, Configuration configuration) throws Exception {
+    this.conf = configuration;
     MetaStoreTestUtils.setConfForStandloneMode(conf);
     MetastoreConf.setVar(conf, ConfVars.THRIFT_BIND_HOST, "localhost");
-    MetastoreConf.setVar(conf, ConfVars.METASTORE_HOUSEKEEPING_LEADER_HOSTNAME, leaderHostName);
     MetastoreConf.setVar(conf, ConfVars.METASTORE_HOUSEKEEPING_LEADER_ELECTION,
-        configuredLeader ? "host" : "lock");
+        leaderHostName != null ? "host" : "lock");

Review Comment:
   please try use enum (i.e Method from LeaderElectionFactory)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2464393374


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/LeaderElectionContext.java:
##########
@@ -97,40 +95,32 @@ private LeaderElectionContext(String servHost, Configuration conf,
   }
 
   public void start() throws Exception {
-    Map<TTYPE, List<LeadershipStateListener>> listenerMap = this.listeners;
-    if (conf.getBoolean(LEADER_IN_TEST, false)) {
-      Map<TTYPE, List<LeadershipStateListener>> newListeners = new HashMap<>();
-      newListeners.put(TTYPE.HOUSEKEEPING, new ArrayList<>());
-      listenerMap.forEach((k, v) -> newListeners.get(TTYPE.HOUSEKEEPING).addAll(v));
-      listenerMap = newListeners;
-    }
-    for (Map.Entry<TTYPE, List<LeadershipStateListener>> entry :
-        listenerMap.entrySet()) {
-      List<LeadershipStateListener> listenerList = entry.getValue();
+    List<TTYPE> ttypes = new ArrayList<>(listeners.keySet());
+    Collections.shuffle(ttypes);

Review Comment:
   there are two types of leaders, shuffling the types let other HMS instances have a chance to win the election on different types.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2464395030


##########
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/leader/TestLeaderElection.java:
##########
@@ -98,60 +90,21 @@ public void testLeaseLeaderElection() throws Exception {
     // elect1 as a leader now
     assertTrue(flag1.get() && instance1.isLeader());
 
-    configuration.setBoolean(LeaseLeaderElection.METASTORE_RENEW_LEASE, true);
     LeaseLeaderElection instance2 = new LeaseLeaderElection();
     AtomicBoolean flag2 = new AtomicBoolean(false);
     instance2.addStateListener(new TestLeaderListener(flag2));
     instance2.tryBeLeader(configuration, mutex);
     // instance2 should not be leader as elect1 holds the lease
     assertFalse(flag2.get() || instance2.isLeader());
 
-    ExecutorService service = Executors.newFixedThreadPool(4);
-    wait(service, flag1, flag2);
-    // now instance1 lease is timeout, the instance2 should be leader now
+    instance1.close();
+    synchronized (flag2) {

Review Comment:
   we need to fetch the flag2 value back in the `main` thread, which set by the leader in background



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2462921757


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java:
##########
@@ -388,6 +388,7 @@ LockResponse lock(LockRequest rqst)
    * @throws TxnAbortedException
    * @throws MetaException
    */
+  @SqlRetry(lockInternally = true)

Review Comment:
   could you please move it below `@RetrySemantics.SafeToRetry`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2462919003


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/LeaderElectionContext.java:
##########
@@ -44,8 +45,6 @@ public class LeaderElectionContext {
   public enum TTYPE {
     HOUSEKEEPING(new TableName(Warehouse.DEFAULT_CATALOG_NAME, "__METASTORE_LEADER_ELECTION__",
         "metastore_housekeeping"), "housekeeping"),
-    WORKER(new TableName(Warehouse.DEFAULT_CATALOG_NAME, "__METASTORE_LEADER_ELECTION__",

Review Comment:
   nice! we forgot to drop this



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2453534698


##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetastoreHousekeepingLeader.java:
##########
@@ -34,7 +35,7 @@ public class TestMetastoreHousekeepingLeader extends MetastoreHousekeepingLeader
 
   @Before
   public void setUp() throws Exception {
-    internalSetup("localhost", true);
+    internalSetup("localhost", MetastoreConf.newMetastoreConf());

Review Comment:
   Interesting approach to instantiate and use a junit TEST class within another test. 
   https://github.com/apache/hive/pull/6142/files#diff-bd73307ca500a5ceb6b862f1819eb919a4f03f8db1737fad425a1b684353c713R46 
   It might be cleaner to extract the service logic into a separate class.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2459022644


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/LeaderElectionContext.java:
##########
@@ -100,8 +101,14 @@ public void start() throws Exception {
     Map<TTYPE, List<LeadershipStateListener>> listenerMap = this.listeners;
     if (conf.getBoolean(LEADER_IN_TEST, false)) {
       Map<TTYPE, List<LeadershipStateListener>> newListeners = new HashMap<>();
-      newListeners.put(TTYPE.HOUSEKEEPING, new ArrayList<>());
-      listenerMap.forEach((k, v) -> newListeners.get(TTYPE.HOUSEKEEPING).addAll(v));
+      List<LeadershipStateListener> listeners = new ArrayList<>();
+      listenerMap.forEach((k, v) -> listeners.addAll(v));
+      String testListenerCls = conf.get("metastore.leader.test.listener");

Review Comment:
   Move all the test codes and configs from the production



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6142:
URL: https://github.com/apache/hive/pull/6142#issuecomment-3418425767

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6142) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [5 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6142&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6142&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6142&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6142&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6142&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6142)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2444160416


##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/MetastoreHousekeepingLeaderTestBase.java:
##########
@@ -54,19 +56,24 @@ class MetastoreHousekeepingLeaderTestBase {
   static Map<String, Boolean> threadNames = new HashMap<>();
   static Map<Class<? extends Thread>, Boolean> threadClasses = new HashMap<>();
 
-  void internalSetup(final String leaderHostName, boolean configuredLeader) throws Exception {
+  void internalSetup(final String leaderHostName, Configuration configuration) throws Exception {
+    this.conf = configuration;
     MetaStoreTestUtils.setConfForStandloneMode(conf);
     MetastoreConf.setVar(conf, ConfVars.THRIFT_BIND_HOST, "localhost");
-    MetastoreConf.setVar(conf, ConfVars.METASTORE_HOUSEKEEPING_LEADER_HOSTNAME, leaderHostName);
     MetastoreConf.setVar(conf, ConfVars.METASTORE_HOUSEKEEPING_LEADER_ELECTION,
-        configuredLeader ? "host" : "lock");
+        leaderHostName != null ? "host" : "lock");
+    if (leaderHostName != null) {
+      MetastoreConf.setVar(conf, ConfVars.METASTORE_HOUSEKEEPING_LEADER_HOSTNAME, leaderHostName);
+    }
     MetastoreConf.setBoolVar(conf, MetastoreConf.ConfVars.COMPACTOR_INITIATOR_ON, true);
     MetastoreConf.setBoolVar(conf, MetastoreConf.ConfVars.COMPACTOR_CLEANER_ON, true);
 
     addHouseKeepingThreadConfigs();
 
     warehouse = new Warehouse(conf);
 
+    conf.set("metastore.leader.test.listener", TestLeaderNotification.class.getName());

Review Comment:
   could we move this to constants?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2454837515


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/LeaderElectionContext.java:
##########
@@ -100,8 +101,14 @@ public void start() throws Exception {
     Map<TTYPE, List<LeadershipStateListener>> listenerMap = this.listeners;
     if (conf.getBoolean(LEADER_IN_TEST, false)) {
       Map<TTYPE, List<LeadershipStateListener>> newListeners = new HashMap<>();
-      newListeners.put(TTYPE.HOUSEKEEPING, new ArrayList<>());
-      listenerMap.forEach((k, v) -> newListeners.get(TTYPE.HOUSEKEEPING).addAll(v));
+      List<LeadershipStateListener> listeners = new ArrayList<>();
+      listenerMap.forEach((k, v) -> listeners.addAll(v));
+      String testListenerCls = conf.get("metastore.leader.test.listener");

Review Comment:
   > We also have some properties such as hive.in*.test for testing purpose
   
   True — those are legacy and should be removed once all tests are refactored (I’ve already done this in several cases). 
   However, that’s not a reason to introduce more such configs. The code should be structured in a way that makes it easy to test using the existing test framework tools.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2444205194


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PersistenceManagerProvider.java:
##########
@@ -185,10 +185,10 @@ private static void logPropChanges(Properties newProps) {
               oldVal = MetaStoreServerUtils.anonymizeConnectionURL(oldVal);
               newVal = MetaStoreServerUtils.anonymizeConnectionURL(newVal);
             }
-            LOG.debug("Found {} to be different. Old val : {} : New Val : {}", key,
+            LOG.info("Found {} to be different. Old val : {} : New Val : {}", key,

Review Comment:
   maybe warn?



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PersistenceManagerProvider.java:
##########
@@ -185,10 +185,10 @@ private static void logPropChanges(Properties newProps) {
               oldVal = MetaStoreServerUtils.anonymizeConnectionURL(oldVal);
               newVal = MetaStoreServerUtils.anonymizeConnectionURL(newVal);
             }
-            LOG.debug("Found {} to be different. Old val : {} : New Val : {}", key,
+            LOG.info("Found {} to be different. Old val : {} : New Val : {}", key,
                 oldVal, newVal);
           } else {
-            LOG.debug("Found masked property {} to be different", key);
+            LOG.info("Found masked property {} to be different", key);

Review Comment:
   same



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 merged PR #6142:
URL: https://github.com/apache/hive/pull/6142


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2444163396


##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/MetastoreHousekeepingLeaderTestBase.java:
##########
@@ -195,5 +202,30 @@ private void resetThreadStatus() {
     threadNames.forEach((name, status) -> threadNames.put(name, false));
     threadClasses.forEach((thread, status) -> threadClasses.put(thread, false));
   }
+
+  public static class TestLeaderNotification implements LeaderElection.LeadershipStateListener {

Review Comment:
   maybe package-private



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2444631316


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/LeaderElectionContext.java:
##########
@@ -100,8 +101,14 @@ public void start() throws Exception {
     Map<TTYPE, List<LeadershipStateListener>> listenerMap = this.listeners;
     if (conf.getBoolean(LEADER_IN_TEST, false)) {
       Map<TTYPE, List<LeadershipStateListener>> newListeners = new HashMap<>();
-      newListeners.put(TTYPE.HOUSEKEEPING, new ArrayList<>());
-      listenerMap.forEach((k, v) -> newListeners.get(TTYPE.HOUSEKEEPING).addAll(v));
+      List<LeadershipStateListener> listeners = new ArrayList<>();
+      listenerMap.forEach((k, v) -> listeners.addAll(v));
+      String testListenerCls = conf.get("metastore.leader.test.listener");

Review Comment:
   This is for the integration test, the mocking/stubbing I think couldn't imitate a HMS similar as the one deployed in  production. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2444201875


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/LeaseLeaderElection.java:
##########
@@ -178,23 +186,22 @@ public void tryBeLeader(Configuration conf, TableName table) throws LeaderExcept
         LockResponse res = store.lock(req);
         if (res.getState() == LockState.WAITING || res.getState() == LockState.ACQUIRED) {
           lockable = true;
+          LOG.info("{} Spent {}ms to take part in election, retries: {}", getName(), System.currentTimeMillis() - start, i);

Review Comment:
   same



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6142:
URL: https://github.com/apache/hive/pull/6142#issuecomment-3424445070

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6142) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [7 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6142&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6142&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6142&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6142&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6142&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6142)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2465530889


##########
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/leader/TestLeaderElection.java:
##########
@@ -98,60 +90,21 @@ public void testLeaseLeaderElection() throws Exception {
     // elect1 as a leader now
     assertTrue(flag1.get() && instance1.isLeader());
 
-    configuration.setBoolean(LeaseLeaderElection.METASTORE_RENEW_LEASE, true);
     LeaseLeaderElection instance2 = new LeaseLeaderElection();
     AtomicBoolean flag2 = new AtomicBoolean(false);
     instance2.addStateListener(new TestLeaderListener(flag2));
     instance2.tryBeLeader(configuration, mutex);
     // instance2 should not be leader as elect1 holds the lease
     assertFalse(flag2.get() || instance2.isLeader());
 
-    ExecutorService service = Executors.newFixedThreadPool(4);
-    wait(service, flag1, flag2);
-    // now instance1 lease is timeout, the instance2 should be leader now
+    instance1.close();
+    synchronized (flag2) {

Review Comment:
   sorry, I meant `Ref<Boolean>`, but it's seem only available in hive modules



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2462924920


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/LeaderElectionContext.java:
##########
@@ -100,8 +101,14 @@ public void start() throws Exception {
     Map<TTYPE, List<LeadershipStateListener>> listenerMap = this.listeners;
     if (conf.getBoolean(LEADER_IN_TEST, false)) {
       Map<TTYPE, List<LeadershipStateListener>> newListeners = new HashMap<>();
-      newListeners.put(TTYPE.HOUSEKEEPING, new ArrayList<>());
-      listenerMap.forEach((k, v) -> newListeners.get(TTYPE.HOUSEKEEPING).addAll(v));
+      List<LeadershipStateListener> listeners = new ArrayList<>();
+      listenerMap.forEach((k, v) -> listeners.addAll(v));
+      String testListenerCls = conf.get("metastore.leader.test.listener");

Review Comment:
   appreciate the refactor, @dengzhhu653! the code is much easier to follow now



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2465521731


##########
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/leader/TestLeaderElection.java:
##########
@@ -98,60 +90,21 @@ public void testLeaseLeaderElection() throws Exception {
     // elect1 as a leader now
     assertTrue(flag1.get() && instance1.isLeader());
 
-    configuration.setBoolean(LeaseLeaderElection.METASTORE_RENEW_LEASE, true);
     LeaseLeaderElection instance2 = new LeaseLeaderElection();
     AtomicBoolean flag2 = new AtomicBoolean(false);
     instance2.addStateListener(new TestLeaderListener(flag2));
     instance2.tryBeLeader(configuration, mutex);
     // instance2 should not be leader as elect1 holds the lease
     assertFalse(flag2.get() || instance2.isLeader());
 
-    ExecutorService service = Executors.newFixedThreadPool(4);
-    wait(service, flag1, flag2);
-    // now instance1 lease is timeout, the instance2 should be leader now
+    instance1.close();
+    synchronized (flag2) {

Review Comment:
   > why do you need AtomicBoolean then here, isn't simple Boolean sufficient?
   
   No, the Boolean will lost the update set by the background



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2465500144


##########
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/leader/TestLeaderElection.java:
##########
@@ -98,60 +90,21 @@ public void testLeaseLeaderElection() throws Exception {
     // elect1 as a leader now
     assertTrue(flag1.get() && instance1.isLeader());
 
-    configuration.setBoolean(LeaseLeaderElection.METASTORE_RENEW_LEASE, true);
     LeaseLeaderElection instance2 = new LeaseLeaderElection();
     AtomicBoolean flag2 = new AtomicBoolean(false);
     instance2.addStateListener(new TestLeaderListener(flag2));
     instance2.tryBeLeader(configuration, mutex);
     // instance2 should not be leader as elect1 holds the lease
     assertFalse(flag2.get() || instance2.isLeader());
 
-    ExecutorService service = Executors.newFixedThreadPool(4);
-    wait(service, flag1, flag2);
-    // now instance1 lease is timeout, the instance2 should be leader now
+    instance1.close();
+    synchronized (flag2) {

Review Comment:
   why do you need synchronized on AtomicBoolean?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2453528512


##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetastoreHousekeepingLeader.java:
##########
@@ -34,7 +35,7 @@ public class TestMetastoreHousekeepingLeader extends MetastoreHousekeepingLeader
 
   @Before
   public void setUp() throws Exception {
-    internalSetup("localhost", true);
+    internalSetup("localhost", MetastoreConf.newMetastoreConf());

Review Comment:
   `internalSetup` method name implies it’s intended for internal use rather than being a publicly exposed method



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2444636533


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/LeaseLeaderElection.java:
##########
@@ -131,7 +139,7 @@ private synchronized void doWork(LockResponse resp, Configuration conf,
     default:
       throw new IllegalStateException("Unexpected lock state: " + resp.getState());
     }
-    LOG.debug("Spent {}ms to notify the listeners, isLeader: {}", System.currentTimeMillis() - start, isLeader);
+    LOG.info("Spent {}ms to notify the listeners, isLeader: {}", System.currentTimeMillis() - start, isLeader);

Review Comment:
   This is for debug purpose, before that I couldn't see how much time spent on the notifying the listener. will revert it back if we don't want it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2444197787


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/LeaseLeaderElection.java:
##########
@@ -111,14 +111,22 @@ private synchronized void doWork(LockResponse resp, Configuration conf,
 
     switch (resp.getState()) {
     case ACQUIRED:
-      boolean renewLease = conf.getBoolean(METASTORE_RENEW_LEASE, true);
-      heartbeater = renewLease ?
-          new Heartbeater(conf, tableName) : new ReleaseAndRequireWatcher(conf, tableName);
+      heartbeater = new Heartbeater(conf, tableName);
       heartbeater.perform();
       if (!isLeader) {
         isLeader = true;
         notifyListener();
       }
+      if (!conf.getBoolean(METASTORE_RENEW_LEASE, true)) {
+        // For test only.

Review Comment:
   again, same thing. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6142:
URL: https://github.com/apache/hive/pull/6142#issuecomment-3441627292

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6142) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [22 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6142&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6142&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6142&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6142&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6142&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6142)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2453516375


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/LeaderElectionContext.java:
##########
@@ -100,8 +101,14 @@ public void start() throws Exception {
     Map<TTYPE, List<LeadershipStateListener>> listenerMap = this.listeners;
     if (conf.getBoolean(LEADER_IN_TEST, false)) {
       Map<TTYPE, List<LeadershipStateListener>> newListeners = new HashMap<>();
-      newListeners.put(TTYPE.HOUSEKEEPING, new ArrayList<>());
-      listenerMap.forEach((k, v) -> newListeners.get(TTYPE.HOUSEKEEPING).addAll(v));
+      List<LeadershipStateListener> listeners = new ArrayList<>();
+      listenerMap.forEach((k, v) -> listeners.addAll(v));
+      String testListenerCls = conf.get("metastore.leader.test.listener");

Review Comment:
   Including test-specific workarounds in production code is generally discouraged, as it’s considered a code smell and not a best practice.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2453524984


##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/MetastoreHousekeepingLeaderTestBase.java:
##########
@@ -195,5 +202,30 @@ private void resetThreadStatus() {
     threadNames.forEach((name, status) -> threadNames.put(name, false));
     threadClasses.forEach((thread, status) -> threadClasses.put(thread, false));
   }
+
+  public static class TestLeaderNotification implements LeaderElection.LeadershipStateListener {

Review Comment:
   using reflection in tests is typically not the best approach, as it often indicates underlying design issues



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2462923240


##########
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/leader/TestLeaderElection.java:
##########
@@ -98,60 +90,21 @@ public void testLeaseLeaderElection() throws Exception {
     // elect1 as a leader now
     assertTrue(flag1.get() && instance1.isLeader());
 
-    configuration.setBoolean(LeaseLeaderElection.METASTORE_RENEW_LEASE, true);
     LeaseLeaderElection instance2 = new LeaseLeaderElection();
     AtomicBoolean flag2 = new AtomicBoolean(false);
     instance2.addStateListener(new TestLeaderListener(flag2));
     instance2.tryBeLeader(configuration, mutex);
     // instance2 should not be leader as elect1 holds the lease
     assertFalse(flag2.get() || instance2.isLeader());
 
-    ExecutorService service = Executors.newFixedThreadPool(4);
-    wait(service, flag1, flag2);
-    // now instance1 lease is timeout, the instance2 should be leader now
+    instance1.close();
+    synchronized (flag2) {

Review Comment:
   why do you need synchromized here? AtomicBoolean is designed for lock-free atomic updates.
   ````
   while (!flag2.get()) {
       Thread.onSpinWait(); // JDK 9+
   }
   ````



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2462924324


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/LeaseLeaderElection.java:
##########
@@ -64,10 +64,10 @@ public class LeaseLeaderElection implements LeaderElection<TableName> {
 
   private static final Logger LOG = LoggerFactory.getLogger(LeaseLeaderElection.class);
 
-  private static final AtomicLong ID = new AtomicLong();
+  protected static final AtomicLong ID = new AtomicLong();

Review Comment:
   won't it be better if we expose protected fields via getters?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2453521464


##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/MetastoreHousekeepingLeaderTestBase.java:
##########
@@ -54,19 +56,24 @@ class MetastoreHousekeepingLeaderTestBase {
   static Map<String, Boolean> threadNames = new HashMap<>();
   static Map<Class<? extends Thread>, Boolean> threadClasses = new HashMap<>();
 
-  void internalSetup(final String leaderHostName, boolean configuredLeader) throws Exception {
+  void internalSetup(final String leaderHostName, Configuration configuration) throws Exception {
+    this.conf = configuration;
     MetaStoreTestUtils.setConfForStandloneMode(conf);
     MetastoreConf.setVar(conf, ConfVars.THRIFT_BIND_HOST, "localhost");
-    MetastoreConf.setVar(conf, ConfVars.METASTORE_HOUSEKEEPING_LEADER_HOSTNAME, leaderHostName);
     MetastoreConf.setVar(conf, ConfVars.METASTORE_HOUSEKEEPING_LEADER_ELECTION,
-        configuredLeader ? "host" : "lock");
+        leaderHostName != null ? "host" : "lock");
+    if (leaderHostName != null) {
+      MetastoreConf.setVar(conf, ConfVars.METASTORE_HOUSEKEEPING_LEADER_HOSTNAME, leaderHostName);
+    }
     MetastoreConf.setBoolVar(conf, MetastoreConf.ConfVars.COMPACTOR_INITIATOR_ON, true);
     MetastoreConf.setBoolVar(conf, MetastoreConf.ConfVars.COMPACTOR_CLEANER_ON, true);
 
     addHouseKeepingThreadConfigs();
 
     warehouse = new Warehouse(conf);
 
+    conf.set("metastore.leader.test.listener", TestLeaderNotification.class.getName());

Review Comment:
   how is that different from `LEADER_IN_TEST`: https://github.com/apache/hive/pull/6142/files#diff-d697cd0d86def896a111982bc9a1c1d4cdd1175fcc3a21599ce3f0e228613895R80



##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/MetastoreHousekeepingLeaderTestBase.java:
##########
@@ -54,19 +56,24 @@ class MetastoreHousekeepingLeaderTestBase {
   static Map<String, Boolean> threadNames = new HashMap<>();
   static Map<Class<? extends Thread>, Boolean> threadClasses = new HashMap<>();
 
-  void internalSetup(final String leaderHostName, boolean configuredLeader) throws Exception {
+  void internalSetup(final String leaderHostName, Configuration configuration) throws Exception {
+    this.conf = configuration;
     MetaStoreTestUtils.setConfForStandloneMode(conf);
     MetastoreConf.setVar(conf, ConfVars.THRIFT_BIND_HOST, "localhost");
-    MetastoreConf.setVar(conf, ConfVars.METASTORE_HOUSEKEEPING_LEADER_HOSTNAME, leaderHostName);
     MetastoreConf.setVar(conf, ConfVars.METASTORE_HOUSEKEEPING_LEADER_ELECTION,
-        configuredLeader ? "host" : "lock");
+        leaderHostName != null ? "host" : "lock");
+    if (leaderHostName != null) {
+      MetastoreConf.setVar(conf, ConfVars.METASTORE_HOUSEKEEPING_LEADER_HOSTNAME, leaderHostName);
+    }
     MetastoreConf.setBoolVar(conf, MetastoreConf.ConfVars.COMPACTOR_INITIATOR_ON, true);
     MetastoreConf.setBoolVar(conf, MetastoreConf.ConfVars.COMPACTOR_CLEANER_ON, true);
 
     addHouseKeepingThreadConfigs();
 
     warehouse = new Warehouse(conf);
 
+    conf.set("metastore.leader.test.listener", TestLeaderNotification.class.getName());

Review Comment:
   how is that different from `LEADER_IN_TEST`?: https://github.com/apache/hive/pull/6142/files#diff-d697cd0d86def896a111982bc9a1c1d4cdd1175fcc3a21599ce3f0e228613895R80



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2453516375


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/LeaderElectionContext.java:
##########
@@ -100,8 +101,14 @@ public void start() throws Exception {
     Map<TTYPE, List<LeadershipStateListener>> listenerMap = this.listeners;
     if (conf.getBoolean(LEADER_IN_TEST, false)) {
       Map<TTYPE, List<LeadershipStateListener>> newListeners = new HashMap<>();
-      newListeners.put(TTYPE.HOUSEKEEPING, new ArrayList<>());
-      listenerMap.forEach((k, v) -> newListeners.get(TTYPE.HOUSEKEEPING).addAll(v));
+      List<LeadershipStateListener> listeners = new ArrayList<>();
+      listenerMap.forEach((k, v) -> listeners.addAll(v));
+      String testListenerCls = conf.get("metastore.leader.test.listener");

Review Comment:
   Including test-specific workarounds in production code is generally discouraged, as it’s considered a code smell and not a best practice.
   same applies `conf.getBoolean(LEADER_IN_TEST, false)`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2453528512


##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetastoreHousekeepingLeader.java:
##########
@@ -34,7 +35,7 @@ public class TestMetastoreHousekeepingLeader extends MetastoreHousekeepingLeader
 
   @Before
   public void setUp() throws Exception {
-    internalSetup("localhost", true);
+    internalSetup("localhost", MetastoreConf.newMetastoreConf());

Review Comment:
   `internalSetup` method name implies it’s intended for internal use rather than being a publicly exposed method



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2462923240


##########
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/leader/TestLeaderElection.java:
##########
@@ -98,60 +90,21 @@ public void testLeaseLeaderElection() throws Exception {
     // elect1 as a leader now
     assertTrue(flag1.get() && instance1.isLeader());
 
-    configuration.setBoolean(LeaseLeaderElection.METASTORE_RENEW_LEASE, true);
     LeaseLeaderElection instance2 = new LeaseLeaderElection();
     AtomicBoolean flag2 = new AtomicBoolean(false);
     instance2.addStateListener(new TestLeaderListener(flag2));
     instance2.tryBeLeader(configuration, mutex);
     // instance2 should not be leader as elect1 holds the lease
     assertFalse(flag2.get() || instance2.isLeader());
 
-    ExecutorService service = Executors.newFixedThreadPool(4);
-    wait(service, flag1, flag2);
-    // now instance1 lease is timeout, the instance2 should be leader now
+    instance1.close();
+    synchronized (flag2) {

Review Comment:
   why do you need synchromized here? AtomicBoolean is designed for lock-free atomic updates



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2462924324


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/LeaseLeaderElection.java:
##########
@@ -64,10 +64,10 @@ public class LeaseLeaderElection implements LeaderElection<TableName> {
 
   private static final Logger LOG = LoggerFactory.getLogger(LeaseLeaderElection.class);
 
-  private static final AtomicLong ID = new AtomicLong();
+  protected static final AtomicLong ID = new AtomicLong();

Review Comment:
   minor: won't it be better if we expose protected fields via getters?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2465500144


##########
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/leader/TestLeaderElection.java:
##########
@@ -98,60 +90,21 @@ public void testLeaseLeaderElection() throws Exception {
     // elect1 as a leader now
     assertTrue(flag1.get() && instance1.isLeader());
 
-    configuration.setBoolean(LeaseLeaderElection.METASTORE_RENEW_LEASE, true);
     LeaseLeaderElection instance2 = new LeaseLeaderElection();
     AtomicBoolean flag2 = new AtomicBoolean(false);
     instance2.addStateListener(new TestLeaderListener(flag2));
     instance2.tryBeLeader(configuration, mutex);
     // instance2 should not be leader as elect1 holds the lease
     assertFalse(flag2.get() || instance2.isLeader());
 
-    ExecutorService service = Executors.newFixedThreadPool(4);
-    wait(service, flag1, flag2);
-    // now instance1 lease is timeout, the instance2 should be leader now
+    instance1.close();
+    synchronized (flag2) {

Review Comment:
   why do you need AtomicBoolean then here, isn't simple Boolean sufficient?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2465530889


##########
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/leader/TestLeaderElection.java:
##########
@@ -98,60 +90,21 @@ public void testLeaseLeaderElection() throws Exception {
     // elect1 as a leader now
     assertTrue(flag1.get() && instance1.isLeader());
 
-    configuration.setBoolean(LeaseLeaderElection.METASTORE_RENEW_LEASE, true);
     LeaseLeaderElection instance2 = new LeaseLeaderElection();
     AtomicBoolean flag2 = new AtomicBoolean(false);
     instance2.addStateListener(new TestLeaderListener(flag2));
     instance2.tryBeLeader(configuration, mutex);
     // instance2 should not be leader as elect1 holds the lease
     assertFalse(flag2.get() || instance2.isLeader());
 
-    ExecutorService service = Executors.newFixedThreadPool(4);
-    wait(service, flag1, flag2);
-    // now instance1 lease is timeout, the instance2 should be leader now
+    instance1.close();
+    synchronized (flag2) {

Review Comment:
   sorry, I meant Ref<Boolean>



##########
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/leader/TestLeaderElection.java:
##########
@@ -98,60 +90,21 @@ public void testLeaseLeaderElection() throws Exception {
     // elect1 as a leader now
     assertTrue(flag1.get() && instance1.isLeader());
 
-    configuration.setBoolean(LeaseLeaderElection.METASTORE_RENEW_LEASE, true);
     LeaseLeaderElection instance2 = new LeaseLeaderElection();
     AtomicBoolean flag2 = new AtomicBoolean(false);
     instance2.addStateListener(new TestLeaderListener(flag2));
     instance2.tryBeLeader(configuration, mutex);
     // instance2 should not be leader as elect1 holds the lease
     assertFalse(flag2.get() || instance2.isLeader());
 
-    ExecutorService service = Executors.newFixedThreadPool(4);
-    wait(service, flag1, flag2);
-    // now instance1 lease is timeout, the instance2 should be leader now
+    instance1.close();
+    synchronized (flag2) {

Review Comment:
   sorry, I meant `Ref<Boolean>`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2444201427


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/LeaseLeaderElection.java:
##########
@@ -131,7 +139,7 @@ private synchronized void doWork(LockResponse resp, Configuration conf,
     default:
       throw new IllegalStateException("Unexpected lock state: " + resp.getState());
     }
-    LOG.debug("Spent {}ms to notify the listeners, isLeader: {}", System.currentTimeMillis() - start, isLeader);
+    LOG.info("Spent {}ms to notify the listeners, isLeader: {}", System.currentTimeMillis() - start, isLeader);

Review Comment:
   why do we need this under info?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2444180152


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/LeaderElectionContext.java:
##########
@@ -100,8 +101,14 @@ public void start() throws Exception {
     Map<TTYPE, List<LeadershipStateListener>> listenerMap = this.listeners;
     if (conf.getBoolean(LEADER_IN_TEST, false)) {
       Map<TTYPE, List<LeadershipStateListener>> newListeners = new HashMap<>();
-      newListeners.put(TTYPE.HOUSEKEEPING, new ArrayList<>());
-      listenerMap.forEach((k, v) -> newListeners.get(TTYPE.HOUSEKEEPING).addAll(v));
+      List<LeadershipStateListener> listeners = new ArrayList<>();
+      listenerMap.forEach((k, v) -> listeners.addAll(v));
+      String testListenerCls = conf.get("metastore.leader.test.listener");

Review Comment:
   That is a very bad practice to bring test related things into production code! Try mocking/stubbing, etc.. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2444170233


##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetastoreHousekeepingLeader.java:
##########
@@ -34,7 +35,7 @@ public class TestMetastoreHousekeepingLeader extends MetastoreHousekeepingLeader
 
   @Before
   public void setUp() throws Exception {
-    internalSetup("localhost", true);
+    internalSetup("localhost", MetastoreConf.newMetastoreConf());

Review Comment:
   do you need extra conf param, it seems every tests passes the same thing



##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetastoreHousekeepingLeader.java:
##########
@@ -34,7 +35,7 @@ public class TestMetastoreHousekeepingLeader extends MetastoreHousekeepingLeader
 
   @Before
   public void setUp() throws Exception {
-    internalSetup("localhost", true);
+    internalSetup("localhost", MetastoreConf.newMetastoreConf());

Review Comment:
   do you need extra conf param? it seems every tests passes the same thing



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2444576095


##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/MetastoreHousekeepingLeaderTestBase.java:
##########
@@ -34,15 +35,16 @@
 import java.util.HashMap;
 import java.util.Map;
 import java.util.Set;
+import java.util.concurrent.CountDownLatch;
 import java.util.concurrent.TimeUnit;
 
 /**
  * Base class for HMS leader config testing.
  */
-class MetastoreHousekeepingLeaderTestBase {
+public class MetastoreHousekeepingLeaderTestBase {

Review Comment:
   ack



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2453851043


##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetastoreHousekeepingLeader.java:
##########
@@ -34,7 +35,7 @@ public class TestMetastoreHousekeepingLeader extends MetastoreHousekeepingLeader
 
   @Before
   public void setUp() throws Exception {
-    internalSetup("localhost", true);
+    internalSetup("localhost", MetastoreConf.newMetastoreConf());

Review Comment:
   Move the check to the parent class, https://github.com/apache/hive/pull/6142/files#diff-afdd9d2455383922b9eb4eaa0bae23dc39cd01a081100d00acb16b2e41322635R234



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2465514031


##########
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/leader/TestLeaderElection.java:
##########
@@ -98,60 +90,21 @@ public void testLeaseLeaderElection() throws Exception {
     // elect1 as a leader now
     assertTrue(flag1.get() && instance1.isLeader());
 
-    configuration.setBoolean(LeaseLeaderElection.METASTORE_RENEW_LEASE, true);
     LeaseLeaderElection instance2 = new LeaseLeaderElection();
     AtomicBoolean flag2 = new AtomicBoolean(false);
     instance2.addStateListener(new TestLeaderListener(flag2));
     instance2.tryBeLeader(configuration, mutex);
     // instance2 should not be leader as elect1 holds the lease
     assertFalse(flag2.get() || instance2.isLeader());
 
-    ExecutorService service = Executors.newFixedThreadPool(4);
-    wait(service, flag1, flag2);
-    // now instance1 lease is timeout, the instance2 should be leader now
+    instance1.close();
+    synchronized (flag2) {

Review Comment:
   `wait()` requires synchronization on the object, https://stackoverflow.com/questions/2779484/why-must-wait-always-be-in-synchronized-block



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2462921757


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java:
##########
@@ -388,6 +388,7 @@ LockResponse lock(LockRequest rqst)
    * @throws TxnAbortedException
    * @throws MetaException
    */
+  @SqlRetry(lockInternally = true)

Review Comment:
   could you please move it below `@RetrySemantics.SafeToRetry`



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java:
##########
@@ -425,7 +426,7 @@ void unlock(UnlockRequest rqst)
    * @throws TxnAbortedException
    * @throws MetaException
    */
-  @SqlRetry
+  @SqlRetry(lockInternally = true)

Review Comment:
   same



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6142:
URL: https://github.com/apache/hive/pull/6142#issuecomment-3436708888

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6142) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [10 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6142&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6142&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6142&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6142&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6142&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6142)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6142:
URL: https://github.com/apache/hive/pull/6142#issuecomment-3450499811

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6142) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [23 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6142&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6142&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6142&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6142&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6142&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6142)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2464395030


##########
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/leader/TestLeaderElection.java:
##########
@@ -98,60 +90,21 @@ public void testLeaseLeaderElection() throws Exception {
     // elect1 as a leader now
     assertTrue(flag1.get() && instance1.isLeader());
 
-    configuration.setBoolean(LeaseLeaderElection.METASTORE_RENEW_LEASE, true);
     LeaseLeaderElection instance2 = new LeaseLeaderElection();
     AtomicBoolean flag2 = new AtomicBoolean(false);
     instance2.addStateListener(new TestLeaderListener(flag2));
     instance2.tryBeLeader(configuration, mutex);
     // instance2 should not be leader as elect1 holds the lease
     assertFalse(flag2.get() || instance2.isLeader());
 
-    ExecutorService service = Executors.newFixedThreadPool(4);
-    wait(service, flag1, flag2);
-    // now instance1 lease is timeout, the instance2 should be leader now
+    instance1.close();
+    synchronized (flag2) {

Review Comment:
   we need to fetch the flag2 value back and check it in the `main` thread, which set by the leader in background



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2453566270


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/LeaderElectionContext.java:
##########
@@ -100,8 +101,14 @@ public void start() throws Exception {
     Map<TTYPE, List<LeadershipStateListener>> listenerMap = this.listeners;
     if (conf.getBoolean(LEADER_IN_TEST, false)) {
       Map<TTYPE, List<LeadershipStateListener>> newListeners = new HashMap<>();
-      newListeners.put(TTYPE.HOUSEKEEPING, new ArrayList<>());
-      listenerMap.forEach((k, v) -> newListeners.get(TTYPE.HOUSEKEEPING).addAll(v));
+      List<LeadershipStateListener> listeners = new ArrayList<>();
+      listenerMap.forEach((k, v) -> listeners.addAll(v));
+      String testListenerCls = conf.get("metastore.leader.test.listener");

Review Comment:
   Agreed,  IMO we need to trade off between the test coverage and the maintenance(code smell), and reach a balance. We also have some properties such as `hive.in*.test` for testing purpose.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2462919812


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/LeaderElectionContext.java:
##########
@@ -97,40 +95,32 @@ private LeaderElectionContext(String servHost, Configuration conf,
   }
 
   public void start() throws Exception {
-    Map<TTYPE, List<LeadershipStateListener>> listenerMap = this.listeners;
-    if (conf.getBoolean(LEADER_IN_TEST, false)) {
-      Map<TTYPE, List<LeadershipStateListener>> newListeners = new HashMap<>();
-      newListeners.put(TTYPE.HOUSEKEEPING, new ArrayList<>());
-      listenerMap.forEach((k, v) -> newListeners.get(TTYPE.HOUSEKEEPING).addAll(v));
-      listenerMap = newListeners;
-    }
-    for (Map.Entry<TTYPE, List<LeadershipStateListener>> entry :
-        listenerMap.entrySet()) {
-      List<LeadershipStateListener> listenerList = entry.getValue();
+    List<TTYPE> ttypes = new ArrayList<>(listeners.keySet());
+    Collections.shuffle(ttypes);

Review Comment:
   why do we need shuffle?



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/LeaderElectionContext.java:
##########
@@ -97,40 +95,32 @@ private LeaderElectionContext(String servHost, Configuration conf,
   }
 
   public void start() throws Exception {
-    Map<TTYPE, List<LeadershipStateListener>> listenerMap = this.listeners;
-    if (conf.getBoolean(LEADER_IN_TEST, false)) {
-      Map<TTYPE, List<LeadershipStateListener>> newListeners = new HashMap<>();
-      newListeners.put(TTYPE.HOUSEKEEPING, new ArrayList<>());
-      listenerMap.forEach((k, v) -> newListeners.get(TTYPE.HOUSEKEEPING).addAll(v));
-      listenerMap = newListeners;
-    }
-    for (Map.Entry<TTYPE, List<LeadershipStateListener>> entry :
-        listenerMap.entrySet()) {
-      List<LeadershipStateListener> listenerList = entry.getValue();
+    List<TTYPE> ttypes = new ArrayList<>(listeners.keySet());
+    Collections.shuffle(ttypes);

Review Comment:
   why do we need to shuffle?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2462919003


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/LeaderElectionContext.java:
##########
@@ -44,8 +45,6 @@ public class LeaderElectionContext {
   public enum TTYPE {
     HOUSEKEEPING(new TableName(Warehouse.DEFAULT_CATALOG_NAME, "__METASTORE_LEADER_ELECTION__",
         "metastore_housekeeping"), "housekeeping"),
-    WORKER(new TableName(Warehouse.DEFAULT_CATALOG_NAME, "__METASTORE_LEADER_ELECTION__",

Review Comment:
   was it used when we allowed Workers on HMS?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2453534698


##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetastoreHousekeepingLeader.java:
##########
@@ -34,7 +35,7 @@ public class TestMetastoreHousekeepingLeader extends MetastoreHousekeepingLeader
 
   @Before
   public void setUp() throws Exception {
-    internalSetup("localhost", true);
+    internalSetup("localhost", MetastoreConf.newMetastoreConf());

Review Comment:
   Interesting approach to instantiate and use a base test class within another test. 
   https://github.com/apache/hive/pull/6142/files#diff-bd73307ca500a5ceb6b862f1819eb919a4f03f8db1737fad425a1b684353c713R46 
   It might be cleaner to extract the service logic into a separate class.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2462923240


##########
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/leader/TestLeaderElection.java:
##########
@@ -98,60 +90,21 @@ public void testLeaseLeaderElection() throws Exception {
     // elect1 as a leader now
     assertTrue(flag1.get() && instance1.isLeader());
 
-    configuration.setBoolean(LeaseLeaderElection.METASTORE_RENEW_LEASE, true);
     LeaseLeaderElection instance2 = new LeaseLeaderElection();
     AtomicBoolean flag2 = new AtomicBoolean(false);
     instance2.addStateListener(new TestLeaderListener(flag2));
     instance2.tryBeLeader(configuration, mutex);
     // instance2 should not be leader as elect1 holds the lease
     assertFalse(flag2.get() || instance2.isLeader());
 
-    ExecutorService service = Executors.newFixedThreadPool(4);
-    wait(service, flag1, flag2);
-    // now instance1 lease is timeout, the instance2 should be leader now
+    instance1.close();
+    synchronized (flag2) {

Review Comment:
   why do you need synchronized here? AtomicBoolean is designed for lock-free atomic updates.
   ````
   while (!flag2.get()) {
       Thread.onSpinWait(); // JDK 9+
   }
   ````



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2462924920


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/leader/LeaderElectionContext.java:
##########
@@ -100,8 +101,14 @@ public void start() throws Exception {
     Map<TTYPE, List<LeadershipStateListener>> listenerMap = this.listeners;
     if (conf.getBoolean(LEADER_IN_TEST, false)) {
       Map<TTYPE, List<LeadershipStateListener>> newListeners = new HashMap<>();
-      newListeners.put(TTYPE.HOUSEKEEPING, new ArrayList<>());
-      listenerMap.forEach((k, v) -> newListeners.get(TTYPE.HOUSEKEEPING).addAll(v));
+      List<LeadershipStateListener> listeners = new ArrayList<>();
+      listenerMap.forEach((k, v) -> listeners.addAll(v));
+      String testListenerCls = conf.get("metastore.leader.test.listener");

Review Comment:
   appreciate the refactor. thank you, @dengzhhu653! the code is much easier to follow now



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2462923918


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java:
##########
@@ -425,7 +426,7 @@ void unlock(UnlockRequest rqst)
    * @throws TxnAbortedException
    * @throws MetaException
    */
-  @SqlRetry
+  @SqlRetry(lockInternally = true)

Review Comment:
   same



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29274: Flaky TestMetastoreLeaseLeader#testHouseKeepingThreads [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6142:
URL: https://github.com/apache/hive/pull/6142#discussion_r2444155536


##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/MetastoreHousekeepingLeaderTestBase.java:
##########
@@ -34,15 +35,16 @@
 import java.util.HashMap;
 import java.util.Map;
 import java.util.Set;
+import java.util.concurrent.CountDownLatch;
 import java.util.concurrent.TimeUnit;
 
 /**
  * Base class for HMS leader config testing.
  */
-class MetastoreHousekeepingLeaderTestBase {
+public class MetastoreHousekeepingLeaderTestBase {

Review Comment:
   can it be abstract?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org