You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "kevinrr888 (via GitHub)" <gi...@apache.org> on 2025/08/21 17:03:16 UTC

[PR] Small FATE improvements [accumulo]

kevinrr888 opened a new pull request, #5813:
URL: https://github.com/apache/accumulo/pull/5813

   This PR partially addresses #5787
   
   I have reached a dead end with debugging this test.
   
   The test logic has no issues as far as I can tell and the FATE logic (as far as I can tell) has one potential concurrency issue (which I addressed in this PR), but the failure still occurs occassionally. From jstacking the test process in a failure case, it appears that the thread is either getting stuck on the `workQueue.poll(100, MILLISECONDS)` call or it is repeatedly retrying it, neither of which should be possible given the shutdown logic. Here is the code:
   ```
   while (fate.getKeepRunning().get() && !stop.get()) {
       FateId unreservedFateId = workQueue.poll(100, MILLISECONDS);
       ...
   ```
   The jstack trace shows this throughout the time FATE is trying to shutdown:
   ```
   "accumulo.pool.manager.fate.user.commit_compaction.namespace_create.namespace_delete.namespace_rename.shutdown_tserver.system_split.system_merge.table_bulk_import2.table_cancel_compact.table_clone.table_compact-Worker-1" #57 daemon prio=5 os_prio=0 cpu=82600.00ms elapsed=86.73s tid=0x00007693e00058f0 nid=0x2304f runnable  [0x00007694a5ef9000]
      java.lang.Thread.State: RUNNABLE
           at java.util.concurrent.LinkedTransferQueue.awaitMatch(java.base@17.0.15/LinkedTransferQueue.java:652)
           at java.util.concurrent.LinkedTransferQueue.xfer(java.base@17.0.15/LinkedTransferQueue.java:616)
           at java.util.concurrent.LinkedTransferQueue.poll(java.base@17.0.15/LinkedTransferQueue.java:1294)
           at org.apache.accumulo.core.fate.FateExecutor$TransactionRunner.reserveFateTx(FateExecutor.java:349)
           at org.apache.accumulo.core.fate.FateExecutor$TransactionRunner.run(FateExecutor.java:378)
           at org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
           at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.15/ThreadPoolExecutor.java:1136)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.15/ThreadPoolExecutor.java:635)
           at org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
           at java.lang.Thread.run(java.base@17.0.15/Thread.java:840)
   ```
   This doesn't make sense as:
   1) When we shutdown FATE, we first set keepRunning to false, so the while loop should terminate
   2) The poll will return after, at most, 100ms
   
   I have run out of ideas. This could use another set of eyes, if anyone has the time. I can explain anything in regards to test logic or the fate logic, if needed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Small FATE improvements [accumulo]

Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #5813:
URL: https://github.com/apache/accumulo/pull/5813#discussion_r2294065958


##########
core/src/main/java/org/apache/accumulo/core/fate/FateExecutor.java:
##########
@@ -306,18 +306,19 @@ public void run() {
             }
           });
         } catch (Exception e) {
-          if (!fate.getKeepRunning().get() || isShutdown()) {
-            log.debug("Expected failure while attempting to find work for fate: either fate is "
-                + "being shutdown and therefore all fate threads are being shutdown or the "
-                + "fate threads assigned to work on {} were invalidated by config changes "
-                + "and are being shutdown", fateOps, e);
-          } else {
-            log.warn("Unexpected failure while attempting to find work for fate", e);
-          }
-
+          log.warn("Unexpected failure while attempting to find work for fate", e);
           workQueue.clear();
         }
       }
+
+      if (!fate.getKeepRunning().get() || isShutdown()) {

Review Comment:
   Catching the exception is inside the while loop and this if is outside the while loop, so doe snot seem like it would reach the else in case of exception.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Small FATE improvements [accumulo]

Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on PR #5813:
URL: https://github.com/apache/accumulo/pull/5813#issuecomment-3212168676

   So, I'm seeing it fail like this currently:
   ```
   org.opentest4j.AssertionFailedError: expected: <0> but was: <3>
   	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
   	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
   	at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
   	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:150)
   	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:145)
   	at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:531)
   	at org.apache.accumulo.test.fate.FatePoolsWatcherITBase.testIncrease1(FatePoolsWatcherITBase.java:177)
   	at org.apache.accumulo.test.fate.user.UserFatePoolsWatcherIT_SimpleSuite.executeTest(UserFatePoolsWatcherIT_SimpleSuite.java:56)
   	at org.apache.accumulo.test.fate.FateTestRunner.executeTest(FateTestRunner.java:33)
   	at org.apache.accumulo.test.fate.FatePoolsWatcherITBase.testIncrease1(FatePoolsWatcherITBase.java:73)
   	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
   	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
   	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
   	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
   	at java.base/java.lang.Thread.run(Thread.java:840)
   ```
   
   I do have some changes locally, but they haven't made much of a difference. What's interesting here is that this is saying there are 3 tasks and fails this check in the finally block:
   ```
         assertEquals(0, fate.getTotalTxRunnersActive());
   ```
   
   You only create two tasks in the `testIncrease1` method...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Small FATE improvements [accumulo]

Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on PR #5813:
URL: https://github.com/apache/accumulo/pull/5813#issuecomment-3211873528

   > Could add some prints or logging around the poll call to try to see which one of those is happening.
   
   Yeah, I tried this, but was never able to get it to fail with added logging here


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Small FATE improvements [accumulo]

Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #5813:
URL: https://github.com/apache/accumulo/pull/5813#discussion_r2293987349


##########
core/src/main/java/org/apache/accumulo/core/fate/FateExecutor.java:
##########
@@ -205,14 +205,14 @@ protected Set<Fate.FateOperation> getFateOps() {
    * Initiates the shutdown of this FateExecutor. This means the pool executing TransactionRunners
    * will no longer accept new TransactionRunners, the currently running TransactionRunners will
    * terminate after they are done with their current transaction, if applicable, and the work
-   * finder is interrupted. {@link #isShutdown()} returns true after this is called.
+   * finder is shutdown. {@link #isShutdown()} returns true after this is called.
    */
   protected void initiateShutdown() {
     transactionExecutor.shutdown();
     synchronized (runningTxRunners) {
       runningTxRunners.forEach(TransactionRunner::flagStop);
     }
-    workFinder.interrupt();

Review Comment:
   Interrupting  threads can be useful for the case when they are stuck on IO and not checking the boolean.  Was this interrupt causing a problem?  It does seem inconsistent, we at not interrupting other threads, so removing the interrupt makes it consistent.



##########
core/src/main/java/org/apache/accumulo/core/fate/FateExecutor.java:
##########
@@ -306,18 +306,19 @@ public void run() {
             }
           });
         } catch (Exception e) {
-          if (!fate.getKeepRunning().get() || isShutdown()) {
-            log.debug("Expected failure while attempting to find work for fate: either fate is "
-                + "being shutdown and therefore all fate threads are being shutdown or the "
-                + "fate threads assigned to work on {} were invalidated by config changes "
-                + "and are being shutdown", fateOps, e);
-          } else {
-            log.warn("Unexpected failure while attempting to find work for fate", e);
-          }
-
+          log.warn("Unexpected failure while attempting to find work for fate", e);
           workQueue.clear();
         }
       }
+
+      if (!fate.getKeepRunning().get() || isShutdown()) {

Review Comment:
   Maybe this if is not needed.  Seems like these should always be true the way the code is structured.  Was wondering if there should be else for this if, looking at the code not sure what the else should be.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Small FATE improvements [accumulo]

Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5813:
URL: https://github.com/apache/accumulo/pull/5813#discussion_r2294012052


##########
core/src/main/java/org/apache/accumulo/core/fate/FateExecutor.java:
##########
@@ -306,18 +306,19 @@ public void run() {
             }
           });
         } catch (Exception e) {
-          if (!fate.getKeepRunning().get() || isShutdown()) {
-            log.debug("Expected failure while attempting to find work for fate: either fate is "
-                + "being shutdown and therefore all fate threads are being shutdown or the "
-                + "fate threads assigned to work on {} were invalidated by config changes "
-                + "and are being shutdown", fateOps, e);
-          } else {
-            log.warn("Unexpected failure while attempting to find work for fate", e);
-          }
-
+          log.warn("Unexpected failure while attempting to find work for fate", e);
           workQueue.clear();
         }
       }
+
+      if (!fate.getKeepRunning().get() || isShutdown()) {

Review Comment:
   If an unexpected failure occurred (ie the catch Exception block), we could reach this code and this statement would not be true. Could add an else



##########
core/src/main/java/org/apache/accumulo/core/fate/FateExecutor.java:
##########
@@ -205,14 +205,14 @@ protected Set<Fate.FateOperation> getFateOps() {
    * Initiates the shutdown of this FateExecutor. This means the pool executing TransactionRunners
    * will no longer accept new TransactionRunners, the currently running TransactionRunners will
    * terminate after they are done with their current transaction, if applicable, and the work
-   * finder is interrupted. {@link #isShutdown()} returns true after this is called.
+   * finder is shutdown. {@link #isShutdown()} returns true after this is called.
    */
   protected void initiateShutdown() {
     transactionExecutor.shutdown();
     synchronized (runningTxRunners) {
       runningTxRunners.forEach(TransactionRunner::flagStop);
     }
-    workFinder.interrupt();

Review Comment:
   It seems cleaner to allow the work finder to gracefully stop via the check isShutdown() in the while loop. It also removes some error handling where we handle an error as expected, which isn't great and makes the logs confusing



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Small FATE improvements [accumulo]

Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on PR #5813:
URL: https://github.com/apache/accumulo/pull/5813#issuecomment-3211804113

   > it appears that the thread is either getting stuck on the workQueue.poll(100, MILLISECONDS) call or it is repeatedly retrying it
   
   Could add some prints or logging around the poll call to try to see which one of those is happening.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Small FATE improvements [accumulo]

Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on PR #5813:
URL: https://github.com/apache/accumulo/pull/5813#issuecomment-3214771967

   Created #5818


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Small FATE improvements [accumulo]

Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on PR #5813:
URL: https://github.com/apache/accumulo/pull/5813#issuecomment-3214086577

   Created #5817 from something I found yesterday. It doesn't fix the test though...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Small FATE improvements [accumulo]

Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on PR #5813:
URL: https://github.com/apache/accumulo/pull/5813#issuecomment-3212637373

   Opened #5816 after looking into the failure and making the following changes locally from https://github.com/apache/accumulo/commit/3a0d8e5c72915fb4c7ce26d6156d15b0a444fb71.  Ran the test and got it time out and saw the same odd stack traces again the first time it printed, but they were not there the second time.  Also w/ the extra logging for the catch saw a test failure that I have not looked into yet.  The exception from the assertion in the finallly was obscuring any exceptions from the test.
   
   ```diff
   diff --git a/test/src/main/java/org/apache/accumulo/test/fate/FatePoolsWatcherITBase.java b/test/src/main/java/org/apache/accumulo/test/fate/FatePoolsWatcherITBase.java
   index a4fb0b8f9e..72fc00f57f 100644
   --- a/test/src/main/java/org/apache/accumulo/test/fate/FatePoolsWatcherITBase.java
   +++ b/test/src/main/java/org/apache/accumulo/test/fate/FatePoolsWatcherITBase.java
   @@ -36,6 +36,7 @@ import org.apache.accumulo.core.fate.FateId;
    import org.apache.accumulo.core.fate.FateInstanceType;
    import org.apache.accumulo.core.fate.FateStore;
    import org.apache.accumulo.core.fate.Repo;
   +import org.apache.accumulo.core.util.UtilWaitThread;
    import org.apache.accumulo.harness.SharedMiniClusterBase;
    import org.apache.accumulo.server.ServerContext;
    import org.apache.accumulo.test.util.Wait;
   @@ -171,8 +172,31 @@ public abstract class FatePoolsWatcherITBase extends SharedMiniClusterBase
          assertEquals(0, fate.getTxRunnersActive(set2));
          assertEquals(numWorkersSet3, fate.getTxRunnersActive(set3));
          assertEquals(numWorkersSet4, fate.getTxRunnersActive(set4));
   +    } catch (Throwable e) {
   +      System.err.println("SAW ERROR");
   +      e.printStackTrace();
   +      throw e;
        } finally {
          fate.shutdown(30, TimeUnit.SECONDS);
   +      if (fate.getTotalTxRunnersActive() > 0) {
   +        System.out.println("BEGIN STACKS 1");
   +        Thread.getAllStackTraces().forEach((t, se) -> {
   +          System.out.println(" " + t.getName());
   +          for (var e : se) {
   +            System.out.println("\t" + e);
   +          }
   +        });
   +        System.out.println("END STACKS 1");
   +        UtilWaitThread.sleep(2000);
   +        System.out.println("BEGIN STACKS 2");
   +        Thread.getAllStackTraces().forEach((t, se) -> {
   +          System.out.println(" " + t.getName());
   +          for (var e : se) {
   +            System.out.println("\t" + e);
   +          }
   +        });
   +        System.out.println("END STACKS 2");
   +      }
          assertEquals(0, fate.getTotalTxRunnersActive());
        }
      }
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Small FATE improvements [accumulo]

Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5813:
URL: https://github.com/apache/accumulo/pull/5813#discussion_r2294108992


##########
core/src/main/java/org/apache/accumulo/core/fate/FateExecutor.java:
##########
@@ -306,18 +306,19 @@ public void run() {
             }
           });
         } catch (Exception e) {
-          if (!fate.getKeepRunning().get() || isShutdown()) {
-            log.debug("Expected failure while attempting to find work for fate: either fate is "
-                + "being shutdown and therefore all fate threads are being shutdown or the "
-                + "fate threads assigned to work on {} were invalidated by config changes "
-                + "and are being shutdown", fateOps, e);
-          } else {
-            log.warn("Unexpected failure while attempting to find work for fate", e);
-          }
-
+          log.warn("Unexpected failure while attempting to find work for fate", e);
           workQueue.clear();
         }
       }
+
+      if (!fate.getKeepRunning().get() || isShutdown()) {

Review Comment:
   You are correct. I'll remove the if statement



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Small FATE improvements [accumulo]

Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on PR #5813:
URL: https://github.com/apache/accumulo/pull/5813#issuecomment-3212500139

   Made the following changes against 3a0d8e5c72915fb4c7ce26d6156d15b0a444fb71 which is the current head of main.  
   
   ```diff
   diff --git a/test/src/main/java/org/apache/accumulo/test/fate/FatePoolsWatcherITBase.java b/test/src/main/java/org/apache/accumulo/test/fate/FatePoolsWatcherITBase.java
   index a4fb0b8f9e..70f6341246 100644
   --- a/test/src/main/java/org/apache/accumulo/test/fate/FatePoolsWatcherITBase.java
   +++ b/test/src/main/java/org/apache/accumulo/test/fate/FatePoolsWatcherITBase.java
   @@ -173,6 +173,14 @@ public abstract class FatePoolsWatcherITBase extends SharedMiniClusterBase
          assertEquals(numWorkersSet4, fate.getTxRunnersActive(set4));
        } finally {
          fate.shutdown(30, TimeUnit.SECONDS);
   +      if (fate.getTotalTxRunnersActive() > 0) {
   +        Thread.getAllStackTraces().forEach((t, se) -> {
   +          System.out.println(" " + t.getName());
   +          for (var e : se) {
   +            System.out.println("\t" + e);
   +          }
   +        });
   +      }
          assertEquals(0, fate.getTotalTxRunnersActive());
        }
      }
   ```
   
   Ran the test a few times and then it finally failed and saw the following odd stack traces in the output in the failsafe dir.  These are all of the stacks that contained FateExecutor.  Not sure what is going on w/ the class loader stuff.
   
   ```
    accumulo.pool.manager.fate.user.table_create.table_delete.table_delete_range.table_export.table_import.table_merge.table_offline.table_online.table_rename.table_split.table_tablet_availability-Worker-1
           java.base@17.0.16/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:651)
           java.base@17.0.16/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:639)
           java.base@17.0.16/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
           java.base@17.0.16/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
           app//org.apache.accumulo.core.fate.FateExecutor$TransactionRunner.blockIfHadoopShutdown(FateExecutor.java:457)
           app//org.apache.accumulo.core.fate.FateExecutor$TransactionRunner.run(FateExecutor.java:390)
           app//org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
           java.base@17.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
           java.base@17.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
           app//org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
           java.base@17.0.16/java.lang.Thread.run(Thread.java:840)
    accumulo.pool.manager.fate.user.table_create.table_delete.table_delete_range.table_export.table_import.table_merge.table_offline.table_online.table_rename.table_split.table_tablet_availability-Worker-4
           app//org.apache.logging.log4j.core.pattern.DatePatternConverter$CachedTime.<init>(DatePatternConverter.java:166)
           app//org.apache.logging.log4j.core.pattern.DatePatternConverter.formatWithoutThreadLocals(DatePatternConverter.java:330)
           app//org.apache.logging.log4j.core.pattern.DatePatternConverter.format(DatePatternConverter.java:309)
           app//org.apache.logging.log4j.core.pattern.DatePatternConverter.format(DatePatternConverter.java:284)
           app//org.apache.logging.log4j.core.pattern.PatternFormatter.format(PatternFormatter.java:44)
           app//org.apache.logging.log4j.core.layout.PatternLayout$PatternFormatterPatternSerializer.toSerializable(PatternLayout.java:397)
           app//org.apache.logging.log4j.core.layout.PatternLayout.toText(PatternLayout.java:252)
           app//org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:238)
           app//org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:58)
           app//org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:227)
           app//org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:220)
           app//org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:211)
           app//org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:160)
           app//org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:133)
           app//org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:124)
           app//org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:88)
           app//org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:714)
           app//org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:672)
           app//org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:648)
           app//org.apache.logging.log4j.core.config.LoggerConfig.logParent(LoggerConfig.java:705)
           app//org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:674)
           app//org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:648)
           app//org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:584)
           app//org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:92)
           app//org.apache.logging.log4j.core.Logger.log(Logger.java:187)
           app//org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2904)
           app//org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2857)
           app//org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2839)
           app//org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2618)
           app//org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:2565)
           app//org.apache.logging.slf4j.Log4jLogger.error(Log4jLogger.java:313)
           app//org.apache.accumulo.core.fate.FateExecutor$TransactionRunner.run(FateExecutor.java:406)
           app//org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
           java.base@17.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
           java.base@17.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
           app//org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
           java.base@17.0.16/java.lang.Thread.run(Thread.java:840)
    accumulo.pool.manager.fate.user.commit_compaction.namespace_create.namespace_delete.namespace_rename.shutdown_tserver.system_split.system_merge.table_bulk_import2.table_cancel_compact.table_clone.table_compact-Worker-1
           java.base@17.0.16/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:842)
           java.base@17.0.16/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:760)
           java.base@17.0.16/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:681)
           java.base@17.0.16/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:639)
           java.base@17.0.16/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
           java.base@17.0.16/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
           app//org.apache.accumulo.core.fate.FateExecutor$TransactionRunner.blockIfHadoopShutdown(FateExecutor.java:457)
           app//org.apache.accumulo.core.fate.FateExecutor$TransactionRunner.run(FateExecutor.java:390)
           app//org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
           java.base@17.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
           java.base@17.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
           app//org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
           java.base@17.0.16/java.lang.Thread.run(Thread.java:840)
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Small FATE improvements [accumulo]

Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on PR #5813:
URL: https://github.com/apache/accumulo/pull/5813#issuecomment-3214328197

   @keith-turner @dlmarion - Some more details that I didn't mention in my initial comment:
   
   * This test fails in several different possible ways:
       * One is the 3 tasks are still executing when we expect 0 that Dave mentioned. I confirmed that only 2 FATE ops were created and reserved in this case, so my only idea is that a TransactionRunner is somehow stuck/still running (which again, does not seem possible given the shutdown logic). The other 2 threads may be stuck because the test failed before the latch countdown, I can't remember when the failure occurs for this case. Either way, it seems at least one TransactionRunner is stuck in the while loop.
       * Another failure occurs at this `waitFor`: `fate.getTotalTxRunnersActive() == newNumWorkersSet1 + 1 + numWorkersSet3 + numWorkersSet4`. For the 30 second period, it will see that the pool which is shutdown due to config changes has 2 threads still running instead of the 1 expected (the only 1 stuck should be the worker working on the transaction). This causes the `waitFor` to fail, then the shutdown will fail and hide the `waitFor` exception since the shutdown is in a `finally`.
       * Another failure occurs after ALL checks and waitFors in the try occur (confirmed with prints). So the very last assertion in the try passes. Then when it tries to shutdown fate, one thread is stuck (even though we've confirmed our FATE work has completed successfully).
   
   All these failure cases appear to show that a thread is working on some phantom FATE transaction (seems impossible given how FATE and this test are setup), is continuously executing the `while` (seems impossible given the FATE shutdown logic), or got blocked on some operation in the `while` (from jstacking the process, the thread was stuck on/repeatedly retrying the `poll()`, which again seems impossible)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Small FATE improvements [accumulo]

Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 merged PR #5813:
URL: https://github.com/apache/accumulo/pull/5813


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org