You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "kevinrr888 (via GitHub)" <gi...@apache.org> on 2024/01/31 19:22:40 UTC
[I] Elasticity Broken or Flaky test: ReadWriteIT.sunnyDay() [accumulo]
kevinrr888 opened a new issue, #4209:
URL: https://github.com/apache/accumulo/issues/4209
**Test name(s)**
- ReadWriteIT.sunnyDay()
**Describe the failure observed**
The test does not always pass. The manager lock is sometimes held until the test times out.
The message logged until timeout: `[functional.ReadWriteIT] INFO : Manager lock is still held`
The error the test fails with: `org.opentest4j.AssertionFailedError: execution timed out after 360000 ms`
Error is occurring in the elasticity branch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Elasticity Broken or Flaky test: ReadWriteIT.sunnyDay() [accumulo]
Posted by "cshannon (via GitHub)" <gi...@apache.org>.
cshannon commented on issue #4209:
URL: https://github.com/apache/accumulo/issues/4209#issuecomment-1925875714
I took a look at fixing this today and I still could not get the Manager to shutdown when the Tservers are offline. I tried the following to try and make it shutdown but this did not work:
```java
public void shutdown() {
keepRunning.set(false);
fatePoolWatcher.shutdownNow();
try {
if (executor != null) {
executor.shutdown();
if (!executor.awaitTermination(5, SECONDS)) {
executor.shutdownNow();
}
}
while (workFinder.isAlive()) {
workFinder.interrupt();
workFinder.join(100);
}
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
}
```
I was hoping by calling shtudown now on the executors and then interrupting in a loop it would stop always but it still was hung on the same threads trying to scan.
The test itself could be fixed by simply not shutting down the Tservers until after the Manager shuts down but that doesn't help if the Manager and Tservers go offline in a real cluster at the same time (Manager can't talk to the Tservers when shutting down).
One option could be to set a timeout in TabletServerBatchReaderIterator on the scan to make sure it doesn't wait forever but not sure if that's a great idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Elasticity Broken or Flaky test: ReadWriteIT.sunnyDay() [accumulo]
Posted by "cshannon (via GitHub)" <gi...@apache.org>.
cshannon closed issue #4209: Elasticity Broken or Flaky test: ReadWriteIT.sunnyDay()
URL: https://github.com/apache/accumulo/issues/4209
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Elasticity Broken or Flaky test: ReadWriteIT.sunnyDay() [accumulo]
Posted by "DomGarguilo (via GitHub)" <gi...@apache.org>.
DomGarguilo commented on issue #4209:
URL: https://github.com/apache/accumulo/issues/4209#issuecomment-1919781601
I am able to reproduce this test failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Elasticity Broken or Flaky test: ReadWriteIT.sunnyDay() [accumulo]
Posted by "cshannon (via GitHub)" <gi...@apache.org>.
cshannon commented on issue #4209:
URL: https://github.com/apache/accumulo/issues/4209#issuecomment-1924920177
I looked into this a little bit with @keith-turner today as I saw the same thing. A thread dump shows stuck threads because the Manager is trying to scan the Tservers but they were shut down.
```
"manager" #42 prio=5 os_prio=0 cpu=168.68ms elapsed=176.53s tid=0x00007f938860de60 nid=0x3305b in Object.wait() [0x00007f93cef47000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base@17.0.10/Native Method)
- waiting on <no object reference available>
at java.lang.Thread.join(java.base@17.0.10/Thread.java:1313)
- locked <0x00000000f14f5d60> (a org.apache.accumulo.core.util.threads.Threads$AccumuloDaemonThread)
at java.lang.Thread.join(java.base@17.0.10/Thread.java:1381)
at org.apache.accumulo.core.fate.Fate.shutdown(Fate.java:472)
at org.apache.accumulo.manager.Manager.lambda$run$7(Manager.java:1148)
at org.apache.accumulo.manager.Manager$$Lambda$743/0x00007f93ec480248.accept(Unknown Source)
at java.lang.Iterable.forEach(java.base@17.0.10/Iterable.java:75)
at org.apache.accumulo.manager.Manager.run(Manager.java:1148)
at org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
at java.lang.Thread.run(java.base@17.0.10/Thread.java:840)
Locked ownable synchronizers:
- None
```
```
"Fate work finder" #67 daemon prio=5 os_prio=0 cpu=45.49ms elapsed=176.24s tid=0x00007f93540dfc20 nid=0x3309e waiting on condition [0x00007f93cd82f000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@17.0.10/Native Method)
- parking to wait for <0x00000000f14e2a48> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@17.0.10/LockSupport.java:211)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@17.0.10/AbstractQueuedSynchronizer.java:715)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(java.base@17.0.10/AbstractQueuedSynchronizer.java:1027)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(java.base@17.0.10/ReentrantReadWriteLock.java:738)
at org.apache.accumulo.core.clientImpl.ClientTabletCacheImpl._findTablet(ClientTabletCacheImpl.java:860)
at org.apache.accumulo.core.clientImpl.ClientTabletCacheImpl.findTablet(ClientTabletCacheImpl.java:552)
at org.apache.accumulo.core.clientImpl.ClientTabletCache.findTablet(ClientTabletCache.java:120)
at org.apache.accumulo.core.clientImpl.ClientTabletCacheImpl.lookupTablet(ClientTabletCacheImpl.java:712)
at org.apache.accumulo.core.clientImpl.ClientTabletCacheImpl.lookupTabletLocationAndCheckLock(ClientTabletCacheImpl.java:891)
at org.apache.accumulo.core.clientImpl.ClientTabletCacheImpl._findTablet(ClientTabletCacheImpl.java:876)
at org.apache.accumulo.core.clientImpl.ClientTabletCacheImpl.findTablet(ClientTabletCacheImpl.java:552)
at org.apache.accumulo.core.clientImpl.ThriftScanner.getNextScanAddress(ThriftScanner.java:559)
at org.apache.accumulo.core.clientImpl.ThriftScanner.scan(ThriftScanner.java:662)
at org.apache.accumulo.core.clientImpl.ScannerIterator.readBatch(ScannerIterator.java:159)
- locked <0x00000000f43e0938> (a org.apache.accumulo.core.clientImpl.ThriftScanner$ScanState)
at org.apache.accumulo.core.clientImpl.ScannerIterator.getNextBatch(ScannerIterator.java:177)
at org.apache.accumulo.core.clientImpl.ScannerIterator.hasNext(ScannerIterator.java:111)
at java.util.Iterator.forEachRemaining(java.base@17.0.10/Iterator.java:132)
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(java.base@17.0.10/Spliterators.java:1845)
at java.util.stream.AbstractPipeline.copyInto(java.base@17.0.10/AbstractPipeline.java:509)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(java.base@17.0.10/AbstractPipeline.java:499)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(java.base@17.0.10/ForEachOps.java:150)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(java.base@17.0.10/ForEachOps.java:173)
at java.util.stream.AbstractPipeline.evaluate(java.base@17.0.10/AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(java.base@17.0.10/ReferencePipeline.java:596)
at org.apache.accumulo.core.fate.AbstractFateStore.runnable(AbstractFateStore.java:163)
at org.apache.accumulo.core.logging.FateLogger$1.runnable(FateLogger.java:124)
at org.apache.accumulo.core.fate.Fate$WorkFinder.run(Fate.java:93)
at org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
at java.lang.Thread.run(java.base@17.0.10/Thread.java:840)
Locked ownable synchronizers:
- <0x00000000f1534078> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
```
```
"batch scanner 27--Worker-1" #194 daemon prio=5 os_prio=0 cpu=178.97ms elapsed=158.34s tid=0x00007f931c00ed70 nid=0x33334 sleeping [0x00007f9297dfb000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@17.0.10/Native Method)
at java.lang.Thread.sleep(java.base@17.0.10/Thread.java:344)
at java.util.concurrent.TimeUnit.sleep(java.base@17.0.10/TimeUnit.java:446)
at com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly(Uninterruptibles.java:406)
at org.apache.accumulo.core.clientImpl.RootClientTabletCache.findTablet(RootClientTabletCache.java:149)
at org.apache.accumulo.core.clientImpl.ClientTabletCache.findTablet(ClientTabletCache.java:120)
at org.apache.accumulo.core.clientImpl.ClientTabletCacheImpl.lookupTablet(ClientTabletCacheImpl.java:712)
at org.apache.accumulo.core.clientImpl.ClientTabletCacheImpl.lookupTabletLocationAndCheckLock(ClientTabletCacheImpl.java:891)
at org.apache.accumulo.core.clientImpl.ClientTabletCacheImpl._findTablet(ClientTabletCacheImpl.java:881)
at org.apache.accumulo.core.clientImpl.ClientTabletCacheImpl.findTablets(ClientTabletCacheImpl.java:340)
at org.apache.accumulo.core.clientImpl.ClientTabletCacheImpl.findTablets(ClientTabletCacheImpl.java:450)
at org.apache.accumulo.core.clientImpl.ClientTabletCache.binRanges(ClientTabletCache.java:174)
at org.apache.accumulo.core.clientImpl.SyncingClientTabletCache.binRanges(SyncingClientTabletCache.java:97)
at org.apache.accumulo.core.clientImpl.TimeoutClientTabletCache.binRanges(TimeoutClientTabletCache.java:110)
at org.apache.accumulo.core.clientImpl.TabletServerBatchReaderIterator.binRanges(TabletServerBatchReaderIterator.java:275)
at org.apache.accumulo.core.clientImpl.TabletServerBatchReaderIterator.processFailures(TabletServerBatchReaderIterator.java:375)
at org.apache.accumulo.core.clientImpl.TabletServerBatchReaderIterator$QueryTask.run(TabletServerBatchReaderIterator.java:485)
at org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.10/ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.10/ThreadPoolExecutor.java:635)
at org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
at java.lang.Thread.run(java.base@17.0.10/Thread.java:840)
Locked ownable synchronizers:
- <0x00000000f14e2a48> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
- <0x00000000f3dad138> (a java.util.concurrent.ThreadPoolExecutor$Worker)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org