You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "cshannon (via GitHub)" <gi...@apache.org> on 2024/02/03 00:00:45 UTC
Re: [I] Elasticity Broken or Flaky test: ReadWriteIT.sunnyDay() [accumulo]
cshannon commented on issue #4209:
URL: https://github.com/apache/accumulo/issues/4209#issuecomment-1924920177
I looked into this a little bit with @keith-turner today as I saw the same thing. A thread dump shows stuck threads because the Manager is trying to scan the Tservers but they were shut down.
```
"manager" #42 prio=5 os_prio=0 cpu=168.68ms elapsed=176.53s tid=0x00007f938860de60 nid=0x3305b in Object.wait() [0x00007f93cef47000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base@17.0.10/Native Method)
- waiting on <no object reference available>
at java.lang.Thread.join(java.base@17.0.10/Thread.java:1313)
- locked <0x00000000f14f5d60> (a org.apache.accumulo.core.util.threads.Threads$AccumuloDaemonThread)
at java.lang.Thread.join(java.base@17.0.10/Thread.java:1381)
at org.apache.accumulo.core.fate.Fate.shutdown(Fate.java:472)
at org.apache.accumulo.manager.Manager.lambda$run$7(Manager.java:1148)
at org.apache.accumulo.manager.Manager$$Lambda$743/0x00007f93ec480248.accept(Unknown Source)
at java.lang.Iterable.forEach(java.base@17.0.10/Iterable.java:75)
at org.apache.accumulo.manager.Manager.run(Manager.java:1148)
at org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
at java.lang.Thread.run(java.base@17.0.10/Thread.java:840)
Locked ownable synchronizers:
- None
```
```
"Fate work finder" #67 daemon prio=5 os_prio=0 cpu=45.49ms elapsed=176.24s tid=0x00007f93540dfc20 nid=0x3309e waiting on condition [0x00007f93cd82f000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@17.0.10/Native Method)
- parking to wait for <0x00000000f14e2a48> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@17.0.10/LockSupport.java:211)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@17.0.10/AbstractQueuedSynchronizer.java:715)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(java.base@17.0.10/AbstractQueuedSynchronizer.java:1027)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(java.base@17.0.10/ReentrantReadWriteLock.java:738)
at org.apache.accumulo.core.clientImpl.ClientTabletCacheImpl._findTablet(ClientTabletCacheImpl.java:860)
at org.apache.accumulo.core.clientImpl.ClientTabletCacheImpl.findTablet(ClientTabletCacheImpl.java:552)
at org.apache.accumulo.core.clientImpl.ClientTabletCache.findTablet(ClientTabletCache.java:120)
at org.apache.accumulo.core.clientImpl.ClientTabletCacheImpl.lookupTablet(ClientTabletCacheImpl.java:712)
at org.apache.accumulo.core.clientImpl.ClientTabletCacheImpl.lookupTabletLocationAndCheckLock(ClientTabletCacheImpl.java:891)
at org.apache.accumulo.core.clientImpl.ClientTabletCacheImpl._findTablet(ClientTabletCacheImpl.java:876)
at org.apache.accumulo.core.clientImpl.ClientTabletCacheImpl.findTablet(ClientTabletCacheImpl.java:552)
at org.apache.accumulo.core.clientImpl.ThriftScanner.getNextScanAddress(ThriftScanner.java:559)
at org.apache.accumulo.core.clientImpl.ThriftScanner.scan(ThriftScanner.java:662)
at org.apache.accumulo.core.clientImpl.ScannerIterator.readBatch(ScannerIterator.java:159)
- locked <0x00000000f43e0938> (a org.apache.accumulo.core.clientImpl.ThriftScanner$ScanState)
at org.apache.accumulo.core.clientImpl.ScannerIterator.getNextBatch(ScannerIterator.java:177)
at org.apache.accumulo.core.clientImpl.ScannerIterator.hasNext(ScannerIterator.java:111)
at java.util.Iterator.forEachRemaining(java.base@17.0.10/Iterator.java:132)
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(java.base@17.0.10/Spliterators.java:1845)
at java.util.stream.AbstractPipeline.copyInto(java.base@17.0.10/AbstractPipeline.java:509)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(java.base@17.0.10/AbstractPipeline.java:499)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(java.base@17.0.10/ForEachOps.java:150)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(java.base@17.0.10/ForEachOps.java:173)
at java.util.stream.AbstractPipeline.evaluate(java.base@17.0.10/AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(java.base@17.0.10/ReferencePipeline.java:596)
at org.apache.accumulo.core.fate.AbstractFateStore.runnable(AbstractFateStore.java:163)
at org.apache.accumulo.core.logging.FateLogger$1.runnable(FateLogger.java:124)
at org.apache.accumulo.core.fate.Fate$WorkFinder.run(Fate.java:93)
at org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
at java.lang.Thread.run(java.base@17.0.10/Thread.java:840)
Locked ownable synchronizers:
- <0x00000000f1534078> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
```
```
"batch scanner 27--Worker-1" #194 daemon prio=5 os_prio=0 cpu=178.97ms elapsed=158.34s tid=0x00007f931c00ed70 nid=0x33334 sleeping [0x00007f9297dfb000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@17.0.10/Native Method)
at java.lang.Thread.sleep(java.base@17.0.10/Thread.java:344)
at java.util.concurrent.TimeUnit.sleep(java.base@17.0.10/TimeUnit.java:446)
at com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly(Uninterruptibles.java:406)
at org.apache.accumulo.core.clientImpl.RootClientTabletCache.findTablet(RootClientTabletCache.java:149)
at org.apache.accumulo.core.clientImpl.ClientTabletCache.findTablet(ClientTabletCache.java:120)
at org.apache.accumulo.core.clientImpl.ClientTabletCacheImpl.lookupTablet(ClientTabletCacheImpl.java:712)
at org.apache.accumulo.core.clientImpl.ClientTabletCacheImpl.lookupTabletLocationAndCheckLock(ClientTabletCacheImpl.java:891)
at org.apache.accumulo.core.clientImpl.ClientTabletCacheImpl._findTablet(ClientTabletCacheImpl.java:881)
at org.apache.accumulo.core.clientImpl.ClientTabletCacheImpl.findTablets(ClientTabletCacheImpl.java:340)
at org.apache.accumulo.core.clientImpl.ClientTabletCacheImpl.findTablets(ClientTabletCacheImpl.java:450)
at org.apache.accumulo.core.clientImpl.ClientTabletCache.binRanges(ClientTabletCache.java:174)
at org.apache.accumulo.core.clientImpl.SyncingClientTabletCache.binRanges(SyncingClientTabletCache.java:97)
at org.apache.accumulo.core.clientImpl.TimeoutClientTabletCache.binRanges(TimeoutClientTabletCache.java:110)
at org.apache.accumulo.core.clientImpl.TabletServerBatchReaderIterator.binRanges(TabletServerBatchReaderIterator.java:275)
at org.apache.accumulo.core.clientImpl.TabletServerBatchReaderIterator.processFailures(TabletServerBatchReaderIterator.java:375)
at org.apache.accumulo.core.clientImpl.TabletServerBatchReaderIterator$QueryTask.run(TabletServerBatchReaderIterator.java:485)
at org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.10/ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.10/ThreadPoolExecutor.java:635)
at org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
at java.lang.Thread.run(java.base@17.0.10/Thread.java:840)
Locked ownable synchronizers:
- <0x00000000f14e2a48> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
- <0x00000000f3dad138> (a java.util.concurrent.ThreadPoolExecutor$Worker)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org