Re: Build failed in Jenkins: HBase-1.2 » latest1.7,Hadoop #397

[Stack <st...@duboce.net>]

This is interesting fail. We seem to be super slow downloading artifacts
and then when we run, we fail in thrift... the zombie detector finds a
bunch of tests that have been running for a while but we are not clean in
our reporting... might be hard to be clean in the reporting here.

This 1.7 suite has not failed in a good while on 1.2. Lets see if an
anomaly.

St.Ack


On Tue, Nov 24, 2015 at 9:43 AM, Apache Jenkins Server <
jenkins@builds.apache.org> wrote:

> See <
> https://builds.apache.org/job/HBase-1.2/jdk=latest1.7,label=Hadoop/397/>
>
> ------------------------------------------
> [...truncated 52380 lines...]
>         - parking to wait for  <0x00000007c6e5e470> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1198)
>         at java.lang.Thread.run(Thread.java:745)
>
> "sync.0" prio=10 tid=0x00007fc28003e000 nid=0x1e34 waiting on condition
> [0x00007fc2c07ad000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000007c6e5e178> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1198)
>         at java.lang.Thread.run(Thread.java:745)
>
> "main.append-pool2-t1" prio=10 tid=0x00007fc304325000 nid=0x1e33 waiting
> on condition [0x00007fc2c0ab0000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000007c6ddd810> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>         at
> com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:45)
>         at
> com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:55)
>         at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:123)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>
> "LruBlockCacheStatsExecutor" daemon prio=10 tid=0x00007fc28002c000
> nid=0x1a06 waiting on condition [0x00007fc2c08ae000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000007d1818068> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>         at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
>         at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>         at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>
> "StoreOpener-e05706f03e597b6b832634d9d43c4f1c-1.LruBlockCache.EvictionThread"
> daemon prio=10 tid=0x00007fc280026000 nid=0x1a05 in Object.wait()
> [0x00007fc2c09af000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x00000007d1820068> (a
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
>         at
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:824)
>         - locked <0x00000007d1820068> (a
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
>         at java.lang.Thread.run(Thread.java:745)
>
> "sync.4" prio=10 tid=0x00007fc298007800 nid=0x19fd waiting on condition
> [0x00007fc2c0bb1000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000007d1895cc0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1198)
>         at java.lang.Thread.run(Thread.java:745)
>
> "sync.3" prio=10 tid=0x00007fc298005800 nid=0x19fc waiting on condition
> [0x00007fc2c0cb2000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000007d1830068> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1198)
>         at java.lang.Thread.run(Thread.java:745)
>
> "sync.2" prio=10 tid=0x00007fc298004000 nid=0x19fb waiting on condition
> [0x00007fc2c0db3000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000007d1838068> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1198)
>         at java.lang.Thread.run(Thread.java:745)
>
> "sync.1" prio=10 tid=0x00007fc298002800 nid=0x19fa waiting on condition
> [0x00007fc2c0eb4000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000007d1810520> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1198)
>         at java.lang.Thread.run(Thread.java:745)
>
> "sync.0" prio=10 tid=0x00007fc298001000 nid=0x19f9 waiting on condition
> [0x00007fc2c1329000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000007d18200e8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1198)
>         at java.lang.Thread.run(Thread.java:745)
>
> "main.append-pool1-t1" prio=10 tid=0x00007fc30500c000 nid=0x19f8 waiting
> on condition [0x00007fc2c142a000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000007d1840068> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>         at
> com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:45)
>         at
> com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:55)
>         at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:123)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>
> "Timer for 'HBase' metrics system" daemon prio=10 tid=0x00007fc304fd7000
> nid=0x19f1 in Object.wait() [0x00007fc2c11b9000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x00000007d1810638> (a java.util.TaskQueue)
>         at java.util.TimerThread.mainLoop(Timer.java:552)
>         - locked <0x00000007d1810638> (a java.util.TaskQueue)
>         at java.util.TimerThread.run(Timer.java:505)
>
> "process reaper" daemon prio=10 tid=0x00007fc304ce2800 nid=0x19c5 waiting
> on condition [0x00007fc2c1463000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000007d18618f8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>         at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>         at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>         at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
>         at
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
>         at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>
> "Service Thread" daemon prio=10 tid=0x00007fc30429b800 nid=0x1967 runnable
> [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread1" daemon prio=10 tid=0x00007fc304299800 nid=0x1966
> waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" daemon prio=10 tid=0x00007fc304296800 nid=0x1965
> waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" daemon prio=10 tid=0x00007fc304294000 nid=0x1964
> runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Finalizer" daemon prio=10 tid=0x00007fc30426b800 nid=0x1960 in
> Object.wait() [0x00007fc2c2bc9000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x00000007d1810760> (a
> java.lang.ref.ReferenceQueue$Lock)
>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
>         - locked <0x00000007d1810760> (a java.lang.ref.ReferenceQueue$Lock)
>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
>         at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>
> "Reference Handler" daemon prio=10 tid=0x00007fc304269800 nid=0x195e in
> Object.wait() [0x00007fc2c2cca000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x00000007d1838178> (a java.lang.ref.Reference$Lock)
>         at java.lang.Object.wait(Object.java:503)
>         at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
>         - locked <0x00000007d1838178> (a java.lang.ref.Reference$Lock)
>
> "main" prio=10 tid=0x00007fc30400a800 nid=0x194b waiting on condition
> [0x00007fc30b5ee000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>         at java.lang.Thread.sleep(Native Method)
>         at
> org.apache.hadoop.hbase.util.Threads.sleepWithoutInterrupt(Threads.java:164)
>         at
> org.apache.hadoop.hbase.io.hfile.TestScannerSelectionUsingTTL.testScannerSelection(TestScannerSelectionUsingTTL.java:117)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>         at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>         at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>         at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>         at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>         at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>         at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>         at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>         at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>         at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>         at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>         at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>         at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>         at org.junit.runners.Suite.runChild(Suite.java:128)
>         at org.junit.runners.Suite.runChild(Suite.java:27)
>         at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>         at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>         at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>         at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>         at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>         at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>         at org.junit.runners.Suite.runChild(Suite.java:128)
>         at org.junit.runners.Suite.runChild(Suite.java:27)
>         at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>         at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>         at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>         at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>         at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>         at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>         at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
>         at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
>         at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:108)
>         at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:78)
>         at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:54)
>         at
> org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:144)
>         at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
>         at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
>         at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
>
> "VM Thread" prio=10 tid=0x00007fc304265000 nid=0x1959 runnable
>
> "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007fc304020800 nid=0x194c
> runnable
>
> "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007fc304022800 nid=0x194d
> runnable
>
> "GC task thread#2 (ParallelGC)" prio=10 tid=0x00007fc304024000 nid=0x194e
> runnable
>
> "GC task thread#3 (ParallelGC)" prio=10 tid=0x00007fc304026000 nid=0x194f
> runnable
>
> "GC task thread#4 (ParallelGC)" prio=10 tid=0x00007fc304028000 nid=0x1950
> runnable
>
> "GC task thread#5 (ParallelGC)" prio=10 tid=0x00007fc304029800 nid=0x1951
> runnable
>
> "GC task thread#6 (ParallelGC)" prio=10 tid=0x00007fc30402b800 nid=0x1952
> runnable
>
> "GC task thread#7 (ParallelGC)" prio=10 tid=0x00007fc30402d800 nid=0x1953
> runnable
>
> "GC task thread#8 (ParallelGC)" prio=10 tid=0x00007fc30402f800 nid=0x1954
> runnable
>
> "GC task thread#9 (ParallelGC)" prio=10 tid=0x00007fc304031000 nid=0x1955
> runnable
>
> "GC task thread#10 (ParallelGC)" prio=10 tid=0x00007fc304033000 nid=0x1956
> runnable
>
> "GC task thread#11 (ParallelGC)" prio=10 tid=0x00007fc304035000 nid=0x1957
> runnable
>
> "GC task thread#12 (ParallelGC)" prio=10 tid=0x00007fc304036800 nid=0x1958
> runnable
>
> "VM Periodic Task Thread" prio=10 tid=0x00007fc3042a6800 nid=0x1968
> waiting on condition
>
> JNI global references: 242
>
> + exit 1
> POST BUILD TASK : FAILURE
> END OF POST BUILD TASK : 0
> Archiving artifacts
> Recording test results
>