You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by triumvir <pe...@alfa.de> on 2013/11/07 13:27:17 UTC
WARN | Store update waiting on 1 replica(s) to catch up to log
position 0
Hi,
I have installed a Replicated LevelDB Store with 3 zookeeper Server and 3
activemq Server. It works well, but if I test it with jmeter and kill a
activemq-master with kill -9 ( simulate power blackout ), often it works
well but at two times following message was logged endless to the logfile of
the slave which should become master ( controlled by the width variable in
the activemq.xml ).
*Store update waiting on 1 replica(s) to catch up to log position 0*
At this time, no activemq Instance is started and the activemq-cluster
doesn't work :-(
After I start the killed activemq server, the cluster works well again and
the slave become master.
--
View this message in context: http://activemq.2283324.n4.nabble.com/WARN-Store-update-waiting-on-1-replica-s-to-catch-up-to-log-position-0-tp4674001.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: WARN | Store update waiting on 1 replica(s) to catch up to log
position 0
Posted by triumvir <pe...@alfa.de>.
Hi,
i tried it with the version apache-activemq-5.10-20131106.134045-17-bin
today, no success :-(
WARN | Store update waiting on 1 replica(s) to catch up to log position
12184056. 1 slave nodes attaching.
I use jdk1.6.0_22 (I will try it with the newest version again)
Look at the stacktrace:
2013-11-08 14:39:55
Full thread dump Java HotSpot(TM) Server VM (17.1-b03 mixed mode):
"Attach Listener" daemon prio=3 tid=0x08f73800 nid=0x44 waiting on condition
[0x00000000]
java.lang.Thread.State: RUNNABLE
"LevelDB store io write" daemon prio=3 tid=0x08212c00 nid=0x43 waiting on
condition [0xd53a1000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xf835bd60> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
"ActiveMQ BrokerService[omACS-test-FS] Task-2" daemon prio=3 tid=0x08b2d400
nid=0x36 waiting on condition [0xd53f2000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xf34449c8> (a
java.util.concurrent.CountDownLatch$Sync)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)
at
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:253)
at
org.apache.activemq.leveldb.replicated.MasterLevelDBStore.wal_sync_to(MasterLevelDBStore.scala:429)
at
org.apache.activemq.leveldb.replicated.MasterLevelDBStore.doStart(MasterLevelDBStore.scala:113)
at
org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
at
org.apache.activemq.leveldb.replicated.ElectingLevelDBStore$$anonfun$start_master$1.apply$mcV$sp(ElectingLevelDBStore.scala:226)
at
org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:357)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
"hawtdispatch timer" daemon prio=3 tid=0x088ee000 nid=0x34 in Object.wait()
[0xd5494000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xf8318348> (a java.lang.Object)
at java.lang.Object.wait(Object.java:443)
at
org.fusesource.hawtdispatch.internal.TimerThread.run(TimerThread.java:188)
- locked <0xf8318348> (a java.lang.Object)
"hawtdispatch-DEFAULT-8" daemon prio=3 tid=0x08ca5800 nid=0x33 runnable
[0xd54e5000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:170)
at
sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0xf8307d50> (a sun.nio.ch.Util$1)
- locked <0xf8307d60> (a java.util.Collections$UnmodifiableSet)
- locked <0xf8307d08> (a sun.nio.ch.DevPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
at
org.fusesource.hawtdispatch.internal.NioManager$SelectStrategy.select(NioManager.java:52)
at
org.fusesource.hawtdispatch.internal.NioManager.select(NioManager.java:182)
at
org.fusesource.hawtdispatch.internal.pool.SimplePool.park(SimplePool.java:130)
at
org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:75)
"hawtdispatch-DEFAULT-7" daemon prio=3 tid=0x08ee5400 nid=0x32 runnable
[0xd5536000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:170)
at
sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0xf8310188> (a sun.nio.ch.Util$1)
- locked <0xf8310198> (a java.util.Collections$UnmodifiableSet)
- locked <0xf8310140> (a sun.nio.ch.DevPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
at
org.fusesource.hawtdispatch.internal.NioManager$SelectStrategy.select(NioManager.java:52)
at
org.fusesource.hawtdispatch.internal.NioManager.select(NioManager.java:182)
at
org.fusesource.hawtdispatch.internal.pool.SimplePool.park(SimplePool.java:130)
at
org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:75)
"hawtdispatch-DEFAULT-6" daemon prio=3 tid=0x08f29800 nid=0x31 runnable
[0xd5587000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:170)
at
sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0xf830daf0> (a sun.nio.ch.Util$1)
- locked <0xf830dae0> (a java.util.Collections$UnmodifiableSet)
- locked <0xf830d8e0> (a sun.nio.ch.DevPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
at
org.fusesource.hawtdispatch.internal.NioManager$SelectStrategy.select(NioManager.java:52)
at
org.fusesource.hawtdispatch.internal.NioManager.select(NioManager.java:182)
at
org.fusesource.hawtdispatch.internal.pool.SimplePool.park(SimplePool.java:130)
at
org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:75)
"hawtdispatch-DEFAULT-5" daemon prio=3 tid=0x08452400 nid=0x30 runnable
[0xd55d8000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:170)
at
sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0xf830e138> (a sun.nio.ch.Util$1)
- locked <0xf830e128> (a java.util.Collections$UnmodifiableSet)
- locked <0xf830df28> (a sun.nio.ch.DevPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
at
org.fusesource.hawtdispatch.internal.NioManager$SelectStrategy.select(NioManager.java:52)
at
org.fusesource.hawtdispatch.internal.NioManager.select(NioManager.java:182)
at
org.fusesource.hawtdispatch.internal.pool.SimplePool.park(SimplePool.java:130)
at
org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:75)
"hawtdispatch-DEFAULT-4" daemon prio=3 tid=0x08db3c00 nid=0x2f runnable
[0xd5629000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:170)
at
sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0xf830e828> (a sun.nio.ch.Util$1)
- locked <0xf830e818> (a java.util.Collections$UnmodifiableSet)
- locked <0xf830e618> (a sun.nio.ch.DevPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
at
org.fusesource.hawtdispatch.internal.NioManager$SelectStrategy.select(NioManager.java:52)
at
org.fusesource.hawtdispatch.internal.NioManager.select(NioManager.java:182)
at
org.fusesource.hawtdispatch.internal.pool.SimplePool.park(SimplePool.java:130)
at
org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:75)
"hawtdispatch-DEFAULT-3" daemon prio=3 tid=0x08f94800 nid=0x2e runnable
[0xd567a000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:170)
at
sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0xf831c458> (a sun.nio.ch.Util$1)
- locked <0xf831c468> (a java.util.Collections$UnmodifiableSet)
- locked <0xf831c410> (a sun.nio.ch.DevPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
at
org.fusesource.hawtdispatch.internal.NioManager$SelectStrategy.select(NioManager.java:52)
at
org.fusesource.hawtdispatch.internal.NioManager.select(NioManager.java:182)
at
org.fusesource.hawtdispatch.internal.pool.SimplePool.park(SimplePool.java:130)
at
org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:75)
"hawtdispatch-DEFAULT-2" daemon prio=3 tid=0x08bad800 nid=0x2d runnable
[0xd56cb000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:170)
at
sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0xf83187e0> (a sun.nio.ch.Util$1)
- locked <0xf83187f0> (a java.util.Collections$UnmodifiableSet)
- locked <0xf8318798> (a sun.nio.ch.DevPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
at
org.fusesource.hawtdispatch.internal.NioManager$SelectStrategy.select(NioManager.java:52)
at
org.fusesource.hawtdispatch.internal.NioManager.select(NioManager.java:182)
at
org.fusesource.hawtdispatch.internal.pool.SimplePool.park(SimplePool.java:130)
at
org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:75)
"hawtdispatch-DEFAULT-1" daemon prio=3 tid=0x08db0000 nid=0x2c runnable
[0xd571c000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:170)
at
sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0xf8314368> (a sun.nio.ch.Util$1)
- locked <0xf8314378> (a java.util.Collections$UnmodifiableSet)
- locked <0xf8314320> (a sun.nio.ch.DevPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
at
org.fusesource.hawtdispatch.internal.NioManager$SelectStrategy.select(NioManager.java:52)
at
org.fusesource.hawtdispatch.internal.NioManager.select(NioManager.java:182)
at
org.fusesource.hawtdispatch.internal.pool.SimplePool.park(SimplePool.java:130)
at
org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:75)
"main-EventThread" daemon prio=3 tid=0x08f5f800 nid=0x2b waiting on
condition [0xd576d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xf832c080> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
"main-SendThread(localhost:2182)" daemon prio=3 tid=0x08f5f000 nid=0x2a
runnable [0xd57be000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:170)
at
sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0xf8318c50> (a sun.nio.ch.Util$1)
- locked <0xf8318c60> (a java.util.Collections$UnmodifiableSet)
- locked <0xf8318c08> (a sun.nio.ch.DevPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
"ZooKeeper state change dispatcher thread" daemon prio=3 tid=0x09036c00
nid=0x29 waiting on condition [0xd5aad000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xf8303ea8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at
org.apache.activemq.leveldb.replicated.groups.ZKClient$StateChangeDispatcher.run(ZKClient.java:347)
"RMI RenewClean-[172.25.0.67:38828]" daemon prio=3 tid=0x08f72000 nid=0x27
in Object.wait() [0xd587e000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xf8330070> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0xf8330070> (a java.lang.ref.ReferenceQueue$Lock)
at
sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:516)
at java.lang.Thread.run(Thread.java:662)
"RMI Scheduler(0)" daemon prio=3 tid=0x093d0800 nid=0x26 waiting on
condition [0xd58eb000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xdae08408> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
"GC Daemon" daemon prio=3 tid=0x09027000 nid=0x24 in Object.wait()
[0xd598d000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xdac08048> (a sun.misc.GC$LatencyLock)
at sun.misc.GC$Daemon.run(GC.java:100)
- locked <0xdac08048> (a sun.misc.GC$LatencyLock)
"RMI Reaper" prio=3 tid=0x09026000 nid=0x23 in Object.wait() [0xd59de000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xdac0c028> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0xdac0c028> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:333)
at java.lang.Thread.run(Thread.java:662)
"RMI TCP Accept-0" daemon prio=3 tid=0x08f20800 nid=0x22 runnable
[0xd5a5c000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
- locked <0xdac10020> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
at java.lang.Thread.run(Thread.java:662)
"RMI TCP Accept-1099" daemon prio=3 tid=0x08f35400 nid=0x20 runnable
[0xd5afe000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
- locked <0xdac17fd8> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
at java.lang.Thread.run(Thread.java:662)
"qtp22041176-21" prio=3 tid=0x08c50000 nid=0x1f waiting on condition
[0xd5bab000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xdac00108> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:662)
"qtp22041176-20" prio=3 tid=0x08b3f000 nid=0x1e waiting on condition
[0xd5bfc000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xdac00108> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:662)
"qtp22041176-19" prio=3 tid=0x08f76000 nid=0x1d waiting on condition
[0xd5c4d000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xdac00108> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:662)
"qtp22041176-18" prio=3 tid=0x08b8f400 nid=0x1c waiting on condition
[0xd5c9e000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xdac00108> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:662)
"qtp22041176-17 Acceptor1 SelectChannelConnector@omACS-test-FS:8161" prio=3
tid=0x08e3fc00 nid=0x1b runnable [0xd5cef000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
- locked <0xdac10258> (a java.lang.Object)
at
org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
at
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:662)
"qtp22041176-16 Acceptor0 SelectChannelConnector@omACS-test-FS:8161" prio=3
tid=0x09028c00 nid=0x1a waiting for monitor entry [0xd5d40000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:129)
- waiting to lock <0xdac10258> (a java.lang.Object)
at
org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
at
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:662)
"qtp22041176-15 Selector1" prio=3 tid=0x091eec00 nid=0x19 runnable
[0xd5d91000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:170)
at
sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0xdac082d8> (a sun.nio.ch.Util$1)
- locked <0xdac082e8> (a java.util.Collections$UnmodifiableSet)
- locked <0xdac08290> (a sun.nio.ch.DevPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
at
org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:662)
"qtp22041176-14 Selector0" prio=3 tid=0x08c53800 nid=0x18 runnable
[0xd5de2000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:170)
at
sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0xdac0c1d0> (a sun.nio.ch.Util$1)
- locked <0xdac0c1e0> (a java.util.Collections$UnmodifiableSet)
- locked <0xdac0c188> (a sun.nio.ch.DevPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
at
org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:662)
"HashSessionScavenger-3" daemon prio=3 tid=0x08dcd000 nid=0x17 in
Object.wait() [0xd5e33000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xdac14128> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0xdac14128> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"HashSessionScavenger-2" daemon prio=3 tid=0x091eb400 nid=0x16 in
Object.wait() [0xd5e84000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xdac04110> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0xdac04110> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"HashSessionScavenger-1" daemon prio=3 tid=0x08bc4400 nid=0x15 in
Object.wait() [0xd5efd000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xdac001b8> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0xdac001b8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"HashSessionScavenger-0" daemon prio=3 tid=0x0899cc00 nid=0x14 in
Object.wait() [0xd5f4e000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xdac141e0> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0xdac141e0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"RMI TCP Accept-0" daemon prio=3 tid=0x0833f000 nid=0x12 runnable
[0xd634e000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
- locked <0xdac142a0> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at
sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
at java.lang.Thread.run(Thread.java:662)
"Low Memory Detector" daemon prio=3 tid=0x081a5c00 nid=0x11 runnable
[0x00000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread1" daemon prio=3 tid=0x081a3000 nid=0x10 waiting on condition
[0x00000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread0" daemon prio=3 tid=0x081a1400 nid=0xf waiting on condition
[0x00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=3 tid=0x0819fc00 nid=0xe runnable
[0x00000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=3 tid=0x0818b400 nid=0xd in Object.wait()
[0xd665e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xdac00428> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0xdac00428> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=3 tid=0x08189c00 nid=0xc in Object.wait()
[0xfae5a000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xdac08038> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0xdac08038> (a java.lang.ref.Reference$Lock)
"main" prio=3 tid=0x08070000 nid=0x2 waiting on condition [0xfe30e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xdac56970> (a
java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
at
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
at
org.apache.activemq.leveldb.replicated.ElectingLevelDBStore$$anon$1.start(ElectingLevelDBStore.scala:207)
at
org.apache.activemq.broker.LockableServiceSupport.preStart(LockableServiceSupport.java:94)
at
org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:54)
at
org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:613)
at
org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:602)
at
org.apache.activemq.broker.BrokerService.start(BrokerService.java:567)
at
org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:88)
at
org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
at
org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:150)
at
org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
at
org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.activemq.console.Main.runTaskClass(Main.java:262)
at org.apache.activemq.console.Main.main(Main.java:115)
"VM Thread" prio=3 tid=0x08187400 nid=0xb runnable
"GC task thread#0 (ParallelGC)" prio=3 tid=0x08078400 nid=0x3 runnable
"GC task thread#1 (ParallelGC)" prio=3 tid=0x08079400 nid=0x4 runnable
"GC task thread#2 (ParallelGC)" prio=3 tid=0x0807a800 nid=0x5 runnable
"GC task thread#3 (ParallelGC)" prio=3 tid=0x0807bc00 nid=0x6 runnable
"GC task thread#4 (ParallelGC)" prio=3 tid=0x0807d000 nid=0x7 runnable
"GC task thread#5 (ParallelGC)" prio=3 tid=0x0807e400 nid=0x8 runnable
"GC task thread#6 (ParallelGC)" prio=3 tid=0x0807fc00 nid=0x9 runnable
"GC task thread#7 (ParallelGC)" prio=3 tid=0x08081000 nid=0xa runnable
"VM Periodic Task Thread" prio=3 tid=0x08383c00 nid=0x13 waiting on
condition
JNI global references: 1229
--
View this message in context: http://activemq.2283324.n4.nabble.com/WARN-Store-update-waiting-on-1-replica-s-to-catch-up-to-log-position-0-tp4674001p4674072.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: WARN | Store update waiting on 1 replica(s) to catch up to log
position 0
Posted by Raj Khanna <kh...@gmail.com>.
Thanks guys!! I found my problem with the help of Peter, many thanks to him.
I had a issue in my amq config. My slave were trying to write in the same
dir where master was writing and hence the contention. But I was able to
move forward. Thanks!
--
View this message in context: http://activemq.2283324.n4.nabble.com/WARN-Store-update-waiting-on-1-replica-s-to-catch-up-to-log-position-0-tp4674001p4674598.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: WARN | Store update waiting on 1 replica(s) to catch up to log
position 0
Posted by Hiram Chirino <hi...@hiramchirino.com>.
Raj,
Your slave probably can't connect to your master. Could be a firewall
issue. Or it could be that the master is not advertising the right
hostname for clients to connect to in ZooKeeper. Try configuring the
hostname attribute on the replicatedLevelDB attribute. That might fix
it for you.
On Sun, Nov 10, 2013 at 11:28 AM, Raj Khanna <kh...@gmail.com> wrote:
> I am trying to use the setup as explained in the post with JDK "1.7.0_21",
> ZooKepper 3.4.3 and AMQ 5.10 SNAPSHOT version. But I can not get rid of the
> following error
>
> WARN | Store update waiting on 1 replica(s) to catch up to log position 0.
>
> I think the problem is that I dont see the Slave started message in the
> console.
>
> Could you please give me a pointer or solution to this.
>
> Thanks in advance,
> Raj
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/WARN-Store-update-waiting-on-1-replica-s-to-catch-up-to-log-position-0-tp4674001p4674128.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
--
Hiram Chirino
Engineering | Red Hat, Inc.
hchirino@redhat.com | fusesource.com | redhat.com
skype: hiramchirino | twitter: @hiramchirino
blog: Hiram Chirino's Bit Mojo
Re: WARN | Store update waiting on 1 replica(s) to catch up to log
position 0
Posted by triumvir <pe...@alfa.de>.
Raj,
I am execute "*netstat -an | grep <openwire-port>*" at the activemq-Servers.
If a slave becomes Master, the activemq transportConnectors-Port(s) (you
have configured at the activemq.xml file) will run up and has the state
"LISTEN".
Only the activemq-Master starts the transportConnectors (openwire,...)
Port(s), on the slaves the connection-Port(s) aren't start.
This methode helps me, to see which Server ist the Master without analyse
Logfiles :-)
--
View this message in context: http://activemq.2283324.n4.nabble.com/WARN-Store-update-waiting-on-1-replica-s-to-catch-up-to-log-position-0-tp4674001p4674163.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: WARN | Store update waiting on 1 replica(s) to catch up to log
position 0
Posted by Raj Khanna <kh...@gmail.com>.
I am trying to use the setup as explained in the post with JDK "1.7.0_21",
ZooKepper 3.4.3 and AMQ 5.10 SNAPSHOT version. But I can not get rid of the
following error
WARN | Store update waiting on 1 replica(s) to catch up to log position 0.
I think the problem is that I dont see the Slave started message in the
console.
Could you please give me a pointer or solution to this.
Thanks in advance,
Raj
--
View this message in context: http://activemq.2283324.n4.nabble.com/WARN-Store-update-waiting-on-1-replica-s-to-catch-up-to-log-position-0-tp4674001p4674128.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: WARN | Store update waiting on 1 replica(s) to catch up to log
position 0
Posted by Christian Posta <ch...@gmail.com>.
Thanks for the update... that's weird though... i'll give it a shot
later today on 1.6 and 1.7 and see if i can reproduce...
On Fri, Nov 8, 2013 at 7:12 AM, triumvir <pe...@alfa.de> wrote:
> Hi,
>
> with the java version jdk1.7.0_21 it works well. I tried it since 5 minutes
> and can't produce the error. Perhapse this was the solution :-)
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/WARN-Store-update-waiting-on-1-replica-s-to-catch-up-to-log-position-0-tp4674001p4674074.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
--
Christian Posta
http://www.christianposta.com/blog
twitter: @christianposta
Re: WARN | Store update waiting on 1 replica(s) to catch up to log
position 0
Posted by triumvir <pe...@alfa.de>.
Hi,
with the java version jdk1.7.0_21 it works well. I tried it since 5 minutes
and can't produce the error. Perhapse this was the solution :-)
--
View this message in context: http://activemq.2283324.n4.nabble.com/WARN-Store-update-waiting-on-1-replica-s-to-catch-up-to-log-position-0-tp4674001p4674074.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: WARN | Store update waiting on 1 replica(s) to catch up to log
position 0
Posted by Christian Posta <ch...@gmail.com>.
Would it be possible to try with a 5.10 snapshot? There were some
recent fixes around leveldb replication, and before trying to
reproduce on my side, would be good to know you see it in the latest:
https://repository.apache.org/content/repositories/snapshots/org/apache/activemq/apache-activemq/5.10-SNAPSHOT/
On Thu, Nov 7, 2013 at 5:27 AM, triumvir <pe...@alfa.de> wrote:
> Hi,
>
> I have installed a Replicated LevelDB Store with 3 zookeeper Server and 3
> activemq Server. It works well, but if I test it with jmeter and kill a
> activemq-master with kill -9 ( simulate power blackout ), often it works
> well but at two times following message was logged endless to the logfile of
> the slave which should become master ( controlled by the width variable in
> the activemq.xml ).
>
> *Store update waiting on 1 replica(s) to catch up to log position 0*
>
> At this time, no activemq Instance is started and the activemq-cluster
> doesn't work :-(
>
> After I start the killed activemq server, the cluster works well again and
> the slave become master.
>
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/WARN-Store-update-waiting-on-1-replica-s-to-catch-up-to-log-position-0-tp4674001.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
--
Christian Posta
http://www.christianposta.com/blog
twitter: @christianposta