You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by kal123 <kp...@gmail.com> on 2013/11/15 14:46:19 UTC
replicatedLevelDB errors after failover
I am using the latest changes from trunk and i see following error after
failover (still getting corrupted?)
2013-11-15 06:44:53,606 | INFO | jolokia-agent: Using access restrictor
classpath:/jolokia-access.xml | /hawtio | main
2013-11-15 06:44:53,743 | INFO | ActiveMQ WebConsole available at
http://localhost:8161/ | org.apache.activemq.web.WebConsoleStarter | main
2013-11-15 06:44:53,754 | INFO | Initializing Spring FrameworkServlet
'dispatcher' | /admin | main
2013-11-15 06:44:56,969 | INFO | Stopping BrokerService[largeamq] due to
exception, java.io.IOException |
org.apache.activemq.util.DefaultIOExceptionHandler | LevelDB IOException
handler.
java.io.IOException
at
org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:39)
at
org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:554)
at
org.apache.activemq.leveldb.LevelDBClient.might_fail_using_index(LevelDBClient.scala:1021)
at
org.apache.activemq.leveldb.LevelDBClient.collectionCursor(LevelDBClient.scala:1320)
at
org.apache.activemq.leveldb.LevelDBClient.queueCursor(LevelDBClient.scala:1244)
at
org.apache.activemq.leveldb.DBManager.cursorMessages(DBManager.scala:708)
at
org.apache.activemq.leveldb.LevelDBStore$LevelDBMessageStore.recoverNextMessages(LevelDBStore.scala:756)
at
org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:106)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:258)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:145)
at
org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
at
org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1876)
at
org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2086)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1581)
at org.apache.activemq.broker.region.Queue.wakeup(Queue.java:1803)
at
org.apache.activemq.broker.region.Queue.addSubscription(Queue.java:464)
at
org.apache.activemq.broker.region.AbstractRegion.addConsumer(AbstractRegion.java:314)
at
org.apache.activemq.broker.region.RegionBroker.addConsumer(RegionBroker.java:400)
at
org.apache.activemq.broker.jmx.ManagedRegionBroker.addConsumer(ManagedRegionBroker.java:230)
at
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:97)
at
org.apache.activemq.advisory.AdvisoryBroker.addConsumer(AdvisoryBroker.java:102)
at
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:97)
at
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:97)
at
org.apache.activemq.broker.MutableBrokerFilter.addConsumer(MutableBrokerFilter.java:102)
at
org.apache.activemq.broker.TransportConnection.processAddConsumer(TransportConnection.java:587)
at
org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo.java:349)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292)
--
View this message in context: http://activemq.2283324.n4.nabble.com/replicatedLevelDB-errors-after-failover-tp4674550.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: replicatedLevelDB errors after failover
Posted by mtod <mt...@thetods.net>.
Was this ever resolved?
I'm testing a leveldb cluster and was performing some fail over and I now
have the same problem.
<http://activemq.2283324.n4.nabble.com/file/n4705479/12-29-2015_7-03-15_AM.bmp>
--
View this message in context: http://activemq.2283324.n4.nabble.com/replicatedLevelDB-errors-after-failover-tp4674550p4705479.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: replicatedLevelDB errors after failover
Posted by kal123 <kp...@gmail.com>.
The leveldb seems to be corrupted after 10 failovers, broker was not able to
load records from leveldb, similar to the issue described in early post but
now i see these additional error msg
--
View this message in context: http://activemq.2283324.n4.nabble.com/replicatedLevelDB-errors-after-failover-tp4674550p4675532.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: replicatedLevelDB errors after failover
Posted by Hiram Chirino <hi...@hiramchirino.com>.
I suspect those 'Could not load message seq' warnings can be ignored.
Can you verify that there has been no message loss?
On Fri, Dec 6, 2013 at 12:19 PM, kal123 <kp...@gmail.com> wrote:
> Tried testing with dec. 5 snapshot and getting following errors after about
> 10 failovers:
>
> 2013-12-06 12:06:39,673 | WARN | Could not load message seq: 45760 from
> DataLocator(630f8b1, 2262) | org.apache.activemq.leveldb.LevelDBClient |
> ActiveMQ NIO Worker 2
> 2013-12-06 12:06:39,673 | WARN | No reader available for position: 63101c1,
> log_infos:
> {104859611=LogInfo(/solidstate/ldb/00000000064007db.log,104859611,104859172),
> 209718783=LogInfo(/solidstate/ldb/000000000c800dff.log,209718783,104859220),
> 314578003=LogInfo(/solidstate/ldb/0000000012c01453.log,314578003,104858557),
> 419436560=LogInfo(/solidstate/ldb/0000000019001810.log,419436560,104858088),
> 524294648=LogInfo(/solidstate/ldb/000000001f4019f8.log,524294648,0)} |
> org.apache.activemq.leveldb.RecordLog | ActiveMQ NIO Worker 2
> 2013-12-06 12:06:39,673 | WARN | Could not load message seq: 45761 from
> DataLocator(63101c1, 2262) | org.apache.activemq.leveldb.LevelDBClient |
> ActiveMQ NIO Worker 2
> 2013-12-06 12:06:39,673 | WARN | No reader available for position: 6310ad1,
> log_infos:
> {104859611=LogInfo(/solidstate/ldb/00000000064007db.log,104859611,104859172),
> 209718783=LogInfo(/solidstate/ldb/000000000c800dff.log,209718783,104859220),
> 314578003=LogInfo(/solidstate/ldb/0000000012c01453.log,314578003,104858557),
> 419436560=LogInfo(/solidstate/ldb/0000000019001810.log,419436560,104858088),
> 524294648=LogInfo(/solidstate/ldb/000000001f4019f8.log,524294648,0)} |
> org.apache.activemq.leveldb.RecordLog | ActiveMQ NIO Worker 2
> 2013-12-06 12:06:39,673 | WARN | Could not load message seq: 45762 from
> DataLocator(6310ad1, 2262) | org.apache.activemq.leveldb.LevelDBClient |
> ActiveMQ NIO Worker 2
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/replicatedLevelDB-errors-after-failover-tp4674550p4675255.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
--
Hiram Chirino
Engineering | Red Hat, Inc.
hchirino@redhat.com | fusesource.com | redhat.com
skype: hiramchirino | twitter: @hiramchirino
blog: Hiram Chirino's Bit Mojo
Re: replicatedLevelDB errors after failover
Posted by kal123 <kp...@gmail.com>.
Tried testing with dec. 5 snapshot and getting following errors after about
10 failovers:
2013-12-06 12:06:39,673 | WARN | Could not load message seq: 45760 from
DataLocator(630f8b1, 2262) | org.apache.activemq.leveldb.LevelDBClient |
ActiveMQ NIO Worker 2
2013-12-06 12:06:39,673 | WARN | No reader available for position: 63101c1,
log_infos:
{104859611=LogInfo(/solidstate/ldb/00000000064007db.log,104859611,104859172),
209718783=LogInfo(/solidstate/ldb/000000000c800dff.log,209718783,104859220),
314578003=LogInfo(/solidstate/ldb/0000000012c01453.log,314578003,104858557),
419436560=LogInfo(/solidstate/ldb/0000000019001810.log,419436560,104858088),
524294648=LogInfo(/solidstate/ldb/000000001f4019f8.log,524294648,0)} |
org.apache.activemq.leveldb.RecordLog | ActiveMQ NIO Worker 2
2013-12-06 12:06:39,673 | WARN | Could not load message seq: 45761 from
DataLocator(63101c1, 2262) | org.apache.activemq.leveldb.LevelDBClient |
ActiveMQ NIO Worker 2
2013-12-06 12:06:39,673 | WARN | No reader available for position: 6310ad1,
log_infos:
{104859611=LogInfo(/solidstate/ldb/00000000064007db.log,104859611,104859172),
209718783=LogInfo(/solidstate/ldb/000000000c800dff.log,209718783,104859220),
314578003=LogInfo(/solidstate/ldb/0000000012c01453.log,314578003,104858557),
419436560=LogInfo(/solidstate/ldb/0000000019001810.log,419436560,104858088),
524294648=LogInfo(/solidstate/ldb/000000001f4019f8.log,524294648,0)} |
org.apache.activemq.leveldb.RecordLog | ActiveMQ NIO Worker 2
2013-12-06 12:06:39,673 | WARN | Could not load message seq: 45762 from
DataLocator(6310ad1, 2262) | org.apache.activemq.leveldb.LevelDBClient |
ActiveMQ NIO Worker 2
--
View this message in context: http://activemq.2283324.n4.nabble.com/replicatedLevelDB-errors-after-failover-tp4674550p4675255.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: replicatedLevelDB errors after failover
Posted by kal123 <kp...@gmail.com>.
Ran failover test again and after about 25 or so fail overs I got the below
errors on the master. I did not see any other errors. I will try to run
with debug turned on next week to see it helps with debugging.
Note: even after this error i was able to do few more failover before things
stopped working properly.
After few more failovers, the system was failing over automatically
continuously within the cluster and producer and consumer stopped working.
ACTIVEMQ_DATA: /activemq_5_10/apache-activemq-5.10-SNAPSHOT/data
Loading message broker from: xbean:activemq.xml
java.io.IOException: invalid record position 1809753339 (file:
/solidstate/ldb/000000006a404a30.log, offset: 27155147)
at
org.apache.activemq.leveldb.RecordLog$LogReader.read(RecordLog.scala:316)
at
org.apache.activemq.leveldb.RecordLog$$anonfun$read$1.apply(RecordLog.scala:560)
at
org.apache.activemq.leveldb.RecordLog$$anonfun$read$1.apply(RecordLog.scala:560)
at
org.apache.activemq.leveldb.RecordLog$$anonfun$get_reader$1.apply(RecordLog.scala:552)
at
org.apache.activemq.leveldb.RecordLog$$anonfun$get_reader$1.apply(RecordLog.scala:534)
at scala.Option.map(Option.scala:133)
at
org.apache.activemq.leveldb.RecordLog.get_reader(RecordLog.scala:534)
at org.apache.activemq.leveldb.RecordLog.read(RecordLog.scala:560)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$replay_from$1.apply$mcV$sp(LevelDBClient.scala:734)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$replay_from$1.apply(LevelDBClient.scala:701)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$replay_from$1.apply(LevelDBClient.scala:701)
at
org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:551)
at
org.apache.activemq.leveldb.LevelDBClient.replay_from(LevelDBClient.scala:700)
at
org.apache.activemq.leveldb.replicated.SlaveLevelDBStore$$anonfun$send_wal_ack$1.apply$mcV$sp(SlaveLevelDBStore.scala:175)
at
org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:357)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Restarting broker
Loading message broker from: xbean:activemq.xml
also see the followng during failover:
2013-11-22 11:55:32,192 | INFO | Stopping BrokerService[largeamq] due to
except ion, java.io.IOException |
org.apache.activemq.util.DefaultIOExceptionHandler |
LevelDB IOException handler.
java.io.IOException
at
org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport
.java:39)
at
org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.sc
ala:554)
at
org.apache.activemq.leveldb.LevelDBClient.might_fail_using_index(Leve
lDBClient.scala:1021)
at
org.apache.activemq.leveldb.LevelDBClient.collectionCursor(LevelDBCli
ent.scala:1320)
at
org.apache.activemq.leveldb.LevelDBClient.queueCursor(LevelDBClient.s
cala:1244)
at
org.apache.activemq.leveldb.DBManager.cursorMessages(DBManager.scala:
708)
at
org.apache.activemq.leveldb.LevelDBStore$LevelDBMessageStore.recoverN
extMessages(LevelDBStore.scala:756)
at
org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBa
tch(QueueStorePrefetch.java:106)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBat
ch(AbstractStoreCursor.java:258)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext
(AbstractStoreCursor.java:145)
at
org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(St
oreQueueCursor.java:131)
at
org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.jav
a:1876)
at
org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:208
6)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1581)
at org.apache.activemq.broker.region.Queue.wakeup(Queue.java:1803)
at
org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(Pr
efetchSubscription.java:409)
at
org.apache.activemq.broker.region.AbstractRegion.acknowledge(Abstract
Region.java:412)
at
org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBrok
er.java:457)
at
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java
:82)
at
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java
:82)
at
org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionB
roker.java:277)
at
org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBro
kerFilter.java:92)
at
org.apache.activemq.broker.TransportConnection.processMessageAck(Tran
sportConnection.java:476)
at org.apache.activemq.command.MessageAck.visit(MessageAck.java:236)
at
org.apache.activemq.broker.TransportConnection.service(TransportConne
ction.java:292)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportC
onnection.java:149)
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport
.java:50)
at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireForm
atNegotiator.java:113)
at
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(Abs
tractInactivityMonitor.java:270)
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSup
port.java:83)
at
org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTranspo
rt.java:138)
at
org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTranspor
t.java:69)
at
org.apache.activemq.transport.nio.SelectorSelection.onSelect(Selector
Selection.java:94)
at
org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker
.java:119)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.lang.NullPointerException
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$queueCursor$1.appl
y(LevelDBClient.scala:1248)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$queueCursor$1.appl
y(LevelDBClient.scala:1244)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCursor$1
$$anonfun$apply$mcV$sp$12.apply(LevelDBClient.scala:1322)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCursor$1
$$anonfun$apply$mcV$sp$12.apply(LevelDBClient.scala:1321)
at
org.apache.activemq.leveldb.LevelDBClient$RichDB.check$4(LevelDBClien
t.scala:326)
at
org.apache.activemq.leveldb.LevelDBClient$RichDB.cursorRange(LevelDBC
lient.scala:328)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCursor$1
.apply$mcV$sp(LevelDBClient.scala:1321)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCursor$1
.apply(LevelDBClient.scala:1321)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCursor$1
.apply(LevelDBClient.scala:1321)
at
org.apache.activemq.leveldb.LevelDBClient.usingIndex(LevelDBClient.sc
ala:1015)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$might_fail_using_i
ndex$1.apply(LevelDBClient.scala:1021)
at
org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.sc
ala:551)
--
View this message in context: http://activemq.2283324.n4.nabble.com/replicatedLevelDB-errors-after-failover-tp4674550p4674773.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: replicatedLevelDB errors after failover
Posted by Hiram Chirino <hi...@hiramchirino.com>.
Yep that helps a little. So that seems like an error on the slave.
Where there any errors on the master?
On Mon, Nov 18, 2013 at 5:37 PM, kal123 <kp...@gmail.com> wrote:
>
> Today after the exception occurred, I updated the activemq start script to
> redirect output to file instead of null and at the start i see the following
> error. Does this help?
>
> INFO | Attaching... Downloaded 295.12/295.12 kb and 4/4 files
> INFO | Attached
> java.io.IOException: invalid record position 314575339 (file:
> /solidstate/ldb/000000000c80071d.log, offset: 104858318)
> at
> org.apache.activemq.leveldb.RecordLog$LogReader.read(RecordLog.scala:316)
> at
> org.apache.activemq.leveldb.RecordLog$$anonfun$read$1.apply(RecordLog.scala:560)
> at
> org.apache.activemq.leveldb.RecordLog$$anonfun$read$1.apply(RecordLog.scala:560)
> at
> org.apache.activemq.leveldb.RecordLog$$anonfun$get_reader$1.apply(RecordLog.scala:552)
> at
> org.apache.activemq.leveldb.RecordLog$$anonfun$get_reader$1.apply(RecordLog.scala:534)
> at scala.Option.map(Option.scala:133)
> at
> org.apache.activemq.leveldb.RecordLog.get_reader(RecordLog.scala:534)
> at org.apache.activemq.leveldb.RecordLog.read(RecordLog.scala:560)
> at
> org.apache.activemq.leveldb.LevelDBClient$$anonfun$replay_from$1.apply$mcV$sp(LevelDBClient.scala:734)
> at
> org.apache.activemq.leveldb.LevelDBClient$$anonfun$replay_from$1.apply(LevelDBClient.scala:701)
> at
> org.apache.activemq.leveldb.LevelDBClient$$anonfun$replay_from$1.apply(LevelDBClient.scala:701)
> at
> org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:551)
> at
> org.apache.activemq.leveldb.LevelDBClient.replay_from(LevelDBClient.scala:700)
> at
> org.apache.activemq.leveldb.replicated.SlaveLevelDBStore$$anonfun$send_wal_ack$1.apply$mcV$sp(SlaveLevelDBStore.scala:175)
> at
> org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:357)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/replicatedLevelDB-errors-after-failover-tp4674550p4674643.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
--
Hiram Chirino
Engineering | Red Hat, Inc.
hchirino@redhat.com | fusesource.com | redhat.com
skype: hiramchirino | twitter: @hiramchirino
blog: Hiram Chirino's Bit Mojo
Re: replicatedLevelDB errors after failover
Posted by kal123 <kp...@gmail.com>.
Today after the exception occurred, I updated the activemq start script to
redirect output to file instead of null and at the start i see the following
error. Does this help?
INFO | Attaching... Downloaded 295.12/295.12 kb and 4/4 files
INFO | Attached
java.io.IOException: invalid record position 314575339 (file:
/solidstate/ldb/000000000c80071d.log, offset: 104858318)
at
org.apache.activemq.leveldb.RecordLog$LogReader.read(RecordLog.scala:316)
at
org.apache.activemq.leveldb.RecordLog$$anonfun$read$1.apply(RecordLog.scala:560)
at
org.apache.activemq.leveldb.RecordLog$$anonfun$read$1.apply(RecordLog.scala:560)
at
org.apache.activemq.leveldb.RecordLog$$anonfun$get_reader$1.apply(RecordLog.scala:552)
at
org.apache.activemq.leveldb.RecordLog$$anonfun$get_reader$1.apply(RecordLog.scala:534)
at scala.Option.map(Option.scala:133)
at
org.apache.activemq.leveldb.RecordLog.get_reader(RecordLog.scala:534)
at org.apache.activemq.leveldb.RecordLog.read(RecordLog.scala:560)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$replay_from$1.apply$mcV$sp(LevelDBClient.scala:734)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$replay_from$1.apply(LevelDBClient.scala:701)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$replay_from$1.apply(LevelDBClient.scala:701)
at
org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:551)
at
org.apache.activemq.leveldb.LevelDBClient.replay_from(LevelDBClient.scala:700)
at
org.apache.activemq.leveldb.replicated.SlaveLevelDBStore$$anonfun$send_wal_ack$1.apply$mcV$sp(SlaveLevelDBStore.scala:175)
at
org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:357)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java
--
View this message in context: http://activemq.2283324.n4.nabble.com/replicatedLevelDB-errors-after-failover-tp4674550p4674643.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: replicatedLevelDB errors after failover
Posted by kal123 <kp...@gmail.com>.
I believe there was no other errors prior to that exception, i will try to
reproduce and update you on it.
I downloaded the source yesterday form git:
parent: e57aeb3) | patch
See AMQ-4886. Updated tearDown so it can't hang, reduced timeouts, updated
to JUnit4
--
View this message in context: http://activemq.2283324.n4.nabble.com/replicatedLevelDB-errors-after-failover-tp4674550p4674583.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: replicatedLevelDB errors after failover
Posted by Hiram Chirino <hi...@hiramchirino.com>.
Hi that exception should have also printed out a cause. Could you
include that too? If you built it yourself, which git commit did you
build? If you downloaded a snapshot whats the file name of the
download?
On Fri, Nov 15, 2013 at 8:46 AM, kal123 <kp...@gmail.com> wrote:
> I am using the latest changes from trunk and i see following error after
> failover (still getting corrupted?)
>
> 2013-11-15 06:44:53,606 | INFO | jolokia-agent: Using access restrictor
> classpath:/jolokia-access.xml | /hawtio | main
> 2013-11-15 06:44:53,743 | INFO | ActiveMQ WebConsole available at
> http://localhost:8161/ | org.apache.activemq.web.WebConsoleStarter | main
> 2013-11-15 06:44:53,754 | INFO | Initializing Spring FrameworkServlet
> 'dispatcher' | /admin | main
> 2013-11-15 06:44:56,969 | INFO | Stopping BrokerService[largeamq] due to
> exception, java.io.IOException |
> org.apache.activemq.util.DefaultIOExceptionHandler | LevelDB IOException
> handler.
> java.io.IOException
> at
> org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:39)
> at
> org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:554)
> at
> org.apache.activemq.leveldb.LevelDBClient.might_fail_using_index(LevelDBClient.scala:1021)
> at
> org.apache.activemq.leveldb.LevelDBClient.collectionCursor(LevelDBClient.scala:1320)
> at
> org.apache.activemq.leveldb.LevelDBClient.queueCursor(LevelDBClient.scala:1244)
> at
> org.apache.activemq.leveldb.DBManager.cursorMessages(DBManager.scala:708)
> at
> org.apache.activemq.leveldb.LevelDBStore$LevelDBMessageStore.recoverNextMessages(LevelDBStore.scala:756)
> at
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:106)
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:258)
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:145)
> at
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
> at
> org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1876)
> at
> org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2086)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1581)
> at org.apache.activemq.broker.region.Queue.wakeup(Queue.java:1803)
> at
> org.apache.activemq.broker.region.Queue.addSubscription(Queue.java:464)
> at
> org.apache.activemq.broker.region.AbstractRegion.addConsumer(AbstractRegion.java:314)
> at
> org.apache.activemq.broker.region.RegionBroker.addConsumer(RegionBroker.java:400)
> at
> org.apache.activemq.broker.jmx.ManagedRegionBroker.addConsumer(ManagedRegionBroker.java:230)
> at
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:97)
> at
> org.apache.activemq.advisory.AdvisoryBroker.addConsumer(AdvisoryBroker.java:102)
> at
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:97)
> at
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:97)
> at
> org.apache.activemq.broker.MutableBrokerFilter.addConsumer(MutableBrokerFilter.java:102)
> at
> org.apache.activemq.broker.TransportConnection.processAddConsumer(TransportConnection.java:587)
> at
> org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo.java:349)
> at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292)
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/replicatedLevelDB-errors-after-failover-tp4674550.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
--
Hiram Chirino
Engineering | Red Hat, Inc.
hchirino@redhat.com | fusesource.com | redhat.com
skype: hiramchirino | twitter: @hiramchirino
blog: Hiram Chirino's Bit Mojo
Re: replicatedLevelDB errors after failover
Posted by kal123 <kp...@gmail.com>.
the issue seems to happen when broker that is being stopped generated the
following exception.
Store getting corrupted due to the way the stop is handled?
2013-11-15 09:26:52,881 | WARN | Transport Connection to:
tcp://10.44.173.146:47163 failed: java.io.IOException: Connection reset by
peer | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
NIO Worker 3
2013-11-15 09:27:04,959 | INFO | Apache ActiveMQ 5.10-SNAPSHOT (largeamq,
ID:dl360x2832-38143-1384525531423-0:1) is shutting down |
org.apache.activemq.broker.BrokerService | ActiveMQ ShutdownHook
2013-11-15 09:27:04,982 | INFO | Stopping BrokerService[largeamq] due to
exception, java.io.IOException: Store replication stopped |
org.apache.activemq.util.DefaultIOExceptionHandler | LevelDB IOException
handler.
java.io.IOException: Store replication stopped
at
org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:39)
at
org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:554)
at
org.apache.activemq.leveldb.LevelDBClient.might_fail_using_index(LevelDBClient.scala:1021)
at
org.apache.activemq.leveldb.LevelDBClient.store(LevelDBClient.scala:1352)
at
org.apache.activemq.leveldb.DBManager$$anonfun$drainFlushes$1.apply$mcV$sp(DBManager.scala:600)
at
org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:357)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.IllegalStateException: Store replication stopped
at
org.apache.activemq.leveldb.replicated.MasterLevelDBStore.wal_sync_to(MasterLevelDBStore.scala:420)
at
org.apache.activemq.leveldb.replicated.MasterLevelDBClient$$anon$2$$anon$1.force(MasterLevelDBClient.scala:147)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$store$1$$anonfun$apply$mcV$sp$14.apply(LevelDBClient.scala:1358)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$store$1$$anonfun$apply$mcV$sp$14.apply(LevelDBClient.scala:1353)
at
org.apache.activemq.leveldb.RecordLog$$anonfun$appender$1.apply(RecordLog.scala:479)
at
org.apache.activemq.leveldb.util.TimeMetric.apply(TimeMetric.scala:43)
at
org.apache.activemq.leveldb.RecordLog.appender(RecordLog.scala:478)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$store$1.apply$mcV$sp(LevelDBClient.scala:1353)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$store$1.apply(LevelDBClient.scala:1352)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$store$1.apply(LevelDBClient.scala:1352)
at
org.apache.activemq.leveldb.LevelDBClient.usingIndex(LevelDBClient.scala:1015)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$might_fail_using_index$1.apply(LevelDBClient.scala:1021)
at
org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:551)
... 7 more
2013-11-15 09:27:05,011 | INFO | Stopped LevelDB[/solidstate/ldb] |
org.apache.activemq.leveldb.LevelDBStore | LevelDB IOException handler.
2013-11-15 09:27:06,879 | INFO | Connector openwire stopped |
org.apache.activemq.broker.TransportConnector | ActiveMQ ShutdownHook
2013-11-15 09:27:07,768 | INFO | Connector amqp stopped |
org.apache.activemq.broker.TransportConnector | ActiveMQ ShutdownHook
--
View this message in context: http://activemq.2283324.n4.nabble.com/replicatedLevelDB-errors-after-failover-tp4674550p4674556.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.