Re: Activemq 5.9 leveldb replication issue

[Hiram Chirino <hi...@hiramchirino.com>]

Hi Freddy,

That 'short record at position' is odd.  Basically it means it tired
to read a message from the log file and the read call read fewer bytes
than what we requested.  I guess that's valid for an OS to do and we
just need to issue an additional read to load the remaining bytes.
I've just committed a change that should help with that:
http://is.gd/0nMmLH

So try out a new snapshot and keep an eye for that exception and let
me know you see it happen on your system again.

Thanks!


On Fri, May 31, 2013 at 1:28 PM, heimdull <fr...@cfandersen.com> wrote:
> I tried this morning to replicate the issue on these two servers and was not able to do so. The only issue that I got was a few messages that got stuck in the queue with a consumer holding them hostage. either restarting that consumer or moving them to a different queue and back resolved that.
>
> I did look at older errors that I had and found there recovery mode errors:
>
> INFO   | jvm 1    | 2013/05/19 12:15:22 |  INFO | DB recovered from failure.
> INFO   | jvm 1    | 2013/05/19 12:15:22 |  WARN | DB operation failed. (entering recovery mode)
> INFO   | jvm 1    | 2013/05/19 12:15:22 | java.io.IOException: short record at position: 2467052 in file: /opt/activemq/data/0000000000000000.log, offset: 2467052
> INFO   | jvm 1    | 2013/05/19 12:15:22 |       at org.apache.activemq.leveldb.RecordLog$LogReader.read(RecordLog.scala:287)


--
Hiram Chirino

Engineering | Red Hat, Inc.

hchirino@redhat.com | fusesource.com | redhat.com

skype: hiramchirino | twitter: @hiramchirino

blog: Hiram Chirino's Bit Mojo

Re: Activemq 5.9 leveldb replication issue

[linuraj <li...@gmail.com>]

Hi, We have the same issue with activemq version 5.13.1. After a failover, we
see some messages stuck in the queue. We see a valid number as queue-size;
however when we browse the queue via hawtio we will not see those messages.
Also, it is the same number of messages that we miss after a failover. Do we
know the cause? Do we know the fix? Really appreciate your help.ThanksLinu
Raj.



--
View this message in context: http://activemq.2283324.n4.nabble.com/Activemq-5-9-leveldb-replication-issue-tp4667495p4709236.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Activemq 5.9 leveldb replication issue

[heimdull <fr...@cfandersen.com>]

great! I have tested latest snapshot and have not seen this error when I
failover. I do however see that I get a handful of messages stuck in a queue
when I do and I need to move them out and back in to the queue before they
get consumed.

Could this have something todo with that:

2013-06-03 23:50:53,523 | WARN  | Async error occurred:
java.lang.IllegalArgumentException: The subscription does not exist:
ID:tn11-6552-1369955706360-1:2:1:1 |
org.apache.activemq.broker.TransportConnection.Service | ActiveMQ NIO Worker
14
java.lang.IllegalArgumentException: The subscription does not exist:
ID:tn11-6552-1369955706360-1:2:1:1
	at
org.apache.activemq.broker.region.AbstractRegion.messagePull(AbstractRegion.java:432)
	at
org.apache.activemq.broker.region.RegionBroker.messagePull(RegionBroker.java:468)
	at
org.apache.activemq.broker.BrokerFilter.messagePull(BrokerFilter.java:87)
	at
org.apache.activemq.broker.BrokerFilter.messagePull(BrokerFilter.java:87)
	at
org.apache.activemq.broker.BrokerFilter.messagePull(BrokerFilter.java:87)
	at
org.apache.activemq.broker.MutableBrokerFilter.messagePull(MutableBrokerFilter.java:289)
	at
org.apache.activemq.broker.TransportConnection.processMessagePull(TransportConnection.java:515)
	at org.apache.activemq.command.MessagePull.visit(MessagePull.java:43)
	at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:329)
	at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:184)
	at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
	at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
	at
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:288)
	at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
	at
org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:138)
	at
org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:69)
	at
org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:94)
	at
org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:119)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:722)

If I look at my web console (Active Consumers list) I see that 7 consumers
have 1=Enqueues, 0=Dequeues, 1=Dispatched, 1=Dispatched Queue which happen
to be the same number of stuck messages other consumers have even numbers.

We are also using activemq-5.7 jar for our consumers could that be a
problem?
 



--
View this message in context: http://activemq.2283324.n4.nabble.com/Activemq-5-9-leveldb-replication-issue-tp4667495p4667791.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.