You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by yaussy <ya...@cboe.com> on 2008/04/14 15:07:55 UTC
Kaha persistence issue w/5.1 (upto 4/8 SNAPSHOT)
I have not been able to reproduce this problem outside of our application,
but I'm still trying.
Anyway, I'm using regular Kaha persistence (not AMQMessageStore), and during
a Durable Topic test, I'm eventually getting the following exception in the
AMQBroker, after which the broker stops forwarding events to the consumer.
Anyone seen this?
<org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch>
<Thread[ActiveMQ Transport: tcp:///127.0.0.1:43595,4,main]> Failed to fill
batch
Stack Trace follows:
java.lang.RuntimeException: Failed to get next index from
IndexManager:(index-topic-subs) for offset=7681977, key=(2, 20080006, 47),
value=(2, 20080058, 165), previousItem=7681212, nextItem=7682895
at
org.apache.activemq.kaha.impl.index.DiskIndexLinkedList.getNextEntry(DiskIndexLinkedList.java:267)
at
org.apache.activemq.kaha.impl.container.MapContainerImpl.getNext(MapContainerImpl.java:449)
at
org.apache.activemq.store.kahadaptor.TopicSubContainer.getNextEntry(TopicSubContainer.java:95)
at
org.apache.activemq.store.kahadaptor.KahaTopicMessageStore.recoverNextMessages(KahaTopicMessageStore.java:165)
at
org.apache.activemq.store.ProxyTopicMessageStore.recoverNextMessages(ProxyTopicMessageStore.java:97)
at
org.apache.activemq.broker.region.cursors.TopicStorePrefetch.doFillBatch(TopicStorePrefetch.java:107)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:188)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:104)
at
org.apache.activemq.broker.region.cursors.StoreDurableSubscriberCursor.hasNext(StoreDurableSubscriberCursor.java:210
)
at
org.apache.activemq.broker.region.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:479)
at
org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:357)
at
org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:349)
at
org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:474)
at
org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:194)
at
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:73)
at
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:73)
at
org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:84)
at
org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:444)
at org.apache.activemq.command.MessageAck.visit(MessageAck.java:196)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:293)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:181)
at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:143)
at
org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:196)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:183)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.EOFException
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
at
org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:46)
at
org.apache.activemq.kaha.impl.index.IndexManager.getIndex(IndexManager.java:66)
at
org.apache.activemq.kaha.impl.index.DiskIndexLinkedList.getNextEntry(DiskIndexLinkedList.java:265)
... 27 more
--
View this message in context: http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16677512.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Kaha persistence issue w/5.1 (upto 4/8 SNAPSHOT)
Posted by Rob Davies <ra...@gmail.com>.
Thanks for letting us know it works!
cheers,
Rob
On 17 Apr 2008, at 18:51, yaussy wrote:
>
> Rob,
>
> This looks to have done the trick. I've tested with our
> application, which
> was hitting the problem every time, and it is fine.
>
> Thanks for your attentiveness to this issue!
>
> Kevin
>
>
> rajdavies wrote:
>>
>> Hopefully!
>> On 16 Apr 2008, at 18:39, yaussy wrote:
>>
>>>
>>> Rob,
>>>
>>> So, I would find this under the 5.2 SNAPSHOT tomorrow morning (USA
>>> central
>>> standard time)?
>>>
>>> Kevin
>>>
>>>
>>> rajdavies wrote:
>>>>
>>>> yep - being able to reproduce - this is now fixed in trunk - but
>>>> too
>>>> late for the 5.1 release
>>>>
>>>> thanks,
>>>>
>>>> Rob
>>>>
>>>> On 16 Apr 2008, at 13:54, yaussy wrote:
>>>>
>>>>>
>>>>> Rob,
>>>>>
>>>>> While I don't yet have a solid / consistent test case for you, our
>>>>> application, using a particular test scenario the app guys have,
>>>>> can
>>>>> reproduce the problem every time. If you have anything you want
>>>>> me
>>>>> to try
>>>>> out, let me know.
>>>>>
>>>>> I'm still working with my test code to get something more
>>>>> reproduceable,
>>>>> while adding some debug code to IndexManager.
>>>>>
>>>>> Kevin
>>>>>
>>>>>
>>>>> yaussy wrote:
>>>>>>
>>>>>> Still working on a consistent test case. But, I've run into
>>>>>> something
>>>>>> along the way, and I'm not sure if it's related to the original
>>>>>> exception
>>>>>> I've logged here. I was able to get the problem to happen when I
>>>>>> expanded
>>>>>> my test case to a publisher of 6 topics, at 100+/sec/topic, and a
>>>>>> consumer
>>>>>> that consumes all those topics. I have this configured to use a
>>>>>> connection for each topic, for both the publisher and the
>>>>>> consumer. I've
>>>>>> only been able to get the original exception to happen once, but
>>>>>> I've more
>>>>>> consistently hit a problem wherein no exception is thrown, but
>>>>>> the
>>>>>> broker
>>>>>> just stops forwarding events for one of the topics. After this
>>>>>> point, the
>>>>>> publisher continues, but all events for this one topic are queued
>>>>>> to disk.
>>>>>>
>>>>>> I'm trying to see if I can reduce this test case down right now.
>>>>>> But,
>>>>>> maybe you could give this a go with the supplied consumer /
>>>>>> supplier test
>>>>>> code?
>>>>>>
>>>>>> Kevin
>>>>>>
>>>>>>
>>>>>> rajdavies wrote:
>>>>>>>
>>>>>>> ok - thx for checking! - let me know when you have a test case
>>>>>>>
>>>>>>> On 15 Apr 2008, at 15:06, yaussy wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> Bad news - problem still happens with "5.2" snapshot dated
>>>>>>>> 4/13. I
>>>>>>>> did not
>>>>>>>> see anymore 5.1 snapshots, but saw the 5.2 directory. The
>>>>>>>> archive
>>>>>>>> still has
>>>>>>>> 5.1 for the path names, so I figured this is what I should
>>>>>>>> take.
>>>>>>>>
>>>>>>>> I'm still working with it and am trying to see if I can get any
>>>>>>>> more
>>>>>>>> debug
>>>>>>>> information. I have not been able to reproduce it with a test
>>>>>>>> program yet.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> yaussy wrote:
>>>>>>>>>
>>>>>>>>> Rob,
>>>>>>>>>
>>>>>>>>> This is good news - I will try using last nights snapshot.
>>>>>>>>>
>>>>>>>>> I started out using AMQMessageStore, but I've never had good
>>>>>>>>> performance
>>>>>>>>> luck with AMQs Journal - it always seems to have this
>>>>>>>>> frequent,
>>>>>>>>> high CPU
>>>>>>>>> cost as it checkpoints. Our Durable rate requirements are
>>>>>>>>> fairly
>>>>>>>>> significant, such as a few hundred events per second to a
>>>>>>>>> handful of
>>>>>>>>> consumers, on one of our clusters. It has not performed well,
>>>>>>>>> whereas
>>>>>>>>> Kaha by itself is excellent.
>>>>>>>>>
>>>>>>>>> Kevin
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> rajdavies wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 14 Apr 2008, at 14:07, yaussy wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I have not been able to reproduce this problem outside of
>>>>>>>>>>> our
>>>>>>>>>>> application,
>>>>>>>>>>> but I'm still trying.
>>>>>>>>>>>
>>>>>>>>>>> Anyway, I'm using regular Kaha persistence (not
>>>>>>>>>>> AMQMessageStore),
>>>>>>>>>>> and during
>>>>>>>>>>> a Durable Topic test, I'm eventually getting the following
>>>>>>>>>>> exception
>>>>>>>>>>> in the
>>>>>>>>>>> AMQBroker, after which the broker stops forwarding events to
>>>>>>>>>>> the
>>>>>>>>>>> consumer.
>>>>>>>>>>>
>>>>>>>>>>> Anyone seen this?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> <
>>>>>>>>>>> org
>>>>>>>>>>> .apache
>>>>>>>>>>> .activemq
>>>>>>>>>>> .broker.region.cursors.AbstractStoreCursor.fillBatch>
>>>>>>>>>>> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:43595,4,main]>
>>>>>>>>>>> Failed
>>>>>>>>>>> to fill
>>>>>>>>>>> batch
>>>>>>>>>>> Stack Trace follows:
>>>>>>>>>>> java.lang.RuntimeException: Failed to get next index from
>>>>>>>>>>> IndexManager:(index-topic-subs) for offset=7681977, key=(2,
>>>>>>>>>>> 20080006, 47),
>>>>>>>>>>> value=(2, 20080058, 165), previousItem=7681212,
>>>>>>>>>>> nextItem=7682895
>>>>>>>>>>> at
>>>>>>>>>>> org
>>>>>>>>>>> .apache
>>>>>>>>>>> .activemq
>>>>>>>>>>> .kaha
>>>>>>>>>>> .impl
>>>>>>>>>>> .index
>>>>>>>>>>> .DiskIndexLinkedList.getNextEntry(DiskIndexLinkedList.java:
>>>>>>>>>>> 267)
>>>>>>>>>>> at
>>>>>>>>>>> org
>>>>>>>>>>> .apache
>>>>>>>>>>> .activemq
>>>>>>>>>>> .kaha
>>>>>>>>>>> .impl
>>>>>>>>>>> .container.MapContainerImpl.getNext(MapContainerImpl.java:
>>>>>>>>>>> 449)
>>>>>>>>>>> at
>>>>>>>>>>> org
>>>>>>>>>>> .apache
>>>>>>>>>>> .activemq
>>>>>>>>>>> .store
>>>>>>>>>>> .kahadaptor
>>>>>>>>>>> .TopicSubContainer.getNextEntry(TopicSubContainer.java:
>>>>>>>>>>> 95)
>>>>>>>>>>> at
>>>>>>>>>>> org
>>>>>>>>>>> .apache
>>>>>>>>>>> .activemq
>>>>>>>>>>> .store
>>>>>>>>>>> .kahadaptor
>>>>>>>>>>> .KahaTopicMessageStore
>>>>>>>>>>> .recoverNextMessages(KahaTopicMessageStore.java:165)
>>>>>>>>>>> at
>>>>>>>>>>> org
>>>>>>>>>>> .apache
>>>>>>>>>>> .activemq
>>>>>>>>>>> .store
>>>>>>>>>>> .ProxyTopicMessageStore
>>>>>>>>>>> .recoverNextMessages(ProxyTopicMessageStore.java:97)
>>>>>>>>>>> at
>>>>>>>>>>> org
>>>>>>>>>>> .apache
>>>>>>>>>>> .activemq
>>>>>>>>>>> .broker
>>>>>>>>>>> .region
>>>>>>>>>>> .cursors
>>>>>>>>>>> .TopicStorePrefetch.doFillBatch(TopicStorePrefetch.java:
>>>>>>>>>>> 107)
>>>>>>>>>>> at
>>>>>>>>>>> org
>>>>>>>>>>> .apache
>>>>>>>>>>> .activemq
>>>>>>>>>>> .broker
>>>>>>>>>>> .region
>>>>>>>>>>> .cursors
>>>>>>>>>>> .AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:
>>>>>>>>>>> 188)
>>>>>>>>>>> at
>>>>>>>>>>> org
>>>>>>>>>>> .apache
>>>>>>>>>>> .activemq
>>>>>>>>>>> .broker
>>>>>>>>>>> .region
>>>>>>>>>>> .cursors
>>>>>>>>>>> .AbstractStoreCursor.hasNext(AbstractStoreCursor.java:
>>>>>>>>>>> 104)
>>>>>>>>>>> at
>>>>>>>>>>> org
>>>>>>>>>>> .apache
>>>>>>>>>>> .activemq
>>>>>>>>>>> .broker
>>>>>>>>>>> .region
>>>>>>>>>>> .cursors
>>>>>>>>>>> .StoreDurableSubscriberCursor
>>>>>>>>>>> .hasNext(StoreDurableSubscriberCursor.java:210
>>>>>>>>>>> )
>>>>>>>>>>> at
>>>>>>>>>>> org
>>>>>>>>>>> .apache
>>>>>>>>>>> .activemq
>>>>>>>>>>> .broker
>>>>>>>>>>> .region
>>>>>>>>>>> .PrefetchSubscription
>>>>>>>>>>> .dispatchPending(PrefetchSubscription.java:
>>>>>>>>>>> 479)
>>>>>>>>>>> at
>>>>>>>>>>> org
>>>>>>>>>>> .apache
>>>>>>>>>>> .activemq
>>>>>>>>>>> .broker
>>>>>>>>>>> .region
>>>>>>>>>>> .PrefetchSubscription.acknowledge(PrefetchSubscription.java:
>>>>>>>>>>> 357)
>>>>>>>>>>> at
>>>>>>>>>>> org
>>>>>>>>>>> .apache
>>>>>>>>>>> .activemq
>>>>>>>>>>> .broker
>>>>>>>>>>> .region.AbstractRegion.acknowledge(AbstractRegion.java:
>>>>>>>>>>> 349)
>>>>>>>>>>> at
>>>>>>>>>>> org
>>>>>>>>>>> .apache
>>>>>>>>>>> .activemq
>>>>>>>>>>> .broker.region.RegionBroker.acknowledge(RegionBroker.java:
>>>>>>>>>>> 474)
>>>>>>>>>>> at
>>>>>>>>>>> org
>>>>>>>>>>> .apache
>>>>>>>>>>> .activemq
>>>>>>>>>>> .broker
>>>>>>>>>>> .TransactionBroker.acknowledge(TransactionBroker.java:
>>>>>>>>>>> 194)
>>>>>>>>>>> at
>>>>>>>>>>> org
>>>>>>>>>>> .apache
>>>>>>>>>>> .activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:
>>>>>>>>>>> 73)
>>>>>>>>>>> at
>>>>>>>>>>> org
>>>>>>>>>>> .apache
>>>>>>>>>>> .activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:
>>>>>>>>>>> 73)
>>>>>>>>>>> at
>>>>>>>>>>> org
>>>>>>>>>>> .apache
>>>>>>>>>>> .activemq
>>>>>>>>>>> .broker
>>>>>>>>>>> .MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:
>>>>>>>>>>> 84)
>>>>>>>>>>> at
>>>>>>>>>>> org
>>>>>>>>>>> .apache
>>>>>>>>>>> .activemq
>>>>>>>>>>> .broker
>>>>>>>>>>> .TransportConnection
>>>>>>>>>>> .processMessageAck(TransportConnection.java:
>>>>>>>>>>> 444)
>>>>>>>>>>> at
>>>>>>>>>>> org
>>>>>>>>>>> .apache.activemq.command.MessageAck.visit(MessageAck.java:
>>>>>>>>>>> 196)
>>>>>>>>>>> at
>>>>>>>>>>> org
>>>>>>>>>>> .apache
>>>>>>>>>>> .activemq
>>>>>>>>>>> .broker
>>>>>>>>>>> .TransportConnection.service(TransportConnection.java:
>>>>>>>>>>> 293)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.activemq.broker.TransportConnection
>>>>>>>>>>> $1.onCommand(TransportConnection.java:181)
>>>>>>>>>>> at
>>>>>>>>>>> org
>>>>>>>>>>> .apache
>>>>>>>>>>> .activemq
>>>>>>>>>>> .transport.TransportFilter.onCommand(TransportFilter.java:
>>>>>>>>>>> 68)
>>>>>>>>>>> at
>>>>>>>>>>> org
>>>>>>>>>>> .apache
>>>>>>>>>>> .activemq
>>>>>>>>>>> .transport
>>>>>>>>>>> .WireFormatNegotiator.onCommand(WireFormatNegotiator.java:
>>>>>>>>>>> 143)
>>>>>>>>>>> at
>>>>>>>>>>> org
>>>>>>>>>>> .apache
>>>>>>>>>>> .activemq
>>>>>>>>>>> .transport
>>>>>>>>>>> .InactivityMonitor.onCommand(InactivityMonitor.java:
>>>>>>>>>>> 206)
>>>>>>>>>>> at
>>>>>>>>>>> org
>>>>>>>>>>> .apache
>>>>>>>>>>> .activemq
>>>>>>>>>>> .transport.TransportSupport.doConsume(TransportSupport.java:
>>>>>>>>>>> 84)
>>>>>>>>>>> at
>>>>>>>>>>> org
>>>>>>>>>>> .apache
>>>>>>>>>>> .activemq
>>>>>>>>>>> .transport.tcp.TcpTransport.doRun(TcpTransport.java:
>>>>>>>>>>> 196)
>>>>>>>>>>> at
>>>>>>>>>>> org
>>>>>>>>>>> .apache
>>>>>>>>>>> .activemq.transport.tcp.TcpTransport.run(TcpTransport.java:
>>>>>>>>>>> 183)
>>>>>>>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>>>>>>>> Caused by: java.io.EOFException
>>>>>>>>>>> at
>>>>>>>>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:
>>>>>>>>>>> 383)
>>>>>>>>>>> at
>>>>>>>>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:
>>>>>>>>>>> 361)
>>>>>>>>>>> at
>>>>>>>>>>> org
>>>>>>>>>>> .apache
>>>>>>>>>>> .activemq
>>>>>>>>>>> .kaha
>>>>>>>>>>> .impl.index.StoreIndexReader.readItem(StoreIndexReader.java:
>>>>>>>>>>> 46)
>>>>>>>>>>> at
>>>>>>>>>>> org
>>>>>>>>>>> .apache
>>>>>>>>>>> .activemq
>>>>>>>>>>> .kaha.impl.index.IndexManager.getIndex(IndexManager.java:
>>>>>>>>>>> 66)
>>>>>>>>>>> at
>>>>>>>>>>> org
>>>>>>>>>>> .apache
>>>>>>>>>>> .activemq
>>>>>>>>>>> .kaha
>>>>>>>>>>> .impl
>>>>>>>>>>> .index
>>>>>>>>>>> .DiskIndexLinkedList.getNextEntry(DiskIndexLinkedList.java:
>>>>>>>>>>> 265)
>>>>>>>>>>> ... 27 more
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> View this message in context:
>>>>>>>>>>> http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16677512.html
>>>>>>>>>>> Sent from the ActiveMQ - User mailing list archive at
>>>>>>>>>>> Nabble.com.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> There was a bug fixed in this area last Friday - would be
>>>>>>>>>> interested
>>>>>>>>>> if this is still a problem for you with a later version?
>>>>>>>>>> If it is - Ill digg a little deeper.
>>>>>>>>>>
>>>>>>>>>> btw - why are you using Kaha instead of AMQStore ?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> cheers,
>>>>>>>>>>
>>>>>>>>>> Rob
>>>>>>>>>>
>>>>>>>>>> http://open.iona.com/ -Enterprise Open Integration
>>>>>>>>>> http://rajdavies.blogspot.com/
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> View this message in context:
>>>>>>>> http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16700798.html
>>>>>>>> Sent from the ActiveMQ - User mailing list archive at
>>>>>>>> Nabble.com.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16721669.html
>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16725769.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16748908.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Kaha persistence issue w/5.1 (upto 4/8 SNAPSHOT)
Posted by yaussy <ya...@cboe.com>.
Rob,
This looks to have done the trick. I've tested with our application, which
was hitting the problem every time, and it is fine.
Thanks for your attentiveness to this issue!
Kevin
rajdavies wrote:
>
> Hopefully!
> On 16 Apr 2008, at 18:39, yaussy wrote:
>
>>
>> Rob,
>>
>> So, I would find this under the 5.2 SNAPSHOT tomorrow morning (USA
>> central
>> standard time)?
>>
>> Kevin
>>
>>
>> rajdavies wrote:
>>>
>>> yep - being able to reproduce - this is now fixed in trunk - but too
>>> late for the 5.1 release
>>>
>>> thanks,
>>>
>>> Rob
>>>
>>> On 16 Apr 2008, at 13:54, yaussy wrote:
>>>
>>>>
>>>> Rob,
>>>>
>>>> While I don't yet have a solid / consistent test case for you, our
>>>> application, using a particular test scenario the app guys have, can
>>>> reproduce the problem every time. If you have anything you want me
>>>> to try
>>>> out, let me know.
>>>>
>>>> I'm still working with my test code to get something more
>>>> reproduceable,
>>>> while adding some debug code to IndexManager.
>>>>
>>>> Kevin
>>>>
>>>>
>>>> yaussy wrote:
>>>>>
>>>>> Still working on a consistent test case. But, I've run into
>>>>> something
>>>>> along the way, and I'm not sure if it's related to the original
>>>>> exception
>>>>> I've logged here. I was able to get the problem to happen when I
>>>>> expanded
>>>>> my test case to a publisher of 6 topics, at 100+/sec/topic, and a
>>>>> consumer
>>>>> that consumes all those topics. I have this configured to use a
>>>>> connection for each topic, for both the publisher and the
>>>>> consumer. I've
>>>>> only been able to get the original exception to happen once, but
>>>>> I've more
>>>>> consistently hit a problem wherein no exception is thrown, but the
>>>>> broker
>>>>> just stops forwarding events for one of the topics. After this
>>>>> point, the
>>>>> publisher continues, but all events for this one topic are queued
>>>>> to disk.
>>>>>
>>>>> I'm trying to see if I can reduce this test case down right now.
>>>>> But,
>>>>> maybe you could give this a go with the supplied consumer /
>>>>> supplier test
>>>>> code?
>>>>>
>>>>> Kevin
>>>>>
>>>>>
>>>>> rajdavies wrote:
>>>>>>
>>>>>> ok - thx for checking! - let me know when you have a test case
>>>>>>
>>>>>> On 15 Apr 2008, at 15:06, yaussy wrote:
>>>>>>
>>>>>>>
>>>>>>> Bad news - problem still happens with "5.2" snapshot dated
>>>>>>> 4/13. I
>>>>>>> did not
>>>>>>> see anymore 5.1 snapshots, but saw the 5.2 directory. The
>>>>>>> archive
>>>>>>> still has
>>>>>>> 5.1 for the path names, so I figured this is what I should take.
>>>>>>>
>>>>>>> I'm still working with it and am trying to see if I can get any
>>>>>>> more
>>>>>>> debug
>>>>>>> information. I have not been able to reproduce it with a test
>>>>>>> program yet.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> yaussy wrote:
>>>>>>>>
>>>>>>>> Rob,
>>>>>>>>
>>>>>>>> This is good news - I will try using last nights snapshot.
>>>>>>>>
>>>>>>>> I started out using AMQMessageStore, but I've never had good
>>>>>>>> performance
>>>>>>>> luck with AMQs Journal - it always seems to have this frequent,
>>>>>>>> high CPU
>>>>>>>> cost as it checkpoints. Our Durable rate requirements are
>>>>>>>> fairly
>>>>>>>> significant, such as a few hundred events per second to a
>>>>>>>> handful of
>>>>>>>> consumers, on one of our clusters. It has not performed well,
>>>>>>>> whereas
>>>>>>>> Kaha by itself is excellent.
>>>>>>>>
>>>>>>>> Kevin
>>>>>>>>
>>>>>>>>
>>>>>>>> rajdavies wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 14 Apr 2008, at 14:07, yaussy wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I have not been able to reproduce this problem outside of our
>>>>>>>>>> application,
>>>>>>>>>> but I'm still trying.
>>>>>>>>>>
>>>>>>>>>> Anyway, I'm using regular Kaha persistence (not
>>>>>>>>>> AMQMessageStore),
>>>>>>>>>> and during
>>>>>>>>>> a Durable Topic test, I'm eventually getting the following
>>>>>>>>>> exception
>>>>>>>>>> in the
>>>>>>>>>> AMQBroker, after which the broker stops forwarding events to
>>>>>>>>>> the
>>>>>>>>>> consumer.
>>>>>>>>>>
>>>>>>>>>> Anyone seen this?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> <
>>>>>>>>>> org
>>>>>>>>>> .apache
>>>>>>>>>> .activemq.broker.region.cursors.AbstractStoreCursor.fillBatch>
>>>>>>>>>> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:43595,4,main]>
>>>>>>>>>> Failed
>>>>>>>>>> to fill
>>>>>>>>>> batch
>>>>>>>>>> Stack Trace follows:
>>>>>>>>>> java.lang.RuntimeException: Failed to get next index from
>>>>>>>>>> IndexManager:(index-topic-subs) for offset=7681977, key=(2,
>>>>>>>>>> 20080006, 47),
>>>>>>>>>> value=(2, 20080058, 165), previousItem=7681212,
>>>>>>>>>> nextItem=7682895
>>>>>>>>>> at
>>>>>>>>>> org
>>>>>>>>>> .apache
>>>>>>>>>> .activemq
>>>>>>>>>> .kaha
>>>>>>>>>> .impl
>>>>>>>>>> .index
>>>>>>>>>> .DiskIndexLinkedList.getNextEntry(DiskIndexLinkedList.java:
>>>>>>>>>> 267)
>>>>>>>>>> at
>>>>>>>>>> org
>>>>>>>>>> .apache
>>>>>>>>>> .activemq
>>>>>>>>>> .kaha
>>>>>>>>>> .impl
>>>>>>>>>> .container.MapContainerImpl.getNext(MapContainerImpl.java:
>>>>>>>>>> 449)
>>>>>>>>>> at
>>>>>>>>>> org
>>>>>>>>>> .apache
>>>>>>>>>> .activemq
>>>>>>>>>> .store
>>>>>>>>>> .kahadaptor
>>>>>>>>>> .TopicSubContainer.getNextEntry(TopicSubContainer.java:
>>>>>>>>>> 95)
>>>>>>>>>> at
>>>>>>>>>> org
>>>>>>>>>> .apache
>>>>>>>>>> .activemq
>>>>>>>>>> .store
>>>>>>>>>> .kahadaptor
>>>>>>>>>> .KahaTopicMessageStore
>>>>>>>>>> .recoverNextMessages(KahaTopicMessageStore.java:165)
>>>>>>>>>> at
>>>>>>>>>> org
>>>>>>>>>> .apache
>>>>>>>>>> .activemq
>>>>>>>>>> .store
>>>>>>>>>> .ProxyTopicMessageStore
>>>>>>>>>> .recoverNextMessages(ProxyTopicMessageStore.java:97)
>>>>>>>>>> at
>>>>>>>>>> org
>>>>>>>>>> .apache
>>>>>>>>>> .activemq
>>>>>>>>>> .broker
>>>>>>>>>> .region
>>>>>>>>>> .cursors
>>>>>>>>>> .TopicStorePrefetch.doFillBatch(TopicStorePrefetch.java:
>>>>>>>>>> 107)
>>>>>>>>>> at
>>>>>>>>>> org
>>>>>>>>>> .apache
>>>>>>>>>> .activemq
>>>>>>>>>> .broker
>>>>>>>>>> .region
>>>>>>>>>> .cursors
>>>>>>>>>> .AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:
>>>>>>>>>> 188)
>>>>>>>>>> at
>>>>>>>>>> org
>>>>>>>>>> .apache
>>>>>>>>>> .activemq
>>>>>>>>>> .broker
>>>>>>>>>> .region
>>>>>>>>>> .cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:
>>>>>>>>>> 104)
>>>>>>>>>> at
>>>>>>>>>> org
>>>>>>>>>> .apache
>>>>>>>>>> .activemq
>>>>>>>>>> .broker
>>>>>>>>>> .region
>>>>>>>>>> .cursors
>>>>>>>>>> .StoreDurableSubscriberCursor
>>>>>>>>>> .hasNext(StoreDurableSubscriberCursor.java:210
>>>>>>>>>> )
>>>>>>>>>> at
>>>>>>>>>> org
>>>>>>>>>> .apache
>>>>>>>>>> .activemq
>>>>>>>>>> .broker
>>>>>>>>>> .region
>>>>>>>>>> .PrefetchSubscription
>>>>>>>>>> .dispatchPending(PrefetchSubscription.java:
>>>>>>>>>> 479)
>>>>>>>>>> at
>>>>>>>>>> org
>>>>>>>>>> .apache
>>>>>>>>>> .activemq
>>>>>>>>>> .broker
>>>>>>>>>> .region
>>>>>>>>>> .PrefetchSubscription.acknowledge(PrefetchSubscription.java:
>>>>>>>>>> 357)
>>>>>>>>>> at
>>>>>>>>>> org
>>>>>>>>>> .apache
>>>>>>>>>> .activemq
>>>>>>>>>> .broker.region.AbstractRegion.acknowledge(AbstractRegion.java:
>>>>>>>>>> 349)
>>>>>>>>>> at
>>>>>>>>>> org
>>>>>>>>>> .apache
>>>>>>>>>> .activemq
>>>>>>>>>> .broker.region.RegionBroker.acknowledge(RegionBroker.java:
>>>>>>>>>> 474)
>>>>>>>>>> at
>>>>>>>>>> org
>>>>>>>>>> .apache
>>>>>>>>>> .activemq
>>>>>>>>>> .broker.TransactionBroker.acknowledge(TransactionBroker.java:
>>>>>>>>>> 194)
>>>>>>>>>> at
>>>>>>>>>> org
>>>>>>>>>> .apache
>>>>>>>>>> .activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:
>>>>>>>>>> 73)
>>>>>>>>>> at
>>>>>>>>>> org
>>>>>>>>>> .apache
>>>>>>>>>> .activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:
>>>>>>>>>> 73)
>>>>>>>>>> at
>>>>>>>>>> org
>>>>>>>>>> .apache
>>>>>>>>>> .activemq
>>>>>>>>>> .broker
>>>>>>>>>> .MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:
>>>>>>>>>> 84)
>>>>>>>>>> at
>>>>>>>>>> org
>>>>>>>>>> .apache
>>>>>>>>>> .activemq
>>>>>>>>>> .broker
>>>>>>>>>> .TransportConnection
>>>>>>>>>> .processMessageAck(TransportConnection.java:
>>>>>>>>>> 444)
>>>>>>>>>> at
>>>>>>>>>> org.apache.activemq.command.MessageAck.visit(MessageAck.java:
>>>>>>>>>> 196)
>>>>>>>>>> at
>>>>>>>>>> org
>>>>>>>>>> .apache
>>>>>>>>>> .activemq
>>>>>>>>>> .broker.TransportConnection.service(TransportConnection.java:
>>>>>>>>>> 293)
>>>>>>>>>> at
>>>>>>>>>> org.apache.activemq.broker.TransportConnection
>>>>>>>>>> $1.onCommand(TransportConnection.java:181)
>>>>>>>>>> at
>>>>>>>>>> org
>>>>>>>>>> .apache
>>>>>>>>>> .activemq
>>>>>>>>>> .transport.TransportFilter.onCommand(TransportFilter.java:68)
>>>>>>>>>> at
>>>>>>>>>> org
>>>>>>>>>> .apache
>>>>>>>>>> .activemq
>>>>>>>>>> .transport
>>>>>>>>>> .WireFormatNegotiator.onCommand(WireFormatNegotiator.java:
>>>>>>>>>> 143)
>>>>>>>>>> at
>>>>>>>>>> org
>>>>>>>>>> .apache
>>>>>>>>>> .activemq
>>>>>>>>>> .transport.InactivityMonitor.onCommand(InactivityMonitor.java:
>>>>>>>>>> 206)
>>>>>>>>>> at
>>>>>>>>>> org
>>>>>>>>>> .apache
>>>>>>>>>> .activemq
>>>>>>>>>> .transport.TransportSupport.doConsume(TransportSupport.java:
>>>>>>>>>> 84)
>>>>>>>>>> at
>>>>>>>>>> org
>>>>>>>>>> .apache
>>>>>>>>>> .activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:
>>>>>>>>>> 196)
>>>>>>>>>> at
>>>>>>>>>> org
>>>>>>>>>> .apache
>>>>>>>>>> .activemq.transport.tcp.TcpTransport.run(TcpTransport.java:
>>>>>>>>>> 183)
>>>>>>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>>>>>>> Caused by: java.io.EOFException
>>>>>>>>>> at
>>>>>>>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:
>>>>>>>>>> 383)
>>>>>>>>>> at
>>>>>>>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:
>>>>>>>>>> 361)
>>>>>>>>>> at
>>>>>>>>>> org
>>>>>>>>>> .apache
>>>>>>>>>> .activemq
>>>>>>>>>> .kaha
>>>>>>>>>> .impl.index.StoreIndexReader.readItem(StoreIndexReader.java:
>>>>>>>>>> 46)
>>>>>>>>>> at
>>>>>>>>>> org
>>>>>>>>>> .apache
>>>>>>>>>> .activemq
>>>>>>>>>> .kaha.impl.index.IndexManager.getIndex(IndexManager.java:
>>>>>>>>>> 66)
>>>>>>>>>> at
>>>>>>>>>> org
>>>>>>>>>> .apache
>>>>>>>>>> .activemq
>>>>>>>>>> .kaha
>>>>>>>>>> .impl
>>>>>>>>>> .index
>>>>>>>>>> .DiskIndexLinkedList.getNextEntry(DiskIndexLinkedList.java:
>>>>>>>>>> 265)
>>>>>>>>>> ... 27 more
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> View this message in context:
>>>>>>>>>> http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16677512.html
>>>>>>>>>> Sent from the ActiveMQ - User mailing list archive at
>>>>>>>>>> Nabble.com.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> There was a bug fixed in this area last Friday - would be
>>>>>>>>> interested
>>>>>>>>> if this is still a problem for you with a later version?
>>>>>>>>> If it is - Ill digg a little deeper.
>>>>>>>>>
>>>>>>>>> btw - why are you using Kaha instead of AMQStore ?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> cheers,
>>>>>>>>>
>>>>>>>>> Rob
>>>>>>>>>
>>>>>>>>> http://open.iona.com/ -Enterprise Open Integration
>>>>>>>>> http://rajdavies.blogspot.com/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> View this message in context:
>>>>>>> http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16700798.html
>>>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16721669.html
>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>
>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16725769.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>
--
View this message in context: http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16748908.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Kaha persistence issue w/5.1 (upto 4/8 SNAPSHOT)
Posted by Rob Davies <ra...@gmail.com>.
Hopefully!
On 16 Apr 2008, at 18:39, yaussy wrote:
>
> Rob,
>
> So, I would find this under the 5.2 SNAPSHOT tomorrow morning (USA
> central
> standard time)?
>
> Kevin
>
>
> rajdavies wrote:
>>
>> yep - being able to reproduce - this is now fixed in trunk - but too
>> late for the 5.1 release
>>
>> thanks,
>>
>> Rob
>>
>> On 16 Apr 2008, at 13:54, yaussy wrote:
>>
>>>
>>> Rob,
>>>
>>> While I don't yet have a solid / consistent test case for you, our
>>> application, using a particular test scenario the app guys have, can
>>> reproduce the problem every time. If you have anything you want me
>>> to try
>>> out, let me know.
>>>
>>> I'm still working with my test code to get something more
>>> reproduceable,
>>> while adding some debug code to IndexManager.
>>>
>>> Kevin
>>>
>>>
>>> yaussy wrote:
>>>>
>>>> Still working on a consistent test case. But, I've run into
>>>> something
>>>> along the way, and I'm not sure if it's related to the original
>>>> exception
>>>> I've logged here. I was able to get the problem to happen when I
>>>> expanded
>>>> my test case to a publisher of 6 topics, at 100+/sec/topic, and a
>>>> consumer
>>>> that consumes all those topics. I have this configured to use a
>>>> connection for each topic, for both the publisher and the
>>>> consumer. I've
>>>> only been able to get the original exception to happen once, but
>>>> I've more
>>>> consistently hit a problem wherein no exception is thrown, but the
>>>> broker
>>>> just stops forwarding events for one of the topics. After this
>>>> point, the
>>>> publisher continues, but all events for this one topic are queued
>>>> to disk.
>>>>
>>>> I'm trying to see if I can reduce this test case down right now.
>>>> But,
>>>> maybe you could give this a go with the supplied consumer /
>>>> supplier test
>>>> code?
>>>>
>>>> Kevin
>>>>
>>>>
>>>> rajdavies wrote:
>>>>>
>>>>> ok - thx for checking! - let me know when you have a test case
>>>>>
>>>>> On 15 Apr 2008, at 15:06, yaussy wrote:
>>>>>
>>>>>>
>>>>>> Bad news - problem still happens with "5.2" snapshot dated
>>>>>> 4/13. I
>>>>>> did not
>>>>>> see anymore 5.1 snapshots, but saw the 5.2 directory. The
>>>>>> archive
>>>>>> still has
>>>>>> 5.1 for the path names, so I figured this is what I should take.
>>>>>>
>>>>>> I'm still working with it and am trying to see if I can get any
>>>>>> more
>>>>>> debug
>>>>>> information. I have not been able to reproduce it with a test
>>>>>> program yet.
>>>>>>
>>>>>>
>>>>>>
>>>>>> yaussy wrote:
>>>>>>>
>>>>>>> Rob,
>>>>>>>
>>>>>>> This is good news - I will try using last nights snapshot.
>>>>>>>
>>>>>>> I started out using AMQMessageStore, but I've never had good
>>>>>>> performance
>>>>>>> luck with AMQs Journal - it always seems to have this frequent,
>>>>>>> high CPU
>>>>>>> cost as it checkpoints. Our Durable rate requirements are
>>>>>>> fairly
>>>>>>> significant, such as a few hundred events per second to a
>>>>>>> handful of
>>>>>>> consumers, on one of our clusters. It has not performed well,
>>>>>>> whereas
>>>>>>> Kaha by itself is excellent.
>>>>>>>
>>>>>>> Kevin
>>>>>>>
>>>>>>>
>>>>>>> rajdavies wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 14 Apr 2008, at 14:07, yaussy wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I have not been able to reproduce this problem outside of our
>>>>>>>>> application,
>>>>>>>>> but I'm still trying.
>>>>>>>>>
>>>>>>>>> Anyway, I'm using regular Kaha persistence (not
>>>>>>>>> AMQMessageStore),
>>>>>>>>> and during
>>>>>>>>> a Durable Topic test, I'm eventually getting the following
>>>>>>>>> exception
>>>>>>>>> in the
>>>>>>>>> AMQBroker, after which the broker stops forwarding events to
>>>>>>>>> the
>>>>>>>>> consumer.
>>>>>>>>>
>>>>>>>>> Anyone seen this?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> <
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq.broker.region.cursors.AbstractStoreCursor.fillBatch>
>>>>>>>>> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:43595,4,main]>
>>>>>>>>> Failed
>>>>>>>>> to fill
>>>>>>>>> batch
>>>>>>>>> Stack Trace follows:
>>>>>>>>> java.lang.RuntimeException: Failed to get next index from
>>>>>>>>> IndexManager:(index-topic-subs) for offset=7681977, key=(2,
>>>>>>>>> 20080006, 47),
>>>>>>>>> value=(2, 20080058, 165), previousItem=7681212,
>>>>>>>>> nextItem=7682895
>>>>>>>>> at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .kaha
>>>>>>>>> .impl
>>>>>>>>> .index
>>>>>>>>> .DiskIndexLinkedList.getNextEntry(DiskIndexLinkedList.java:
>>>>>>>>> 267)
>>>>>>>>> at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .kaha
>>>>>>>>> .impl
>>>>>>>>> .container.MapContainerImpl.getNext(MapContainerImpl.java:
>>>>>>>>> 449)
>>>>>>>>> at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .store
>>>>>>>>> .kahadaptor
>>>>>>>>> .TopicSubContainer.getNextEntry(TopicSubContainer.java:
>>>>>>>>> 95)
>>>>>>>>> at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .store
>>>>>>>>> .kahadaptor
>>>>>>>>> .KahaTopicMessageStore
>>>>>>>>> .recoverNextMessages(KahaTopicMessageStore.java:165)
>>>>>>>>> at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .store
>>>>>>>>> .ProxyTopicMessageStore
>>>>>>>>> .recoverNextMessages(ProxyTopicMessageStore.java:97)
>>>>>>>>> at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .broker
>>>>>>>>> .region
>>>>>>>>> .cursors
>>>>>>>>> .TopicStorePrefetch.doFillBatch(TopicStorePrefetch.java:
>>>>>>>>> 107)
>>>>>>>>> at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .broker
>>>>>>>>> .region
>>>>>>>>> .cursors
>>>>>>>>> .AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:
>>>>>>>>> 188)
>>>>>>>>> at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .broker
>>>>>>>>> .region
>>>>>>>>> .cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:
>>>>>>>>> 104)
>>>>>>>>> at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .broker
>>>>>>>>> .region
>>>>>>>>> .cursors
>>>>>>>>> .StoreDurableSubscriberCursor
>>>>>>>>> .hasNext(StoreDurableSubscriberCursor.java:210
>>>>>>>>> )
>>>>>>>>> at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .broker
>>>>>>>>> .region
>>>>>>>>> .PrefetchSubscription
>>>>>>>>> .dispatchPending(PrefetchSubscription.java:
>>>>>>>>> 479)
>>>>>>>>> at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .broker
>>>>>>>>> .region
>>>>>>>>> .PrefetchSubscription.acknowledge(PrefetchSubscription.java:
>>>>>>>>> 357)
>>>>>>>>> at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .broker.region.AbstractRegion.acknowledge(AbstractRegion.java:
>>>>>>>>> 349)
>>>>>>>>> at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .broker.region.RegionBroker.acknowledge(RegionBroker.java:
>>>>>>>>> 474)
>>>>>>>>> at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .broker.TransactionBroker.acknowledge(TransactionBroker.java:
>>>>>>>>> 194)
>>>>>>>>> at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:
>>>>>>>>> 73)
>>>>>>>>> at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:
>>>>>>>>> 73)
>>>>>>>>> at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .broker
>>>>>>>>> .MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:
>>>>>>>>> 84)
>>>>>>>>> at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .broker
>>>>>>>>> .TransportConnection
>>>>>>>>> .processMessageAck(TransportConnection.java:
>>>>>>>>> 444)
>>>>>>>>> at
>>>>>>>>> org.apache.activemq.command.MessageAck.visit(MessageAck.java:
>>>>>>>>> 196)
>>>>>>>>> at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .broker.TransportConnection.service(TransportConnection.java:
>>>>>>>>> 293)
>>>>>>>>> at
>>>>>>>>> org.apache.activemq.broker.TransportConnection
>>>>>>>>> $1.onCommand(TransportConnection.java:181)
>>>>>>>>> at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .transport.TransportFilter.onCommand(TransportFilter.java:68)
>>>>>>>>> at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .transport
>>>>>>>>> .WireFormatNegotiator.onCommand(WireFormatNegotiator.java:
>>>>>>>>> 143)
>>>>>>>>> at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .transport.InactivityMonitor.onCommand(InactivityMonitor.java:
>>>>>>>>> 206)
>>>>>>>>> at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .transport.TransportSupport.doConsume(TransportSupport.java:
>>>>>>>>> 84)
>>>>>>>>> at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:
>>>>>>>>> 196)
>>>>>>>>> at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq.transport.tcp.TcpTransport.run(TcpTransport.java:
>>>>>>>>> 183)
>>>>>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>>>>>> Caused by: java.io.EOFException
>>>>>>>>> at
>>>>>>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:
>>>>>>>>> 383)
>>>>>>>>> at
>>>>>>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:
>>>>>>>>> 361)
>>>>>>>>> at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .kaha
>>>>>>>>> .impl.index.StoreIndexReader.readItem(StoreIndexReader.java:
>>>>>>>>> 46)
>>>>>>>>> at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .kaha.impl.index.IndexManager.getIndex(IndexManager.java:
>>>>>>>>> 66)
>>>>>>>>> at
>>>>>>>>> org
>>>>>>>>> .apache
>>>>>>>>> .activemq
>>>>>>>>> .kaha
>>>>>>>>> .impl
>>>>>>>>> .index
>>>>>>>>> .DiskIndexLinkedList.getNextEntry(DiskIndexLinkedList.java:
>>>>>>>>> 265)
>>>>>>>>> ... 27 more
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> View this message in context:
>>>>>>>>> http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16677512.html
>>>>>>>>> Sent from the ActiveMQ - User mailing list archive at
>>>>>>>>> Nabble.com.
>>>>>>>>>
>>>>>>>>
>>>>>>>> There was a bug fixed in this area last Friday - would be
>>>>>>>> interested
>>>>>>>> if this is still a problem for you with a later version?
>>>>>>>> If it is - Ill digg a little deeper.
>>>>>>>>
>>>>>>>> btw - why are you using Kaha instead of AMQStore ?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> cheers,
>>>>>>>>
>>>>>>>> Rob
>>>>>>>>
>>>>>>>> http://open.iona.com/ -Enterprise Open Integration
>>>>>>>> http://rajdavies.blogspot.com/
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16700798.html
>>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16721669.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16725769.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Kaha persistence issue w/5.1 (upto 4/8 SNAPSHOT)
Posted by yaussy <ya...@cboe.com>.
Rob,
So, I would find this under the 5.2 SNAPSHOT tomorrow morning (USA central
standard time)?
Kevin
rajdavies wrote:
>
> yep - being able to reproduce - this is now fixed in trunk - but too
> late for the 5.1 release
>
> thanks,
>
> Rob
>
> On 16 Apr 2008, at 13:54, yaussy wrote:
>
>>
>> Rob,
>>
>> While I don't yet have a solid / consistent test case for you, our
>> application, using a particular test scenario the app guys have, can
>> reproduce the problem every time. If you have anything you want me
>> to try
>> out, let me know.
>>
>> I'm still working with my test code to get something more
>> reproduceable,
>> while adding some debug code to IndexManager.
>>
>> Kevin
>>
>>
>> yaussy wrote:
>>>
>>> Still working on a consistent test case. But, I've run into
>>> something
>>> along the way, and I'm not sure if it's related to the original
>>> exception
>>> I've logged here. I was able to get the problem to happen when I
>>> expanded
>>> my test case to a publisher of 6 topics, at 100+/sec/topic, and a
>>> consumer
>>> that consumes all those topics. I have this configured to use a
>>> connection for each topic, for both the publisher and the
>>> consumer. I've
>>> only been able to get the original exception to happen once, but
>>> I've more
>>> consistently hit a problem wherein no exception is thrown, but the
>>> broker
>>> just stops forwarding events for one of the topics. After this
>>> point, the
>>> publisher continues, but all events for this one topic are queued
>>> to disk.
>>>
>>> I'm trying to see if I can reduce this test case down right now.
>>> But,
>>> maybe you could give this a go with the supplied consumer /
>>> supplier test
>>> code?
>>>
>>> Kevin
>>>
>>>
>>> rajdavies wrote:
>>>>
>>>> ok - thx for checking! - let me know when you have a test case
>>>>
>>>> On 15 Apr 2008, at 15:06, yaussy wrote:
>>>>
>>>>>
>>>>> Bad news - problem still happens with "5.2" snapshot dated 4/13. I
>>>>> did not
>>>>> see anymore 5.1 snapshots, but saw the 5.2 directory. The archive
>>>>> still has
>>>>> 5.1 for the path names, so I figured this is what I should take.
>>>>>
>>>>> I'm still working with it and am trying to see if I can get any
>>>>> more
>>>>> debug
>>>>> information. I have not been able to reproduce it with a test
>>>>> program yet.
>>>>>
>>>>>
>>>>>
>>>>> yaussy wrote:
>>>>>>
>>>>>> Rob,
>>>>>>
>>>>>> This is good news - I will try using last nights snapshot.
>>>>>>
>>>>>> I started out using AMQMessageStore, but I've never had good
>>>>>> performance
>>>>>> luck with AMQs Journal - it always seems to have this frequent,
>>>>>> high CPU
>>>>>> cost as it checkpoints. Our Durable rate requirements are fairly
>>>>>> significant, such as a few hundred events per second to a
>>>>>> handful of
>>>>>> consumers, on one of our clusters. It has not performed well,
>>>>>> whereas
>>>>>> Kaha by itself is excellent.
>>>>>>
>>>>>> Kevin
>>>>>>
>>>>>>
>>>>>> rajdavies wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 14 Apr 2008, at 14:07, yaussy wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> I have not been able to reproduce this problem outside of our
>>>>>>>> application,
>>>>>>>> but I'm still trying.
>>>>>>>>
>>>>>>>> Anyway, I'm using regular Kaha persistence (not
>>>>>>>> AMQMessageStore),
>>>>>>>> and during
>>>>>>>> a Durable Topic test, I'm eventually getting the following
>>>>>>>> exception
>>>>>>>> in the
>>>>>>>> AMQBroker, after which the broker stops forwarding events to the
>>>>>>>> consumer.
>>>>>>>>
>>>>>>>> Anyone seen this?
>>>>>>>>
>>>>>>>>
>>>>>>>> <
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq.broker.region.cursors.AbstractStoreCursor.fillBatch>
>>>>>>>> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:43595,4,main]>
>>>>>>>> Failed
>>>>>>>> to fill
>>>>>>>> batch
>>>>>>>> Stack Trace follows:
>>>>>>>> java.lang.RuntimeException: Failed to get next index from
>>>>>>>> IndexManager:(index-topic-subs) for offset=7681977, key=(2,
>>>>>>>> 20080006, 47),
>>>>>>>> value=(2, 20080058, 165), previousItem=7681212, nextItem=7682895
>>>>>>>> at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .kaha
>>>>>>>> .impl
>>>>>>>> .index
>>>>>>>> .DiskIndexLinkedList.getNextEntry(DiskIndexLinkedList.java:
>>>>>>>> 267)
>>>>>>>> at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .kaha
>>>>>>>> .impl.container.MapContainerImpl.getNext(MapContainerImpl.java:
>>>>>>>> 449)
>>>>>>>> at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .store
>>>>>>>> .kahadaptor
>>>>>>>> .TopicSubContainer.getNextEntry(TopicSubContainer.java:
>>>>>>>> 95)
>>>>>>>> at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .store
>>>>>>>> .kahadaptor
>>>>>>>> .KahaTopicMessageStore
>>>>>>>> .recoverNextMessages(KahaTopicMessageStore.java:165)
>>>>>>>> at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .store
>>>>>>>> .ProxyTopicMessageStore
>>>>>>>> .recoverNextMessages(ProxyTopicMessageStore.java:97)
>>>>>>>> at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .broker
>>>>>>>> .region
>>>>>>>> .cursors.TopicStorePrefetch.doFillBatch(TopicStorePrefetch.java:
>>>>>>>> 107)
>>>>>>>> at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .broker
>>>>>>>> .region
>>>>>>>> .cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:
>>>>>>>> 188)
>>>>>>>> at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .broker
>>>>>>>> .region
>>>>>>>> .cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:
>>>>>>>> 104)
>>>>>>>> at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .broker
>>>>>>>> .region
>>>>>>>> .cursors
>>>>>>>> .StoreDurableSubscriberCursor
>>>>>>>> .hasNext(StoreDurableSubscriberCursor.java:210
>>>>>>>> )
>>>>>>>> at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .broker
>>>>>>>> .region
>>>>>>>> .PrefetchSubscription.dispatchPending(PrefetchSubscription.java:
>>>>>>>> 479)
>>>>>>>> at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .broker
>>>>>>>> .region
>>>>>>>> .PrefetchSubscription.acknowledge(PrefetchSubscription.java:
>>>>>>>> 357)
>>>>>>>> at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .broker.region.AbstractRegion.acknowledge(AbstractRegion.java:
>>>>>>>> 349)
>>>>>>>> at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .broker.region.RegionBroker.acknowledge(RegionBroker.java:
>>>>>>>> 474)
>>>>>>>> at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .broker.TransactionBroker.acknowledge(TransactionBroker.java:
>>>>>>>> 194)
>>>>>>>> at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:73)
>>>>>>>> at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:73)
>>>>>>>> at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .broker
>>>>>>>> .MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:
>>>>>>>> 84)
>>>>>>>> at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .broker
>>>>>>>> .TransportConnection.processMessageAck(TransportConnection.java:
>>>>>>>> 444)
>>>>>>>> at
>>>>>>>> org.apache.activemq.command.MessageAck.visit(MessageAck.java:
>>>>>>>> 196)
>>>>>>>> at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .broker.TransportConnection.service(TransportConnection.java:
>>>>>>>> 293)
>>>>>>>> at
>>>>>>>> org.apache.activemq.broker.TransportConnection
>>>>>>>> $1.onCommand(TransportConnection.java:181)
>>>>>>>> at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .transport.TransportFilter.onCommand(TransportFilter.java:68)
>>>>>>>> at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .transport
>>>>>>>> .WireFormatNegotiator.onCommand(WireFormatNegotiator.java:
>>>>>>>> 143)
>>>>>>>> at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .transport.InactivityMonitor.onCommand(InactivityMonitor.java:
>>>>>>>> 206)
>>>>>>>> at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .transport.TransportSupport.doConsume(TransportSupport.java:
>>>>>>>> 84)
>>>>>>>> at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:
>>>>>>>> 196)
>>>>>>>> at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq.transport.tcp.TcpTransport.run(TcpTransport.java:
>>>>>>>> 183)
>>>>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>>>>> Caused by: java.io.EOFException
>>>>>>>> at
>>>>>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:
>>>>>>>> 383)
>>>>>>>> at
>>>>>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:
>>>>>>>> 361)
>>>>>>>> at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .kaha
>>>>>>>> .impl.index.StoreIndexReader.readItem(StoreIndexReader.java:
>>>>>>>> 46)
>>>>>>>> at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .kaha.impl.index.IndexManager.getIndex(IndexManager.java:
>>>>>>>> 66)
>>>>>>>> at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .kaha
>>>>>>>> .impl
>>>>>>>> .index
>>>>>>>> .DiskIndexLinkedList.getNextEntry(DiskIndexLinkedList.java:
>>>>>>>> 265)
>>>>>>>> ... 27 more
>>>>>>>>
>>>>>>>> --
>>>>>>>> View this message in context:
>>>>>>>> http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16677512.html
>>>>>>>> Sent from the ActiveMQ - User mailing list archive at
>>>>>>>> Nabble.com.
>>>>>>>>
>>>>>>>
>>>>>>> There was a bug fixed in this area last Friday - would be
>>>>>>> interested
>>>>>>> if this is still a problem for you with a later version?
>>>>>>> If it is - Ill digg a little deeper.
>>>>>>>
>>>>>>> btw - why are you using Kaha instead of AMQStore ?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> cheers,
>>>>>>>
>>>>>>> Rob
>>>>>>>
>>>>>>> http://open.iona.com/ -Enterprise Open Integration
>>>>>>> http://rajdavies.blogspot.com/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16700798.html
>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16721669.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>
--
View this message in context: http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16725769.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Kaha persistence issue w/5.1 (upto 4/8 SNAPSHOT)
Posted by Rob Davies <ra...@gmail.com>.
yep - being able to reproduce - this is now fixed in trunk - but too
late for the 5.1 release
thanks,
Rob
On 16 Apr 2008, at 13:54, yaussy wrote:
>
> Rob,
>
> While I don't yet have a solid / consistent test case for you, our
> application, using a particular test scenario the app guys have, can
> reproduce the problem every time. If you have anything you want me
> to try
> out, let me know.
>
> I'm still working with my test code to get something more
> reproduceable,
> while adding some debug code to IndexManager.
>
> Kevin
>
>
> yaussy wrote:
>>
>> Still working on a consistent test case. But, I've run into
>> something
>> along the way, and I'm not sure if it's related to the original
>> exception
>> I've logged here. I was able to get the problem to happen when I
>> expanded
>> my test case to a publisher of 6 topics, at 100+/sec/topic, and a
>> consumer
>> that consumes all those topics. I have this configured to use a
>> connection for each topic, for both the publisher and the
>> consumer. I've
>> only been able to get the original exception to happen once, but
>> I've more
>> consistently hit a problem wherein no exception is thrown, but the
>> broker
>> just stops forwarding events for one of the topics. After this
>> point, the
>> publisher continues, but all events for this one topic are queued
>> to disk.
>>
>> I'm trying to see if I can reduce this test case down right now.
>> But,
>> maybe you could give this a go with the supplied consumer /
>> supplier test
>> code?
>>
>> Kevin
>>
>>
>> rajdavies wrote:
>>>
>>> ok - thx for checking! - let me know when you have a test case
>>>
>>> On 15 Apr 2008, at 15:06, yaussy wrote:
>>>
>>>>
>>>> Bad news - problem still happens with "5.2" snapshot dated 4/13. I
>>>> did not
>>>> see anymore 5.1 snapshots, but saw the 5.2 directory. The archive
>>>> still has
>>>> 5.1 for the path names, so I figured this is what I should take.
>>>>
>>>> I'm still working with it and am trying to see if I can get any
>>>> more
>>>> debug
>>>> information. I have not been able to reproduce it with a test
>>>> program yet.
>>>>
>>>>
>>>>
>>>> yaussy wrote:
>>>>>
>>>>> Rob,
>>>>>
>>>>> This is good news - I will try using last nights snapshot.
>>>>>
>>>>> I started out using AMQMessageStore, but I've never had good
>>>>> performance
>>>>> luck with AMQs Journal - it always seems to have this frequent,
>>>>> high CPU
>>>>> cost as it checkpoints. Our Durable rate requirements are fairly
>>>>> significant, such as a few hundred events per second to a
>>>>> handful of
>>>>> consumers, on one of our clusters. It has not performed well,
>>>>> whereas
>>>>> Kaha by itself is excellent.
>>>>>
>>>>> Kevin
>>>>>
>>>>>
>>>>> rajdavies wrote:
>>>>>>
>>>>>>
>>>>>> On 14 Apr 2008, at 14:07, yaussy wrote:
>>>>>>
>>>>>>>
>>>>>>> I have not been able to reproduce this problem outside of our
>>>>>>> application,
>>>>>>> but I'm still trying.
>>>>>>>
>>>>>>> Anyway, I'm using regular Kaha persistence (not
>>>>>>> AMQMessageStore),
>>>>>>> and during
>>>>>>> a Durable Topic test, I'm eventually getting the following
>>>>>>> exception
>>>>>>> in the
>>>>>>> AMQBroker, after which the broker stops forwarding events to the
>>>>>>> consumer.
>>>>>>>
>>>>>>> Anyone seen this?
>>>>>>>
>>>>>>>
>>>>>>> <
>>>>>>> org
>>>>>>> .apache
>>>>>>> .activemq.broker.region.cursors.AbstractStoreCursor.fillBatch>
>>>>>>> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:43595,4,main]>
>>>>>>> Failed
>>>>>>> to fill
>>>>>>> batch
>>>>>>> Stack Trace follows:
>>>>>>> java.lang.RuntimeException: Failed to get next index from
>>>>>>> IndexManager:(index-topic-subs) for offset=7681977, key=(2,
>>>>>>> 20080006, 47),
>>>>>>> value=(2, 20080058, 165), previousItem=7681212, nextItem=7682895
>>>>>>> at
>>>>>>> org
>>>>>>> .apache
>>>>>>> .activemq
>>>>>>> .kaha
>>>>>>> .impl
>>>>>>> .index
>>>>>>> .DiskIndexLinkedList.getNextEntry(DiskIndexLinkedList.java:
>>>>>>> 267)
>>>>>>> at
>>>>>>> org
>>>>>>> .apache
>>>>>>> .activemq
>>>>>>> .kaha
>>>>>>> .impl.container.MapContainerImpl.getNext(MapContainerImpl.java:
>>>>>>> 449)
>>>>>>> at
>>>>>>> org
>>>>>>> .apache
>>>>>>> .activemq
>>>>>>> .store
>>>>>>> .kahadaptor
>>>>>>> .TopicSubContainer.getNextEntry(TopicSubContainer.java:
>>>>>>> 95)
>>>>>>> at
>>>>>>> org
>>>>>>> .apache
>>>>>>> .activemq
>>>>>>> .store
>>>>>>> .kahadaptor
>>>>>>> .KahaTopicMessageStore
>>>>>>> .recoverNextMessages(KahaTopicMessageStore.java:165)
>>>>>>> at
>>>>>>> org
>>>>>>> .apache
>>>>>>> .activemq
>>>>>>> .store
>>>>>>> .ProxyTopicMessageStore
>>>>>>> .recoverNextMessages(ProxyTopicMessageStore.java:97)
>>>>>>> at
>>>>>>> org
>>>>>>> .apache
>>>>>>> .activemq
>>>>>>> .broker
>>>>>>> .region
>>>>>>> .cursors.TopicStorePrefetch.doFillBatch(TopicStorePrefetch.java:
>>>>>>> 107)
>>>>>>> at
>>>>>>> org
>>>>>>> .apache
>>>>>>> .activemq
>>>>>>> .broker
>>>>>>> .region
>>>>>>> .cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:
>>>>>>> 188)
>>>>>>> at
>>>>>>> org
>>>>>>> .apache
>>>>>>> .activemq
>>>>>>> .broker
>>>>>>> .region
>>>>>>> .cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:
>>>>>>> 104)
>>>>>>> at
>>>>>>> org
>>>>>>> .apache
>>>>>>> .activemq
>>>>>>> .broker
>>>>>>> .region
>>>>>>> .cursors
>>>>>>> .StoreDurableSubscriberCursor
>>>>>>> .hasNext(StoreDurableSubscriberCursor.java:210
>>>>>>> )
>>>>>>> at
>>>>>>> org
>>>>>>> .apache
>>>>>>> .activemq
>>>>>>> .broker
>>>>>>> .region
>>>>>>> .PrefetchSubscription.dispatchPending(PrefetchSubscription.java:
>>>>>>> 479)
>>>>>>> at
>>>>>>> org
>>>>>>> .apache
>>>>>>> .activemq
>>>>>>> .broker
>>>>>>> .region
>>>>>>> .PrefetchSubscription.acknowledge(PrefetchSubscription.java:
>>>>>>> 357)
>>>>>>> at
>>>>>>> org
>>>>>>> .apache
>>>>>>> .activemq
>>>>>>> .broker.region.AbstractRegion.acknowledge(AbstractRegion.java:
>>>>>>> 349)
>>>>>>> at
>>>>>>> org
>>>>>>> .apache
>>>>>>> .activemq
>>>>>>> .broker.region.RegionBroker.acknowledge(RegionBroker.java:
>>>>>>> 474)
>>>>>>> at
>>>>>>> org
>>>>>>> .apache
>>>>>>> .activemq
>>>>>>> .broker.TransactionBroker.acknowledge(TransactionBroker.java:
>>>>>>> 194)
>>>>>>> at
>>>>>>> org
>>>>>>> .apache
>>>>>>> .activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:73)
>>>>>>> at
>>>>>>> org
>>>>>>> .apache
>>>>>>> .activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:73)
>>>>>>> at
>>>>>>> org
>>>>>>> .apache
>>>>>>> .activemq
>>>>>>> .broker
>>>>>>> .MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:
>>>>>>> 84)
>>>>>>> at
>>>>>>> org
>>>>>>> .apache
>>>>>>> .activemq
>>>>>>> .broker
>>>>>>> .TransportConnection.processMessageAck(TransportConnection.java:
>>>>>>> 444)
>>>>>>> at
>>>>>>> org.apache.activemq.command.MessageAck.visit(MessageAck.java:
>>>>>>> 196)
>>>>>>> at
>>>>>>> org
>>>>>>> .apache
>>>>>>> .activemq
>>>>>>> .broker.TransportConnection.service(TransportConnection.java:
>>>>>>> 293)
>>>>>>> at
>>>>>>> org.apache.activemq.broker.TransportConnection
>>>>>>> $1.onCommand(TransportConnection.java:181)
>>>>>>> at
>>>>>>> org
>>>>>>> .apache
>>>>>>> .activemq
>>>>>>> .transport.TransportFilter.onCommand(TransportFilter.java:68)
>>>>>>> at
>>>>>>> org
>>>>>>> .apache
>>>>>>> .activemq
>>>>>>> .transport
>>>>>>> .WireFormatNegotiator.onCommand(WireFormatNegotiator.java:
>>>>>>> 143)
>>>>>>> at
>>>>>>> org
>>>>>>> .apache
>>>>>>> .activemq
>>>>>>> .transport.InactivityMonitor.onCommand(InactivityMonitor.java:
>>>>>>> 206)
>>>>>>> at
>>>>>>> org
>>>>>>> .apache
>>>>>>> .activemq
>>>>>>> .transport.TransportSupport.doConsume(TransportSupport.java:
>>>>>>> 84)
>>>>>>> at
>>>>>>> org
>>>>>>> .apache
>>>>>>> .activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:
>>>>>>> 196)
>>>>>>> at
>>>>>>> org
>>>>>>> .apache
>>>>>>> .activemq.transport.tcp.TcpTransport.run(TcpTransport.java:
>>>>>>> 183)
>>>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>>>> Caused by: java.io.EOFException
>>>>>>> at
>>>>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:
>>>>>>> 383)
>>>>>>> at
>>>>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:
>>>>>>> 361)
>>>>>>> at
>>>>>>> org
>>>>>>> .apache
>>>>>>> .activemq
>>>>>>> .kaha
>>>>>>> .impl.index.StoreIndexReader.readItem(StoreIndexReader.java:
>>>>>>> 46)
>>>>>>> at
>>>>>>> org
>>>>>>> .apache
>>>>>>> .activemq
>>>>>>> .kaha.impl.index.IndexManager.getIndex(IndexManager.java:
>>>>>>> 66)
>>>>>>> at
>>>>>>> org
>>>>>>> .apache
>>>>>>> .activemq
>>>>>>> .kaha
>>>>>>> .impl
>>>>>>> .index
>>>>>>> .DiskIndexLinkedList.getNextEntry(DiskIndexLinkedList.java:
>>>>>>> 265)
>>>>>>> ... 27 more
>>>>>>>
>>>>>>> --
>>>>>>> View this message in context:
>>>>>>> http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16677512.html
>>>>>>> Sent from the ActiveMQ - User mailing list archive at
>>>>>>> Nabble.com.
>>>>>>>
>>>>>>
>>>>>> There was a bug fixed in this area last Friday - would be
>>>>>> interested
>>>>>> if this is still a problem for you with a later version?
>>>>>> If it is - Ill digg a little deeper.
>>>>>>
>>>>>> btw - why are you using Kaha instead of AMQStore ?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> cheers,
>>>>>>
>>>>>> Rob
>>>>>>
>>>>>> http://open.iona.com/ -Enterprise Open Integration
>>>>>> http://rajdavies.blogspot.com/
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16700798.html
>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>
>>>
>>>
>>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16721669.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Kaha persistence issue w/5.1 (upto 4/8 SNAPSHOT)
Posted by yaussy <ya...@cboe.com>.
Rob,
While I don't yet have a solid / consistent test case for you, our
application, using a particular test scenario the app guys have, can
reproduce the problem every time. If you have anything you want me to try
out, let me know.
I'm still working with my test code to get something more reproduceable,
while adding some debug code to IndexManager.
Kevin
yaussy wrote:
>
> Still working on a consistent test case. But, I've run into something
> along the way, and I'm not sure if it's related to the original exception
> I've logged here. I was able to get the problem to happen when I expanded
> my test case to a publisher of 6 topics, at 100+/sec/topic, and a consumer
> that consumes all those topics. I have this configured to use a
> connection for each topic, for both the publisher and the consumer. I've
> only been able to get the original exception to happen once, but I've more
> consistently hit a problem wherein no exception is thrown, but the broker
> just stops forwarding events for one of the topics. After this point, the
> publisher continues, but all events for this one topic are queued to disk.
>
> I'm trying to see if I can reduce this test case down right now. But,
> maybe you could give this a go with the supplied consumer / supplier test
> code?
>
> Kevin
>
>
> rajdavies wrote:
>>
>> ok - thx for checking! - let me know when you have a test case
>>
>> On 15 Apr 2008, at 15:06, yaussy wrote:
>>
>>>
>>> Bad news - problem still happens with "5.2" snapshot dated 4/13. I
>>> did not
>>> see anymore 5.1 snapshots, but saw the 5.2 directory. The archive
>>> still has
>>> 5.1 for the path names, so I figured this is what I should take.
>>>
>>> I'm still working with it and am trying to see if I can get any more
>>> debug
>>> information. I have not been able to reproduce it with a test
>>> program yet.
>>>
>>>
>>>
>>> yaussy wrote:
>>>>
>>>> Rob,
>>>>
>>>> This is good news - I will try using last nights snapshot.
>>>>
>>>> I started out using AMQMessageStore, but I've never had good
>>>> performance
>>>> luck with AMQs Journal - it always seems to have this frequent,
>>>> high CPU
>>>> cost as it checkpoints. Our Durable rate requirements are fairly
>>>> significant, such as a few hundred events per second to a handful of
>>>> consumers, on one of our clusters. It has not performed well,
>>>> whereas
>>>> Kaha by itself is excellent.
>>>>
>>>> Kevin
>>>>
>>>>
>>>> rajdavies wrote:
>>>>>
>>>>>
>>>>> On 14 Apr 2008, at 14:07, yaussy wrote:
>>>>>
>>>>>>
>>>>>> I have not been able to reproduce this problem outside of our
>>>>>> application,
>>>>>> but I'm still trying.
>>>>>>
>>>>>> Anyway, I'm using regular Kaha persistence (not AMQMessageStore),
>>>>>> and during
>>>>>> a Durable Topic test, I'm eventually getting the following
>>>>>> exception
>>>>>> in the
>>>>>> AMQBroker, after which the broker stops forwarding events to the
>>>>>> consumer.
>>>>>>
>>>>>> Anyone seen this?
>>>>>>
>>>>>>
>>>>>> <
>>>>>> org
>>>>>> .apache
>>>>>> .activemq.broker.region.cursors.AbstractStoreCursor.fillBatch>
>>>>>> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:43595,4,main]> Failed
>>>>>> to fill
>>>>>> batch
>>>>>> Stack Trace follows:
>>>>>> java.lang.RuntimeException: Failed to get next index from
>>>>>> IndexManager:(index-topic-subs) for offset=7681977, key=(2,
>>>>>> 20080006, 47),
>>>>>> value=(2, 20080058, 165), previousItem=7681212, nextItem=7682895
>>>>>> at
>>>>>> org
>>>>>> .apache
>>>>>> .activemq
>>>>>> .kaha
>>>>>> .impl
>>>>>> .index.DiskIndexLinkedList.getNextEntry(DiskIndexLinkedList.java:
>>>>>> 267)
>>>>>> at
>>>>>> org
>>>>>> .apache
>>>>>> .activemq
>>>>>> .kaha
>>>>>> .impl.container.MapContainerImpl.getNext(MapContainerImpl.java:
>>>>>> 449)
>>>>>> at
>>>>>> org
>>>>>> .apache
>>>>>> .activemq
>>>>>> .store
>>>>>> .kahadaptor.TopicSubContainer.getNextEntry(TopicSubContainer.java:
>>>>>> 95)
>>>>>> at
>>>>>> org
>>>>>> .apache
>>>>>> .activemq
>>>>>> .store
>>>>>> .kahadaptor
>>>>>> .KahaTopicMessageStore
>>>>>> .recoverNextMessages(KahaTopicMessageStore.java:165)
>>>>>> at
>>>>>> org
>>>>>> .apache
>>>>>> .activemq
>>>>>> .store
>>>>>> .ProxyTopicMessageStore
>>>>>> .recoverNextMessages(ProxyTopicMessageStore.java:97)
>>>>>> at
>>>>>> org
>>>>>> .apache
>>>>>> .activemq
>>>>>> .broker
>>>>>> .region
>>>>>> .cursors.TopicStorePrefetch.doFillBatch(TopicStorePrefetch.java:
>>>>>> 107)
>>>>>> at
>>>>>> org
>>>>>> .apache
>>>>>> .activemq
>>>>>> .broker
>>>>>> .region
>>>>>> .cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:
>>>>>> 188)
>>>>>> at
>>>>>> org
>>>>>> .apache
>>>>>> .activemq
>>>>>> .broker
>>>>>> .region
>>>>>> .cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:
>>>>>> 104)
>>>>>> at
>>>>>> org
>>>>>> .apache
>>>>>> .activemq
>>>>>> .broker
>>>>>> .region
>>>>>> .cursors
>>>>>> .StoreDurableSubscriberCursor
>>>>>> .hasNext(StoreDurableSubscriberCursor.java:210
>>>>>> )
>>>>>> at
>>>>>> org
>>>>>> .apache
>>>>>> .activemq
>>>>>> .broker
>>>>>> .region
>>>>>> .PrefetchSubscription.dispatchPending(PrefetchSubscription.java:
>>>>>> 479)
>>>>>> at
>>>>>> org
>>>>>> .apache
>>>>>> .activemq
>>>>>> .broker
>>>>>> .region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:
>>>>>> 357)
>>>>>> at
>>>>>> org
>>>>>> .apache
>>>>>> .activemq
>>>>>> .broker.region.AbstractRegion.acknowledge(AbstractRegion.java:349)
>>>>>> at
>>>>>> org
>>>>>> .apache
>>>>>> .activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:
>>>>>> 474)
>>>>>> at
>>>>>> org
>>>>>> .apache
>>>>>> .activemq
>>>>>> .broker.TransactionBroker.acknowledge(TransactionBroker.java:194)
>>>>>> at
>>>>>> org
>>>>>> .apache
>>>>>> .activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:73)
>>>>>> at
>>>>>> org
>>>>>> .apache
>>>>>> .activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:73)
>>>>>> at
>>>>>> org
>>>>>> .apache
>>>>>> .activemq
>>>>>> .broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:
>>>>>> 84)
>>>>>> at
>>>>>> org
>>>>>> .apache
>>>>>> .activemq
>>>>>> .broker
>>>>>> .TransportConnection.processMessageAck(TransportConnection.java:
>>>>>> 444)
>>>>>> at
>>>>>> org.apache.activemq.command.MessageAck.visit(MessageAck.java:196)
>>>>>> at
>>>>>> org
>>>>>> .apache
>>>>>> .activemq
>>>>>> .broker.TransportConnection.service(TransportConnection.java:293)
>>>>>> at
>>>>>> org.apache.activemq.broker.TransportConnection
>>>>>> $1.onCommand(TransportConnection.java:181)
>>>>>> at
>>>>>> org
>>>>>> .apache
>>>>>> .activemq
>>>>>> .transport.TransportFilter.onCommand(TransportFilter.java:68)
>>>>>> at
>>>>>> org
>>>>>> .apache
>>>>>> .activemq
>>>>>> .transport
>>>>>> .WireFormatNegotiator.onCommand(WireFormatNegotiator.java:
>>>>>> 143)
>>>>>> at
>>>>>> org
>>>>>> .apache
>>>>>> .activemq
>>>>>> .transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
>>>>>> at
>>>>>> org
>>>>>> .apache
>>>>>> .activemq
>>>>>> .transport.TransportSupport.doConsume(TransportSupport.java:
>>>>>> 84)
>>>>>> at
>>>>>> org
>>>>>> .apache
>>>>>> .activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:
>>>>>> 196)
>>>>>> at
>>>>>> org
>>>>>> .apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:
>>>>>> 183)
>>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>>> Caused by: java.io.EOFException
>>>>>> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:
>>>>>> 383)
>>>>>> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:
>>>>>> 361)
>>>>>> at
>>>>>> org
>>>>>> .apache
>>>>>> .activemq
>>>>>> .kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:
>>>>>> 46)
>>>>>> at
>>>>>> org
>>>>>> .apache
>>>>>> .activemq.kaha.impl.index.IndexManager.getIndex(IndexManager.java:
>>>>>> 66)
>>>>>> at
>>>>>> org
>>>>>> .apache
>>>>>> .activemq
>>>>>> .kaha
>>>>>> .impl
>>>>>> .index.DiskIndexLinkedList.getNextEntry(DiskIndexLinkedList.java:
>>>>>> 265)
>>>>>> ... 27 more
>>>>>>
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16677512.html
>>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>>>
>>>>>
>>>>> There was a bug fixed in this area last Friday - would be interested
>>>>> if this is still a problem for you with a later version?
>>>>> If it is - Ill digg a little deeper.
>>>>>
>>>>> btw - why are you using Kaha instead of AMQStore ?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> cheers,
>>>>>
>>>>> Rob
>>>>>
>>>>> http://open.iona.com/ -Enterprise Open Integration
>>>>> http://rajdavies.blogspot.com/
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16700798.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16721669.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Kaha persistence issue w/5.1 (upto 4/8 SNAPSHOT)
Posted by yaussy <ya...@cboe.com>.
Still working on a consistent test case. But, I've run into something along
the way, and I'm not sure if it's related to the original exception I've
logged here. I was able to get the problem to happen when I expanded my
test case to a publisher of 6 topics, at 100+/sec/topic, and a consumer that
consumes all those topics. I have this configured to use a connection for
each topic, for both the publisher and the consumer. I've only been able to
get the original exception to happen once, but I've more consistently hit a
problem wherein no exception is thrown, but the broker just stops forwarding
events for one of the topics. After this point, the publisher continues,
but all events for this one topic are queued to disk.
I'm trying to see if I can reduce this test case down right now. But, maybe
you could give this a go with the supplied consumer / supplier test code?
Kevin
rajdavies wrote:
>
> ok - thx for checking! - let me know when you have a test case
>
> On 15 Apr 2008, at 15:06, yaussy wrote:
>
>>
>> Bad news - problem still happens with "5.2" snapshot dated 4/13. I
>> did not
>> see anymore 5.1 snapshots, but saw the 5.2 directory. The archive
>> still has
>> 5.1 for the path names, so I figured this is what I should take.
>>
>> I'm still working with it and am trying to see if I can get any more
>> debug
>> information. I have not been able to reproduce it with a test
>> program yet.
>>
>>
>>
>> yaussy wrote:
>>>
>>> Rob,
>>>
>>> This is good news - I will try using last nights snapshot.
>>>
>>> I started out using AMQMessageStore, but I've never had good
>>> performance
>>> luck with AMQs Journal - it always seems to have this frequent,
>>> high CPU
>>> cost as it checkpoints. Our Durable rate requirements are fairly
>>> significant, such as a few hundred events per second to a handful of
>>> consumers, on one of our clusters. It has not performed well,
>>> whereas
>>> Kaha by itself is excellent.
>>>
>>> Kevin
>>>
>>>
>>> rajdavies wrote:
>>>>
>>>>
>>>> On 14 Apr 2008, at 14:07, yaussy wrote:
>>>>
>>>>>
>>>>> I have not been able to reproduce this problem outside of our
>>>>> application,
>>>>> but I'm still trying.
>>>>>
>>>>> Anyway, I'm using regular Kaha persistence (not AMQMessageStore),
>>>>> and during
>>>>> a Durable Topic test, I'm eventually getting the following
>>>>> exception
>>>>> in the
>>>>> AMQBroker, after which the broker stops forwarding events to the
>>>>> consumer.
>>>>>
>>>>> Anyone seen this?
>>>>>
>>>>>
>>>>> <
>>>>> org
>>>>> .apache
>>>>> .activemq.broker.region.cursors.AbstractStoreCursor.fillBatch>
>>>>> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:43595,4,main]> Failed
>>>>> to fill
>>>>> batch
>>>>> Stack Trace follows:
>>>>> java.lang.RuntimeException: Failed to get next index from
>>>>> IndexManager:(index-topic-subs) for offset=7681977, key=(2,
>>>>> 20080006, 47),
>>>>> value=(2, 20080058, 165), previousItem=7681212, nextItem=7682895
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .kaha
>>>>> .impl
>>>>> .index.DiskIndexLinkedList.getNextEntry(DiskIndexLinkedList.java:
>>>>> 267)
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .kaha
>>>>> .impl.container.MapContainerImpl.getNext(MapContainerImpl.java:
>>>>> 449)
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .store
>>>>> .kahadaptor.TopicSubContainer.getNextEntry(TopicSubContainer.java:
>>>>> 95)
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .store
>>>>> .kahadaptor
>>>>> .KahaTopicMessageStore
>>>>> .recoverNextMessages(KahaTopicMessageStore.java:165)
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .store
>>>>> .ProxyTopicMessageStore
>>>>> .recoverNextMessages(ProxyTopicMessageStore.java:97)
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .broker
>>>>> .region
>>>>> .cursors.TopicStorePrefetch.doFillBatch(TopicStorePrefetch.java:
>>>>> 107)
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .broker
>>>>> .region
>>>>> .cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:
>>>>> 188)
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .broker
>>>>> .region
>>>>> .cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:
>>>>> 104)
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .broker
>>>>> .region
>>>>> .cursors
>>>>> .StoreDurableSubscriberCursor
>>>>> .hasNext(StoreDurableSubscriberCursor.java:210
>>>>> )
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .broker
>>>>> .region
>>>>> .PrefetchSubscription.dispatchPending(PrefetchSubscription.java:
>>>>> 479)
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .broker
>>>>> .region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:
>>>>> 357)
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .broker.region.AbstractRegion.acknowledge(AbstractRegion.java:349)
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:
>>>>> 474)
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .broker.TransactionBroker.acknowledge(TransactionBroker.java:194)
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:73)
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:73)
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:
>>>>> 84)
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .broker
>>>>> .TransportConnection.processMessageAck(TransportConnection.java:
>>>>> 444)
>>>>> at
>>>>> org.apache.activemq.command.MessageAck.visit(MessageAck.java:196)
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .broker.TransportConnection.service(TransportConnection.java:293)
>>>>> at
>>>>> org.apache.activemq.broker.TransportConnection
>>>>> $1.onCommand(TransportConnection.java:181)
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .transport.TransportFilter.onCommand(TransportFilter.java:68)
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .transport
>>>>> .WireFormatNegotiator.onCommand(WireFormatNegotiator.java:
>>>>> 143)
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .transport.TransportSupport.doConsume(TransportSupport.java:
>>>>> 84)
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:
>>>>> 196)
>>>>> at
>>>>> org
>>>>> .apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:
>>>>> 183)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> Caused by: java.io.EOFException
>>>>> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:
>>>>> 383)
>>>>> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:
>>>>> 361)
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:
>>>>> 46)
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .activemq.kaha.impl.index.IndexManager.getIndex(IndexManager.java:
>>>>> 66)
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .kaha
>>>>> .impl
>>>>> .index.DiskIndexLinkedList.getNextEntry(DiskIndexLinkedList.java:
>>>>> 265)
>>>>> ... 27 more
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16677512.html
>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>>
>>>>
>>>> There was a bug fixed in this area last Friday - would be interested
>>>> if this is still a problem for you with a later version?
>>>> If it is - Ill digg a little deeper.
>>>>
>>>> btw - why are you using Kaha instead of AMQStore ?
>>>>
>>>>
>>>>
>>>>
>>>> cheers,
>>>>
>>>> Rob
>>>>
>>>> http://open.iona.com/ -Enterprise Open Integration
>>>> http://rajdavies.blogspot.com/
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16700798.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>
--
View this message in context: http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16711062.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Kaha persistence issue w/5.1 (upto 4/8 SNAPSHOT)
Posted by Rob Davies <ra...@gmail.com>.
ok - thx for checking! - let me know when you have a test case
On 15 Apr 2008, at 15:06, yaussy wrote:
>
> Bad news - problem still happens with "5.2" snapshot dated 4/13. I
> did not
> see anymore 5.1 snapshots, but saw the 5.2 directory. The archive
> still has
> 5.1 for the path names, so I figured this is what I should take.
>
> I'm still working with it and am trying to see if I can get any more
> debug
> information. I have not been able to reproduce it with a test
> program yet.
>
>
>
> yaussy wrote:
>>
>> Rob,
>>
>> This is good news - I will try using last nights snapshot.
>>
>> I started out using AMQMessageStore, but I've never had good
>> performance
>> luck with AMQs Journal - it always seems to have this frequent,
>> high CPU
>> cost as it checkpoints. Our Durable rate requirements are fairly
>> significant, such as a few hundred events per second to a handful of
>> consumers, on one of our clusters. It has not performed well,
>> whereas
>> Kaha by itself is excellent.
>>
>> Kevin
>>
>>
>> rajdavies wrote:
>>>
>>>
>>> On 14 Apr 2008, at 14:07, yaussy wrote:
>>>
>>>>
>>>> I have not been able to reproduce this problem outside of our
>>>> application,
>>>> but I'm still trying.
>>>>
>>>> Anyway, I'm using regular Kaha persistence (not AMQMessageStore),
>>>> and during
>>>> a Durable Topic test, I'm eventually getting the following
>>>> exception
>>>> in the
>>>> AMQBroker, after which the broker stops forwarding events to the
>>>> consumer.
>>>>
>>>> Anyone seen this?
>>>>
>>>>
>>>> <
>>>> org
>>>> .apache
>>>> .activemq.broker.region.cursors.AbstractStoreCursor.fillBatch>
>>>> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:43595,4,main]> Failed
>>>> to fill
>>>> batch
>>>> Stack Trace follows:
>>>> java.lang.RuntimeException: Failed to get next index from
>>>> IndexManager:(index-topic-subs) for offset=7681977, key=(2,
>>>> 20080006, 47),
>>>> value=(2, 20080058, 165), previousItem=7681212, nextItem=7682895
>>>> at
>>>> org
>>>> .apache
>>>> .activemq
>>>> .kaha
>>>> .impl
>>>> .index.DiskIndexLinkedList.getNextEntry(DiskIndexLinkedList.java:
>>>> 267)
>>>> at
>>>> org
>>>> .apache
>>>> .activemq
>>>> .kaha
>>>> .impl.container.MapContainerImpl.getNext(MapContainerImpl.java:
>>>> 449)
>>>> at
>>>> org
>>>> .apache
>>>> .activemq
>>>> .store
>>>> .kahadaptor.TopicSubContainer.getNextEntry(TopicSubContainer.java:
>>>> 95)
>>>> at
>>>> org
>>>> .apache
>>>> .activemq
>>>> .store
>>>> .kahadaptor
>>>> .KahaTopicMessageStore
>>>> .recoverNextMessages(KahaTopicMessageStore.java:165)
>>>> at
>>>> org
>>>> .apache
>>>> .activemq
>>>> .store
>>>> .ProxyTopicMessageStore
>>>> .recoverNextMessages(ProxyTopicMessageStore.java:97)
>>>> at
>>>> org
>>>> .apache
>>>> .activemq
>>>> .broker
>>>> .region
>>>> .cursors.TopicStorePrefetch.doFillBatch(TopicStorePrefetch.java:
>>>> 107)
>>>> at
>>>> org
>>>> .apache
>>>> .activemq
>>>> .broker
>>>> .region
>>>> .cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:
>>>> 188)
>>>> at
>>>> org
>>>> .apache
>>>> .activemq
>>>> .broker
>>>> .region
>>>> .cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:
>>>> 104)
>>>> at
>>>> org
>>>> .apache
>>>> .activemq
>>>> .broker
>>>> .region
>>>> .cursors
>>>> .StoreDurableSubscriberCursor
>>>> .hasNext(StoreDurableSubscriberCursor.java:210
>>>> )
>>>> at
>>>> org
>>>> .apache
>>>> .activemq
>>>> .broker
>>>> .region
>>>> .PrefetchSubscription.dispatchPending(PrefetchSubscription.java:
>>>> 479)
>>>> at
>>>> org
>>>> .apache
>>>> .activemq
>>>> .broker
>>>> .region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:
>>>> 357)
>>>> at
>>>> org
>>>> .apache
>>>> .activemq
>>>> .broker.region.AbstractRegion.acknowledge(AbstractRegion.java:349)
>>>> at
>>>> org
>>>> .apache
>>>> .activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:
>>>> 474)
>>>> at
>>>> org
>>>> .apache
>>>> .activemq
>>>> .broker.TransactionBroker.acknowledge(TransactionBroker.java:194)
>>>> at
>>>> org
>>>> .apache
>>>> .activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:73)
>>>> at
>>>> org
>>>> .apache
>>>> .activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:73)
>>>> at
>>>> org
>>>> .apache
>>>> .activemq
>>>> .broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:
>>>> 84)
>>>> at
>>>> org
>>>> .apache
>>>> .activemq
>>>> .broker
>>>> .TransportConnection.processMessageAck(TransportConnection.java:
>>>> 444)
>>>> at
>>>> org.apache.activemq.command.MessageAck.visit(MessageAck.java:196)
>>>> at
>>>> org
>>>> .apache
>>>> .activemq
>>>> .broker.TransportConnection.service(TransportConnection.java:293)
>>>> at
>>>> org.apache.activemq.broker.TransportConnection
>>>> $1.onCommand(TransportConnection.java:181)
>>>> at
>>>> org
>>>> .apache
>>>> .activemq
>>>> .transport.TransportFilter.onCommand(TransportFilter.java:68)
>>>> at
>>>> org
>>>> .apache
>>>> .activemq
>>>> .transport
>>>> .WireFormatNegotiator.onCommand(WireFormatNegotiator.java:
>>>> 143)
>>>> at
>>>> org
>>>> .apache
>>>> .activemq
>>>> .transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
>>>> at
>>>> org
>>>> .apache
>>>> .activemq
>>>> .transport.TransportSupport.doConsume(TransportSupport.java:
>>>> 84)
>>>> at
>>>> org
>>>> .apache
>>>> .activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:
>>>> 196)
>>>> at
>>>> org
>>>> .apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:
>>>> 183)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> Caused by: java.io.EOFException
>>>> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:
>>>> 383)
>>>> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:
>>>> 361)
>>>> at
>>>> org
>>>> .apache
>>>> .activemq
>>>> .kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:
>>>> 46)
>>>> at
>>>> org
>>>> .apache
>>>> .activemq.kaha.impl.index.IndexManager.getIndex(IndexManager.java:
>>>> 66)
>>>> at
>>>> org
>>>> .apache
>>>> .activemq
>>>> .kaha
>>>> .impl
>>>> .index.DiskIndexLinkedList.getNextEntry(DiskIndexLinkedList.java:
>>>> 265)
>>>> ... 27 more
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16677512.html
>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>
>>>
>>> There was a bug fixed in this area last Friday - would be interested
>>> if this is still a problem for you with a later version?
>>> If it is - Ill digg a little deeper.
>>>
>>> btw - why are you using Kaha instead of AMQStore ?
>>>
>>>
>>>
>>>
>>> cheers,
>>>
>>> Rob
>>>
>>> http://open.iona.com/ -Enterprise Open Integration
>>> http://rajdavies.blogspot.com/
>>>
>>>
>>>
>>>
>>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16700798.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Kaha persistence issue w/5.1 (upto 4/8 SNAPSHOT)
Posted by yaussy <ya...@cboe.com>.
Bad news - problem still happens with "5.2" snapshot dated 4/13. I did not
see anymore 5.1 snapshots, but saw the 5.2 directory. The archive still has
5.1 for the path names, so I figured this is what I should take.
I'm still working with it and am trying to see if I can get any more debug
information. I have not been able to reproduce it with a test program yet.
yaussy wrote:
>
> Rob,
>
> This is good news - I will try using last nights snapshot.
>
> I started out using AMQMessageStore, but I've never had good performance
> luck with AMQs Journal - it always seems to have this frequent, high CPU
> cost as it checkpoints. Our Durable rate requirements are fairly
> significant, such as a few hundred events per second to a handful of
> consumers, on one of our clusters. It has not performed well, whereas
> Kaha by itself is excellent.
>
> Kevin
>
>
> rajdavies wrote:
>>
>>
>> On 14 Apr 2008, at 14:07, yaussy wrote:
>>
>>>
>>> I have not been able to reproduce this problem outside of our
>>> application,
>>> but I'm still trying.
>>>
>>> Anyway, I'm using regular Kaha persistence (not AMQMessageStore),
>>> and during
>>> a Durable Topic test, I'm eventually getting the following exception
>>> in the
>>> AMQBroker, after which the broker stops forwarding events to the
>>> consumer.
>>>
>>> Anyone seen this?
>>>
>>>
>>> <
>>> org
>>> .apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch>
>>> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:43595,4,main]> Failed
>>> to fill
>>> batch
>>> Stack Trace follows:
>>> java.lang.RuntimeException: Failed to get next index from
>>> IndexManager:(index-topic-subs) for offset=7681977, key=(2,
>>> 20080006, 47),
>>> value=(2, 20080058, 165), previousItem=7681212, nextItem=7682895
>>> at
>>> org
>>> .apache
>>> .activemq
>>> .kaha
>>> .impl
>>> .index.DiskIndexLinkedList.getNextEntry(DiskIndexLinkedList.java:267)
>>> at
>>> org
>>> .apache
>>> .activemq
>>> .kaha.impl.container.MapContainerImpl.getNext(MapContainerImpl.java:
>>> 449)
>>> at
>>> org
>>> .apache
>>> .activemq
>>> .store
>>> .kahadaptor.TopicSubContainer.getNextEntry(TopicSubContainer.java:95)
>>> at
>>> org
>>> .apache
>>> .activemq
>>> .store
>>> .kahadaptor
>>> .KahaTopicMessageStore
>>> .recoverNextMessages(KahaTopicMessageStore.java:165)
>>> at
>>> org
>>> .apache
>>> .activemq
>>> .store
>>> .ProxyTopicMessageStore
>>> .recoverNextMessages(ProxyTopicMessageStore.java:97)
>>> at
>>> org
>>> .apache
>>> .activemq
>>> .broker
>>> .region
>>> .cursors.TopicStorePrefetch.doFillBatch(TopicStorePrefetch.java:107)
>>> at
>>> org
>>> .apache
>>> .activemq
>>> .broker
>>> .region
>>> .cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:188)
>>> at
>>> org
>>> .apache
>>> .activemq
>>> .broker
>>> .region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:
>>> 104)
>>> at
>>> org
>>> .apache
>>> .activemq
>>> .broker
>>> .region
>>> .cursors
>>> .StoreDurableSubscriberCursor
>>> .hasNext(StoreDurableSubscriberCursor.java:210
>>> )
>>> at
>>> org
>>> .apache
>>> .activemq
>>> .broker
>>> .region
>>> .PrefetchSubscription.dispatchPending(PrefetchSubscription.java:479)
>>> at
>>> org
>>> .apache
>>> .activemq
>>> .broker
>>> .region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:
>>> 357)
>>> at
>>> org
>>> .apache
>>> .activemq
>>> .broker.region.AbstractRegion.acknowledge(AbstractRegion.java:349)
>>> at
>>> org
>>> .apache
>>> .activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:
>>> 474)
>>> at
>>> org
>>> .apache
>>> .activemq
>>> .broker.TransactionBroker.acknowledge(TransactionBroker.java:194)
>>> at
>>> org
>>> .apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:73)
>>> at
>>> org
>>> .apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:73)
>>> at
>>> org
>>> .apache
>>> .activemq
>>> .broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:84)
>>> at
>>> org
>>> .apache
>>> .activemq
>>> .broker
>>> .TransportConnection.processMessageAck(TransportConnection.java:444)
>>> at
>>> org.apache.activemq.command.MessageAck.visit(MessageAck.java:196)
>>> at
>>> org
>>> .apache
>>> .activemq
>>> .broker.TransportConnection.service(TransportConnection.java:293)
>>> at
>>> org.apache.activemq.broker.TransportConnection
>>> $1.onCommand(TransportConnection.java:181)
>>> at
>>> org
>>> .apache
>>> .activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
>>> at
>>> org
>>> .apache
>>> .activemq
>>> .transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:
>>> 143)
>>> at
>>> org
>>> .apache
>>> .activemq
>>> .transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
>>> at
>>> org
>>> .apache
>>> .activemq.transport.TransportSupport.doConsume(TransportSupport.java:
>>> 84)
>>> at
>>> org
>>> .apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:
>>> 196)
>>> at
>>> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:
>>> 183)
>>> at java.lang.Thread.run(Thread.java:619)
>>> Caused by: java.io.EOFException
>>> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:
>>> 383)
>>> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:
>>> 361)
>>> at
>>> org
>>> .apache
>>> .activemq
>>> .kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:46)
>>> at
>>> org
>>> .apache
>>> .activemq.kaha.impl.index.IndexManager.getIndex(IndexManager.java:66)
>>> at
>>> org
>>> .apache
>>> .activemq
>>> .kaha
>>> .impl
>>> .index.DiskIndexLinkedList.getNextEntry(DiskIndexLinkedList.java:265)
>>> ... 27 more
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16677512.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>
>> There was a bug fixed in this area last Friday - would be interested
>> if this is still a problem for you with a later version?
>> If it is - Ill digg a little deeper.
>>
>> btw - why are you using Kaha instead of AMQStore ?
>>
>>
>>
>>
>> cheers,
>>
>> Rob
>>
>> http://open.iona.com/ -Enterprise Open Integration
>> http://rajdavies.blogspot.com/
>>
>>
>>
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16700798.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Kaha persistence issue w/5.1 (upto 4/8 SNAPSHOT)
Posted by yaussy <ya...@cboe.com>.
Rob,
This is good news - I will try using last nights snapshot.
I started out using AMQMessageStore, but I've never had good performance
luck with AMQs Journal - it always seems to have this frequent, high CPU
cost as it checkpoints. Our Durable rate requirements are fairly
significant, such as a few hundred events per second to a handful of
consumers, on one of our clusters. It has not performed well, whereas Kaha
by itself is excellent.
Kevin
rajdavies wrote:
>
>
> On 14 Apr 2008, at 14:07, yaussy wrote:
>
>>
>> I have not been able to reproduce this problem outside of our
>> application,
>> but I'm still trying.
>>
>> Anyway, I'm using regular Kaha persistence (not AMQMessageStore),
>> and during
>> a Durable Topic test, I'm eventually getting the following exception
>> in the
>> AMQBroker, after which the broker stops forwarding events to the
>> consumer.
>>
>> Anyone seen this?
>>
>>
>> <
>> org
>> .apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch>
>> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:43595,4,main]> Failed
>> to fill
>> batch
>> Stack Trace follows:
>> java.lang.RuntimeException: Failed to get next index from
>> IndexManager:(index-topic-subs) for offset=7681977, key=(2,
>> 20080006, 47),
>> value=(2, 20080058, 165), previousItem=7681212, nextItem=7682895
>> at
>> org
>> .apache
>> .activemq
>> .kaha
>> .impl
>> .index.DiskIndexLinkedList.getNextEntry(DiskIndexLinkedList.java:267)
>> at
>> org
>> .apache
>> .activemq
>> .kaha.impl.container.MapContainerImpl.getNext(MapContainerImpl.java:
>> 449)
>> at
>> org
>> .apache
>> .activemq
>> .store
>> .kahadaptor.TopicSubContainer.getNextEntry(TopicSubContainer.java:95)
>> at
>> org
>> .apache
>> .activemq
>> .store
>> .kahadaptor
>> .KahaTopicMessageStore
>> .recoverNextMessages(KahaTopicMessageStore.java:165)
>> at
>> org
>> .apache
>> .activemq
>> .store
>> .ProxyTopicMessageStore
>> .recoverNextMessages(ProxyTopicMessageStore.java:97)
>> at
>> org
>> .apache
>> .activemq
>> .broker
>> .region
>> .cursors.TopicStorePrefetch.doFillBatch(TopicStorePrefetch.java:107)
>> at
>> org
>> .apache
>> .activemq
>> .broker
>> .region
>> .cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:188)
>> at
>> org
>> .apache
>> .activemq
>> .broker
>> .region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:
>> 104)
>> at
>> org
>> .apache
>> .activemq
>> .broker
>> .region
>> .cursors
>> .StoreDurableSubscriberCursor
>> .hasNext(StoreDurableSubscriberCursor.java:210
>> )
>> at
>> org
>> .apache
>> .activemq
>> .broker
>> .region
>> .PrefetchSubscription.dispatchPending(PrefetchSubscription.java:479)
>> at
>> org
>> .apache
>> .activemq
>> .broker
>> .region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:
>> 357)
>> at
>> org
>> .apache
>> .activemq
>> .broker.region.AbstractRegion.acknowledge(AbstractRegion.java:349)
>> at
>> org
>> .apache
>> .activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:
>> 474)
>> at
>> org
>> .apache
>> .activemq
>> .broker.TransactionBroker.acknowledge(TransactionBroker.java:194)
>> at
>> org
>> .apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:73)
>> at
>> org
>> .apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:73)
>> at
>> org
>> .apache
>> .activemq
>> .broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:84)
>> at
>> org
>> .apache
>> .activemq
>> .broker
>> .TransportConnection.processMessageAck(TransportConnection.java:444)
>> at
>> org.apache.activemq.command.MessageAck.visit(MessageAck.java:196)
>> at
>> org
>> .apache
>> .activemq
>> .broker.TransportConnection.service(TransportConnection.java:293)
>> at
>> org.apache.activemq.broker.TransportConnection
>> $1.onCommand(TransportConnection.java:181)
>> at
>> org
>> .apache
>> .activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
>> at
>> org
>> .apache
>> .activemq
>> .transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:
>> 143)
>> at
>> org
>> .apache
>> .activemq
>> .transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
>> at
>> org
>> .apache
>> .activemq.transport.TransportSupport.doConsume(TransportSupport.java:
>> 84)
>> at
>> org
>> .apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:
>> 196)
>> at
>> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:
>> 183)
>> at java.lang.Thread.run(Thread.java:619)
>> Caused by: java.io.EOFException
>> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:
>> 383)
>> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:
>> 361)
>> at
>> org
>> .apache
>> .activemq
>> .kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:46)
>> at
>> org
>> .apache
>> .activemq.kaha.impl.index.IndexManager.getIndex(IndexManager.java:66)
>> at
>> org
>> .apache
>> .activemq
>> .kaha
>> .impl
>> .index.DiskIndexLinkedList.getNextEntry(DiskIndexLinkedList.java:265)
>> ... 27 more
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16677512.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
> There was a bug fixed in this area last Friday - would be interested
> if this is still a problem for you with a later version?
> If it is - Ill digg a little deeper.
>
> btw - why are you using Kaha instead of AMQStore ?
>
>
>
>
> cheers,
>
> Rob
>
> http://open.iona.com/ -Enterprise Open Integration
> http://rajdavies.blogspot.com/
>
>
>
>
>
--
View this message in context: http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16700153.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Kaha persistence issue w/5.1 (upto 4/8 SNAPSHOT)
Posted by Rob Davies <ra...@gmail.com>.
On 14 Apr 2008, at 14:07, yaussy wrote:
>
> I have not been able to reproduce this problem outside of our
> application,
> but I'm still trying.
>
> Anyway, I'm using regular Kaha persistence (not AMQMessageStore),
> and during
> a Durable Topic test, I'm eventually getting the following exception
> in the
> AMQBroker, after which the broker stops forwarding events to the
> consumer.
>
> Anyone seen this?
>
>
> <
> org
> .apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch>
> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:43595,4,main]> Failed
> to fill
> batch
> Stack Trace follows:
> java.lang.RuntimeException: Failed to get next index from
> IndexManager:(index-topic-subs) for offset=7681977, key=(2,
> 20080006, 47),
> value=(2, 20080058, 165), previousItem=7681212, nextItem=7682895
> at
> org
> .apache
> .activemq
> .kaha
> .impl
> .index.DiskIndexLinkedList.getNextEntry(DiskIndexLinkedList.java:267)
> at
> org
> .apache
> .activemq
> .kaha.impl.container.MapContainerImpl.getNext(MapContainerImpl.java:
> 449)
> at
> org
> .apache
> .activemq
> .store
> .kahadaptor.TopicSubContainer.getNextEntry(TopicSubContainer.java:95)
> at
> org
> .apache
> .activemq
> .store
> .kahadaptor
> .KahaTopicMessageStore
> .recoverNextMessages(KahaTopicMessageStore.java:165)
> at
> org
> .apache
> .activemq
> .store
> .ProxyTopicMessageStore
> .recoverNextMessages(ProxyTopicMessageStore.java:97)
> at
> org
> .apache
> .activemq
> .broker
> .region
> .cursors.TopicStorePrefetch.doFillBatch(TopicStorePrefetch.java:107)
> at
> org
> .apache
> .activemq
> .broker
> .region
> .cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:188)
> at
> org
> .apache
> .activemq
> .broker
> .region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:
> 104)
> at
> org
> .apache
> .activemq
> .broker
> .region
> .cursors
> .StoreDurableSubscriberCursor
> .hasNext(StoreDurableSubscriberCursor.java:210
> )
> at
> org
> .apache
> .activemq
> .broker
> .region
> .PrefetchSubscription.dispatchPending(PrefetchSubscription.java:479)
> at
> org
> .apache
> .activemq
> .broker
> .region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:
> 357)
> at
> org
> .apache
> .activemq
> .broker.region.AbstractRegion.acknowledge(AbstractRegion.java:349)
> at
> org
> .apache
> .activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:
> 474)
> at
> org
> .apache
> .activemq
> .broker.TransactionBroker.acknowledge(TransactionBroker.java:194)
> at
> org
> .apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:73)
> at
> org
> .apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:73)
> at
> org
> .apache
> .activemq
> .broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:84)
> at
> org
> .apache
> .activemq
> .broker
> .TransportConnection.processMessageAck(TransportConnection.java:444)
> at
> org.apache.activemq.command.MessageAck.visit(MessageAck.java:196)
> at
> org
> .apache
> .activemq
> .broker.TransportConnection.service(TransportConnection.java:293)
> at
> org.apache.activemq.broker.TransportConnection
> $1.onCommand(TransportConnection.java:181)
> at
> org
> .apache
> .activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> at
> org
> .apache
> .activemq
> .transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:
> 143)
> at
> org
> .apache
> .activemq
> .transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
> at
> org
> .apache
> .activemq.transport.TransportSupport.doConsume(TransportSupport.java:
> 84)
> at
> org
> .apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:
> 196)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:
> 183)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.EOFException
> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:
> 383)
> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:
> 361)
> at
> org
> .apache
> .activemq
> .kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:46)
> at
> org
> .apache
> .activemq.kaha.impl.index.IndexManager.getIndex(IndexManager.java:66)
> at
> org
> .apache
> .activemq
> .kaha
> .impl
> .index.DiskIndexLinkedList.getNextEntry(DiskIndexLinkedList.java:265)
> ... 27 more
>
> --
> View this message in context: http://www.nabble.com/Kaha-persistence-issue-w-5.1-%28upto-4-8-SNAPSHOT%29-tp16677512s2354p16677512.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
There was a bug fixed in this area last Friday - would be interested
if this is still a problem for you with a later version?
If it is - Ill digg a little deeper.
btw - why are you using Kaha instead of AMQStore ?
cheers,
Rob
http://open.iona.com/ -Enterprise Open Integration
http://rajdavies.blogspot.com/