You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Daniel Kluesing <dk...@bluekai.com> on 2010/02/01 23:30:41 UTC
RE: OOM with high KahaDB index time
Ran this through some paces, and it works much better. Thanks for the help!
-----Original Message-----
From: Gary Tully [mailto:gary.tully@gmail.com]
Sent: Wednesday, January 27, 2010 2:35 AM
To: users@activemq.apache.org
Subject: Re: OOM with high KahaDB index time
Hi Daniel,
builds from today are there now:
https://repository.apache.org/content/repositories/snapshots/org/apache/activemq/apache-activemq/5.4-SNAPSHOT/
thanks for the validation.
2010/1/26 Daniel Kluesing <dk...@bluekai.com>
> Any eta on the 5.4-snapshot? (
> https://repository.apache.org/content/repositories/snapshots/org/apache/activemq/apache-activemq/5.4-SNAPSHOT/still lists the build date as 1/20) Once it's out, I'll give it a run
> through.
>
> -----Original Message-----
> From: Rob Davies [mailto:rajdavies@gmail.com]
> Sent: Thursday, January 21, 2010 9:42 AM
> To: users@activemq.apache.org
> Subject: Re: OOM with high KahaDB index time
>
> Thanks Daniel! Though please wait for the next valid 5.4 build - looks
> like its missing a dependency at the moment
> On 21 Jan 2010, at 16:48, Daniel Kluesing wrote:
>
> > I'm happy to give 5.4-snapshot a try
> >
> > -----Original Message-----
> > From: Rob Davies [mailto:rajdavies@gmail.com]
> > Sent: Wednesday, January 20, 2010 2:08 PM
> > To: users@activemq.apache.org
> > Subject: Re: OOM with high KahaDB index time
> >
> > I have a potential fix for this problem - hoping someone will try out
> > the next 5.4-snapshot :)
> > On 20 Jan 2010, at 21:41, rideallday wrote:
> >
> >>
> >> I went back to AMQStore. Working fine for me.
> >> --
> >> View this message in context:
> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27249114.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >
> > Rob Davies
> > http://twitter.com/rajdavies
> > I work here: http://fusesource.com
> > My Blog: http://rajdavies.blogspot.com/
> > I'm writing this: http://www.manning.com/snyder/
> >
> >
> >
> >
> >
>
> Rob Davies
> http://twitter.com/rajdavies
> I work here: http://fusesource.com
> My Blog: http://rajdavies.blogspot.com/
> I'm writing this: http://www.manning.com/snyder/
>
>
>
>
>
>
--
http://blog.garytully.com
Open Source Integration
http://fusesource.com
Re: OOM with high KahaDB index time
Posted by rideallday <ni...@uvic.ca>.
That was it. Thanks!
I had to set to indexCacheSize = 100000 to get the performance I'm looking
for.
rajdavies wrote:
>
> ok - it looks like you might need to tune KahaDB - set the
> indexCacheSize = 1000 - see http://activemq.apache.org/kahadb.html
>
>
>
--
View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27657831.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: OOM with high KahaDB index time
Posted by Rob Davies <ra...@gmail.com>.
ok - it looks like you might need to tune KahaDB - set the
indexCacheSize = 1000 - see http://activemq.apache.org/kahadb.html
cheers,
Rob
On 15 Feb 2010, at 18:46, rideallday wrote:
>
> Yes, topics are durable.
>
> Test case:
> 1. publish messages to queue, let several hundred thousand accumulate.
> 2. publish topics to queue, let several hundred thousand accumulate
> 3. begin consuming from queue.
> 4. begin consuming from topic.
>
> You should see a huge degrade in performance and a bunch of indexing
> messages once you start to consume from topic.
>
>
>
>
> rajdavies wrote:
>>
>> could you provide a test case ?
>>
>> cheers,
>>
>> Rob
>> On 11 Feb 2010, at 21:41, rideallday wrote:
>>
>>>
>>> Umm, actually I'm still seeing issues with high Index times for
>>> topics.
>>>
>>> My test case is:
>>>
>>> Send a huge amount of messages into the broker. Let several hundred
>>> thousand
>>> accumulate. Then start consumers and see how long it takes to drain
>>> the
>>> queues and topics.
>>>
>>> The result is:
>>>
>>> For queues- excellent. Performance is amazing. A few 1000ms Journal
>>> append
>>> messages, but recovery performance exceeds my requirements.
>>> For topics- Index updates in the 2-700ms range. This happens every
>>> second or
>>> so and locks the consumer, so recovery is extremely slow.
>>>
>>> This is very similar behaviour to what I saw with Topics and Queues
>>> before
>>> Rob applied the fix for cursors in KahaDB.
>>>
>>> Is it possible that the fix needs to be also applied to topics?
>>>
>>> I'm using 5.4 snapshot from Feb 1.
>>> --
>>> View this message in context:
>>> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27555173.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>
>> Rob Davies
>> http://twitter.com/rajdavies
>> I work here: http://fusesource.com
>> My Blog: http://rajdavies.blogspot.com/
>> I'm writing this: http://www.manning.com/snyder/
>>
>>
>>
>>
>>
>>
>>
>
> --
> View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27598149.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/
Re: OOM with high KahaDB index time
Posted by rideallday <ni...@uvic.ca>.
Yes, topics are durable.
Test case:
1. publish messages to queue, let several hundred thousand accumulate.
2. publish topics to queue, let several hundred thousand accumulate
3. begin consuming from queue.
4. begin consuming from topic.
You should see a huge degrade in performance and a bunch of indexing
messages once you start to consume from topic.
rajdavies wrote:
>
> could you provide a test case ?
>
> cheers,
>
> Rob
> On 11 Feb 2010, at 21:41, rideallday wrote:
>
>>
>> Umm, actually I'm still seeing issues with high Index times for
>> topics.
>>
>> My test case is:
>>
>> Send a huge amount of messages into the broker. Let several hundred
>> thousand
>> accumulate. Then start consumers and see how long it takes to drain
>> the
>> queues and topics.
>>
>> The result is:
>>
>> For queues- excellent. Performance is amazing. A few 1000ms Journal
>> append
>> messages, but recovery performance exceeds my requirements.
>> For topics- Index updates in the 2-700ms range. This happens every
>> second or
>> so and locks the consumer, so recovery is extremely slow.
>>
>> This is very similar behaviour to what I saw with Topics and Queues
>> before
>> Rob applied the fix for cursors in KahaDB.
>>
>> Is it possible that the fix needs to be also applied to topics?
>>
>> I'm using 5.4 snapshot from Feb 1.
>> --
>> View this message in context:
>> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27555173.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
> Rob Davies
> http://twitter.com/rajdavies
> I work here: http://fusesource.com
> My Blog: http://rajdavies.blogspot.com/
> I'm writing this: http://www.manning.com/snyder/
>
>
>
>
>
>
>
--
View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27598149.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: OOM with high KahaDB index time
Posted by Rob Davies <ra...@gmail.com>.
could you provide a test case ?
cheers,
Rob
On 11 Feb 2010, at 21:41, rideallday wrote:
>
> Umm, actually I'm still seeing issues with high Index times for
> topics.
>
> My test case is:
>
> Send a huge amount of messages into the broker. Let several hundred
> thousand
> accumulate. Then start consumers and see how long it takes to drain
> the
> queues and topics.
>
> The result is:
>
> For queues- excellent. Performance is amazing. A few 1000ms Journal
> append
> messages, but recovery performance exceeds my requirements.
> For topics- Index updates in the 2-700ms range. This happens every
> second or
> so and locks the consumer, so recovery is extremely slow.
>
> This is very similar behaviour to what I saw with Topics and Queues
> before
> Rob applied the fix for cursors in KahaDB.
>
> Is it possible that the fix needs to be also applied to topics?
>
> I'm using 5.4 snapshot from Feb 1.
> --
> View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27555173.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/
Re: OOM with high KahaDB index time
Posted by Rob Davies <ra...@gmail.com>.
Are you using Durable Topic Consumers ?
On 11 Feb 2010, at 21:41, rideallday wrote:
>
> Umm, actually I'm still seeing issues with high Index times for
> topics.
>
> My test case is:
>
> Send a huge amount of messages into the broker. Let several hundred
> thousand
> accumulate. Then start consumers and see how long it takes to drain
> the
> queues and topics.
>
> The result is:
>
> For queues- excellent. Performance is amazing. A few 1000ms Journal
> append
> messages, but recovery performance exceeds my requirements.
> For topics- Index updates in the 2-700ms range. This happens every
> second or
> so and locks the consumer, so recovery is extremely slow.
>
> This is very similar behaviour to what I saw with Topics and Queues
> before
> Rob applied the fix for cursors in KahaDB.
>
> Is it possible that the fix needs to be also applied to topics?
>
> I'm using 5.4 snapshot from Feb 1.
> --
> View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27555173.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/
Re: OOM with high KahaDB index time
Posted by rideallday <ni...@uvic.ca>.
Umm, actually I'm still seeing issues with high Index times for topics.
My test case is:
Send a huge amount of messages into the broker. Let several hundred thousand
accumulate. Then start consumers and see how long it takes to drain the
queues and topics.
The result is:
For queues- excellent. Performance is amazing. A few 1000ms Journal append
messages, but recovery performance exceeds my requirements.
For topics- Index updates in the 2-700ms range. This happens every second or
so and locks the consumer, so recovery is extremely slow.
This is very similar behaviour to what I saw with Topics and Queues before
Rob applied the fix for cursors in KahaDB.
Is it possible that the fix needs to be also applied to topics?
I'm using 5.4 snapshot from Feb 1.
--
View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27555173.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: OOM with high KahaDB index time
Posted by rideallday <ni...@uvic.ca>.
I can confirm that has resolved the issue I was seeing. Thanks Rob!
What's ETA for 5.4 or applying this fix in 5.3? I'll need an official stable
release to put into production.
--
View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27413891.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: OOM with high KahaDB index time
Posted by Rob Davies <ra...@gmail.com>.
Good to know! - this was a silly error - When a consumer got so far
behind a producer so that the cache was invalidated, after every
acknowledgement of a message, a cursor would check to see if the index
was empty. To do that it would ask for a count of still outstanding
persistent messages. For a BTreeIndex - a count resulted in every
BTreeNode been paged into memory in the same transaction to be
counted. Its remarkable KahaDB was as fast as it was under the
circumstances :)
On 1 Feb 2010, at 22:30, Daniel Kluesing wrote:
> Ran this through some paces, and it works much better. Thanks for
> the help!
>
> -----Original Message-----
> From: Gary Tully [mailto:gary.tully@gmail.com]
> Sent: Wednesday, January 27, 2010 2:35 AM
> To: users@activemq.apache.org
> Subject: Re: OOM with high KahaDB index time
>
> Hi Daniel,
> builds from today are there now:
> https://repository.apache.org/content/repositories/snapshots/org/apache/activemq/apache-activemq/5.4-SNAPSHOT/
>
> thanks for the validation.
>
> 2010/1/26 Daniel Kluesing <dk...@bluekai.com>
>
>> Any eta on the 5.4-snapshot? (
>> https://repository.apache.org/content/repositories/snapshots/org/apache/activemq/apache-activemq/5.4-SNAPSHOT/still
>> lists the build date as 1/20) Once it's out, I'll give it a run
>> through.
>>
>> -----Original Message-----
>> From: Rob Davies [mailto:rajdavies@gmail.com]
>> Sent: Thursday, January 21, 2010 9:42 AM
>> To: users@activemq.apache.org
>> Subject: Re: OOM with high KahaDB index time
>>
>> Thanks Daniel! Though please wait for the next valid 5.4 build -
>> looks
>> like its missing a dependency at the moment
>> On 21 Jan 2010, at 16:48, Daniel Kluesing wrote:
>>
>>> I'm happy to give 5.4-snapshot a try
>>>
>>> -----Original Message-----
>>> From: Rob Davies [mailto:rajdavies@gmail.com]
>>> Sent: Wednesday, January 20, 2010 2:08 PM
>>> To: users@activemq.apache.org
>>> Subject: Re: OOM with high KahaDB index time
>>>
>>> I have a potential fix for this problem - hoping someone will try
>>> out
>>> the next 5.4-snapshot :)
>>> On 20 Jan 2010, at 21:41, rideallday wrote:
>>>
>>>>
>>>> I went back to AMQStore. Working fine for me.
>>>> --
>>>> View this message in context:
>> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27249114.html
>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>
>>>
>>> Rob Davies
>>> http://twitter.com/rajdavies
>>> I work here: http://fusesource.com
>>> My Blog: http://rajdavies.blogspot.com/
>>> I'm writing this: http://www.manning.com/snyder/
>>>
>>>
>>>
>>>
>>>
>>
>> Rob Davies
>> http://twitter.com/rajdavies
>> I work here: http://fusesource.com
>> My Blog: http://rajdavies.blogspot.com/
>> I'm writing this: http://www.manning.com/snyder/
>>
>>
>>
>>
>>
>>
>
>
> --
> http://blog.garytully.com
>
> Open Source Integration
> http://fusesource.com
Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/