You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Daniel Kluesing <dk...@bluekai.com> on 2010/01/18 23:14:35 UTC
OOM with high KahaDB index time
Hi,
I'm running the 5.3 release as a standalone broker. In one case, a producer is running without a consumer, producing small, persistent messages, with the FileCursor pendingQueuePolicy (per https://issues.apache.org/activemq/browse/AMQ-2512) option and flow control memoryLimit set to 100mb for the queue in question. (Through a policy entry)
As the queue grows above 300k messages, KahaDB indexing starts climbing above 1 second. At around 350k messages, the indexing is taking over 8 seconds. At this point, I start getting java out of heap space errors in essentially random parts of the code. After a while, the producers timeout with a channel inactive for too long error, and the entire broker basically wedges itself. At this point, consumers are generally unable to bind to the broker quitting with timeout errors. When they can connect, consuming a single message triggers an index re-build, which takes 2-8seconds. Turning on verbose garbage collection, the jvm is collecting like mad but reclaiming no space.
If I restart the broker, it comes back up, I can consume the old messages, and can handle another 350k messages until it wedges.
I can reproduce under both default gc and incremental gc.
Two questions:
- It seems like someone is holding onto a handle to the messages after they have been persisted to disk - is this a known issue? Should I open a JIRA for it? (Or is there another explanation?)
- Is there any documentation about the internals of KahaDB - the kind of indices etc? I'd like to get a better understanding of the index performance and in general how KahaDB compares to something like BerkeleyDB.
Thanks
Re: OOM with high KahaDB index time
Posted by Rob Davies <ra...@gmail.com>.
Florida! - nice
On 22 Jan 2010, at 21:57, Joe Fernandez wrote:
>
> Its Miller-time here in Florida, so I'll give it a go on Monday d:)
>
> Joe
>
>
> rajdavies wrote:
>>
>> Hi Joe,
>>
>> any chance you can build from trunk and try it ?
>>
>> cheers,
>>
>> Rob
>>
>> On 22 Jan 2010, at 20:07, Joe Fernandez wrote:
>>
>>>
>>> I ran my 5.3 test with the following
>>>
>>> <systemUsage>
>>> <systemUsage sendFailIfNoSpace="true">
>>> <memoryUsage>
>>> <memoryUsage limit="100mb"/>
>>> </memoryUsage>
>>> <storeUsage>
>>> <storeUsage limit="500 mb" name="foo"/>
>>> </storeUsage>
>>> <tempUsage>
>>> <tempUsage limit="100mb"/>
>>> </tempUsage>
>>> </systemUsage>
>>> </systemUsage>
>>>
>>> ...and had my producer fill up the store; it tried to pump 400k
>>> messages
>>> each at 2k in size. Producer flow control was also enabled. The
>>> broker did
>>> not hurl any OOM exceptions, "just
>>> javax.jms.ResourceAllocationException:
>>> Usage Manager Store is Full. Stopping producer..." exceptions as
>>> expected.
>>> The producer got kicked off its connection.
>>>
>>> The only time I got OOMs was when I used this.
>>>
>>> <systemUsage>
>>> <systemUsage sendFailIfNoSpace="true">
>>> </systemUsage>
>>> </systemUsage>
>>>
>>> Joe
>>>
>>>
>>>
>>> Daniel Kluesing-2 wrote:
>>>>
>>>> So I'm not sure if that's what Rob is talking about being fixed in
>>>> 5.4
>>>> (And I'll try the snapshot as soon as it's ready) but if I don't
>>>> have the
>>>> sendFailIfNoSpace then my understanding is the producers send calls
>>>> block/wait/timeout - as opposed to fail - so it's more difficult to
>>>> get
>>>> into a HA configuration. It's a minor point, not having an OOM is
>>>> much
>>>> more important, but I definitely want the send calls to fail for
>>>> the
>>>> producer if the broker ever does anything funny.
>>>>
>>>> Thanks for the feedback on the config, very helpful.
>>>>
>>>> -----Original Message-----
>>>> From: Joe Fernandez [mailto:joe.fernandez@ttmsolutions.com]
>>>> Sent: Thursday, January 21, 2010 1:01 PM
>>>> To: users@activemq.apache.org
>>>> Subject: RE: OOM with high KahaDB index time
>>>>
>>>>
>>>> Just for grins, I threw your cfg file into our 5.3 testbed and sure
>>>> enough,
>>>> we got the OOMs; I pumped 200k messages with each being 2k in size.
>>>> FWIW,
>>>> taking this out of the cfg file made things run a lot better.
>>>>
>>>> <systemUsage>
>>>> <systemUsage sendFailIfNoSpace="true">
>>>> </systemUsage>
>>>> </systemUsage>
>>>>
>>>> With the above taken out of the cfg file, I was able to pump 400k
>>>> messages
>>>> into the broker, no OOMs and memory utilization looked much better.
>>>> I also
>>>> gave a fully-defined systemUsage a try and that also appeared to do
>>>> the
>>>> trick.
>>>>
>>>> <systemUsage>
>>>> <systemUsage>
>>>> <memoryUsage>
>>>> <memoryUsage limit="100 mb"/>
>>>> </memoryUsage>
>>>> <storeUsage>
>>>> <storeUsage limit="1 gb" name="foo"/>
>>>> </storeUsage>
>>>> <tempUsage>
>>>> <tempUsage limit="100 mb"/>
>>>> </tempUsage>
>>>> </systemUsage>
>>>> </systemUsage>
>>>>
>>>> So may be worth giving it a whirl if you can't scoot over to the
>>>> trunk and
>>>> ride Rob's patch.
>>>>
>>>> Joe
>>>>
>>>>
>>>> Daniel Kluesing-2 wrote:
>>>>>
>>>>> I tried the suggestion of going with the default cursor, but I
>>>>> still get
>>>>> OOM errors. I've included my full config file below, I think I'm
>>>>> running
>>>>> fairly vanilla/default.
>>>>>
>>>>> After about 350k persistent messages, the logs start to look like:
>>>>>
>>>>> INFO | Slow KahaDB access: Journal append took: 10 ms, Index
>>>>> update took
>>>>> 3118 ms
>>>>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
>>>>> took
>>>>> 5118 ms
>>>>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
>>>>> took
>>>>> 2736 ms
>>>>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
>>>>> took
>>>>> 2945 ms
>>>>> INFO | Slow KahaDB access: Journal append took: 33 ms, Index
>>>>> update took
>>>>> 2654 ms
>>>>> INFO | Slow KahaDB access: Journal append took: 82 ms, Index
>>>>> update took
>>>>> 3174 ms
>>>>> INFO | Slow KahaDB access: Journal append took: 1 ms, Index update
>>>>> took
>>>>> 5891 ms
>>>>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
>>>>> took
>>>>> 2906 ms
>>>>> INFO | Slow KahaDB access: Journal append took: 60 ms, Index
>>>>> update took
>>>>> 7619 ms
>>>>> Exception in thread "InactivityMonitor WriteCheck"
>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>> at java.util.jar.Attributes.read(Attributes.java:377)
>>>>> at java.util.jar.Manifest.read(Manifest.java:182)
>>>>> at java.util.jar.Manifest.<init>(Manifest.java:52)
>>>>> at
>>>>> java.util.jar.JarFile.getManifestFromReference(JarFile.java:165)
>>>>> at java.util.jar.JarFile.getManifest(JarFile.java:146)
>>>>> at
>>>>> sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:
>>>>> 693)
>>>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:
>>>>> 221)
>>>>> at java.net.URLClassLoader.access$000(URLClassLoader.java:
>>>>> 56)
>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
>>>>> at java.security.AccessController.doPrivileged(Native
>>>>> Method)
>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:
>>>>> 188)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
>>>>> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:
>>>>> 319)
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .transport.InactivityMonitor.writeCheck(InactivityMonitor.java:
>>>>> 132)
>>>>> at
>>>>> org.apache.activemq.transport.InactivityMonitor
>>>>> $2.run(InactivityMonitor.java:106)
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:
>>>>> 33)
>>>>> at java.util.TimerThread.mainLoop(Timer.java:512)
>>>>> at java.util.TimerThread.run(Timer.java:462)
>>>>>
>>>>> Config file:
>>>>>
>>>>> <beans
>>>>> xmlns="http://www.springframework.org/schema/beans"
>>>>> xmlns:amq="http://activemq.apache.org/schema/core"
>>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>>>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>>>>> http://activemq.apache.org/schema/core
>>>>> http://activemq.apache.org/schema/core/activemq-core.xsd">
>>>>>
>>>>> <bean
>>>>> class
>>>>> =
>>>>> "org
>>>>> .springframework
>>>>> .beans.factory.config.PropertyPlaceholderConfigurer">
>>>>> <property name="locations">
>>>>>
>>>>> <value>file:${activemq.base}/conf/credentials.properties</value>
>>>>> </property>
>>>>> </bean>
>>>>>
>>>>> <broker xmlns="http://activemq.apache.org/schema/core"
>>>>> brokerName="sub01chi" dataDirectory="${activemq.base}/data">
>>>>>
>>>>> <managementContext>
>>>>> <managementContext createConnector="true"/>
>>>>> </managementContext>
>>>>>
>>>>> <persistenceAdapter>
>>>>> <kahaDB directory="${activemq.base}/data/kahadb"/>
>>>>> </persistenceAdapter>
>>>>>
>>>>> <destinationPolicy>
>>>>> <policyMap>
>>>>> <policyEntries>
>>>>> <policyEntry queue="P>" producerFlowControl="true"
>>>>> memoryLimit="10mb"></policyEntry>
>>>>> </policyEntries>
>>>>> </policyMap>
>>>>> </destinationPolicy>
>>>>> <systemUsage>
>>>>> <systemUsage sendFailIfNoSpace="true">
>>>>> </systemUsage>
>>>>> </systemUsage>
>>>>> <transportConnectors>
>>>>> <transportConnector name="openwire"
>>>>> uri="tcp://0.0.0.0:61616"/>
>>>>> </transportConnectors>
>>>>> </broker>
>>>>> <import resource="jetty.xml"/>
>>>>> </beans>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Rob Davies [mailto:rajdavies@gmail.com]
>>>>> Sent: Monday, January 18, 2010 10:42 PM
>>>>> To: users@activemq.apache.org
>>>>> Subject: Re: OOM with high KahaDB index time
>>>>>
>>>>>
>>>>> On 18 Jan 2010, at 22:14, Daniel Kluesing wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I'm running the 5.3 release as a standalone broker. In one
>>>>>> case, a
>>>>>> producer is running without a consumer, producing small,
>>>>>> persistent
>>>>>> messages, with the FileCursor pendingQueuePolicy (per
>>>>>> https://issues.apache.org/activemq/browse/AMQ-2512)
>>>>>> option and flow control memoryLimit set to 100mb for the queue in
>>>>>> question. (Through a policy entry)
>>>>>>
>>>>>> As the queue grows above 300k messages, KahaDB indexing starts
>>>>>> climbing above 1 second. At around 350k messages, the indexing is
>>>>>> taking over 8 seconds. At this point, I start getting java out of
>>>>>> heap space errors in essentially random parts of the code.
>>>>>> After a
>>>>>> while, the producers timeout with a channel inactive for too long
>>>>>> error, and the entire broker basically wedges itself. At this
>>>>>> point,
>>>>>> consumers are generally unable to bind to the broker quitting
>>>>>> with
>>>>>> timeout errors. When they can connect, consuming a single message
>>>>>> triggers an index re-build, which takes 2-8seconds. Turning on
>>>>>> verbose garbage collection, the jvm is collecting like mad but
>>>>>> reclaiming no space.
>>>>>>
>>>>>> If I restart the broker, it comes back up, I can consume the old
>>>>>> messages, and can handle another 350k messages until it wedges.
>>>>>>
>>>>>> I can reproduce under both default gc and incremental gc.
>>>>>>
>>>>>> Two questions:
>>>>>> - It seems like someone is holding onto a handle to the messages
>>>>>> after they have been persisted to disk - is this a known issue?
>>>>>> Should I open a JIRA for it? (Or is there another explanation?)
>>>>>>
>>>>>> - Is there any documentation about the internals of KahaDB - the
>>>>>> kind of indices etc? I'd like to get a better understanding of
>>>>>> the
>>>>>> index performance and in general how KahaDB compares to something
>>>>>> like BerkeleyDB.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> There's is some confusion over naming of our persistence options
>>>>> that
>>>>> doesn't help. There is Kaha - which uses multiple log files and a
>>>>> Hash
>>>>> based index - this is currently used by the FileCursor - whilst
>>>>> KahaDB
>>>>> is a newer implementation, which is more robust and typically
>>>>> uses a
>>>>> BTreeIndex. There is currently a new implementation of the
>>>>> Filecursor
>>>>> btw - but that's a different matter. You can't currently configure
>>>>> the
>>>>> HashIndex via the FileCursor - but it looks like this is the
>>>>> problem
>>>>> you are encountering - as it looks like you need to increase the
>>>>> max
>>>>> hash buckets.
>>>>>
>>>>>
>>>>> So I would recommend the following
>>>>> 1. Use the default pendingQueuePolicy (which only uses a
>>>>> FileCursor
>>>>> for non-persistent messages - and uses the underlying database for
>>>>> persistent messages
>>>>> 2. Try KahaDB - which - with the BTreeIndex - will not hit the
>>>>> problems you are seeing with the Filecursor
>>>>>
>>>>> or - increase the maximum number of hash buckets for the
>>>>> FileCursor
>>>>> index - by setting a Java system property - maximumCapacity to
>>>>> 65536
>>>>> (the default is 16384)
>>>>>
>>>>> cheers,
>>>>>
>>>>> Rob
>>>>>
>>>>> http://twitter.com/rajdavies
>>>>> I work here: http://fusesource.com
>>>>> My Blog: http://rajdavies.blogspot.com/
>>>>> I'm writing this: http://www.manning.com/snyder/
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27264475.html
>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27279301.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>
>> Rob Davies
>> http://twitter.com/rajdavies
>> I work here: http://fusesource.com
>> My Blog: http://rajdavies.blogspot.com/
>> I'm writing this: http://www.manning.com/snyder/
>>
>>
>>
>>
>>
>>
>>
>
> --
> View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27280621.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/
Re: OOM with high KahaDB index time
Posted by Joe Fernandez <jo...@ttmsolutions.com>.
Its Miller-time here in Florida, so I'll give it a go on Monday d:)
Joe
rajdavies wrote:
>
> Hi Joe,
>
> any chance you can build from trunk and try it ?
>
> cheers,
>
> Rob
>
> On 22 Jan 2010, at 20:07, Joe Fernandez wrote:
>
>>
>> I ran my 5.3 test with the following
>>
>> <systemUsage>
>> <systemUsage sendFailIfNoSpace="true">
>> <memoryUsage>
>> <memoryUsage limit="100mb"/>
>> </memoryUsage>
>> <storeUsage>
>> <storeUsage limit="500 mb" name="foo"/>
>> </storeUsage>
>> <tempUsage>
>> <tempUsage limit="100mb"/>
>> </tempUsage>
>> </systemUsage>
>> </systemUsage>
>>
>> ...and had my producer fill up the store; it tried to pump 400k
>> messages
>> each at 2k in size. Producer flow control was also enabled. The
>> broker did
>> not hurl any OOM exceptions, "just
>> javax.jms.ResourceAllocationException:
>> Usage Manager Store is Full. Stopping producer..." exceptions as
>> expected.
>> The producer got kicked off its connection.
>>
>> The only time I got OOMs was when I used this.
>>
>> <systemUsage>
>> <systemUsage sendFailIfNoSpace="true">
>> </systemUsage>
>> </systemUsage>
>>
>> Joe
>>
>>
>>
>> Daniel Kluesing-2 wrote:
>>>
>>> So I'm not sure if that's what Rob is talking about being fixed in
>>> 5.4
>>> (And I'll try the snapshot as soon as it's ready) but if I don't
>>> have the
>>> sendFailIfNoSpace then my understanding is the producers send calls
>>> block/wait/timeout - as opposed to fail - so it's more difficult to
>>> get
>>> into a HA configuration. It's a minor point, not having an OOM is
>>> much
>>> more important, but I definitely want the send calls to fail for the
>>> producer if the broker ever does anything funny.
>>>
>>> Thanks for the feedback on the config, very helpful.
>>>
>>> -----Original Message-----
>>> From: Joe Fernandez [mailto:joe.fernandez@ttmsolutions.com]
>>> Sent: Thursday, January 21, 2010 1:01 PM
>>> To: users@activemq.apache.org
>>> Subject: RE: OOM with high KahaDB index time
>>>
>>>
>>> Just for grins, I threw your cfg file into our 5.3 testbed and sure
>>> enough,
>>> we got the OOMs; I pumped 200k messages with each being 2k in size.
>>> FWIW,
>>> taking this out of the cfg file made things run a lot better.
>>>
>>> <systemUsage>
>>> <systemUsage sendFailIfNoSpace="true">
>>> </systemUsage>
>>> </systemUsage>
>>>
>>> With the above taken out of the cfg file, I was able to pump 400k
>>> messages
>>> into the broker, no OOMs and memory utilization looked much better.
>>> I also
>>> gave a fully-defined systemUsage a try and that also appeared to do
>>> the
>>> trick.
>>>
>>> <systemUsage>
>>> <systemUsage>
>>> <memoryUsage>
>>> <memoryUsage limit="100 mb"/>
>>> </memoryUsage>
>>> <storeUsage>
>>> <storeUsage limit="1 gb" name="foo"/>
>>> </storeUsage>
>>> <tempUsage>
>>> <tempUsage limit="100 mb"/>
>>> </tempUsage>
>>> </systemUsage>
>>> </systemUsage>
>>>
>>> So may be worth giving it a whirl if you can't scoot over to the
>>> trunk and
>>> ride Rob's patch.
>>>
>>> Joe
>>>
>>>
>>> Daniel Kluesing-2 wrote:
>>>>
>>>> I tried the suggestion of going with the default cursor, but I
>>>> still get
>>>> OOM errors. I've included my full config file below, I think I'm
>>>> running
>>>> fairly vanilla/default.
>>>>
>>>> After about 350k persistent messages, the logs start to look like:
>>>>
>>>> INFO | Slow KahaDB access: Journal append took: 10 ms, Index
>>>> update took
>>>> 3118 ms
>>>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
>>>> took
>>>> 5118 ms
>>>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
>>>> took
>>>> 2736 ms
>>>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
>>>> took
>>>> 2945 ms
>>>> INFO | Slow KahaDB access: Journal append took: 33 ms, Index
>>>> update took
>>>> 2654 ms
>>>> INFO | Slow KahaDB access: Journal append took: 82 ms, Index
>>>> update took
>>>> 3174 ms
>>>> INFO | Slow KahaDB access: Journal append took: 1 ms, Index update
>>>> took
>>>> 5891 ms
>>>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
>>>> took
>>>> 2906 ms
>>>> INFO | Slow KahaDB access: Journal append took: 60 ms, Index
>>>> update took
>>>> 7619 ms
>>>> Exception in thread "InactivityMonitor WriteCheck"
>>>> java.lang.OutOfMemoryError: Java heap space
>>>> at java.util.jar.Attributes.read(Attributes.java:377)
>>>> at java.util.jar.Manifest.read(Manifest.java:182)
>>>> at java.util.jar.Manifest.<init>(Manifest.java:52)
>>>> at
>>>> java.util.jar.JarFile.getManifestFromReference(JarFile.java:165)
>>>> at java.util.jar.JarFile.getManifest(JarFile.java:146)
>>>> at
>>>> sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:693)
>>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:
>>>> 221)
>>>> at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
>>>> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:
>>>> 319)
>>>> at
>>>> org
>>>> .apache
>>>> .activemq
>>>> .transport.InactivityMonitor.writeCheck(InactivityMonitor.java:132)
>>>> at
>>>> org.apache.activemq.transport.InactivityMonitor
>>>> $2.run(InactivityMonitor.java:106)
>>>> at
>>>> org
>>>> .apache
>>>> .activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
>>>> at java.util.TimerThread.mainLoop(Timer.java:512)
>>>> at java.util.TimerThread.run(Timer.java:462)
>>>>
>>>> Config file:
>>>>
>>>> <beans
>>>> xmlns="http://www.springframework.org/schema/beans"
>>>> xmlns:amq="http://activemq.apache.org/schema/core"
>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>>>> http://activemq.apache.org/schema/core
>>>> http://activemq.apache.org/schema/core/activemq-core.xsd">
>>>>
>>>> <bean
>>>> class
>>>> =
>>>> "org
>>>> .springframework
>>>> .beans.factory.config.PropertyPlaceholderConfigurer">
>>>> <property name="locations">
>>>>
>>>> <value>file:${activemq.base}/conf/credentials.properties</value>
>>>> </property>
>>>> </bean>
>>>>
>>>> <broker xmlns="http://activemq.apache.org/schema/core"
>>>> brokerName="sub01chi" dataDirectory="${activemq.base}/data">
>>>>
>>>> <managementContext>
>>>> <managementContext createConnector="true"/>
>>>> </managementContext>
>>>>
>>>> <persistenceAdapter>
>>>> <kahaDB directory="${activemq.base}/data/kahadb"/>
>>>> </persistenceAdapter>
>>>>
>>>> <destinationPolicy>
>>>> <policyMap>
>>>> <policyEntries>
>>>> <policyEntry queue="P>" producerFlowControl="true"
>>>> memoryLimit="10mb"></policyEntry>
>>>> </policyEntries>
>>>> </policyMap>
>>>> </destinationPolicy>
>>>> <systemUsage>
>>>> <systemUsage sendFailIfNoSpace="true">
>>>> </systemUsage>
>>>> </systemUsage>
>>>> <transportConnectors>
>>>> <transportConnector name="openwire"
>>>> uri="tcp://0.0.0.0:61616"/>
>>>> </transportConnectors>
>>>> </broker>
>>>> <import resource="jetty.xml"/>
>>>> </beans>
>>>>
>>>> -----Original Message-----
>>>> From: Rob Davies [mailto:rajdavies@gmail.com]
>>>> Sent: Monday, January 18, 2010 10:42 PM
>>>> To: users@activemq.apache.org
>>>> Subject: Re: OOM with high KahaDB index time
>>>>
>>>>
>>>> On 18 Jan 2010, at 22:14, Daniel Kluesing wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I'm running the 5.3 release as a standalone broker. In one case, a
>>>>> producer is running without a consumer, producing small, persistent
>>>>> messages, with the FileCursor pendingQueuePolicy (per
>>>>> https://issues.apache.org/activemq/browse/AMQ-2512)
>>>>> option and flow control memoryLimit set to 100mb for the queue in
>>>>> question. (Through a policy entry)
>>>>>
>>>>> As the queue grows above 300k messages, KahaDB indexing starts
>>>>> climbing above 1 second. At around 350k messages, the indexing is
>>>>> taking over 8 seconds. At this point, I start getting java out of
>>>>> heap space errors in essentially random parts of the code. After a
>>>>> while, the producers timeout with a channel inactive for too long
>>>>> error, and the entire broker basically wedges itself. At this
>>>>> point,
>>>>> consumers are generally unable to bind to the broker quitting with
>>>>> timeout errors. When they can connect, consuming a single message
>>>>> triggers an index re-build, which takes 2-8seconds. Turning on
>>>>> verbose garbage collection, the jvm is collecting like mad but
>>>>> reclaiming no space.
>>>>>
>>>>> If I restart the broker, it comes back up, I can consume the old
>>>>> messages, and can handle another 350k messages until it wedges.
>>>>>
>>>>> I can reproduce under both default gc and incremental gc.
>>>>>
>>>>> Two questions:
>>>>> - It seems like someone is holding onto a handle to the messages
>>>>> after they have been persisted to disk - is this a known issue?
>>>>> Should I open a JIRA for it? (Or is there another explanation?)
>>>>>
>>>>> - Is there any documentation about the internals of KahaDB - the
>>>>> kind of indices etc? I'd like to get a better understanding of the
>>>>> index performance and in general how KahaDB compares to something
>>>>> like BerkeleyDB.
>>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> There's is some confusion over naming of our persistence options
>>>> that
>>>> doesn't help. There is Kaha - which uses multiple log files and a
>>>> Hash
>>>> based index - this is currently used by the FileCursor - whilst
>>>> KahaDB
>>>> is a newer implementation, which is more robust and typically uses a
>>>> BTreeIndex. There is currently a new implementation of the
>>>> Filecursor
>>>> btw - but that's a different matter. You can't currently configure
>>>> the
>>>> HashIndex via the FileCursor - but it looks like this is the
>>>> problem
>>>> you are encountering - as it looks like you need to increase the max
>>>> hash buckets.
>>>>
>>>>
>>>> So I would recommend the following
>>>> 1. Use the default pendingQueuePolicy (which only uses a FileCursor
>>>> for non-persistent messages - and uses the underlying database for
>>>> persistent messages
>>>> 2. Try KahaDB - which - with the BTreeIndex - will not hit the
>>>> problems you are seeing with the Filecursor
>>>>
>>>> or - increase the maximum number of hash buckets for the FileCursor
>>>> index - by setting a Java system property - maximumCapacity to
>>>> 65536
>>>> (the default is 16384)
>>>>
>>>> cheers,
>>>>
>>>> Rob
>>>>
>>>> http://twitter.com/rajdavies
>>>> I work here: http://fusesource.com
>>>> My Blog: http://rajdavies.blogspot.com/
>>>> I'm writing this: http://www.manning.com/snyder/
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27264475.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27279301.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
> Rob Davies
> http://twitter.com/rajdavies
> I work here: http://fusesource.com
> My Blog: http://rajdavies.blogspot.com/
> I'm writing this: http://www.manning.com/snyder/
>
>
>
>
>
>
>
--
View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27280621.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: OOM with high KahaDB index time
Posted by Rob Davies <ra...@gmail.com>.
Great! - thanks for testing this!
On 25 Jan 2010, at 14:41, Joe Fernandez wrote:
>
> Rob,
>
> The trunk (902807) passed my test; no hurling of OOMs. Memory
> utilization
> under JConsole looked much better.
>
> My producer didn't get kicked off when the store filled, but I think
> that's
> because in this case it was issuing async sends.
>
> Joe
>
>
> rajdavies wrote:
>>
>> Hi Joe,
>>
>> any chance you can build from trunk and try it ?
>>
>> cheers,
>>
>> Rob
>>
>> On 22 Jan 2010, at 20:07, Joe Fernandez wrote:
>>
>>>
>>> I ran my 5.3 test with the following
>>>
>>> <systemUsage>
>>> <systemUsage sendFailIfNoSpace="true">
>>> <memoryUsage>
>>> <memoryUsage limit="100mb"/>
>>> </memoryUsage>
>>> <storeUsage>
>>> <storeUsage limit="500 mb" name="foo"/>
>>> </storeUsage>
>>> <tempUsage>
>>> <tempUsage limit="100mb"/>
>>> </tempUsage>
>>> </systemUsage>
>>> </systemUsage>
>>>
>>> ...and had my producer fill up the store; it tried to pump 400k
>>> messages
>>> each at 2k in size. Producer flow control was also enabled. The
>>> broker did
>>> not hurl any OOM exceptions, "just
>>> javax.jms.ResourceAllocationException:
>>> Usage Manager Store is Full. Stopping producer..." exceptions as
>>> expected.
>>> The producer got kicked off its connection.
>>>
>>> The only time I got OOMs was when I used this.
>>>
>>> <systemUsage>
>>> <systemUsage sendFailIfNoSpace="true">
>>> </systemUsage>
>>> </systemUsage>
>>>
>>> Joe
>>>
>>>
>>>
>>> Daniel Kluesing-2 wrote:
>>>>
>>>> So I'm not sure if that's what Rob is talking about being fixed in
>>>> 5.4
>>>> (And I'll try the snapshot as soon as it's ready) but if I don't
>>>> have the
>>>> sendFailIfNoSpace then my understanding is the producers send calls
>>>> block/wait/timeout - as opposed to fail - so it's more difficult to
>>>> get
>>>> into a HA configuration. It's a minor point, not having an OOM is
>>>> much
>>>> more important, but I definitely want the send calls to fail for
>>>> the
>>>> producer if the broker ever does anything funny.
>>>>
>>>> Thanks for the feedback on the config, very helpful.
>>>>
>>>> -----Original Message-----
>>>> From: Joe Fernandez [mailto:joe.fernandez@ttmsolutions.com]
>>>> Sent: Thursday, January 21, 2010 1:01 PM
>>>> To: users@activemq.apache.org
>>>> Subject: RE: OOM with high KahaDB index time
>>>>
>>>>
>>>> Just for grins, I threw your cfg file into our 5.3 testbed and sure
>>>> enough,
>>>> we got the OOMs; I pumped 200k messages with each being 2k in size.
>>>> FWIW,
>>>> taking this out of the cfg file made things run a lot better.
>>>>
>>>> <systemUsage>
>>>> <systemUsage sendFailIfNoSpace="true">
>>>> </systemUsage>
>>>> </systemUsage>
>>>>
>>>> With the above taken out of the cfg file, I was able to pump 400k
>>>> messages
>>>> into the broker, no OOMs and memory utilization looked much better.
>>>> I also
>>>> gave a fully-defined systemUsage a try and that also appeared to do
>>>> the
>>>> trick.
>>>>
>>>> <systemUsage>
>>>> <systemUsage>
>>>> <memoryUsage>
>>>> <memoryUsage limit="100 mb"/>
>>>> </memoryUsage>
>>>> <storeUsage>
>>>> <storeUsage limit="1 gb" name="foo"/>
>>>> </storeUsage>
>>>> <tempUsage>
>>>> <tempUsage limit="100 mb"/>
>>>> </tempUsage>
>>>> </systemUsage>
>>>> </systemUsage>
>>>>
>>>> So may be worth giving it a whirl if you can't scoot over to the
>>>> trunk and
>>>> ride Rob's patch.
>>>>
>>>> Joe
>>>>
>>>>
>>>> Daniel Kluesing-2 wrote:
>>>>>
>>>>> I tried the suggestion of going with the default cursor, but I
>>>>> still get
>>>>> OOM errors. I've included my full config file below, I think I'm
>>>>> running
>>>>> fairly vanilla/default.
>>>>>
>>>>> After about 350k persistent messages, the logs start to look like:
>>>>>
>>>>> INFO | Slow KahaDB access: Journal append took: 10 ms, Index
>>>>> update took
>>>>> 3118 ms
>>>>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
>>>>> took
>>>>> 5118 ms
>>>>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
>>>>> took
>>>>> 2736 ms
>>>>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
>>>>> took
>>>>> 2945 ms
>>>>> INFO | Slow KahaDB access: Journal append took: 33 ms, Index
>>>>> update took
>>>>> 2654 ms
>>>>> INFO | Slow KahaDB access: Journal append took: 82 ms, Index
>>>>> update took
>>>>> 3174 ms
>>>>> INFO | Slow KahaDB access: Journal append took: 1 ms, Index update
>>>>> took
>>>>> 5891 ms
>>>>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
>>>>> took
>>>>> 2906 ms
>>>>> INFO | Slow KahaDB access: Journal append took: 60 ms, Index
>>>>> update took
>>>>> 7619 ms
>>>>> Exception in thread "InactivityMonitor WriteCheck"
>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>> at java.util.jar.Attributes.read(Attributes.java:377)
>>>>> at java.util.jar.Manifest.read(Manifest.java:182)
>>>>> at java.util.jar.Manifest.<init>(Manifest.java:52)
>>>>> at
>>>>> java.util.jar.JarFile.getManifestFromReference(JarFile.java:165)
>>>>> at java.util.jar.JarFile.getManifest(JarFile.java:146)
>>>>> at
>>>>> sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:
>>>>> 693)
>>>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:
>>>>> 221)
>>>>> at java.net.URLClassLoader.access$000(URLClassLoader.java:
>>>>> 56)
>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
>>>>> at java.security.AccessController.doPrivileged(Native
>>>>> Method)
>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:
>>>>> 188)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
>>>>> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:
>>>>> 319)
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .activemq
>>>>> .transport.InactivityMonitor.writeCheck(InactivityMonitor.java:
>>>>> 132)
>>>>> at
>>>>> org.apache.activemq.transport.InactivityMonitor
>>>>> $2.run(InactivityMonitor.java:106)
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:
>>>>> 33)
>>>>> at java.util.TimerThread.mainLoop(Timer.java:512)
>>>>> at java.util.TimerThread.run(Timer.java:462)
>>>>>
>>>>> Config file:
>>>>>
>>>>> <beans
>>>>> xmlns="http://www.springframework.org/schema/beans"
>>>>> xmlns:amq="http://activemq.apache.org/schema/core"
>>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>>>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>>>>> http://activemq.apache.org/schema/core
>>>>> http://activemq.apache.org/schema/core/activemq-core.xsd">
>>>>>
>>>>> <bean
>>>>> class
>>>>> =
>>>>> "org
>>>>> .springframework
>>>>> .beans.factory.config.PropertyPlaceholderConfigurer">
>>>>> <property name="locations">
>>>>>
>>>>> <value>file:${activemq.base}/conf/credentials.properties</value>
>>>>> </property>
>>>>> </bean>
>>>>>
>>>>> <broker xmlns="http://activemq.apache.org/schema/core"
>>>>> brokerName="sub01chi" dataDirectory="${activemq.base}/data">
>>>>>
>>>>> <managementContext>
>>>>> <managementContext createConnector="true"/>
>>>>> </managementContext>
>>>>>
>>>>> <persistenceAdapter>
>>>>> <kahaDB directory="${activemq.base}/data/kahadb"/>
>>>>> </persistenceAdapter>
>>>>>
>>>>> <destinationPolicy>
>>>>> <policyMap>
>>>>> <policyEntries>
>>>>> <policyEntry queue="P>" producerFlowControl="true"
>>>>> memoryLimit="10mb"></policyEntry>
>>>>> </policyEntries>
>>>>> </policyMap>
>>>>> </destinationPolicy>
>>>>> <systemUsage>
>>>>> <systemUsage sendFailIfNoSpace="true">
>>>>> </systemUsage>
>>>>> </systemUsage>
>>>>> <transportConnectors>
>>>>> <transportConnector name="openwire"
>>>>> uri="tcp://0.0.0.0:61616"/>
>>>>> </transportConnectors>
>>>>> </broker>
>>>>> <import resource="jetty.xml"/>
>>>>> </beans>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Rob Davies [mailto:rajdavies@gmail.com]
>>>>> Sent: Monday, January 18, 2010 10:42 PM
>>>>> To: users@activemq.apache.org
>>>>> Subject: Re: OOM with high KahaDB index time
>>>>>
>>>>>
>>>>> On 18 Jan 2010, at 22:14, Daniel Kluesing wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I'm running the 5.3 release as a standalone broker. In one
>>>>>> case, a
>>>>>> producer is running without a consumer, producing small,
>>>>>> persistent
>>>>>> messages, with the FileCursor pendingQueuePolicy (per
>>>>>> https://issues.apache.org/activemq/browse/AMQ-2512)
>>>>>> option and flow control memoryLimit set to 100mb for the queue in
>>>>>> question. (Through a policy entry)
>>>>>>
>>>>>> As the queue grows above 300k messages, KahaDB indexing starts
>>>>>> climbing above 1 second. At around 350k messages, the indexing is
>>>>>> taking over 8 seconds. At this point, I start getting java out of
>>>>>> heap space errors in essentially random parts of the code.
>>>>>> After a
>>>>>> while, the producers timeout with a channel inactive for too long
>>>>>> error, and the entire broker basically wedges itself. At this
>>>>>> point,
>>>>>> consumers are generally unable to bind to the broker quitting
>>>>>> with
>>>>>> timeout errors. When they can connect, consuming a single message
>>>>>> triggers an index re-build, which takes 2-8seconds. Turning on
>>>>>> verbose garbage collection, the jvm is collecting like mad but
>>>>>> reclaiming no space.
>>>>>>
>>>>>> If I restart the broker, it comes back up, I can consume the old
>>>>>> messages, and can handle another 350k messages until it wedges.
>>>>>>
>>>>>> I can reproduce under both default gc and incremental gc.
>>>>>>
>>>>>> Two questions:
>>>>>> - It seems like someone is holding onto a handle to the messages
>>>>>> after they have been persisted to disk - is this a known issue?
>>>>>> Should I open a JIRA for it? (Or is there another explanation?)
>>>>>>
>>>>>> - Is there any documentation about the internals of KahaDB - the
>>>>>> kind of indices etc? I'd like to get a better understanding of
>>>>>> the
>>>>>> index performance and in general how KahaDB compares to something
>>>>>> like BerkeleyDB.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> There's is some confusion over naming of our persistence options
>>>>> that
>>>>> doesn't help. There is Kaha - which uses multiple log files and a
>>>>> Hash
>>>>> based index - this is currently used by the FileCursor - whilst
>>>>> KahaDB
>>>>> is a newer implementation, which is more robust and typically
>>>>> uses a
>>>>> BTreeIndex. There is currently a new implementation of the
>>>>> Filecursor
>>>>> btw - but that's a different matter. You can't currently configure
>>>>> the
>>>>> HashIndex via the FileCursor - but it looks like this is the
>>>>> problem
>>>>> you are encountering - as it looks like you need to increase the
>>>>> max
>>>>> hash buckets.
>>>>>
>>>>>
>>>>> So I would recommend the following
>>>>> 1. Use the default pendingQueuePolicy (which only uses a
>>>>> FileCursor
>>>>> for non-persistent messages - and uses the underlying database for
>>>>> persistent messages
>>>>> 2. Try KahaDB - which - with the BTreeIndex - will not hit the
>>>>> problems you are seeing with the Filecursor
>>>>>
>>>>> or - increase the maximum number of hash buckets for the
>>>>> FileCursor
>>>>> index - by setting a Java system property - maximumCapacity to
>>>>> 65536
>>>>> (the default is 16384)
>>>>>
>>>>> cheers,
>>>>>
>>>>> Rob
>>>>>
>>>>> http://twitter.com/rajdavies
>>>>> I work here: http://fusesource.com
>>>>> My Blog: http://rajdavies.blogspot.com/
>>>>> I'm writing this: http://www.manning.com/snyder/
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27264475.html
>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27279301.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>
>> Rob Davies
>> http://twitter.com/rajdavies
>> I work here: http://fusesource.com
>> My Blog: http://rajdavies.blogspot.com/
>> I'm writing this: http://www.manning.com/snyder/
>>
>>
>>
>>
>>
>>
>>
>
> --
> View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27307657.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/
Re: OOM with high KahaDB index time
Posted by Joe Fernandez <jo...@ttmsolutions.com>.
Rob,
The trunk (902807) passed my test; no hurling of OOMs. Memory utilization
under JConsole looked much better.
My producer didn't get kicked off when the store filled, but I think that's
because in this case it was issuing async sends.
Joe
rajdavies wrote:
>
> Hi Joe,
>
> any chance you can build from trunk and try it ?
>
> cheers,
>
> Rob
>
> On 22 Jan 2010, at 20:07, Joe Fernandez wrote:
>
>>
>> I ran my 5.3 test with the following
>>
>> <systemUsage>
>> <systemUsage sendFailIfNoSpace="true">
>> <memoryUsage>
>> <memoryUsage limit="100mb"/>
>> </memoryUsage>
>> <storeUsage>
>> <storeUsage limit="500 mb" name="foo"/>
>> </storeUsage>
>> <tempUsage>
>> <tempUsage limit="100mb"/>
>> </tempUsage>
>> </systemUsage>
>> </systemUsage>
>>
>> ...and had my producer fill up the store; it tried to pump 400k
>> messages
>> each at 2k in size. Producer flow control was also enabled. The
>> broker did
>> not hurl any OOM exceptions, "just
>> javax.jms.ResourceAllocationException:
>> Usage Manager Store is Full. Stopping producer..." exceptions as
>> expected.
>> The producer got kicked off its connection.
>>
>> The only time I got OOMs was when I used this.
>>
>> <systemUsage>
>> <systemUsage sendFailIfNoSpace="true">
>> </systemUsage>
>> </systemUsage>
>>
>> Joe
>>
>>
>>
>> Daniel Kluesing-2 wrote:
>>>
>>> So I'm not sure if that's what Rob is talking about being fixed in
>>> 5.4
>>> (And I'll try the snapshot as soon as it's ready) but if I don't
>>> have the
>>> sendFailIfNoSpace then my understanding is the producers send calls
>>> block/wait/timeout - as opposed to fail - so it's more difficult to
>>> get
>>> into a HA configuration. It's a minor point, not having an OOM is
>>> much
>>> more important, but I definitely want the send calls to fail for the
>>> producer if the broker ever does anything funny.
>>>
>>> Thanks for the feedback on the config, very helpful.
>>>
>>> -----Original Message-----
>>> From: Joe Fernandez [mailto:joe.fernandez@ttmsolutions.com]
>>> Sent: Thursday, January 21, 2010 1:01 PM
>>> To: users@activemq.apache.org
>>> Subject: RE: OOM with high KahaDB index time
>>>
>>>
>>> Just for grins, I threw your cfg file into our 5.3 testbed and sure
>>> enough,
>>> we got the OOMs; I pumped 200k messages with each being 2k in size.
>>> FWIW,
>>> taking this out of the cfg file made things run a lot better.
>>>
>>> <systemUsage>
>>> <systemUsage sendFailIfNoSpace="true">
>>> </systemUsage>
>>> </systemUsage>
>>>
>>> With the above taken out of the cfg file, I was able to pump 400k
>>> messages
>>> into the broker, no OOMs and memory utilization looked much better.
>>> I also
>>> gave a fully-defined systemUsage a try and that also appeared to do
>>> the
>>> trick.
>>>
>>> <systemUsage>
>>> <systemUsage>
>>> <memoryUsage>
>>> <memoryUsage limit="100 mb"/>
>>> </memoryUsage>
>>> <storeUsage>
>>> <storeUsage limit="1 gb" name="foo"/>
>>> </storeUsage>
>>> <tempUsage>
>>> <tempUsage limit="100 mb"/>
>>> </tempUsage>
>>> </systemUsage>
>>> </systemUsage>
>>>
>>> So may be worth giving it a whirl if you can't scoot over to the
>>> trunk and
>>> ride Rob's patch.
>>>
>>> Joe
>>>
>>>
>>> Daniel Kluesing-2 wrote:
>>>>
>>>> I tried the suggestion of going with the default cursor, but I
>>>> still get
>>>> OOM errors. I've included my full config file below, I think I'm
>>>> running
>>>> fairly vanilla/default.
>>>>
>>>> After about 350k persistent messages, the logs start to look like:
>>>>
>>>> INFO | Slow KahaDB access: Journal append took: 10 ms, Index
>>>> update took
>>>> 3118 ms
>>>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
>>>> took
>>>> 5118 ms
>>>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
>>>> took
>>>> 2736 ms
>>>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
>>>> took
>>>> 2945 ms
>>>> INFO | Slow KahaDB access: Journal append took: 33 ms, Index
>>>> update took
>>>> 2654 ms
>>>> INFO | Slow KahaDB access: Journal append took: 82 ms, Index
>>>> update took
>>>> 3174 ms
>>>> INFO | Slow KahaDB access: Journal append took: 1 ms, Index update
>>>> took
>>>> 5891 ms
>>>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
>>>> took
>>>> 2906 ms
>>>> INFO | Slow KahaDB access: Journal append took: 60 ms, Index
>>>> update took
>>>> 7619 ms
>>>> Exception in thread "InactivityMonitor WriteCheck"
>>>> java.lang.OutOfMemoryError: Java heap space
>>>> at java.util.jar.Attributes.read(Attributes.java:377)
>>>> at java.util.jar.Manifest.read(Manifest.java:182)
>>>> at java.util.jar.Manifest.<init>(Manifest.java:52)
>>>> at
>>>> java.util.jar.JarFile.getManifestFromReference(JarFile.java:165)
>>>> at java.util.jar.JarFile.getManifest(JarFile.java:146)
>>>> at
>>>> sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:693)
>>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:
>>>> 221)
>>>> at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
>>>> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:
>>>> 319)
>>>> at
>>>> org
>>>> .apache
>>>> .activemq
>>>> .transport.InactivityMonitor.writeCheck(InactivityMonitor.java:132)
>>>> at
>>>> org.apache.activemq.transport.InactivityMonitor
>>>> $2.run(InactivityMonitor.java:106)
>>>> at
>>>> org
>>>> .apache
>>>> .activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
>>>> at java.util.TimerThread.mainLoop(Timer.java:512)
>>>> at java.util.TimerThread.run(Timer.java:462)
>>>>
>>>> Config file:
>>>>
>>>> <beans
>>>> xmlns="http://www.springframework.org/schema/beans"
>>>> xmlns:amq="http://activemq.apache.org/schema/core"
>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>>>> http://activemq.apache.org/schema/core
>>>> http://activemq.apache.org/schema/core/activemq-core.xsd">
>>>>
>>>> <bean
>>>> class
>>>> =
>>>> "org
>>>> .springframework
>>>> .beans.factory.config.PropertyPlaceholderConfigurer">
>>>> <property name="locations">
>>>>
>>>> <value>file:${activemq.base}/conf/credentials.properties</value>
>>>> </property>
>>>> </bean>
>>>>
>>>> <broker xmlns="http://activemq.apache.org/schema/core"
>>>> brokerName="sub01chi" dataDirectory="${activemq.base}/data">
>>>>
>>>> <managementContext>
>>>> <managementContext createConnector="true"/>
>>>> </managementContext>
>>>>
>>>> <persistenceAdapter>
>>>> <kahaDB directory="${activemq.base}/data/kahadb"/>
>>>> </persistenceAdapter>
>>>>
>>>> <destinationPolicy>
>>>> <policyMap>
>>>> <policyEntries>
>>>> <policyEntry queue="P>" producerFlowControl="true"
>>>> memoryLimit="10mb"></policyEntry>
>>>> </policyEntries>
>>>> </policyMap>
>>>> </destinationPolicy>
>>>> <systemUsage>
>>>> <systemUsage sendFailIfNoSpace="true">
>>>> </systemUsage>
>>>> </systemUsage>
>>>> <transportConnectors>
>>>> <transportConnector name="openwire"
>>>> uri="tcp://0.0.0.0:61616"/>
>>>> </transportConnectors>
>>>> </broker>
>>>> <import resource="jetty.xml"/>
>>>> </beans>
>>>>
>>>> -----Original Message-----
>>>> From: Rob Davies [mailto:rajdavies@gmail.com]
>>>> Sent: Monday, January 18, 2010 10:42 PM
>>>> To: users@activemq.apache.org
>>>> Subject: Re: OOM with high KahaDB index time
>>>>
>>>>
>>>> On 18 Jan 2010, at 22:14, Daniel Kluesing wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I'm running the 5.3 release as a standalone broker. In one case, a
>>>>> producer is running without a consumer, producing small, persistent
>>>>> messages, with the FileCursor pendingQueuePolicy (per
>>>>> https://issues.apache.org/activemq/browse/AMQ-2512)
>>>>> option and flow control memoryLimit set to 100mb for the queue in
>>>>> question. (Through a policy entry)
>>>>>
>>>>> As the queue grows above 300k messages, KahaDB indexing starts
>>>>> climbing above 1 second. At around 350k messages, the indexing is
>>>>> taking over 8 seconds. At this point, I start getting java out of
>>>>> heap space errors in essentially random parts of the code. After a
>>>>> while, the producers timeout with a channel inactive for too long
>>>>> error, and the entire broker basically wedges itself. At this
>>>>> point,
>>>>> consumers are generally unable to bind to the broker quitting with
>>>>> timeout errors. When they can connect, consuming a single message
>>>>> triggers an index re-build, which takes 2-8seconds. Turning on
>>>>> verbose garbage collection, the jvm is collecting like mad but
>>>>> reclaiming no space.
>>>>>
>>>>> If I restart the broker, it comes back up, I can consume the old
>>>>> messages, and can handle another 350k messages until it wedges.
>>>>>
>>>>> I can reproduce under both default gc and incremental gc.
>>>>>
>>>>> Two questions:
>>>>> - It seems like someone is holding onto a handle to the messages
>>>>> after they have been persisted to disk - is this a known issue?
>>>>> Should I open a JIRA for it? (Or is there another explanation?)
>>>>>
>>>>> - Is there any documentation about the internals of KahaDB - the
>>>>> kind of indices etc? I'd like to get a better understanding of the
>>>>> index performance and in general how KahaDB compares to something
>>>>> like BerkeleyDB.
>>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> There's is some confusion over naming of our persistence options
>>>> that
>>>> doesn't help. There is Kaha - which uses multiple log files and a
>>>> Hash
>>>> based index - this is currently used by the FileCursor - whilst
>>>> KahaDB
>>>> is a newer implementation, which is more robust and typically uses a
>>>> BTreeIndex. There is currently a new implementation of the
>>>> Filecursor
>>>> btw - but that's a different matter. You can't currently configure
>>>> the
>>>> HashIndex via the FileCursor - but it looks like this is the
>>>> problem
>>>> you are encountering - as it looks like you need to increase the max
>>>> hash buckets.
>>>>
>>>>
>>>> So I would recommend the following
>>>> 1. Use the default pendingQueuePolicy (which only uses a FileCursor
>>>> for non-persistent messages - and uses the underlying database for
>>>> persistent messages
>>>> 2. Try KahaDB - which - with the BTreeIndex - will not hit the
>>>> problems you are seeing with the Filecursor
>>>>
>>>> or - increase the maximum number of hash buckets for the FileCursor
>>>> index - by setting a Java system property - maximumCapacity to
>>>> 65536
>>>> (the default is 16384)
>>>>
>>>> cheers,
>>>>
>>>> Rob
>>>>
>>>> http://twitter.com/rajdavies
>>>> I work here: http://fusesource.com
>>>> My Blog: http://rajdavies.blogspot.com/
>>>> I'm writing this: http://www.manning.com/snyder/
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27264475.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27279301.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
> Rob Davies
> http://twitter.com/rajdavies
> I work here: http://fusesource.com
> My Blog: http://rajdavies.blogspot.com/
> I'm writing this: http://www.manning.com/snyder/
>
>
>
>
>
>
>
--
View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27307657.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: OOM with high KahaDB index time
Posted by Rob Davies <ra...@gmail.com>.
Hi Joe,
any chance you can build from trunk and try it ?
cheers,
Rob
On 22 Jan 2010, at 20:07, Joe Fernandez wrote:
>
> I ran my 5.3 test with the following
>
> <systemUsage>
> <systemUsage sendFailIfNoSpace="true">
> <memoryUsage>
> <memoryUsage limit="100mb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="500 mb" name="foo"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="100mb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
>
> ...and had my producer fill up the store; it tried to pump 400k
> messages
> each at 2k in size. Producer flow control was also enabled. The
> broker did
> not hurl any OOM exceptions, "just
> javax.jms.ResourceAllocationException:
> Usage Manager Store is Full. Stopping producer..." exceptions as
> expected.
> The producer got kicked off its connection.
>
> The only time I got OOMs was when I used this.
>
> <systemUsage>
> <systemUsage sendFailIfNoSpace="true">
> </systemUsage>
> </systemUsage>
>
> Joe
>
>
>
> Daniel Kluesing-2 wrote:
>>
>> So I'm not sure if that's what Rob is talking about being fixed in
>> 5.4
>> (And I'll try the snapshot as soon as it's ready) but if I don't
>> have the
>> sendFailIfNoSpace then my understanding is the producers send calls
>> block/wait/timeout - as opposed to fail - so it's more difficult to
>> get
>> into a HA configuration. It's a minor point, not having an OOM is
>> much
>> more important, but I definitely want the send calls to fail for the
>> producer if the broker ever does anything funny.
>>
>> Thanks for the feedback on the config, very helpful.
>>
>> -----Original Message-----
>> From: Joe Fernandez [mailto:joe.fernandez@ttmsolutions.com]
>> Sent: Thursday, January 21, 2010 1:01 PM
>> To: users@activemq.apache.org
>> Subject: RE: OOM with high KahaDB index time
>>
>>
>> Just for grins, I threw your cfg file into our 5.3 testbed and sure
>> enough,
>> we got the OOMs; I pumped 200k messages with each being 2k in size.
>> FWIW,
>> taking this out of the cfg file made things run a lot better.
>>
>> <systemUsage>
>> <systemUsage sendFailIfNoSpace="true">
>> </systemUsage>
>> </systemUsage>
>>
>> With the above taken out of the cfg file, I was able to pump 400k
>> messages
>> into the broker, no OOMs and memory utilization looked much better.
>> I also
>> gave a fully-defined systemUsage a try and that also appeared to do
>> the
>> trick.
>>
>> <systemUsage>
>> <systemUsage>
>> <memoryUsage>
>> <memoryUsage limit="100 mb"/>
>> </memoryUsage>
>> <storeUsage>
>> <storeUsage limit="1 gb" name="foo"/>
>> </storeUsage>
>> <tempUsage>
>> <tempUsage limit="100 mb"/>
>> </tempUsage>
>> </systemUsage>
>> </systemUsage>
>>
>> So may be worth giving it a whirl if you can't scoot over to the
>> trunk and
>> ride Rob's patch.
>>
>> Joe
>>
>>
>> Daniel Kluesing-2 wrote:
>>>
>>> I tried the suggestion of going with the default cursor, but I
>>> still get
>>> OOM errors. I've included my full config file below, I think I'm
>>> running
>>> fairly vanilla/default.
>>>
>>> After about 350k persistent messages, the logs start to look like:
>>>
>>> INFO | Slow KahaDB access: Journal append took: 10 ms, Index
>>> update took
>>> 3118 ms
>>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
>>> took
>>> 5118 ms
>>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
>>> took
>>> 2736 ms
>>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
>>> took
>>> 2945 ms
>>> INFO | Slow KahaDB access: Journal append took: 33 ms, Index
>>> update took
>>> 2654 ms
>>> INFO | Slow KahaDB access: Journal append took: 82 ms, Index
>>> update took
>>> 3174 ms
>>> INFO | Slow KahaDB access: Journal append took: 1 ms, Index update
>>> took
>>> 5891 ms
>>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
>>> took
>>> 2906 ms
>>> INFO | Slow KahaDB access: Journal append took: 60 ms, Index
>>> update took
>>> 7619 ms
>>> Exception in thread "InactivityMonitor WriteCheck"
>>> java.lang.OutOfMemoryError: Java heap space
>>> at java.util.jar.Attributes.read(Attributes.java:377)
>>> at java.util.jar.Manifest.read(Manifest.java:182)
>>> at java.util.jar.Manifest.<init>(Manifest.java:52)
>>> at
>>> java.util.jar.JarFile.getManifestFromReference(JarFile.java:165)
>>> at java.util.jar.JarFile.getManifest(JarFile.java:146)
>>> at
>>> sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:693)
>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:
>>> 221)
>>> at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
>>> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:
>>> 319)
>>> at
>>> org
>>> .apache
>>> .activemq
>>> .transport.InactivityMonitor.writeCheck(InactivityMonitor.java:132)
>>> at
>>> org.apache.activemq.transport.InactivityMonitor
>>> $2.run(InactivityMonitor.java:106)
>>> at
>>> org
>>> .apache
>>> .activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
>>> at java.util.TimerThread.mainLoop(Timer.java:512)
>>> at java.util.TimerThread.run(Timer.java:462)
>>>
>>> Config file:
>>>
>>> <beans
>>> xmlns="http://www.springframework.org/schema/beans"
>>> xmlns:amq="http://activemq.apache.org/schema/core"
>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>>> http://activemq.apache.org/schema/core
>>> http://activemq.apache.org/schema/core/activemq-core.xsd">
>>>
>>> <bean
>>> class
>>> =
>>> "org
>>> .springframework
>>> .beans.factory.config.PropertyPlaceholderConfigurer">
>>> <property name="locations">
>>>
>>> <value>file:${activemq.base}/conf/credentials.properties</value>
>>> </property>
>>> </bean>
>>>
>>> <broker xmlns="http://activemq.apache.org/schema/core"
>>> brokerName="sub01chi" dataDirectory="${activemq.base}/data">
>>>
>>> <managementContext>
>>> <managementContext createConnector="true"/>
>>> </managementContext>
>>>
>>> <persistenceAdapter>
>>> <kahaDB directory="${activemq.base}/data/kahadb"/>
>>> </persistenceAdapter>
>>>
>>> <destinationPolicy>
>>> <policyMap>
>>> <policyEntries>
>>> <policyEntry queue="P>" producerFlowControl="true"
>>> memoryLimit="10mb"></policyEntry>
>>> </policyEntries>
>>> </policyMap>
>>> </destinationPolicy>
>>> <systemUsage>
>>> <systemUsage sendFailIfNoSpace="true">
>>> </systemUsage>
>>> </systemUsage>
>>> <transportConnectors>
>>> <transportConnector name="openwire"
>>> uri="tcp://0.0.0.0:61616"/>
>>> </transportConnectors>
>>> </broker>
>>> <import resource="jetty.xml"/>
>>> </beans>
>>>
>>> -----Original Message-----
>>> From: Rob Davies [mailto:rajdavies@gmail.com]
>>> Sent: Monday, January 18, 2010 10:42 PM
>>> To: users@activemq.apache.org
>>> Subject: Re: OOM with high KahaDB index time
>>>
>>>
>>> On 18 Jan 2010, at 22:14, Daniel Kluesing wrote:
>>>
>>>> Hi,
>>>>
>>>> I'm running the 5.3 release as a standalone broker. In one case, a
>>>> producer is running without a consumer, producing small, persistent
>>>> messages, with the FileCursor pendingQueuePolicy (per
>>>> https://issues.apache.org/activemq/browse/AMQ-2512)
>>>> option and flow control memoryLimit set to 100mb for the queue in
>>>> question. (Through a policy entry)
>>>>
>>>> As the queue grows above 300k messages, KahaDB indexing starts
>>>> climbing above 1 second. At around 350k messages, the indexing is
>>>> taking over 8 seconds. At this point, I start getting java out of
>>>> heap space errors in essentially random parts of the code. After a
>>>> while, the producers timeout with a channel inactive for too long
>>>> error, and the entire broker basically wedges itself. At this
>>>> point,
>>>> consumers are generally unable to bind to the broker quitting with
>>>> timeout errors. When they can connect, consuming a single message
>>>> triggers an index re-build, which takes 2-8seconds. Turning on
>>>> verbose garbage collection, the jvm is collecting like mad but
>>>> reclaiming no space.
>>>>
>>>> If I restart the broker, it comes back up, I can consume the old
>>>> messages, and can handle another 350k messages until it wedges.
>>>>
>>>> I can reproduce under both default gc and incremental gc.
>>>>
>>>> Two questions:
>>>> - It seems like someone is holding onto a handle to the messages
>>>> after they have been persisted to disk - is this a known issue?
>>>> Should I open a JIRA for it? (Or is there another explanation?)
>>>>
>>>> - Is there any documentation about the internals of KahaDB - the
>>>> kind of indices etc? I'd like to get a better understanding of the
>>>> index performance and in general how KahaDB compares to something
>>>> like BerkeleyDB.
>>>>
>>>> Thanks
>>>>
>>>>
>>>>
>>>
>>>
>>> There's is some confusion over naming of our persistence options
>>> that
>>> doesn't help. There is Kaha - which uses multiple log files and a
>>> Hash
>>> based index - this is currently used by the FileCursor - whilst
>>> KahaDB
>>> is a newer implementation, which is more robust and typically uses a
>>> BTreeIndex. There is currently a new implementation of the
>>> Filecursor
>>> btw - but that's a different matter. You can't currently configure
>>> the
>>> HashIndex via the FileCursor - but it looks like this is the
>>> problem
>>> you are encountering - as it looks like you need to increase the max
>>> hash buckets.
>>>
>>>
>>> So I would recommend the following
>>> 1. Use the default pendingQueuePolicy (which only uses a FileCursor
>>> for non-persistent messages - and uses the underlying database for
>>> persistent messages
>>> 2. Try KahaDB - which - with the BTreeIndex - will not hit the
>>> problems you are seeing with the Filecursor
>>>
>>> or - increase the maximum number of hash buckets for the FileCursor
>>> index - by setting a Java system property - maximumCapacity to
>>> 65536
>>> (the default is 16384)
>>>
>>> cheers,
>>>
>>> Rob
>>>
>>> http://twitter.com/rajdavies
>>> I work here: http://fusesource.com
>>> My Blog: http://rajdavies.blogspot.com/
>>> I'm writing this: http://www.manning.com/snyder/
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27264475.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>>
>
> --
> View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27279301.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/
Re: OOM with high KahaDB index time
Posted by Fred Moore <fr...@gmail.com>.
Thanks Joe... no I'm still on 5.3 (cannot go 5.4 SNAPSHOT right now), and
wanted to get the best out of the currently available version.
I'll give it a try.
Cheers,
F.
On Mon, Jan 25, 2010 at 4:14 PM, Joe Fernandez <
joe.fernandez@ttmsolutions.com> wrote:
>
> Here it is - its the xml that Dan posted with some slight mod's. Are you
> using the latest trunk and not the Jan 20 5.4 SNAPSHOT?
>
> <beans
> xmlns="http://www.springframework.org/schema/beans"
> xmlns:amq="http://activemq.apache.org/schema/core"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
> http://activemq.apache.org/schema/core
> http://activemq.apache.org/schema/core/activemq-core.xsd">
>
> <bean
>
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> <property name="locations">
> <value>file:${activemq.base}/conf/credentials.properties</value>
> </property>
> </bean>
>
> <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="danbroker" dataDirectory="${activemq.base}/data">
>
> <managementContext>
> <managementContext createConnector="true"/>
> </managementContext>
>
> <persistenceAdapter>
> <kahaDB directory="${activemq.base}/data/kahadb"/>
> </persistenceAdapter>
>
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry queue=">" producerFlowControl="true"
> memoryLimit="10mb">
> </policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
>
>
> <systemUsage>
> <systemUsage sendFailIfNoSpace="true">
> </systemUsage>
> </systemUsage>
>
>
> <transportConnectors>
> <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
> </transportConnectors>
> </broker>
> <import resource="jetty.xml"/>
> </beans>
>
>
>
>
> Fred Moore-3 wrote:
> >
> > Hi Joe,
> >
> > can you post your whole activemq.xml for this "good enough" test on 5.3?
> >
> > ...I'm still missing something and getting OOMs.
> > TIA,
> > F.
> >
> >
> > On Fri, Jan 22, 2010 at 9:07 PM, Joe Fernandez <
> > joe.fernandez@ttmsolutions.com> wrote:
> >
> >>
> >> I ran my 5.3 test with the following
> >>
> >> <systemUsage>
> >> <systemUsage sendFailIfNoSpace="true">
> >> <memoryUsage>
> >> <memoryUsage limit="100mb"/>
> >> </memoryUsage>
> >> <storeUsage>
> >> <storeUsage limit="500 mb" name="foo"/>
> >> </storeUsage>
> >> <tempUsage>
> >> <tempUsage limit="100mb"/>
> >> </tempUsage>
> >> </systemUsage>
> >> </systemUsage>
> >>
> >> ...and had my producer fill up the store; it tried to pump 400k messages
> >> each at 2k in size. Producer flow control was also enabled. The broker
> >> did
> >> not hurl any OOM exceptions, "just
> javax.jms.ResourceAllocationException:
> >> Usage Manager Store is Full. Stopping producer..." exceptions as
> >> expected.
> >> The producer got kicked off its connection.
> >>
> >> The only time I got OOMs was when I used this.
> >>
> >> <systemUsage>
> >> <systemUsage sendFailIfNoSpace="true">
> >> </systemUsage>
> >> </systemUsage>
> >>
> >> Joe
> >>
> >>
> >>
> >> Daniel Kluesing-2 wrote:
> >> >
> >> > So I'm not sure if that's what Rob is talking about being fixed in 5.4
> >> > (And I'll try the snapshot as soon as it's ready) but if I don't have
> >> the
> >> > sendFailIfNoSpace then my understanding is the producers send calls
> >> > block/wait/timeout - as opposed to fail - so it's more difficult to
> get
> >> > into a HA configuration. It's a minor point, not having an OOM is much
> >> > more important, but I definitely want the send calls to fail for the
> >> > producer if the broker ever does anything funny.
> >> >
> >> > Thanks for the feedback on the config, very helpful.
> >> >
> >> > -----Original Message-----
> >> > From: Joe Fernandez [mailto:joe.fernandez@ttmsolutions.com]
> >> > Sent: Thursday, January 21, 2010 1:01 PM
> >> > To: users@activemq.apache.org
> >> > Subject: RE: OOM with high KahaDB index time
> >> >
> >> >
> >> > Just for grins, I threw your cfg file into our 5.3 testbed and sure
> >> > enough,
> >> > we got the OOMs; I pumped 200k messages with each being 2k in size.
> >> FWIW,
> >> > taking this out of the cfg file made things run a lot better.
> >> >
> >> > <systemUsage>
> >> > <systemUsage sendFailIfNoSpace="true">
> >> > </systemUsage>
> >> > </systemUsage>
> >> >
> >> > With the above taken out of the cfg file, I was able to pump 400k
> >> messages
> >> > into the broker, no OOMs and memory utilization looked much better. I
> >> also
> >> > gave a fully-defined systemUsage a try and that also appeared to do
> the
> >> > trick.
> >> >
> >> > <systemUsage>
> >> > <systemUsage>
> >> > <memoryUsage>
> >> > <memoryUsage limit="100 mb"/>
> >> > </memoryUsage>
> >> > <storeUsage>
> >> > <storeUsage limit="1 gb" name="foo"/>
> >> > </storeUsage>
> >> > <tempUsage>
> >> > <tempUsage limit="100 mb"/>
> >> > </tempUsage>
> >> > </systemUsage>
> >> > </systemUsage>
> >> >
> >> > So may be worth giving it a whirl if you can't scoot over to the trunk
> >> and
> >> > ride Rob's patch.
> >> >
> >> > Joe
> >> >
> >> >
> >> > Daniel Kluesing-2 wrote:
> >> >>
> >> >> I tried the suggestion of going with the default cursor, but I still
> >> get
> >> >> OOM errors. I've included my full config file below, I think I'm
> >> running
> >> >> fairly vanilla/default.
> >> >>
> >> >> After about 350k persistent messages, the logs start to look like:
> >> >>
> >> >> INFO | Slow KahaDB access: Journal append took: 10 ms, Index update
> >> took
> >> >> 3118 ms
> >> >> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
> >> took
> >> >> 5118 ms
> >> >> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
> >> took
> >> >> 2736 ms
> >> >> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
> >> took
> >> >> 2945 ms
> >> >> INFO | Slow KahaDB access: Journal append took: 33 ms, Index update
> >> took
> >> >> 2654 ms
> >> >> INFO | Slow KahaDB access: Journal append took: 82 ms, Index update
> >> took
> >> >> 3174 ms
> >> >> INFO | Slow KahaDB access: Journal append took: 1 ms, Index update
> >> took
> >> >> 5891 ms
> >> >> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
> >> took
> >> >> 2906 ms
> >> >> INFO | Slow KahaDB access: Journal append took: 60 ms, Index update
> >> took
> >> >> 7619 ms
> >> >> Exception in thread "InactivityMonitor WriteCheck"
> >> >> java.lang.OutOfMemoryError: Java heap space
> >> >> at java.util.jar.Attributes.read(Attributes.java:377)
> >> >> at java.util.jar.Manifest.read(Manifest.java:182)
> >> >> at java.util.jar.Manifest.<init>(Manifest.java:52)
> >> >> at
> >> >> java.util.jar.JarFile.getManifestFromReference(JarFile.java:165)
> >> >> at java.util.jar.JarFile.getManifest(JarFile.java:146)
> >> >> at
> >> >> sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:693)
> >> >> at
> >> java.net.URLClassLoader.defineClass(URLClassLoader.java:221)
> >> >> at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
> >> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
> >> >> at java.security.AccessController.doPrivileged(Native Method)
> >> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
> >> >> at
> >> java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
> >> >> at
> >> >>
> >>
> org.apache.activemq.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:132)
> >> >> at
> >> >>
> >>
> org.apache.activemq.transport.InactivityMonitor$2.run(InactivityMonitor.java:106)
> >> >> at
> >> >>
> >>
> org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
> >> >> at java.util.TimerThread.mainLoop(Timer.java:512)
> >> >> at java.util.TimerThread.run(Timer.java:462)
> >> >>
> >> >> Config file:
> >> >>
> >> >> <beans
> >> >> xmlns="http://www.springframework.org/schema/beans"
> >> >> xmlns:amq="http://activemq.apache.org/schema/core"
> >> >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> >> >> xsi:schemaLocation="http://www.springframework.org/schema/beans
> >> >> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
> >> >> http://activemq.apache.org/schema/core
> >> >> http://activemq.apache.org/schema/core/activemq-core.xsd">
> >> >>
> >> >> <bean
> >> >>
> >>
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> >> >> <property name="locations">
> >> >>
> >> >> <value>file:${activemq.base}/conf/credentials.properties</value>
> >> >> </property>
> >> >> </bean>
> >> >>
> >> >> <broker xmlns="http://activemq.apache.org/schema/core"
> >> >> brokerName="sub01chi" dataDirectory="${activemq.base}/data">
> >> >>
> >> >> <managementContext>
> >> >> <managementContext createConnector="true"/>
> >> >> </managementContext>
> >> >>
> >> >> <persistenceAdapter>
> >> >> <kahaDB directory="${activemq.base}/data/kahadb"/>
> >> >> </persistenceAdapter>
> >> >>
> >> >> <destinationPolicy>
> >> >> <policyMap>
> >> >> <policyEntries>
> >> >> <policyEntry queue="P>"
> >> producerFlowControl="true"
> >> >> memoryLimit="10mb"></policyEntry>
> >> >> </policyEntries>
> >> >> </policyMap>
> >> >> </destinationPolicy>
> >> >> <systemUsage>
> >> >> <systemUsage sendFailIfNoSpace="true">
> >> >> </systemUsage>
> >> >> </systemUsage>
> >> >> <transportConnectors>
> >> >> <transportConnector name="openwire"
> >> >> uri="tcp://0.0.0.0:61616"/>
> >> >> </transportConnectors>
> >> >> </broker>
> >> >> <import resource="jetty.xml"/>
> >> >> </beans>
> >> >>
> >> >> -----Original Message-----
> >> >> From: Rob Davies [mailto:rajdavies@gmail.com]
> >> >> Sent: Monday, January 18, 2010 10:42 PM
> >> >> To: users@activemq.apache.org
> >> >> Subject: Re: OOM with high KahaDB index time
> >> >>
> >> >>
> >> >> On 18 Jan 2010, at 22:14, Daniel Kluesing wrote:
> >> >>
> >> >>> Hi,
> >> >>>
> >> >>> I'm running the 5.3 release as a standalone broker. In one case, a
> >> >>> producer is running without a consumer, producing small, persistent
> >> >>> messages, with the FileCursor pendingQueuePolicy (per
> >> >>> https://issues.apache.org/activemq/browse/AMQ-2512)
> >> >>> option and flow control memoryLimit set to 100mb for the queue in
> >> >>> question. (Through a policy entry)
> >> >>>
> >> >>> As the queue grows above 300k messages, KahaDB indexing starts
> >> >>> climbing above 1 second. At around 350k messages, the indexing is
> >> >>> taking over 8 seconds. At this point, I start getting java out of
> >> >>> heap space errors in essentially random parts of the code. After a
> >> >>> while, the producers timeout with a channel inactive for too long
> >> >>> error, and the entire broker basically wedges itself. At this point,
> >> >>> consumers are generally unable to bind to the broker quitting with
> >> >>> timeout errors. When they can connect, consuming a single message
> >> >>> triggers an index re-build, which takes 2-8seconds. Turning on
> >> >>> verbose garbage collection, the jvm is collecting like mad but
> >> >>> reclaiming no space.
> >> >>>
> >> >>> If I restart the broker, it comes back up, I can consume the old
> >> >>> messages, and can handle another 350k messages until it wedges.
> >> >>>
> >> >>> I can reproduce under both default gc and incremental gc.
> >> >>>
> >> >>> Two questions:
> >> >>> - It seems like someone is holding onto a handle to the messages
> >> >>> after they have been persisted to disk - is this a known issue?
> >> >>> Should I open a JIRA for it? (Or is there another explanation?)
> >> >>>
> >> >>> - Is there any documentation about the internals of KahaDB - the
> >> >>> kind of indices etc? I'd like to get a better understanding of the
> >> >>> index performance and in general how KahaDB compares to something
> >> >>> like BerkeleyDB.
> >> >>>
> >> >>> Thanks
> >> >>>
> >> >>>
> >> >>>
> >> >>
> >> >>
> >> >> There's is some confusion over naming of our persistence options that
> >> >> doesn't help. There is Kaha - which uses multiple log files and a
> Hash
> >> >> based index - this is currently used by the FileCursor - whilst
> KahaDB
> >> >> is a newer implementation, which is more robust and typically uses a
> >> >> BTreeIndex. There is currently a new implementation of the Filecursor
> >> >> btw - but that's a different matter. You can't currently configure
> the
> >> >> HashIndex via the FileCursor - but it looks like this is the problem
> >> >> you are encountering - as it looks like you need to increase the max
> >> >> hash buckets.
> >> >>
> >> >>
> >> >> So I would recommend the following
> >> >> 1. Use the default pendingQueuePolicy (which only uses a FileCursor
> >> >> for non-persistent messages - and uses the underlying database for
> >> >> persistent messages
> >> >> 2. Try KahaDB - which - with the BTreeIndex - will not hit the
> >> >> problems you are seeing with the Filecursor
> >> >>
> >> >> or - increase the maximum number of hash buckets for the FileCursor
> >> >> index - by setting a Java system property - maximumCapacity to 65536
> >> >> (the default is 16384)
> >> >>
> >> >> cheers,
> >> >>
> >> >> Rob
> >> >>
> >> >> http://twitter.com/rajdavies
> >> >> I work here: http://fusesource.com
> >> >> My Blog: http://rajdavies.blogspot.com/
> >> >> I'm writing this: http://www.manning.com/snyder/
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >
> >> > --
> >> > View this message in context:
> >> >
> >>
> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27264475.html
> >> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >> >
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >>
> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27279301.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27307957.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
Re: OOM with high KahaDB index time
Posted by Joe Fernandez <jo...@ttmsolutions.com>.
Here it is - its the xml that Dan posted with some slight mod's. Are you
using the latest trunk and not the Jan 20 5.4 SNAPSHOT?
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:amq="http://activemq.apache.org/schema/core"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd">
<bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.base}/conf/credentials.properties</value>
</property>
</bean>
<broker xmlns="http://activemq.apache.org/schema/core"
brokerName="danbroker" dataDirectory="${activemq.base}/data">
<managementContext>
<managementContext createConnector="true"/>
</managementContext>
<persistenceAdapter>
<kahaDB directory="${activemq.base}/data/kahadb"/>
</persistenceAdapter>
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry queue=">" producerFlowControl="true"
memoryLimit="10mb">
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<systemUsage>
<systemUsage sendFailIfNoSpace="true">
</systemUsage>
</systemUsage>
<transportConnectors>
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
</transportConnectors>
</broker>
<import resource="jetty.xml"/>
</beans>
Fred Moore-3 wrote:
>
> Hi Joe,
>
> can you post your whole activemq.xml for this "good enough" test on 5.3?
>
> ...I'm still missing something and getting OOMs.
> TIA,
> F.
>
>
> On Fri, Jan 22, 2010 at 9:07 PM, Joe Fernandez <
> joe.fernandez@ttmsolutions.com> wrote:
>
>>
>> I ran my 5.3 test with the following
>>
>> <systemUsage>
>> <systemUsage sendFailIfNoSpace="true">
>> <memoryUsage>
>> <memoryUsage limit="100mb"/>
>> </memoryUsage>
>> <storeUsage>
>> <storeUsage limit="500 mb" name="foo"/>
>> </storeUsage>
>> <tempUsage>
>> <tempUsage limit="100mb"/>
>> </tempUsage>
>> </systemUsage>
>> </systemUsage>
>>
>> ...and had my producer fill up the store; it tried to pump 400k messages
>> each at 2k in size. Producer flow control was also enabled. The broker
>> did
>> not hurl any OOM exceptions, "just javax.jms.ResourceAllocationException:
>> Usage Manager Store is Full. Stopping producer..." exceptions as
>> expected.
>> The producer got kicked off its connection.
>>
>> The only time I got OOMs was when I used this.
>>
>> <systemUsage>
>> <systemUsage sendFailIfNoSpace="true">
>> </systemUsage>
>> </systemUsage>
>>
>> Joe
>>
>>
>>
>> Daniel Kluesing-2 wrote:
>> >
>> > So I'm not sure if that's what Rob is talking about being fixed in 5.4
>> > (And I'll try the snapshot as soon as it's ready) but if I don't have
>> the
>> > sendFailIfNoSpace then my understanding is the producers send calls
>> > block/wait/timeout - as opposed to fail - so it's more difficult to get
>> > into a HA configuration. It's a minor point, not having an OOM is much
>> > more important, but I definitely want the send calls to fail for the
>> > producer if the broker ever does anything funny.
>> >
>> > Thanks for the feedback on the config, very helpful.
>> >
>> > -----Original Message-----
>> > From: Joe Fernandez [mailto:joe.fernandez@ttmsolutions.com]
>> > Sent: Thursday, January 21, 2010 1:01 PM
>> > To: users@activemq.apache.org
>> > Subject: RE: OOM with high KahaDB index time
>> >
>> >
>> > Just for grins, I threw your cfg file into our 5.3 testbed and sure
>> > enough,
>> > we got the OOMs; I pumped 200k messages with each being 2k in size.
>> FWIW,
>> > taking this out of the cfg file made things run a lot better.
>> >
>> > <systemUsage>
>> > <systemUsage sendFailIfNoSpace="true">
>> > </systemUsage>
>> > </systemUsage>
>> >
>> > With the above taken out of the cfg file, I was able to pump 400k
>> messages
>> > into the broker, no OOMs and memory utilization looked much better. I
>> also
>> > gave a fully-defined systemUsage a try and that also appeared to do the
>> > trick.
>> >
>> > <systemUsage>
>> > <systemUsage>
>> > <memoryUsage>
>> > <memoryUsage limit="100 mb"/>
>> > </memoryUsage>
>> > <storeUsage>
>> > <storeUsage limit="1 gb" name="foo"/>
>> > </storeUsage>
>> > <tempUsage>
>> > <tempUsage limit="100 mb"/>
>> > </tempUsage>
>> > </systemUsage>
>> > </systemUsage>
>> >
>> > So may be worth giving it a whirl if you can't scoot over to the trunk
>> and
>> > ride Rob's patch.
>> >
>> > Joe
>> >
>> >
>> > Daniel Kluesing-2 wrote:
>> >>
>> >> I tried the suggestion of going with the default cursor, but I still
>> get
>> >> OOM errors. I've included my full config file below, I think I'm
>> running
>> >> fairly vanilla/default.
>> >>
>> >> After about 350k persistent messages, the logs start to look like:
>> >>
>> >> INFO | Slow KahaDB access: Journal append took: 10 ms, Index update
>> took
>> >> 3118 ms
>> >> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
>> took
>> >> 5118 ms
>> >> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
>> took
>> >> 2736 ms
>> >> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
>> took
>> >> 2945 ms
>> >> INFO | Slow KahaDB access: Journal append took: 33 ms, Index update
>> took
>> >> 2654 ms
>> >> INFO | Slow KahaDB access: Journal append took: 82 ms, Index update
>> took
>> >> 3174 ms
>> >> INFO | Slow KahaDB access: Journal append took: 1 ms, Index update
>> took
>> >> 5891 ms
>> >> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
>> took
>> >> 2906 ms
>> >> INFO | Slow KahaDB access: Journal append took: 60 ms, Index update
>> took
>> >> 7619 ms
>> >> Exception in thread "InactivityMonitor WriteCheck"
>> >> java.lang.OutOfMemoryError: Java heap space
>> >> at java.util.jar.Attributes.read(Attributes.java:377)
>> >> at java.util.jar.Manifest.read(Manifest.java:182)
>> >> at java.util.jar.Manifest.<init>(Manifest.java:52)
>> >> at
>> >> java.util.jar.JarFile.getManifestFromReference(JarFile.java:165)
>> >> at java.util.jar.JarFile.getManifest(JarFile.java:146)
>> >> at
>> >> sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:693)
>> >> at
>> java.net.URLClassLoader.defineClass(URLClassLoader.java:221)
>> >> at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
>> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
>> >> at java.security.AccessController.doPrivileged(Native Method)
>> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
>> >> at
>> java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
>> >> at
>> >>
>> org.apache.activemq.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:132)
>> >> at
>> >>
>> org.apache.activemq.transport.InactivityMonitor$2.run(InactivityMonitor.java:106)
>> >> at
>> >>
>> org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
>> >> at java.util.TimerThread.mainLoop(Timer.java:512)
>> >> at java.util.TimerThread.run(Timer.java:462)
>> >>
>> >> Config file:
>> >>
>> >> <beans
>> >> xmlns="http://www.springframework.org/schema/beans"
>> >> xmlns:amq="http://activemq.apache.org/schema/core"
>> >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> >> xsi:schemaLocation="http://www.springframework.org/schema/beans
>> >> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>> >> http://activemq.apache.org/schema/core
>> >> http://activemq.apache.org/schema/core/activemq-core.xsd">
>> >>
>> >> <bean
>> >>
>> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
>> >> <property name="locations">
>> >>
>> >> <value>file:${activemq.base}/conf/credentials.properties</value>
>> >> </property>
>> >> </bean>
>> >>
>> >> <broker xmlns="http://activemq.apache.org/schema/core"
>> >> brokerName="sub01chi" dataDirectory="${activemq.base}/data">
>> >>
>> >> <managementContext>
>> >> <managementContext createConnector="true"/>
>> >> </managementContext>
>> >>
>> >> <persistenceAdapter>
>> >> <kahaDB directory="${activemq.base}/data/kahadb"/>
>> >> </persistenceAdapter>
>> >>
>> >> <destinationPolicy>
>> >> <policyMap>
>> >> <policyEntries>
>> >> <policyEntry queue="P>"
>> producerFlowControl="true"
>> >> memoryLimit="10mb"></policyEntry>
>> >> </policyEntries>
>> >> </policyMap>
>> >> </destinationPolicy>
>> >> <systemUsage>
>> >> <systemUsage sendFailIfNoSpace="true">
>> >> </systemUsage>
>> >> </systemUsage>
>> >> <transportConnectors>
>> >> <transportConnector name="openwire"
>> >> uri="tcp://0.0.0.0:61616"/>
>> >> </transportConnectors>
>> >> </broker>
>> >> <import resource="jetty.xml"/>
>> >> </beans>
>> >>
>> >> -----Original Message-----
>> >> From: Rob Davies [mailto:rajdavies@gmail.com]
>> >> Sent: Monday, January 18, 2010 10:42 PM
>> >> To: users@activemq.apache.org
>> >> Subject: Re: OOM with high KahaDB index time
>> >>
>> >>
>> >> On 18 Jan 2010, at 22:14, Daniel Kluesing wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> I'm running the 5.3 release as a standalone broker. In one case, a
>> >>> producer is running without a consumer, producing small, persistent
>> >>> messages, with the FileCursor pendingQueuePolicy (per
>> >>> https://issues.apache.org/activemq/browse/AMQ-2512)
>> >>> option and flow control memoryLimit set to 100mb for the queue in
>> >>> question. (Through a policy entry)
>> >>>
>> >>> As the queue grows above 300k messages, KahaDB indexing starts
>> >>> climbing above 1 second. At around 350k messages, the indexing is
>> >>> taking over 8 seconds. At this point, I start getting java out of
>> >>> heap space errors in essentially random parts of the code. After a
>> >>> while, the producers timeout with a channel inactive for too long
>> >>> error, and the entire broker basically wedges itself. At this point,
>> >>> consumers are generally unable to bind to the broker quitting with
>> >>> timeout errors. When they can connect, consuming a single message
>> >>> triggers an index re-build, which takes 2-8seconds. Turning on
>> >>> verbose garbage collection, the jvm is collecting like mad but
>> >>> reclaiming no space.
>> >>>
>> >>> If I restart the broker, it comes back up, I can consume the old
>> >>> messages, and can handle another 350k messages until it wedges.
>> >>>
>> >>> I can reproduce under both default gc and incremental gc.
>> >>>
>> >>> Two questions:
>> >>> - It seems like someone is holding onto a handle to the messages
>> >>> after they have been persisted to disk - is this a known issue?
>> >>> Should I open a JIRA for it? (Or is there another explanation?)
>> >>>
>> >>> - Is there any documentation about the internals of KahaDB - the
>> >>> kind of indices etc? I'd like to get a better understanding of the
>> >>> index performance and in general how KahaDB compares to something
>> >>> like BerkeleyDB.
>> >>>
>> >>> Thanks
>> >>>
>> >>>
>> >>>
>> >>
>> >>
>> >> There's is some confusion over naming of our persistence options that
>> >> doesn't help. There is Kaha - which uses multiple log files and a Hash
>> >> based index - this is currently used by the FileCursor - whilst KahaDB
>> >> is a newer implementation, which is more robust and typically uses a
>> >> BTreeIndex. There is currently a new implementation of the Filecursor
>> >> btw - but that's a different matter. You can't currently configure the
>> >> HashIndex via the FileCursor - but it looks like this is the problem
>> >> you are encountering - as it looks like you need to increase the max
>> >> hash buckets.
>> >>
>> >>
>> >> So I would recommend the following
>> >> 1. Use the default pendingQueuePolicy (which only uses a FileCursor
>> >> for non-persistent messages - and uses the underlying database for
>> >> persistent messages
>> >> 2. Try KahaDB - which - with the BTreeIndex - will not hit the
>> >> problems you are seeing with the Filecursor
>> >>
>> >> or - increase the maximum number of hash buckets for the FileCursor
>> >> index - by setting a Java system property - maximumCapacity to 65536
>> >> (the default is 16384)
>> >>
>> >> cheers,
>> >>
>> >> Rob
>> >>
>> >> http://twitter.com/rajdavies
>> >> I work here: http://fusesource.com
>> >> My Blog: http://rajdavies.blogspot.com/
>> >> I'm writing this: http://www.manning.com/snyder/
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >
>> > --
>> > View this message in context:
>> >
>> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27264475.html
>> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >
>> >
>> >
>>
>> --
>> View this message in context:
>> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27279301.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
>
--
View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27307957.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: OOM with high KahaDB index time
Posted by Fred Moore <fr...@gmail.com>.
Hi Joe,
can you post your whole activemq.xml for this "good enough" test on 5.3?
...I'm still missing something and getting OOMs.
TIA,
F.
On Fri, Jan 22, 2010 at 9:07 PM, Joe Fernandez <
joe.fernandez@ttmsolutions.com> wrote:
>
> I ran my 5.3 test with the following
>
> <systemUsage>
> <systemUsage sendFailIfNoSpace="true">
> <memoryUsage>
> <memoryUsage limit="100mb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="500 mb" name="foo"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="100mb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
>
> ...and had my producer fill up the store; it tried to pump 400k messages
> each at 2k in size. Producer flow control was also enabled. The broker did
> not hurl any OOM exceptions, "just javax.jms.ResourceAllocationException:
> Usage Manager Store is Full. Stopping producer..." exceptions as expected.
> The producer got kicked off its connection.
>
> The only time I got OOMs was when I used this.
>
> <systemUsage>
> <systemUsage sendFailIfNoSpace="true">
> </systemUsage>
> </systemUsage>
>
> Joe
>
>
>
> Daniel Kluesing-2 wrote:
> >
> > So I'm not sure if that's what Rob is talking about being fixed in 5.4
> > (And I'll try the snapshot as soon as it's ready) but if I don't have the
> > sendFailIfNoSpace then my understanding is the producers send calls
> > block/wait/timeout - as opposed to fail - so it's more difficult to get
> > into a HA configuration. It's a minor point, not having an OOM is much
> > more important, but I definitely want the send calls to fail for the
> > producer if the broker ever does anything funny.
> >
> > Thanks for the feedback on the config, very helpful.
> >
> > -----Original Message-----
> > From: Joe Fernandez [mailto:joe.fernandez@ttmsolutions.com]
> > Sent: Thursday, January 21, 2010 1:01 PM
> > To: users@activemq.apache.org
> > Subject: RE: OOM with high KahaDB index time
> >
> >
> > Just for grins, I threw your cfg file into our 5.3 testbed and sure
> > enough,
> > we got the OOMs; I pumped 200k messages with each being 2k in size. FWIW,
> > taking this out of the cfg file made things run a lot better.
> >
> > <systemUsage>
> > <systemUsage sendFailIfNoSpace="true">
> > </systemUsage>
> > </systemUsage>
> >
> > With the above taken out of the cfg file, I was able to pump 400k
> messages
> > into the broker, no OOMs and memory utilization looked much better. I
> also
> > gave a fully-defined systemUsage a try and that also appeared to do the
> > trick.
> >
> > <systemUsage>
> > <systemUsage>
> > <memoryUsage>
> > <memoryUsage limit="100 mb"/>
> > </memoryUsage>
> > <storeUsage>
> > <storeUsage limit="1 gb" name="foo"/>
> > </storeUsage>
> > <tempUsage>
> > <tempUsage limit="100 mb"/>
> > </tempUsage>
> > </systemUsage>
> > </systemUsage>
> >
> > So may be worth giving it a whirl if you can't scoot over to the trunk
> and
> > ride Rob's patch.
> >
> > Joe
> >
> >
> > Daniel Kluesing-2 wrote:
> >>
> >> I tried the suggestion of going with the default cursor, but I still get
> >> OOM errors. I've included my full config file below, I think I'm running
> >> fairly vanilla/default.
> >>
> >> After about 350k persistent messages, the logs start to look like:
> >>
> >> INFO | Slow KahaDB access: Journal append took: 10 ms, Index update took
> >> 3118 ms
> >> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took
> >> 5118 ms
> >> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took
> >> 2736 ms
> >> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took
> >> 2945 ms
> >> INFO | Slow KahaDB access: Journal append took: 33 ms, Index update
> took
> >> 2654 ms
> >> INFO | Slow KahaDB access: Journal append took: 82 ms, Index update
> took
> >> 3174 ms
> >> INFO | Slow KahaDB access: Journal append took: 1 ms, Index update took
> >> 5891 ms
> >> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took
> >> 2906 ms
> >> INFO | Slow KahaDB access: Journal append took: 60 ms, Index update
> took
> >> 7619 ms
> >> Exception in thread "InactivityMonitor WriteCheck"
> >> java.lang.OutOfMemoryError: Java heap space
> >> at java.util.jar.Attributes.read(Attributes.java:377)
> >> at java.util.jar.Manifest.read(Manifest.java:182)
> >> at java.util.jar.Manifest.<init>(Manifest.java:52)
> >> at
> >> java.util.jar.JarFile.getManifestFromReference(JarFile.java:165)
> >> at java.util.jar.JarFile.getManifest(JarFile.java:146)
> >> at
> >> sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:693)
> >> at java.net.URLClassLoader.defineClass(URLClassLoader.java:221)
> >> at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
> >> at java.security.AccessController.doPrivileged(Native Method)
> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
> >> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
> >> at
> >>
> org.apache.activemq.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:132)
> >> at
> >>
> org.apache.activemq.transport.InactivityMonitor$2.run(InactivityMonitor.java:106)
> >> at
> >>
> org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
> >> at java.util.TimerThread.mainLoop(Timer.java:512)
> >> at java.util.TimerThread.run(Timer.java:462)
> >>
> >> Config file:
> >>
> >> <beans
> >> xmlns="http://www.springframework.org/schema/beans"
> >> xmlns:amq="http://activemq.apache.org/schema/core"
> >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> >> xsi:schemaLocation="http://www.springframework.org/schema/beans
> >> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
> >> http://activemq.apache.org/schema/core
> >> http://activemq.apache.org/schema/core/activemq-core.xsd">
> >>
> >> <bean
> >>
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> >> <property name="locations">
> >>
> >> <value>file:${activemq.base}/conf/credentials.properties</value>
> >> </property>
> >> </bean>
> >>
> >> <broker xmlns="http://activemq.apache.org/schema/core"
> >> brokerName="sub01chi" dataDirectory="${activemq.base}/data">
> >>
> >> <managementContext>
> >> <managementContext createConnector="true"/>
> >> </managementContext>
> >>
> >> <persistenceAdapter>
> >> <kahaDB directory="${activemq.base}/data/kahadb"/>
> >> </persistenceAdapter>
> >>
> >> <destinationPolicy>
> >> <policyMap>
> >> <policyEntries>
> >> <policyEntry queue="P>"
> producerFlowControl="true"
> >> memoryLimit="10mb"></policyEntry>
> >> </policyEntries>
> >> </policyMap>
> >> </destinationPolicy>
> >> <systemUsage>
> >> <systemUsage sendFailIfNoSpace="true">
> >> </systemUsage>
> >> </systemUsage>
> >> <transportConnectors>
> >> <transportConnector name="openwire"
> >> uri="tcp://0.0.0.0:61616"/>
> >> </transportConnectors>
> >> </broker>
> >> <import resource="jetty.xml"/>
> >> </beans>
> >>
> >> -----Original Message-----
> >> From: Rob Davies [mailto:rajdavies@gmail.com]
> >> Sent: Monday, January 18, 2010 10:42 PM
> >> To: users@activemq.apache.org
> >> Subject: Re: OOM with high KahaDB index time
> >>
> >>
> >> On 18 Jan 2010, at 22:14, Daniel Kluesing wrote:
> >>
> >>> Hi,
> >>>
> >>> I'm running the 5.3 release as a standalone broker. In one case, a
> >>> producer is running without a consumer, producing small, persistent
> >>> messages, with the FileCursor pendingQueuePolicy (per
> >>> https://issues.apache.org/activemq/browse/AMQ-2512)
> >>> option and flow control memoryLimit set to 100mb for the queue in
> >>> question. (Through a policy entry)
> >>>
> >>> As the queue grows above 300k messages, KahaDB indexing starts
> >>> climbing above 1 second. At around 350k messages, the indexing is
> >>> taking over 8 seconds. At this point, I start getting java out of
> >>> heap space errors in essentially random parts of the code. After a
> >>> while, the producers timeout with a channel inactive for too long
> >>> error, and the entire broker basically wedges itself. At this point,
> >>> consumers are generally unable to bind to the broker quitting with
> >>> timeout errors. When they can connect, consuming a single message
> >>> triggers an index re-build, which takes 2-8seconds. Turning on
> >>> verbose garbage collection, the jvm is collecting like mad but
> >>> reclaiming no space.
> >>>
> >>> If I restart the broker, it comes back up, I can consume the old
> >>> messages, and can handle another 350k messages until it wedges.
> >>>
> >>> I can reproduce under both default gc and incremental gc.
> >>>
> >>> Two questions:
> >>> - It seems like someone is holding onto a handle to the messages
> >>> after they have been persisted to disk - is this a known issue?
> >>> Should I open a JIRA for it? (Or is there another explanation?)
> >>>
> >>> - Is there any documentation about the internals of KahaDB - the
> >>> kind of indices etc? I'd like to get a better understanding of the
> >>> index performance and in general how KahaDB compares to something
> >>> like BerkeleyDB.
> >>>
> >>> Thanks
> >>>
> >>>
> >>>
> >>
> >>
> >> There's is some confusion over naming of our persistence options that
> >> doesn't help. There is Kaha - which uses multiple log files and a Hash
> >> based index - this is currently used by the FileCursor - whilst KahaDB
> >> is a newer implementation, which is more robust and typically uses a
> >> BTreeIndex. There is currently a new implementation of the Filecursor
> >> btw - but that's a different matter. You can't currently configure the
> >> HashIndex via the FileCursor - but it looks like this is the problem
> >> you are encountering - as it looks like you need to increase the max
> >> hash buckets.
> >>
> >>
> >> So I would recommend the following
> >> 1. Use the default pendingQueuePolicy (which only uses a FileCursor
> >> for non-persistent messages - and uses the underlying database for
> >> persistent messages
> >> 2. Try KahaDB - which - with the BTreeIndex - will not hit the
> >> problems you are seeing with the Filecursor
> >>
> >> or - increase the maximum number of hash buckets for the FileCursor
> >> index - by setting a Java system property - maximumCapacity to 65536
> >> (the default is 16384)
> >>
> >> cheers,
> >>
> >> Rob
> >>
> >> http://twitter.com/rajdavies
> >> I work here: http://fusesource.com
> >> My Blog: http://rajdavies.blogspot.com/
> >> I'm writing this: http://www.manning.com/snyder/
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >
> > --
> > View this message in context:
> >
> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27264475.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
> >
> >
>
> --
> View this message in context:
> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27279301.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
RE: OOM with high KahaDB index time
Posted by Joe Fernandez <jo...@ttmsolutions.com>.
I ran my 5.3 test with the following
<systemUsage>
<systemUsage sendFailIfNoSpace="true">
<memoryUsage>
<memoryUsage limit="100mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="500 mb" name="foo"/>
</storeUsage>
<tempUsage>
<tempUsage limit="100mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
...and had my producer fill up the store; it tried to pump 400k messages
each at 2k in size. Producer flow control was also enabled. The broker did
not hurl any OOM exceptions, "just javax.jms.ResourceAllocationException:
Usage Manager Store is Full. Stopping producer..." exceptions as expected.
The producer got kicked off its connection.
The only time I got OOMs was when I used this.
<systemUsage>
<systemUsage sendFailIfNoSpace="true">
</systemUsage>
</systemUsage>
Joe
Daniel Kluesing-2 wrote:
>
> So I'm not sure if that's what Rob is talking about being fixed in 5.4
> (And I'll try the snapshot as soon as it's ready) but if I don't have the
> sendFailIfNoSpace then my understanding is the producers send calls
> block/wait/timeout - as opposed to fail - so it's more difficult to get
> into a HA configuration. It's a minor point, not having an OOM is much
> more important, but I definitely want the send calls to fail for the
> producer if the broker ever does anything funny.
>
> Thanks for the feedback on the config, very helpful.
>
> -----Original Message-----
> From: Joe Fernandez [mailto:joe.fernandez@ttmsolutions.com]
> Sent: Thursday, January 21, 2010 1:01 PM
> To: users@activemq.apache.org
> Subject: RE: OOM with high KahaDB index time
>
>
> Just for grins, I threw your cfg file into our 5.3 testbed and sure
> enough,
> we got the OOMs; I pumped 200k messages with each being 2k in size. FWIW,
> taking this out of the cfg file made things run a lot better.
>
> <systemUsage>
> <systemUsage sendFailIfNoSpace="true">
> </systemUsage>
> </systemUsage>
>
> With the above taken out of the cfg file, I was able to pump 400k messages
> into the broker, no OOMs and memory utilization looked much better. I also
> gave a fully-defined systemUsage a try and that also appeared to do the
> trick.
>
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="100 mb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="1 gb" name="foo"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="100 mb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
>
> So may be worth giving it a whirl if you can't scoot over to the trunk and
> ride Rob's patch.
>
> Joe
>
>
> Daniel Kluesing-2 wrote:
>>
>> I tried the suggestion of going with the default cursor, but I still get
>> OOM errors. I've included my full config file below, I think I'm running
>> fairly vanilla/default.
>>
>> After about 350k persistent messages, the logs start to look like:
>>
>> INFO | Slow KahaDB access: Journal append took: 10 ms, Index update took
>> 3118 ms
>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took
>> 5118 ms
>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took
>> 2736 ms
>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took
>> 2945 ms
>> INFO | Slow KahaDB access: Journal append took: 33 ms, Index update took
>> 2654 ms
>> INFO | Slow KahaDB access: Journal append took: 82 ms, Index update took
>> 3174 ms
>> INFO | Slow KahaDB access: Journal append took: 1 ms, Index update took
>> 5891 ms
>> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took
>> 2906 ms
>> INFO | Slow KahaDB access: Journal append took: 60 ms, Index update took
>> 7619 ms
>> Exception in thread "InactivityMonitor WriteCheck"
>> java.lang.OutOfMemoryError: Java heap space
>> at java.util.jar.Attributes.read(Attributes.java:377)
>> at java.util.jar.Manifest.read(Manifest.java:182)
>> at java.util.jar.Manifest.<init>(Manifest.java:52)
>> at
>> java.util.jar.JarFile.getManifestFromReference(JarFile.java:165)
>> at java.util.jar.JarFile.getManifest(JarFile.java:146)
>> at
>> sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:693)
>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:221)
>> at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
>> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
>> at
>> org.apache.activemq.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:132)
>> at
>> org.apache.activemq.transport.InactivityMonitor$2.run(InactivityMonitor.java:106)
>> at
>> org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
>> at java.util.TimerThread.mainLoop(Timer.java:512)
>> at java.util.TimerThread.run(Timer.java:462)
>>
>> Config file:
>>
>> <beans
>> xmlns="http://www.springframework.org/schema/beans"
>> xmlns:amq="http://activemq.apache.org/schema/core"
>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>> http://activemq.apache.org/schema/core
>> http://activemq.apache.org/schema/core/activemq-core.xsd">
>>
>> <bean
>> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
>> <property name="locations">
>>
>> <value>file:${activemq.base}/conf/credentials.properties</value>
>> </property>
>> </bean>
>>
>> <broker xmlns="http://activemq.apache.org/schema/core"
>> brokerName="sub01chi" dataDirectory="${activemq.base}/data">
>>
>> <managementContext>
>> <managementContext createConnector="true"/>
>> </managementContext>
>>
>> <persistenceAdapter>
>> <kahaDB directory="${activemq.base}/data/kahadb"/>
>> </persistenceAdapter>
>>
>> <destinationPolicy>
>> <policyMap>
>> <policyEntries>
>> <policyEntry queue="P>" producerFlowControl="true"
>> memoryLimit="10mb"></policyEntry>
>> </policyEntries>
>> </policyMap>
>> </destinationPolicy>
>> <systemUsage>
>> <systemUsage sendFailIfNoSpace="true">
>> </systemUsage>
>> </systemUsage>
>> <transportConnectors>
>> <transportConnector name="openwire"
>> uri="tcp://0.0.0.0:61616"/>
>> </transportConnectors>
>> </broker>
>> <import resource="jetty.xml"/>
>> </beans>
>>
>> -----Original Message-----
>> From: Rob Davies [mailto:rajdavies@gmail.com]
>> Sent: Monday, January 18, 2010 10:42 PM
>> To: users@activemq.apache.org
>> Subject: Re: OOM with high KahaDB index time
>>
>>
>> On 18 Jan 2010, at 22:14, Daniel Kluesing wrote:
>>
>>> Hi,
>>>
>>> I'm running the 5.3 release as a standalone broker. In one case, a
>>> producer is running without a consumer, producing small, persistent
>>> messages, with the FileCursor pendingQueuePolicy (per
>>> https://issues.apache.org/activemq/browse/AMQ-2512)
>>> option and flow control memoryLimit set to 100mb for the queue in
>>> question. (Through a policy entry)
>>>
>>> As the queue grows above 300k messages, KahaDB indexing starts
>>> climbing above 1 second. At around 350k messages, the indexing is
>>> taking over 8 seconds. At this point, I start getting java out of
>>> heap space errors in essentially random parts of the code. After a
>>> while, the producers timeout with a channel inactive for too long
>>> error, and the entire broker basically wedges itself. At this point,
>>> consumers are generally unable to bind to the broker quitting with
>>> timeout errors. When they can connect, consuming a single message
>>> triggers an index re-build, which takes 2-8seconds. Turning on
>>> verbose garbage collection, the jvm is collecting like mad but
>>> reclaiming no space.
>>>
>>> If I restart the broker, it comes back up, I can consume the old
>>> messages, and can handle another 350k messages until it wedges.
>>>
>>> I can reproduce under both default gc and incremental gc.
>>>
>>> Two questions:
>>> - It seems like someone is holding onto a handle to the messages
>>> after they have been persisted to disk - is this a known issue?
>>> Should I open a JIRA for it? (Or is there another explanation?)
>>>
>>> - Is there any documentation about the internals of KahaDB - the
>>> kind of indices etc? I'd like to get a better understanding of the
>>> index performance and in general how KahaDB compares to something
>>> like BerkeleyDB.
>>>
>>> Thanks
>>>
>>>
>>>
>>
>>
>> There's is some confusion over naming of our persistence options that
>> doesn't help. There is Kaha - which uses multiple log files and a Hash
>> based index - this is currently used by the FileCursor - whilst KahaDB
>> is a newer implementation, which is more robust and typically uses a
>> BTreeIndex. There is currently a new implementation of the Filecursor
>> btw - but that's a different matter. You can't currently configure the
>> HashIndex via the FileCursor - but it looks like this is the problem
>> you are encountering - as it looks like you need to increase the max
>> hash buckets.
>>
>>
>> So I would recommend the following
>> 1. Use the default pendingQueuePolicy (which only uses a FileCursor
>> for non-persistent messages - and uses the underlying database for
>> persistent messages
>> 2. Try KahaDB - which - with the BTreeIndex - will not hit the
>> problems you are seeing with the Filecursor
>>
>> or - increase the maximum number of hash buckets for the FileCursor
>> index - by setting a Java system property - maximumCapacity to 65536
>> (the default is 16384)
>>
>> cheers,
>>
>> Rob
>>
>> http://twitter.com/rajdavies
>> I work here: http://fusesource.com
>> My Blog: http://rajdavies.blogspot.com/
>> I'm writing this: http://www.manning.com/snyder/
>>
>>
>>
>>
>>
>>
>>
>
> --
> View this message in context:
> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27264475.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
>
--
View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27279301.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
RE: OOM with high KahaDB index time
Posted by Daniel Kluesing <dk...@bluekai.com>.
So I'm not sure if that's what Rob is talking about being fixed in 5.4 (And I'll try the snapshot as soon as it's ready) but if I don't have the sendFailIfNoSpace then my understanding is the producers send calls block/wait/timeout - as opposed to fail - so it's more difficult to get into a HA configuration. It's a minor point, not having an OOM is much more important, but I definitely want the send calls to fail for the producer if the broker ever does anything funny.
Thanks for the feedback on the config, very helpful.
-----Original Message-----
From: Joe Fernandez [mailto:joe.fernandez@ttmsolutions.com]
Sent: Thursday, January 21, 2010 1:01 PM
To: users@activemq.apache.org
Subject: RE: OOM with high KahaDB index time
Just for grins, I threw your cfg file into our 5.3 testbed and sure enough,
we got the OOMs; I pumped 200k messages with each being 2k in size. FWIW,
taking this out of the cfg file made things run a lot better.
<systemUsage>
<systemUsage sendFailIfNoSpace="true">
</systemUsage>
</systemUsage>
With the above taken out of the cfg file, I was able to pump 400k messages
into the broker, no OOMs and memory utilization looked much better. I also
gave a fully-defined systemUsage a try and that also appeared to do the
trick.
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="100 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="1 gb" name="foo"/>
</storeUsage>
<tempUsage>
<tempUsage limit="100 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
So may be worth giving it a whirl if you can't scoot over to the trunk and
ride Rob's patch.
Joe
Daniel Kluesing-2 wrote:
>
> I tried the suggestion of going with the default cursor, but I still get
> OOM errors. I've included my full config file below, I think I'm running
> fairly vanilla/default.
>
> After about 350k persistent messages, the logs start to look like:
>
> INFO | Slow KahaDB access: Journal append took: 10 ms, Index update took
> 3118 ms
> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took
> 5118 ms
> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took
> 2736 ms
> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took
> 2945 ms
> INFO | Slow KahaDB access: Journal append took: 33 ms, Index update took
> 2654 ms
> INFO | Slow KahaDB access: Journal append took: 82 ms, Index update took
> 3174 ms
> INFO | Slow KahaDB access: Journal append took: 1 ms, Index update took
> 5891 ms
> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took
> 2906 ms
> INFO | Slow KahaDB access: Journal append took: 60 ms, Index update took
> 7619 ms
> Exception in thread "InactivityMonitor WriteCheck"
> java.lang.OutOfMemoryError: Java heap space
> at java.util.jar.Attributes.read(Attributes.java:377)
> at java.util.jar.Manifest.read(Manifest.java:182)
> at java.util.jar.Manifest.<init>(Manifest.java:52)
> at
> java.util.jar.JarFile.getManifestFromReference(JarFile.java:165)
> at java.util.jar.JarFile.getManifest(JarFile.java:146)
> at
> sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:693)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:221)
> at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
> at
> org.apache.activemq.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:132)
> at
> org.apache.activemq.transport.InactivityMonitor$2.run(InactivityMonitor.java:106)
> at
> org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
> at java.util.TimerThread.mainLoop(Timer.java:512)
> at java.util.TimerThread.run(Timer.java:462)
>
> Config file:
>
> <beans
> xmlns="http://www.springframework.org/schema/beans"
> xmlns:amq="http://activemq.apache.org/schema/core"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
> http://activemq.apache.org/schema/core
> http://activemq.apache.org/schema/core/activemq-core.xsd">
>
> <bean
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> <property name="locations">
>
> <value>file:${activemq.base}/conf/credentials.properties</value>
> </property>
> </bean>
>
> <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="sub01chi" dataDirectory="${activemq.base}/data">
>
> <managementContext>
> <managementContext createConnector="true"/>
> </managementContext>
>
> <persistenceAdapter>
> <kahaDB directory="${activemq.base}/data/kahadb"/>
> </persistenceAdapter>
>
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry queue="P>" producerFlowControl="true"
> memoryLimit="10mb"></policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
> <systemUsage>
> <systemUsage sendFailIfNoSpace="true">
> </systemUsage>
> </systemUsage>
> <transportConnectors>
> <transportConnector name="openwire"
> uri="tcp://0.0.0.0:61616"/>
> </transportConnectors>
> </broker>
> <import resource="jetty.xml"/>
> </beans>
>
> -----Original Message-----
> From: Rob Davies [mailto:rajdavies@gmail.com]
> Sent: Monday, January 18, 2010 10:42 PM
> To: users@activemq.apache.org
> Subject: Re: OOM with high KahaDB index time
>
>
> On 18 Jan 2010, at 22:14, Daniel Kluesing wrote:
>
>> Hi,
>>
>> I'm running the 5.3 release as a standalone broker. In one case, a
>> producer is running without a consumer, producing small, persistent
>> messages, with the FileCursor pendingQueuePolicy (per
>> https://issues.apache.org/activemq/browse/AMQ-2512)
>> option and flow control memoryLimit set to 100mb for the queue in
>> question. (Through a policy entry)
>>
>> As the queue grows above 300k messages, KahaDB indexing starts
>> climbing above 1 second. At around 350k messages, the indexing is
>> taking over 8 seconds. At this point, I start getting java out of
>> heap space errors in essentially random parts of the code. After a
>> while, the producers timeout with a channel inactive for too long
>> error, and the entire broker basically wedges itself. At this point,
>> consumers are generally unable to bind to the broker quitting with
>> timeout errors. When they can connect, consuming a single message
>> triggers an index re-build, which takes 2-8seconds. Turning on
>> verbose garbage collection, the jvm is collecting like mad but
>> reclaiming no space.
>>
>> If I restart the broker, it comes back up, I can consume the old
>> messages, and can handle another 350k messages until it wedges.
>>
>> I can reproduce under both default gc and incremental gc.
>>
>> Two questions:
>> - It seems like someone is holding onto a handle to the messages
>> after they have been persisted to disk - is this a known issue?
>> Should I open a JIRA for it? (Or is there another explanation?)
>>
>> - Is there any documentation about the internals of KahaDB - the
>> kind of indices etc? I'd like to get a better understanding of the
>> index performance and in general how KahaDB compares to something
>> like BerkeleyDB.
>>
>> Thanks
>>
>>
>>
>
>
> There's is some confusion over naming of our persistence options that
> doesn't help. There is Kaha - which uses multiple log files and a Hash
> based index - this is currently used by the FileCursor - whilst KahaDB
> is a newer implementation, which is more robust and typically uses a
> BTreeIndex. There is currently a new implementation of the Filecursor
> btw - but that's a different matter. You can't currently configure the
> HashIndex via the FileCursor - but it looks like this is the problem
> you are encountering - as it looks like you need to increase the max
> hash buckets.
>
>
> So I would recommend the following
> 1. Use the default pendingQueuePolicy (which only uses a FileCursor
> for non-persistent messages - and uses the underlying database for
> persistent messages
> 2. Try KahaDB - which - with the BTreeIndex - will not hit the
> problems you are seeing with the Filecursor
>
> or - increase the maximum number of hash buckets for the FileCursor
> index - by setting a Java system property - maximumCapacity to 65536
> (the default is 16384)
>
> cheers,
>
> Rob
>
> http://twitter.com/rajdavies
> I work here: http://fusesource.com
> My Blog: http://rajdavies.blogspot.com/
> I'm writing this: http://www.manning.com/snyder/
>
>
>
>
>
>
>
--
View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27264475.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
RE: OOM with high KahaDB index time
Posted by Joe Fernandez <jo...@ttmsolutions.com>.
Just for grins, I threw your cfg file into our 5.3 testbed and sure enough,
we got the OOMs; I pumped 200k messages with each being 2k in size. FWIW,
taking this out of the cfg file made things run a lot better.
<systemUsage>
<systemUsage sendFailIfNoSpace="true">
</systemUsage>
</systemUsage>
With the above taken out of the cfg file, I was able to pump 400k messages
into the broker, no OOMs and memory utilization looked much better. I also
gave a fully-defined systemUsage a try and that also appeared to do the
trick.
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="100 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="1 gb" name="foo"/>
</storeUsage>
<tempUsage>
<tempUsage limit="100 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
So may be worth giving it a whirl if you can't scoot over to the trunk and
ride Rob's patch.
Joe
Daniel Kluesing-2 wrote:
>
> I tried the suggestion of going with the default cursor, but I still get
> OOM errors. I've included my full config file below, I think I'm running
> fairly vanilla/default.
>
> After about 350k persistent messages, the logs start to look like:
>
> INFO | Slow KahaDB access: Journal append took: 10 ms, Index update took
> 3118 ms
> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took
> 5118 ms
> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took
> 2736 ms
> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took
> 2945 ms
> INFO | Slow KahaDB access: Journal append took: 33 ms, Index update took
> 2654 ms
> INFO | Slow KahaDB access: Journal append took: 82 ms, Index update took
> 3174 ms
> INFO | Slow KahaDB access: Journal append took: 1 ms, Index update took
> 5891 ms
> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took
> 2906 ms
> INFO | Slow KahaDB access: Journal append took: 60 ms, Index update took
> 7619 ms
> Exception in thread "InactivityMonitor WriteCheck"
> java.lang.OutOfMemoryError: Java heap space
> at java.util.jar.Attributes.read(Attributes.java:377)
> at java.util.jar.Manifest.read(Manifest.java:182)
> at java.util.jar.Manifest.<init>(Manifest.java:52)
> at
> java.util.jar.JarFile.getManifestFromReference(JarFile.java:165)
> at java.util.jar.JarFile.getManifest(JarFile.java:146)
> at
> sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:693)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:221)
> at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
> at
> org.apache.activemq.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:132)
> at
> org.apache.activemq.transport.InactivityMonitor$2.run(InactivityMonitor.java:106)
> at
> org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
> at java.util.TimerThread.mainLoop(Timer.java:512)
> at java.util.TimerThread.run(Timer.java:462)
>
> Config file:
>
> <beans
> xmlns="http://www.springframework.org/schema/beans"
> xmlns:amq="http://activemq.apache.org/schema/core"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
> http://activemq.apache.org/schema/core
> http://activemq.apache.org/schema/core/activemq-core.xsd">
>
> <bean
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> <property name="locations">
>
> <value>file:${activemq.base}/conf/credentials.properties</value>
> </property>
> </bean>
>
> <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="sub01chi" dataDirectory="${activemq.base}/data">
>
> <managementContext>
> <managementContext createConnector="true"/>
> </managementContext>
>
> <persistenceAdapter>
> <kahaDB directory="${activemq.base}/data/kahadb"/>
> </persistenceAdapter>
>
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry queue="P>" producerFlowControl="true"
> memoryLimit="10mb"></policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
> <systemUsage>
> <systemUsage sendFailIfNoSpace="true">
> </systemUsage>
> </systemUsage>
> <transportConnectors>
> <transportConnector name="openwire"
> uri="tcp://0.0.0.0:61616"/>
> </transportConnectors>
> </broker>
> <import resource="jetty.xml"/>
> </beans>
>
> -----Original Message-----
> From: Rob Davies [mailto:rajdavies@gmail.com]
> Sent: Monday, January 18, 2010 10:42 PM
> To: users@activemq.apache.org
> Subject: Re: OOM with high KahaDB index time
>
>
> On 18 Jan 2010, at 22:14, Daniel Kluesing wrote:
>
>> Hi,
>>
>> I'm running the 5.3 release as a standalone broker. In one case, a
>> producer is running without a consumer, producing small, persistent
>> messages, with the FileCursor pendingQueuePolicy (per
>> https://issues.apache.org/activemq/browse/AMQ-2512)
>> option and flow control memoryLimit set to 100mb for the queue in
>> question. (Through a policy entry)
>>
>> As the queue grows above 300k messages, KahaDB indexing starts
>> climbing above 1 second. At around 350k messages, the indexing is
>> taking over 8 seconds. At this point, I start getting java out of
>> heap space errors in essentially random parts of the code. After a
>> while, the producers timeout with a channel inactive for too long
>> error, and the entire broker basically wedges itself. At this point,
>> consumers are generally unable to bind to the broker quitting with
>> timeout errors. When they can connect, consuming a single message
>> triggers an index re-build, which takes 2-8seconds. Turning on
>> verbose garbage collection, the jvm is collecting like mad but
>> reclaiming no space.
>>
>> If I restart the broker, it comes back up, I can consume the old
>> messages, and can handle another 350k messages until it wedges.
>>
>> I can reproduce under both default gc and incremental gc.
>>
>> Two questions:
>> - It seems like someone is holding onto a handle to the messages
>> after they have been persisted to disk - is this a known issue?
>> Should I open a JIRA for it? (Or is there another explanation?)
>>
>> - Is there any documentation about the internals of KahaDB - the
>> kind of indices etc? I'd like to get a better understanding of the
>> index performance and in general how KahaDB compares to something
>> like BerkeleyDB.
>>
>> Thanks
>>
>>
>>
>
>
> There's is some confusion over naming of our persistence options that
> doesn't help. There is Kaha - which uses multiple log files and a Hash
> based index - this is currently used by the FileCursor - whilst KahaDB
> is a newer implementation, which is more robust and typically uses a
> BTreeIndex. There is currently a new implementation of the Filecursor
> btw - but that's a different matter. You can't currently configure the
> HashIndex via the FileCursor - but it looks like this is the problem
> you are encountering - as it looks like you need to increase the max
> hash buckets.
>
>
> So I would recommend the following
> 1. Use the default pendingQueuePolicy (which only uses a FileCursor
> for non-persistent messages - and uses the underlying database for
> persistent messages
> 2. Try KahaDB - which - with the BTreeIndex - will not hit the
> problems you are seeing with the Filecursor
>
> or - increase the maximum number of hash buckets for the FileCursor
> index - by setting a Java system property - maximumCapacity to 65536
> (the default is 16384)
>
> cheers,
>
> Rob
>
> http://twitter.com/rajdavies
> I work here: http://fusesource.com
> My Blog: http://rajdavies.blogspot.com/
> I'm writing this: http://www.manning.com/snyder/
>
>
>
>
>
>
>
--
View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27264475.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
RE: OOM with high KahaDB index time
Posted by Joe Fernandez <jo...@ttmsolutions.com>.
To get rid of the INFO messages, set
org.apache.activemq.store.kahadb.LOG_SLOW_ACCESS_TIME to something like 8000
What message size are you using in your tests?
Joe
http://www.ttmsolutions.com
Daniel Kluesing-2 wrote:
>
> I tried the suggestion of going with the default cursor, but I still get
> OOM errors. I've included my full config file below, I think I'm running
> fairly vanilla/default.
>
> After about 350k persistent messages, the logs start to look like:
>
> INFO | Slow KahaDB access: Journal append took: 10 ms, Index update took
> 3118 ms
> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took
> 5118 ms
> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took
> 2736 ms
> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took
> 2945 ms
> INFO | Slow KahaDB access: Journal append took: 33 ms, Index update took
> 2654 ms
> INFO | Slow KahaDB access: Journal append took: 82 ms, Index update took
> 3174 ms
> INFO | Slow KahaDB access: Journal append took: 1 ms, Index update took
> 5891 ms
> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took
> 2906 ms
> INFO | Slow KahaDB access: Journal append took: 60 ms, Index update took
> 7619 ms
> Exception in thread "InactivityMonitor WriteCheck"
> java.lang.OutOfMemoryError: Java heap space
> at java.util.jar.Attributes.read(Attributes.java:377)
> at java.util.jar.Manifest.read(Manifest.java:182)
> at java.util.jar.Manifest.<init>(Manifest.java:52)
> at
> java.util.jar.JarFile.getManifestFromReference(JarFile.java:165)
> at java.util.jar.JarFile.getManifest(JarFile.java:146)
> at
> sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:693)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:221)
> at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
> at
> org.apache.activemq.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:132)
> at
> org.apache.activemq.transport.InactivityMonitor$2.run(InactivityMonitor.java:106)
> at
> org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
> at java.util.TimerThread.mainLoop(Timer.java:512)
> at java.util.TimerThread.run(Timer.java:462)
>
> Config file:
>
> <beans
> xmlns="http://www.springframework.org/schema/beans"
> xmlns:amq="http://activemq.apache.org/schema/core"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
> http://activemq.apache.org/schema/core
> http://activemq.apache.org/schema/core/activemq-core.xsd">
>
> <bean
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> <property name="locations">
>
> <value>file:${activemq.base}/conf/credentials.properties</value>
> </property>
> </bean>
>
> <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="sub01chi" dataDirectory="${activemq.base}/data">
>
> <managementContext>
> <managementContext createConnector="true"/>
> </managementContext>
>
> <persistenceAdapter>
> <kahaDB directory="${activemq.base}/data/kahadb"/>
> </persistenceAdapter>
>
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry queue="P>" producerFlowControl="true"
> memoryLimit="10mb"></policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
> <systemUsage>
> <systemUsage sendFailIfNoSpace="true">
> </systemUsage>
> </systemUsage>
> <transportConnectors>
> <transportConnector name="openwire"
> uri="tcp://0.0.0.0:61616"/>
> </transportConnectors>
> </broker>
> <import resource="jetty.xml"/>
> </beans>
>
> -----Original Message-----
> From: Rob Davies [mailto:rajdavies@gmail.com]
> Sent: Monday, January 18, 2010 10:42 PM
> To: users@activemq.apache.org
> Subject: Re: OOM with high KahaDB index time
>
>
> On 18 Jan 2010, at 22:14, Daniel Kluesing wrote:
>
>> Hi,
>>
>> I'm running the 5.3 release as a standalone broker. In one case, a
>> producer is running without a consumer, producing small, persistent
>> messages, with the FileCursor pendingQueuePolicy (per
>> https://issues.apache.org/activemq/browse/AMQ-2512)
>> option and flow control memoryLimit set to 100mb for the queue in
>> question. (Through a policy entry)
>>
>> As the queue grows above 300k messages, KahaDB indexing starts
>> climbing above 1 second. At around 350k messages, the indexing is
>> taking over 8 seconds. At this point, I start getting java out of
>> heap space errors in essentially random parts of the code. After a
>> while, the producers timeout with a channel inactive for too long
>> error, and the entire broker basically wedges itself. At this point,
>> consumers are generally unable to bind to the broker quitting with
>> timeout errors. When they can connect, consuming a single message
>> triggers an index re-build, which takes 2-8seconds. Turning on
>> verbose garbage collection, the jvm is collecting like mad but
>> reclaiming no space.
>>
>> If I restart the broker, it comes back up, I can consume the old
>> messages, and can handle another 350k messages until it wedges.
>>
>> I can reproduce under both default gc and incremental gc.
>>
>> Two questions:
>> - It seems like someone is holding onto a handle to the messages
>> after they have been persisted to disk - is this a known issue?
>> Should I open a JIRA for it? (Or is there another explanation?)
>>
>> - Is there any documentation about the internals of KahaDB - the
>> kind of indices etc? I'd like to get a better understanding of the
>> index performance and in general how KahaDB compares to something
>> like BerkeleyDB.
>>
>> Thanks
>>
>>
>>
>
>
> There's is some confusion over naming of our persistence options that
> doesn't help. There is Kaha - which uses multiple log files and a Hash
> based index - this is currently used by the FileCursor - whilst KahaDB
> is a newer implementation, which is more robust and typically uses a
> BTreeIndex. There is currently a new implementation of the Filecursor
> btw - but that's a different matter. You can't currently configure the
> HashIndex via the FileCursor - but it looks like this is the problem
> you are encountering - as it looks like you need to increase the max
> hash buckets.
>
>
> So I would recommend the following
> 1. Use the default pendingQueuePolicy (which only uses a FileCursor
> for non-persistent messages - and uses the underlying database for
> persistent messages
> 2. Try KahaDB - which - with the BTreeIndex - will not hit the
> problems you are seeing with the Filecursor
>
> or - increase the maximum number of hash buckets for the FileCursor
> index - by setting a Java system property - maximumCapacity to 65536
> (the default is 16384)
>
> cheers,
>
> Rob
>
> http://twitter.com/rajdavies
> I work here: http://fusesource.com
> My Blog: http://rajdavies.blogspot.com/
> I'm writing this: http://www.manning.com/snyder/
>
>
>
>
>
>
>
--
View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27263696.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: OOM with high KahaDB index time
Posted by Rob Davies <ra...@gmail.com>.
Hi Daniel,
there's a potential cut for this in trunk - the test case passes - but
would like others to try it out before closing the issue.
Unfortunately latest 5.4-SNAPSHOT hasn't been built yet - but when it
is - please give it a try
thanks,
Rob
On 21 Jan 2010, at 19:45, Daniel Kluesing wrote:
> I tried the suggestion of going with the default cursor, but I still
> get OOM errors. I've included my full config file below, I think I'm
> running fairly vanilla/default.
>
> After about 350k persistent messages, the logs start to look like:
>
> INFO | Slow KahaDB access: Journal append took: 10 ms, Index update
> took 3118 ms
> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
> took 5118 ms
> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
> took 2736 ms
> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
> took 2945 ms
> INFO | Slow KahaDB access: Journal append took: 33 ms, Index update
> took 2654 ms
> INFO | Slow KahaDB access: Journal append took: 82 ms, Index update
> took 3174 ms
> INFO | Slow KahaDB access: Journal append took: 1 ms, Index update
> took 5891 ms
> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
> took 2906 ms
> INFO | Slow KahaDB access: Journal append took: 60 ms, Index update
> took 7619 ms
> Exception in thread "InactivityMonitor WriteCheck"
> java.lang.OutOfMemoryError: Java heap space
> at java.util.jar.Attributes.read(Attributes.java:377)
> at java.util.jar.Manifest.read(Manifest.java:182)
> at java.util.jar.Manifest.<init>(Manifest.java:52)
> at
> java.util.jar.JarFile.getManifestFromReference(JarFile.java:165)
> at java.util.jar.JarFile.getManifest(JarFile.java:146)
> at sun.misc.URLClassPath$JarLoader
> $2.getManifest(URLClassPath.java:693)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:221)
> at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:
> 319)
> at
> org
> .apache
> .activemq
> .transport.InactivityMonitor.writeCheck(InactivityMonitor.java:132)
> at org.apache.activemq.transport.InactivityMonitor
> $2.run(InactivityMonitor.java:106)
> at
> org
> .apache
> .activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
> at java.util.TimerThread.mainLoop(Timer.java:512)
> at java.util.TimerThread.run(Timer.java:462)
>
> Config file:
>
> <beans
> xmlns="http://www.springframework.org/schema/beans"
> xmlns:amq="http://activemq.apache.org/schema/core"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
> http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd
> ">
>
> <bean
> class
> =
> "org
> .springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> <property name="locations">
> <value>file:${activemq.base}/conf/credentials.properties</
> value>
> </property>
> </bean>
>
> <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="sub01chi" dataDirectory="${activemq.base}/data">
>
> <managementContext>
> <managementContext createConnector="true"/>
> </managementContext>
>
> <persistenceAdapter>
> <kahaDB directory="${activemq.base}/data/kahadb"/>
> </persistenceAdapter>
>
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry queue="P>" producerFlowControl="true"
> memoryLimit="10mb"></policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
> <systemUsage>
> <systemUsage sendFailIfNoSpace="true">
> </systemUsage>
> </systemUsage>
> <transportConnectors>
> <transportConnector name="openwire" uri="tcp://
> 0.0.0.0:61616"/>
> </transportConnectors>
> </broker>
> <import resource="jetty.xml"/>
> </beans>
>
> -----Original Message-----
> From: Rob Davies [mailto:rajdavies@gmail.com]
> Sent: Monday, January 18, 2010 10:42 PM
> To: users@activemq.apache.org
> Subject: Re: OOM with high KahaDB index time
>
>
> On 18 Jan 2010, at 22:14, Daniel Kluesing wrote:
>
>> Hi,
>>
>> I'm running the 5.3 release as a standalone broker. In one case, a
>> producer is running without a consumer, producing small, persistent
>> messages, with the FileCursor pendingQueuePolicy (per https://issues.apache.org/activemq/browse/AMQ-2512)
>> option and flow control memoryLimit set to 100mb for the queue in
>> question. (Through a policy entry)
>>
>> As the queue grows above 300k messages, KahaDB indexing starts
>> climbing above 1 second. At around 350k messages, the indexing is
>> taking over 8 seconds. At this point, I start getting java out of
>> heap space errors in essentially random parts of the code. After a
>> while, the producers timeout with a channel inactive for too long
>> error, and the entire broker basically wedges itself. At this point,
>> consumers are generally unable to bind to the broker quitting with
>> timeout errors. When they can connect, consuming a single message
>> triggers an index re-build, which takes 2-8seconds. Turning on
>> verbose garbage collection, the jvm is collecting like mad but
>> reclaiming no space.
>>
>> If I restart the broker, it comes back up, I can consume the old
>> messages, and can handle another 350k messages until it wedges.
>>
>> I can reproduce under both default gc and incremental gc.
>>
>> Two questions:
>> - It seems like someone is holding onto a handle to the messages
>> after they have been persisted to disk - is this a known issue?
>> Should I open a JIRA for it? (Or is there another explanation?)
>>
>> - Is there any documentation about the internals of KahaDB - the
>> kind of indices etc? I'd like to get a better understanding of the
>> index performance and in general how KahaDB compares to something
>> like BerkeleyDB.
>>
>> Thanks
>>
>>
>>
>
>
> There's is some confusion over naming of our persistence options that
> doesn't help. There is Kaha - which uses multiple log files and a Hash
> based index - this is currently used by the FileCursor - whilst KahaDB
> is a newer implementation, which is more robust and typically uses a
> BTreeIndex. There is currently a new implementation of the Filecursor
> btw - but that's a different matter. You can't currently configure the
> HashIndex via the FileCursor - but it looks like this is the problem
> you are encountering - as it looks like you need to increase the max
> hash buckets.
>
>
> So I would recommend the following
> 1. Use the default pendingQueuePolicy (which only uses a FileCursor
> for non-persistent messages - and uses the underlying database for
> persistent messages
> 2. Try KahaDB - which - with the BTreeIndex - will not hit the
> problems you are seeing with the Filecursor
>
> or - increase the maximum number of hash buckets for the FileCursor
> index - by setting a Java system property - maximumCapacity to 65536
> (the default is 16384)
>
> cheers,
>
> Rob
>
> http://twitter.com/rajdavies
> I work here: http://fusesource.com
> My Blog: http://rajdavies.blogspot.com/
> I'm writing this: http://www.manning.com/snyder/
>
>
>
>
>
Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/
RE: OOM with high KahaDB index time
Posted by Daniel Kluesing <dk...@bluekai.com>.
I tried the suggestion of going with the default cursor, but I still get OOM errors. I've included my full config file below, I think I'm running fairly vanilla/default.
After about 350k persistent messages, the logs start to look like:
INFO | Slow KahaDB access: Journal append took: 10 ms, Index update took 3118 ms
INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took 5118 ms
INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took 2736 ms
INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took 2945 ms
INFO | Slow KahaDB access: Journal append took: 33 ms, Index update took 2654 ms
INFO | Slow KahaDB access: Journal append took: 82 ms, Index update took 3174 ms
INFO | Slow KahaDB access: Journal append took: 1 ms, Index update took 5891 ms
INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took 2906 ms
INFO | Slow KahaDB access: Journal append took: 60 ms, Index update took 7619 ms
Exception in thread "InactivityMonitor WriteCheck" java.lang.OutOfMemoryError: Java heap space
at java.util.jar.Attributes.read(Attributes.java:377)
at java.util.jar.Manifest.read(Manifest.java:182)
at java.util.jar.Manifest.<init>(Manifest.java:52)
at java.util.jar.JarFile.getManifestFromReference(JarFile.java:165)
at java.util.jar.JarFile.getManifest(JarFile.java:146)
at sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:693)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:221)
at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
at org.apache.activemq.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:132)
at org.apache.activemq.transport.InactivityMonitor$2.run(InactivityMonitor.java:106)
at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
Config file:
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:amq="http://activemq.apache.org/schema/core"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.base}/conf/credentials.properties</value>
</property>
</bean>
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="sub01chi" dataDirectory="${activemq.base}/data">
<managementContext>
<managementContext createConnector="true"/>
</managementContext>
<persistenceAdapter>
<kahaDB directory="${activemq.base}/data/kahadb"/>
</persistenceAdapter>
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry queue="P>" producerFlowControl="true" memoryLimit="10mb"></policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<systemUsage>
<systemUsage sendFailIfNoSpace="true">
</systemUsage>
</systemUsage>
<transportConnectors>
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
</transportConnectors>
</broker>
<import resource="jetty.xml"/>
</beans>
-----Original Message-----
From: Rob Davies [mailto:rajdavies@gmail.com]
Sent: Monday, January 18, 2010 10:42 PM
To: users@activemq.apache.org
Subject: Re: OOM with high KahaDB index time
On 18 Jan 2010, at 22:14, Daniel Kluesing wrote:
> Hi,
>
> I'm running the 5.3 release as a standalone broker. In one case, a
> producer is running without a consumer, producing small, persistent
> messages, with the FileCursor pendingQueuePolicy (per https://issues.apache.org/activemq/browse/AMQ-2512)
> option and flow control memoryLimit set to 100mb for the queue in
> question. (Through a policy entry)
>
> As the queue grows above 300k messages, KahaDB indexing starts
> climbing above 1 second. At around 350k messages, the indexing is
> taking over 8 seconds. At this point, I start getting java out of
> heap space errors in essentially random parts of the code. After a
> while, the producers timeout with a channel inactive for too long
> error, and the entire broker basically wedges itself. At this point,
> consumers are generally unable to bind to the broker quitting with
> timeout errors. When they can connect, consuming a single message
> triggers an index re-build, which takes 2-8seconds. Turning on
> verbose garbage collection, the jvm is collecting like mad but
> reclaiming no space.
>
> If I restart the broker, it comes back up, I can consume the old
> messages, and can handle another 350k messages until it wedges.
>
> I can reproduce under both default gc and incremental gc.
>
> Two questions:
> - It seems like someone is holding onto a handle to the messages
> after they have been persisted to disk - is this a known issue?
> Should I open a JIRA for it? (Or is there another explanation?)
>
> - Is there any documentation about the internals of KahaDB - the
> kind of indices etc? I'd like to get a better understanding of the
> index performance and in general how KahaDB compares to something
> like BerkeleyDB.
>
> Thanks
>
>
>
There's is some confusion over naming of our persistence options that
doesn't help. There is Kaha - which uses multiple log files and a Hash
based index - this is currently used by the FileCursor - whilst KahaDB
is a newer implementation, which is more robust and typically uses a
BTreeIndex. There is currently a new implementation of the Filecursor
btw - but that's a different matter. You can't currently configure the
HashIndex via the FileCursor - but it looks like this is the problem
you are encountering - as it looks like you need to increase the max
hash buckets.
So I would recommend the following
1. Use the default pendingQueuePolicy (which only uses a FileCursor
for non-persistent messages - and uses the underlying database for
persistent messages
2. Try KahaDB - which - with the BTreeIndex - will not hit the
problems you are seeing with the Filecursor
or - increase the maximum number of hash buckets for the FileCursor
index - by setting a Java system property - maximumCapacity to 65536
(the default is 16384)
cheers,
Rob
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/
Re: OOM with high KahaDB index time
Posted by rideallday <ni...@uvic.ca>.
That was it. Thanks!
I had to set to indexCacheSize = 100000 to get the performance I'm looking
for.
rajdavies wrote:
>
> ok - it looks like you might need to tune KahaDB - set the
> indexCacheSize = 1000 - see http://activemq.apache.org/kahadb.html
>
>
>
--
View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27657831.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: OOM with high KahaDB index time
Posted by Rob Davies <ra...@gmail.com>.
ok - it looks like you might need to tune KahaDB - set the
indexCacheSize = 1000 - see http://activemq.apache.org/kahadb.html
cheers,
Rob
On 15 Feb 2010, at 18:46, rideallday wrote:
>
> Yes, topics are durable.
>
> Test case:
> 1. publish messages to queue, let several hundred thousand accumulate.
> 2. publish topics to queue, let several hundred thousand accumulate
> 3. begin consuming from queue.
> 4. begin consuming from topic.
>
> You should see a huge degrade in performance and a bunch of indexing
> messages once you start to consume from topic.
>
>
>
>
> rajdavies wrote:
>>
>> could you provide a test case ?
>>
>> cheers,
>>
>> Rob
>> On 11 Feb 2010, at 21:41, rideallday wrote:
>>
>>>
>>> Umm, actually I'm still seeing issues with high Index times for
>>> topics.
>>>
>>> My test case is:
>>>
>>> Send a huge amount of messages into the broker. Let several hundred
>>> thousand
>>> accumulate. Then start consumers and see how long it takes to drain
>>> the
>>> queues and topics.
>>>
>>> The result is:
>>>
>>> For queues- excellent. Performance is amazing. A few 1000ms Journal
>>> append
>>> messages, but recovery performance exceeds my requirements.
>>> For topics- Index updates in the 2-700ms range. This happens every
>>> second or
>>> so and locks the consumer, so recovery is extremely slow.
>>>
>>> This is very similar behaviour to what I saw with Topics and Queues
>>> before
>>> Rob applied the fix for cursors in KahaDB.
>>>
>>> Is it possible that the fix needs to be also applied to topics?
>>>
>>> I'm using 5.4 snapshot from Feb 1.
>>> --
>>> View this message in context:
>>> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27555173.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>
>> Rob Davies
>> http://twitter.com/rajdavies
>> I work here: http://fusesource.com
>> My Blog: http://rajdavies.blogspot.com/
>> I'm writing this: http://www.manning.com/snyder/
>>
>>
>>
>>
>>
>>
>>
>
> --
> View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27598149.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/
Re: OOM with high KahaDB index time
Posted by rideallday <ni...@uvic.ca>.
Yes, topics are durable.
Test case:
1. publish messages to queue, let several hundred thousand accumulate.
2. publish topics to queue, let several hundred thousand accumulate
3. begin consuming from queue.
4. begin consuming from topic.
You should see a huge degrade in performance and a bunch of indexing
messages once you start to consume from topic.
rajdavies wrote:
>
> could you provide a test case ?
>
> cheers,
>
> Rob
> On 11 Feb 2010, at 21:41, rideallday wrote:
>
>>
>> Umm, actually I'm still seeing issues with high Index times for
>> topics.
>>
>> My test case is:
>>
>> Send a huge amount of messages into the broker. Let several hundred
>> thousand
>> accumulate. Then start consumers and see how long it takes to drain
>> the
>> queues and topics.
>>
>> The result is:
>>
>> For queues- excellent. Performance is amazing. A few 1000ms Journal
>> append
>> messages, but recovery performance exceeds my requirements.
>> For topics- Index updates in the 2-700ms range. This happens every
>> second or
>> so and locks the consumer, so recovery is extremely slow.
>>
>> This is very similar behaviour to what I saw with Topics and Queues
>> before
>> Rob applied the fix for cursors in KahaDB.
>>
>> Is it possible that the fix needs to be also applied to topics?
>>
>> I'm using 5.4 snapshot from Feb 1.
>> --
>> View this message in context:
>> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27555173.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
> Rob Davies
> http://twitter.com/rajdavies
> I work here: http://fusesource.com
> My Blog: http://rajdavies.blogspot.com/
> I'm writing this: http://www.manning.com/snyder/
>
>
>
>
>
>
>
--
View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27598149.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: OOM with high KahaDB index time
Posted by Rob Davies <ra...@gmail.com>.
could you provide a test case ?
cheers,
Rob
On 11 Feb 2010, at 21:41, rideallday wrote:
>
> Umm, actually I'm still seeing issues with high Index times for
> topics.
>
> My test case is:
>
> Send a huge amount of messages into the broker. Let several hundred
> thousand
> accumulate. Then start consumers and see how long it takes to drain
> the
> queues and topics.
>
> The result is:
>
> For queues- excellent. Performance is amazing. A few 1000ms Journal
> append
> messages, but recovery performance exceeds my requirements.
> For topics- Index updates in the 2-700ms range. This happens every
> second or
> so and locks the consumer, so recovery is extremely slow.
>
> This is very similar behaviour to what I saw with Topics and Queues
> before
> Rob applied the fix for cursors in KahaDB.
>
> Is it possible that the fix needs to be also applied to topics?
>
> I'm using 5.4 snapshot from Feb 1.
> --
> View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27555173.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/
Re: OOM with high KahaDB index time
Posted by Rob Davies <ra...@gmail.com>.
Are you using Durable Topic Consumers ?
On 11 Feb 2010, at 21:41, rideallday wrote:
>
> Umm, actually I'm still seeing issues with high Index times for
> topics.
>
> My test case is:
>
> Send a huge amount of messages into the broker. Let several hundred
> thousand
> accumulate. Then start consumers and see how long it takes to drain
> the
> queues and topics.
>
> The result is:
>
> For queues- excellent. Performance is amazing. A few 1000ms Journal
> append
> messages, but recovery performance exceeds my requirements.
> For topics- Index updates in the 2-700ms range. This happens every
> second or
> so and locks the consumer, so recovery is extremely slow.
>
> This is very similar behaviour to what I saw with Topics and Queues
> before
> Rob applied the fix for cursors in KahaDB.
>
> Is it possible that the fix needs to be also applied to topics?
>
> I'm using 5.4 snapshot from Feb 1.
> --
> View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27555173.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/
Re: OOM with high KahaDB index time
Posted by rideallday <ni...@uvic.ca>.
Umm, actually I'm still seeing issues with high Index times for topics.
My test case is:
Send a huge amount of messages into the broker. Let several hundred thousand
accumulate. Then start consumers and see how long it takes to drain the
queues and topics.
The result is:
For queues- excellent. Performance is amazing. A few 1000ms Journal append
messages, but recovery performance exceeds my requirements.
For topics- Index updates in the 2-700ms range. This happens every second or
so and locks the consumer, so recovery is extremely slow.
This is very similar behaviour to what I saw with Topics and Queues before
Rob applied the fix for cursors in KahaDB.
Is it possible that the fix needs to be also applied to topics?
I'm using 5.4 snapshot from Feb 1.
--
View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27555173.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: OOM with high KahaDB index time
Posted by rideallday <ni...@uvic.ca>.
I can confirm that has resolved the issue I was seeing. Thanks Rob!
What's ETA for 5.4 or applying this fix in 5.3? I'll need an official stable
release to put into production.
--
View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27413891.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: OOM with high KahaDB index time
Posted by Rob Davies <ra...@gmail.com>.
Good to know! - this was a silly error - When a consumer got so far
behind a producer so that the cache was invalidated, after every
acknowledgement of a message, a cursor would check to see if the index
was empty. To do that it would ask for a count of still outstanding
persistent messages. For a BTreeIndex - a count resulted in every
BTreeNode been paged into memory in the same transaction to be
counted. Its remarkable KahaDB was as fast as it was under the
circumstances :)
On 1 Feb 2010, at 22:30, Daniel Kluesing wrote:
> Ran this through some paces, and it works much better. Thanks for
> the help!
>
> -----Original Message-----
> From: Gary Tully [mailto:gary.tully@gmail.com]
> Sent: Wednesday, January 27, 2010 2:35 AM
> To: users@activemq.apache.org
> Subject: Re: OOM with high KahaDB index time
>
> Hi Daniel,
> builds from today are there now:
> https://repository.apache.org/content/repositories/snapshots/org/apache/activemq/apache-activemq/5.4-SNAPSHOT/
>
> thanks for the validation.
>
> 2010/1/26 Daniel Kluesing <dk...@bluekai.com>
>
>> Any eta on the 5.4-snapshot? (
>> https://repository.apache.org/content/repositories/snapshots/org/apache/activemq/apache-activemq/5.4-SNAPSHOT/still
>> lists the build date as 1/20) Once it's out, I'll give it a run
>> through.
>>
>> -----Original Message-----
>> From: Rob Davies [mailto:rajdavies@gmail.com]
>> Sent: Thursday, January 21, 2010 9:42 AM
>> To: users@activemq.apache.org
>> Subject: Re: OOM with high KahaDB index time
>>
>> Thanks Daniel! Though please wait for the next valid 5.4 build -
>> looks
>> like its missing a dependency at the moment
>> On 21 Jan 2010, at 16:48, Daniel Kluesing wrote:
>>
>>> I'm happy to give 5.4-snapshot a try
>>>
>>> -----Original Message-----
>>> From: Rob Davies [mailto:rajdavies@gmail.com]
>>> Sent: Wednesday, January 20, 2010 2:08 PM
>>> To: users@activemq.apache.org
>>> Subject: Re: OOM with high KahaDB index time
>>>
>>> I have a potential fix for this problem - hoping someone will try
>>> out
>>> the next 5.4-snapshot :)
>>> On 20 Jan 2010, at 21:41, rideallday wrote:
>>>
>>>>
>>>> I went back to AMQStore. Working fine for me.
>>>> --
>>>> View this message in context:
>> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27249114.html
>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>
>>>
>>> Rob Davies
>>> http://twitter.com/rajdavies
>>> I work here: http://fusesource.com
>>> My Blog: http://rajdavies.blogspot.com/
>>> I'm writing this: http://www.manning.com/snyder/
>>>
>>>
>>>
>>>
>>>
>>
>> Rob Davies
>> http://twitter.com/rajdavies
>> I work here: http://fusesource.com
>> My Blog: http://rajdavies.blogspot.com/
>> I'm writing this: http://www.manning.com/snyder/
>>
>>
>>
>>
>>
>>
>
>
> --
> http://blog.garytully.com
>
> Open Source Integration
> http://fusesource.com
Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/
RE: OOM with high KahaDB index time
Posted by Daniel Kluesing <dk...@bluekai.com>.
Ran this through some paces, and it works much better. Thanks for the help!
-----Original Message-----
From: Gary Tully [mailto:gary.tully@gmail.com]
Sent: Wednesday, January 27, 2010 2:35 AM
To: users@activemq.apache.org
Subject: Re: OOM with high KahaDB index time
Hi Daniel,
builds from today are there now:
https://repository.apache.org/content/repositories/snapshots/org/apache/activemq/apache-activemq/5.4-SNAPSHOT/
thanks for the validation.
2010/1/26 Daniel Kluesing <dk...@bluekai.com>
> Any eta on the 5.4-snapshot? (
> https://repository.apache.org/content/repositories/snapshots/org/apache/activemq/apache-activemq/5.4-SNAPSHOT/still lists the build date as 1/20) Once it's out, I'll give it a run
> through.
>
> -----Original Message-----
> From: Rob Davies [mailto:rajdavies@gmail.com]
> Sent: Thursday, January 21, 2010 9:42 AM
> To: users@activemq.apache.org
> Subject: Re: OOM with high KahaDB index time
>
> Thanks Daniel! Though please wait for the next valid 5.4 build - looks
> like its missing a dependency at the moment
> On 21 Jan 2010, at 16:48, Daniel Kluesing wrote:
>
> > I'm happy to give 5.4-snapshot a try
> >
> > -----Original Message-----
> > From: Rob Davies [mailto:rajdavies@gmail.com]
> > Sent: Wednesday, January 20, 2010 2:08 PM
> > To: users@activemq.apache.org
> > Subject: Re: OOM with high KahaDB index time
> >
> > I have a potential fix for this problem - hoping someone will try out
> > the next 5.4-snapshot :)
> > On 20 Jan 2010, at 21:41, rideallday wrote:
> >
> >>
> >> I went back to AMQStore. Working fine for me.
> >> --
> >> View this message in context:
> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27249114.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >
> > Rob Davies
> > http://twitter.com/rajdavies
> > I work here: http://fusesource.com
> > My Blog: http://rajdavies.blogspot.com/
> > I'm writing this: http://www.manning.com/snyder/
> >
> >
> >
> >
> >
>
> Rob Davies
> http://twitter.com/rajdavies
> I work here: http://fusesource.com
> My Blog: http://rajdavies.blogspot.com/
> I'm writing this: http://www.manning.com/snyder/
>
>
>
>
>
>
--
http://blog.garytully.com
Open Source Integration
http://fusesource.com
Re: OOM with high KahaDB index time
Posted by Gary Tully <ga...@gmail.com>.
Hi Daniel,
builds from today are there now:
https://repository.apache.org/content/repositories/snapshots/org/apache/activemq/apache-activemq/5.4-SNAPSHOT/
thanks for the validation.
2010/1/26 Daniel Kluesing <dk...@bluekai.com>
> Any eta on the 5.4-snapshot? (
> https://repository.apache.org/content/repositories/snapshots/org/apache/activemq/apache-activemq/5.4-SNAPSHOT/still lists the build date as 1/20) Once it's out, I'll give it a run
> through.
>
> -----Original Message-----
> From: Rob Davies [mailto:rajdavies@gmail.com]
> Sent: Thursday, January 21, 2010 9:42 AM
> To: users@activemq.apache.org
> Subject: Re: OOM with high KahaDB index time
>
> Thanks Daniel! Though please wait for the next valid 5.4 build - looks
> like its missing a dependency at the moment
> On 21 Jan 2010, at 16:48, Daniel Kluesing wrote:
>
> > I'm happy to give 5.4-snapshot a try
> >
> > -----Original Message-----
> > From: Rob Davies [mailto:rajdavies@gmail.com]
> > Sent: Wednesday, January 20, 2010 2:08 PM
> > To: users@activemq.apache.org
> > Subject: Re: OOM with high KahaDB index time
> >
> > I have a potential fix for this problem - hoping someone will try out
> > the next 5.4-snapshot :)
> > On 20 Jan 2010, at 21:41, rideallday wrote:
> >
> >>
> >> I went back to AMQStore. Working fine for me.
> >> --
> >> View this message in context:
> http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27249114.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >
> > Rob Davies
> > http://twitter.com/rajdavies
> > I work here: http://fusesource.com
> > My Blog: http://rajdavies.blogspot.com/
> > I'm writing this: http://www.manning.com/snyder/
> >
> >
> >
> >
> >
>
> Rob Davies
> http://twitter.com/rajdavies
> I work here: http://fusesource.com
> My Blog: http://rajdavies.blogspot.com/
> I'm writing this: http://www.manning.com/snyder/
>
>
>
>
>
>
--
http://blog.garytully.com
Open Source Integration
http://fusesource.com
Re: OOM with high KahaDB index time
Posted by Rob Davies <ra...@gmail.com>.
We seem to have a problem with the builds currently. Btw - the fix
also increases message throughput
On 26 Jan 2010, at 17:58, Daniel Kluesing wrote:
> Any eta on the 5.4-snapshot? (https://repository.apache.org/content/repositories/snapshots/org/apache/activemq/apache-activemq/5.4-SNAPSHOT/
> still lists the build date as 1/20) Once it's out, I'll give it a
> run through.
>
> -----Original Message-----
> From: Rob Davies [mailto:rajdavies@gmail.com]
> Sent: Thursday, January 21, 2010 9:42 AM
> To: users@activemq.apache.org
> Subject: Re: OOM with high KahaDB index time
>
> Thanks Daniel! Though please wait for the next valid 5.4 build - looks
> like its missing a dependency at the moment
> On 21 Jan 2010, at 16:48, Daniel Kluesing wrote:
>
>> I'm happy to give 5.4-snapshot a try
>>
>> -----Original Message-----
>> From: Rob Davies [mailto:rajdavies@gmail.com]
>> Sent: Wednesday, January 20, 2010 2:08 PM
>> To: users@activemq.apache.org
>> Subject: Re: OOM with high KahaDB index time
>>
>> I have a potential fix for this problem - hoping someone will try out
>> the next 5.4-snapshot :)
>> On 20 Jan 2010, at 21:41, rideallday wrote:
>>
>>>
>>> I went back to AMQStore. Working fine for me.
>>> --
>>> View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27249114.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>
>> Rob Davies
>> http://twitter.com/rajdavies
>> I work here: http://fusesource.com
>> My Blog: http://rajdavies.blogspot.com/
>> I'm writing this: http://www.manning.com/snyder/
>>
>>
>>
>>
>>
>
> Rob Davies
> http://twitter.com/rajdavies
> I work here: http://fusesource.com
> My Blog: http://rajdavies.blogspot.com/
> I'm writing this: http://www.manning.com/snyder/
>
>
>
>
>
Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/
RE: OOM with high KahaDB index time
Posted by Daniel Kluesing <dk...@bluekai.com>.
Any eta on the 5.4-snapshot? (https://repository.apache.org/content/repositories/snapshots/org/apache/activemq/apache-activemq/5.4-SNAPSHOT/ still lists the build date as 1/20) Once it's out, I'll give it a run through.
-----Original Message-----
From: Rob Davies [mailto:rajdavies@gmail.com]
Sent: Thursday, January 21, 2010 9:42 AM
To: users@activemq.apache.org
Subject: Re: OOM with high KahaDB index time
Thanks Daniel! Though please wait for the next valid 5.4 build - looks
like its missing a dependency at the moment
On 21 Jan 2010, at 16:48, Daniel Kluesing wrote:
> I'm happy to give 5.4-snapshot a try
>
> -----Original Message-----
> From: Rob Davies [mailto:rajdavies@gmail.com]
> Sent: Wednesday, January 20, 2010 2:08 PM
> To: users@activemq.apache.org
> Subject: Re: OOM with high KahaDB index time
>
> I have a potential fix for this problem - hoping someone will try out
> the next 5.4-snapshot :)
> On 20 Jan 2010, at 21:41, rideallday wrote:
>
>>
>> I went back to AMQStore. Working fine for me.
>> --
>> View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27249114.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
> Rob Davies
> http://twitter.com/rajdavies
> I work here: http://fusesource.com
> My Blog: http://rajdavies.blogspot.com/
> I'm writing this: http://www.manning.com/snyder/
>
>
>
>
>
Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/
Re: OOM with high KahaDB index time
Posted by Rob Davies <ra...@gmail.com>.
Thanks Daniel! Though please wait for the next valid 5.4 build - looks
like its missing a dependency at the moment
On 21 Jan 2010, at 16:48, Daniel Kluesing wrote:
> I'm happy to give 5.4-snapshot a try
>
> -----Original Message-----
> From: Rob Davies [mailto:rajdavies@gmail.com]
> Sent: Wednesday, January 20, 2010 2:08 PM
> To: users@activemq.apache.org
> Subject: Re: OOM with high KahaDB index time
>
> I have a potential fix for this problem - hoping someone will try out
> the next 5.4-snapshot :)
> On 20 Jan 2010, at 21:41, rideallday wrote:
>
>>
>> I went back to AMQStore. Working fine for me.
>> --
>> View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27249114.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
> Rob Davies
> http://twitter.com/rajdavies
> I work here: http://fusesource.com
> My Blog: http://rajdavies.blogspot.com/
> I'm writing this: http://www.manning.com/snyder/
>
>
>
>
>
Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/
RE: OOM with high KahaDB index time
Posted by Daniel Kluesing <dk...@bluekai.com>.
I'm happy to give 5.4-snapshot a try
-----Original Message-----
From: Rob Davies [mailto:rajdavies@gmail.com]
Sent: Wednesday, January 20, 2010 2:08 PM
To: users@activemq.apache.org
Subject: Re: OOM with high KahaDB index time
I have a potential fix for this problem - hoping someone will try out
the next 5.4-snapshot :)
On 20 Jan 2010, at 21:41, rideallday wrote:
>
> I went back to AMQStore. Working fine for me.
> --
> View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27249114.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/
Re: OOM with high KahaDB index time
Posted by Rob Davies <ra...@gmail.com>.
I have a potential fix for this problem - hoping someone will try out
the next 5.4-snapshot :)
On 20 Jan 2010, at 21:41, rideallday wrote:
>
> I went back to AMQStore. Working fine for me.
> --
> View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27249114.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/
Re: OOM with high KahaDB index time
Posted by rideallday <ni...@uvic.ca>.
I went back to AMQStore. Working fine for me.
--
View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27249114.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: OOM with high KahaDB index time
Posted by rideallday <ni...@uvic.ca>.
rajdavies wrote:
>
> FileCursor and Kaha (which is
> used by the AMQStore). - can you share the configuration you are using?
> KahaDB isn't the same as Kaha
>
I'm still unsure... I'm using KahaDB, not AMQStore. I was about to go back
to AMQStore as this setup just isn't working, but I really like the way that
it works with the shared disk HA configuration. It's nice and simple.
A quick explanation first:
Data flows unidirectionally on a queue, communications and control are
bi-directional on topics. The test config has 2 brokers, both in a shared
disk HA setup, as described in the book. All 4 configs are the same, except
for the obvious differences in transport and broker name.
The /amq filesystem is NFS4
For my test I'm using the consumer and producer example that ships with AMQ.
I have producer flow control enabled, but it's actually the consumers that
are stalling.
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:amq="http://activemq.apache.org/schema/core"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd">
<!-- Allows us to use system properties as variables in this
configuration file -->
<bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.base}/conf/credentials.properties</value>
</property>
</bean>
<!--
The <broker> element is used to configure the ActiveMQ broker.
-->
<broker xmlns="http://activemq.apache.org/schema/core"
brokerName="test.shore" dataDirectory="/amq/shore" useJmx="true">
<networkConnectors>
<networkConnector
uri="static:(tcp://testmq1:61616,tcp://testmq2:61616)"
name="QtoDC"
duplex="false"
conduitSubscriptions="true"
decreaseNetworkConsumerPriority="false">
<excludedDestinations>
<topic physicalName=">"/>
</excludedDestinations>
</networkConnector>
<networkConnector
uri="static:(tcp://testmq1:61616,tcp://testmq2:61616)"
name="T_comms"
duplex="true">
<excludedDestinations>
<queue physicalName=">"/>
</excludedDestinations>
</networkConnector>
</networkConnectors>
<!--
The managementContext is used to configure how ActiveMQ is
exposed in
JMX. By default, ActiveMQ uses the MBean server that is started
by
the JVM. For more information, see:
http://activemq.apache.org/jmx.html
-->
<managementContext>
<managementContext createConnector="true"/>
</managementContext>
<!--
Configure message persistence for the broker. The default
persistence
mechanism is the KahaDB store (identified by the kahaDB tag).
For more information, see:
http://activemq.apache.org/persistence.html
-->
<persistenceAdapter>
<kahaDB directory="/amq/shore/kahadb"/>
</persistenceAdapter>
<!--
For better performances use VM cursor and small memory limit.
For more information, see:
http://activemq.apache.org/message-cursors.html
Also, if your producer is "hanging", it's probably due to
producer flow control.
For more information, see:
http://activemq.apache.org/producer-flow-control.html
-->
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="false"
memoryLimit="1mb">
<pendingSubscriberPolicy>
<vmCursor />
</pendingSubscriberPolicy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="false"
memoryLimit="1mb">
<!-- Use VM cursor for better latency
For more information, see:
http://activemq.apache.org/message-cursors.html
<pendingQueuePolicy>
<vmQueueCursor/>
</pendingQueuePolicy>
-->
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<!--
The systemUsage controls the maximum amount of space the broker
will
use before slowing down producers. For more information, see:
http://activemq.apache.org/producer-flow-control.html
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="20 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="1 gb" name="foo"/>
</storeUsage>
<tempUsage>
<tempUsage limit="100 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
-->
<!--
The transport connectors expose ActiveMQ over a given protocol
to
clients and other brokers. For more information, see:
http://activemq.apache.org/configuring-transports.html
-->
<transportConnectors>
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
</transportConnectors>
</broker>
<!--
Uncomment to enable Camel
Take a look at activemq-camel.xml for more details
<import resource="camel.xml"/>
-->
<!--
Enable web consoles, REST and Ajax APIs and demos
Take a look at activemq-jetty.xml for more details
-->
<import resource="jetty.xml"/>
</beans>
--
View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27244867.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: OOM with high KahaDB index time
Posted by Rob Davies <ra...@gmail.com>.
Yes - those settings apply to the both FileCursor and Kaha (which is
used by the AMQStore). - can you share the configuration you are using?
KahaDB isn't the same as Kaha - we weren't very imaginative with names
Im afraid
On 19 Jan 2010, at 23:48, rideallday wrote:
>
> Rob,
>
> I'm seeing the exact same behaviour as Daniel,
>
>
>>
>> So I would recommend the following
>>
>> 2. Try KahaDB - which - with the BTreeIndex - will not hit the
>> problems you are seeing with the Filecursor
>>
>>
>
> however I am already using KahaDB as the persistenceAdapter.
>
> Your advice seems to apply only to FileCursor, right? Can you please
> clarify.
>
> Most settings are at default values, as I am only in the evaluation
> stage.
>
> Thanks.
> Nic.
> --
> View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27234786.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/
Re: OOM with high KahaDB index time
Posted by rideallday <ni...@uvic.ca>.
Rob,
I'm seeing the exact same behaviour as Daniel,
>
> So I would recommend the following
>
> 2. Try KahaDB - which - with the BTreeIndex - will not hit the
> problems you are seeing with the Filecursor
>
>
however I am already using KahaDB as the persistenceAdapter.
Your advice seems to apply only to FileCursor, right? Can you please
clarify.
Most settings are at default values, as I am only in the evaluation stage.
Thanks.
Nic.
--
View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27234786.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: OOM with high KahaDB index time
Posted by Rob Davies <ra...@gmail.com>.
On 18 Jan 2010, at 22:14, Daniel Kluesing wrote:
> Hi,
>
> I'm running the 5.3 release as a standalone broker. In one case, a
> producer is running without a consumer, producing small, persistent
> messages, with the FileCursor pendingQueuePolicy (per https://issues.apache.org/activemq/browse/AMQ-2512)
> option and flow control memoryLimit set to 100mb for the queue in
> question. (Through a policy entry)
>
> As the queue grows above 300k messages, KahaDB indexing starts
> climbing above 1 second. At around 350k messages, the indexing is
> taking over 8 seconds. At this point, I start getting java out of
> heap space errors in essentially random parts of the code. After a
> while, the producers timeout with a channel inactive for too long
> error, and the entire broker basically wedges itself. At this point,
> consumers are generally unable to bind to the broker quitting with
> timeout errors. When they can connect, consuming a single message
> triggers an index re-build, which takes 2-8seconds. Turning on
> verbose garbage collection, the jvm is collecting like mad but
> reclaiming no space.
>
> If I restart the broker, it comes back up, I can consume the old
> messages, and can handle another 350k messages until it wedges.
>
> I can reproduce under both default gc and incremental gc.
>
> Two questions:
> - It seems like someone is holding onto a handle to the messages
> after they have been persisted to disk - is this a known issue?
> Should I open a JIRA for it? (Or is there another explanation?)
>
> - Is there any documentation about the internals of KahaDB - the
> kind of indices etc? I'd like to get a better understanding of the
> index performance and in general how KahaDB compares to something
> like BerkeleyDB.
>
> Thanks
>
>
>
There's is some confusion over naming of our persistence options that
doesn't help. There is Kaha - which uses multiple log files and a Hash
based index - this is currently used by the FileCursor - whilst KahaDB
is a newer implementation, which is more robust and typically uses a
BTreeIndex. There is currently a new implementation of the Filecursor
btw - but that's a different matter. You can't currently configure the
HashIndex via the FileCursor - but it looks like this is the problem
you are encountering - as it looks like you need to increase the max
hash buckets.
So I would recommend the following
1. Use the default pendingQueuePolicy (which only uses a FileCursor
for non-persistent messages - and uses the underlying database for
persistent messages
2. Try KahaDB - which - with the BTreeIndex - will not hit the
problems you are seeing with the Filecursor
or - increase the maximum number of hash buckets for the FileCursor
index - by setting a Java system property - maximumCapacity to 65536
(the default is 16384)
cheers,
Rob
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/