You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Maarten Dirkse <Ma...@filterworks.com> on 2008/05/02 09:01:56 UTC
Broker hitting max memory with large persistent queue
Hi,
I've spent quite a bit of time trying to find a solution to this problem, but nothing that I've come across in the documentation or the mailing list seems to help, so here goes:
I've got ActiveMQ set up to provide messaging between a chain of four components, connected by three queues like so:
Component 1 --> queue A --> component 2 --> queue B --> component 3 --> queue C --> component 4
Component 1 gets things going by messages and putting them on queue A. Component 2 then processes these, and, for every message received, puts a message on queue B. Component three then does the same thing as 2, generating an equal amount of message for component 4. Beside the queue pipeline, components also communicate with each other using a "command" topic which produces 2 messages for every queue message that goes through the pipeline.
All queues are (pure JDBC) persistent, topics are non-persistent.
This setup works very well for low volumes of messages. At a certain point though, the first component has to start producing messages at rates of several thousand per minute. Component 2 can't consume the messages at the rate they're produced, so the size of queue A slowly grows into the thousands.
The problem is that, once the size of queue A gets large enough, the broker runs out of memory and everything will screech to a halt (ie messages will be processed at a rate of 1 per minute or something). At around 256mb of broker memory for example, it reaches 100% usage when queue A contains around 16.000 messages. The number of messages that queue A can contain before activemq becomes unusably slow grows and shrinks according to the amount of memory the broker has.
What I don't get is why the broker slowly uses up all its memory for a queue that persists its messages to the database (this actually happens, the database does fill up). I've read a number of times in the documentation and mailing list that AMQ is capable of dealing with queues that contain millions of messages, but in my setup it grinds to a halt at a number far below that.
Can anyone tell me what I'm doing wrong?
Below is the setup used for the broker. Queues are defined in Spring using <amq:queue> and producers and consumers use the Spring JMS template and JMS Listeners respectively. I'm using AMQ 5.1 with Postgresql for persistence and Spring 2.5.3 to configure everything.
Broker config:
<broker id="broker" useJmx="true" brokerName="${brokerA.name}" xmlns="http://activemq.apache.org/schema/core" dataDirectory="${activemq.data}">
<persistenceAdapter>
<jdbcPersistenceAdapter dataDirectory="${activemq.data}/jdbc" dataSource="#postgres-ds" createTablesOnStartup="true" />
</persistenceAdapter>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="128mb" />
</memoryUsage>
</systemUsage>
</systemUsage>
<transportConnectors>
<transportConnector uri="tcp://localhost:61616" />
</transportConnectors>
</broker>
Template and queue/topic config:
<beans>
<amq:connectionFactory brokerURL="${broker.jmsUrl}" id="jmsFactory" dispatchAsync="true" />
<bean id="scf" class="org.springframework.jms.connection.SingleConnectionFactory" p:targetConnectionFactory-ref="jmsFactory" />
<bean class="org.springframework.jms.core.JmsTemplate" id="jmsTemplate" p:receiveTimeout="${jms.receivetimeout}" p:connectionFactory-ref="scf" />
<bean class="org.springframework.jms.core.JmsTemplate" id="commandTemplate" p:receiveTimeout="${jms.receivetimeout}" p:explicitQosEnabled="true"
p:priority="4" p:deliveryMode="2" p:timeToLive="120000" p:connectionFactory-ref="scf" p:defaultDestination-ref="commandTopic" />
<amq:queue id="A" physicalName="queue.A" />
<amq:queue id="B" physicalName="queue.B" />
<amq:queue id="C" physicalName="queue.C" />
<amq:topic id="commandTopic" physicalName="command.topic" />
</beans>
Thanks in advance,
Maarten
Re: Broker hitting max memory with large persistent queue
Posted by tpounds <tr...@gmail.com>.
Maarten Dirkse-2 wrote:
>
> Hi,
> I've spent quite a bit of time trying to find a solution to this problem,
> but nothing that I've come across in the documentation or the mailing list
> seems to help, so here goes:
>
> I've got ActiveMQ set up to provide messaging between a chain of four
> components, connected by three queues like so:
> Component 1 --> queue A --> component 2 --> queue B --> component 3 -->
> queue C --> component 4
>
> Component 1 gets things going by messages and putting them on queue A.
> Component 2 then processes these, and, for every message received, puts a
> message on queue B. Component three then does the same thing as 2,
> generating an equal amount of message for component 4. Beside the queue
> pipeline, components also communicate with each other using a "command"
> topic which produces 2 messages for every queue message that goes through
> the pipeline.
> All queues are (pure JDBC) persistent, topics are non-persistent.
>
> This setup works very well for low volumes of messages. At a certain point
> though, the first component has to start producing messages at rates of
> several thousand per minute. Component 2 can't consume the messages at the
> rate they're produced, so the size of queue A slowly grows into the
> thousands.
>
> The problem is that, once the size of queue A gets large enough, the
> broker runs out of memory and everything will screech to a halt (ie
> messages will be processed at a rate of 1 per minute or something). At
> around 256mb of broker memory for example, it reaches 100% usage when
> queue A contains around 16.000 messages. The number of messages that queue
> A can contain before activemq becomes unusably slow grows and shrinks
> according to the amount of memory the broker has.
>
> What I don't get is why the broker slowly uses up all its memory for a
> queue that persists its messages to the database (this actually happens,
> the database does fill up). I've read a number of times in the
> documentation and mailing list that AMQ is capable of dealing with queues
> that contain millions of messages, but in my setup it grinds to a halt at
> a number far below that.
>
> Can anyone tell me what I'm doing wrong?
>
> Below is the setup used for the broker. Queues are defined in Spring using
> <amq:queue> and producers and consumers use the Spring JMS template and
> JMS Listeners respectively. I'm using AMQ 5.1 with Postgresql for
> persistence and Spring 2.5.3 to configure everything.
>
> Broker config:
> <broker id="broker" useJmx="true" brokerName="${brokerA.name}"
> xmlns="http://activemq.apache.org/schema/core"
> dataDirectory="${activemq.data}">
>
> <persistenceAdapter>
> <jdbcPersistenceAdapter dataDirectory="${activemq.data}/jdbc"
> dataSource="#postgres-ds" createTablesOnStartup="true" />
> </persistenceAdapter>
>
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="128mb" />
> </memoryUsage>
> </systemUsage>
> </systemUsage>
>
> <transportConnectors>
> <transportConnector uri="tcp://localhost:61616" />
> </transportConnectors>
> </broker>
>
> Template and queue/topic config:
>
> <beans>
> <amq:connectionFactory brokerURL="${broker.jmsUrl}" id="jmsFactory"
> dispatchAsync="true" />
>
> <bean id="scf"
> class="org.springframework.jms.connection.SingleConnectionFactory"
> p:targetConnectionFactory-ref="jmsFactory" />
>
> <bean class="org.springframework.jms.core.JmsTemplate" id="jmsTemplate"
> p:receiveTimeout="${jms.receivetimeout}" p:connectionFactory-ref="scf" />
>
> <bean class="org.springframework.jms.core.JmsTemplate"
> id="commandTemplate" p:receiveTimeout="${jms.receivetimeout}"
> p:explicitQosEnabled="true"
> p:priority="4" p:deliveryMode="2" p:timeToLive="120000"
> p:connectionFactory-ref="scf" p:defaultDestination-ref="commandTopic" />
>
> <amq:queue id="A" physicalName="queue.A" />
> <amq:queue id="B" physicalName="queue.B" />
> <amq:queue id="C" physicalName="queue.C" />
>
> <amq:topic id="commandTopic" physicalName="command.topic" />
> </beans>
>
> Thanks in advance,
> Maarten
>
>
>
>
--
View this message in context: http://www.nabble.com/Broker-hitting-max-memory-with-large-persistent-queue-tp17014805s2354p17606223.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
RE: Broker hitting max memory with large persistent queue
Posted by Vinod Venkatraman <vi...@gmail.com>.
Have you checked out
http://activemq.apache.org/message-cursors.html
4.1.1 seems to store some kind of message reference in memory so broker may
run out of memory even if persistence is enabled.
Maarten Dirkse-2 wrote:
>
> Allright, the sleuthing continues. I did a heap dump of the broker vm
> when the broker was using 100% of its memory and everything had come to
> a standstill. I openend the heapdum in visualvm (see pic), but I have to
> confess it doesn't mean that much to me. There seem to be an awful lot
> of ActiveMQTextMesssages hanging around, and ObjectMessages as well. Not
> sure what they're all doing in memory, as I'd rather see then on disk
> (in the case of queues) or discarded (in the case of topics).
> Anyone have any ideas why my broker is maxing out?
> Regards,
> Maarten
>
> -----Original Message-----
> From: Maarten Dirkse [mailto:Maarten.Dirkse@filterworks.com]
> Sent: 2 mei 2008 16:45
> To: users@activemq.apache.org
> Subject: RE: Broker hitting max memory with large persistent queue
>
> Hi Joe,
>
> Using the Spring listeners, we've actually got about 10-20 threads on
> each stage of the pipeline, all consuming messages. The problem is that
> we'd like to be able to deal with a situation where there are
> potentially millions of messages waiting in to be processed without
> having to concede ever more ram to the broker (or ever more threads to
> the consumers). New messages should just get in line to be processed,
> whether that's 100 milliseconds or 100 minutes from the time they
> arrive. I thought persistence would do this for us, as it writes the
> messages to disk instead of keeping them in ram, but the broker is still
> running out of ram.
>
> It seems as though, for some reason, it's keeping something in memory
> for every message in a persistent queue, and once it hits 100% memory
> usage it's game over.
>
> Regards,
> Maarten
>
> -----Original Message-----
> From: ttmdev [mailto:joe.fernandez@ttmsolutions.com]
> Sent: 2 mei 2008 16:22
> To: users@activemq.apache.org
> Subject: RE: Broker hitting max memory with large persistent queue
>
>
> What about creating clusters of consumers for each of your processing
> stages to better-handle the load spikes? The messages will get
> load-balanced across the consumers in a cluster.
>
> Joe
> Get a free ActiveMQ user guide at www.ttmsolutions.com
>
>
>
> Maarten Dirkse-2 wrote:
>>
>> An update:
>> I've now tried it with the standard amqpersistence config:
>>
>> <persistenceAdapter>
>> <amqPersistenceAdapter directory="/var/filterworks/amq/store"
>> maxFileLength="32mb"/>
>> </persistenceAdapter>
>>
>> And it still results in the same problem. When I give the broker 128mb
>
>> of memory, everything runs fine until it has processed about 126,000
>> messages. Then the memory usage hits 100%, and everything slows to a
>> crawl.
>>
>> Anyone have any ideas as to what's going on?
>>
>> Regards,
>> Maarten
>>
>> -----Original Message-----
>> From: Maarten Dirkse [mailto:Maarten.Dirkse@filterworks.com]
>> Sent: 2 mei 2008 9:02
>> To: users@activemq.apache.org
>> Subject: Broker hitting max memory with large persistent queue
>>
>> Hi,
>> I've spent quite a bit of time trying to find a solution to this
>> problem, but nothing that I've come across in the documentation or the
>
>> mailing list seems to help, so here goes:
>>
>> I've got ActiveMQ set up to provide messaging between a chain of four
>> components, connected by three queues like so:
>> Component 1 --> queue A --> component 2 --> queue B --> component 3
>> --> queue C --> component 4
>>
>> Component 1 gets things going by messages and putting them on queue A.
>> Component 2 then processes these, and, for every message received,
>> puts a message on queue B. Component three then does the same thing as
>
>> 2, generating an equal amount of message for component 4. Beside the
>> queue pipeline, components also communicate with each other using a
> "command"
>> topic which produces 2 messages for every queue message that goes
>> through the pipeline.
>> All queues are (pure JDBC) persistent, topics are non-persistent.
>>
>> This setup works very well for low volumes of messages. At a certain
>> point though, the first component has to start producing messages at
>> rates of several thousand per minute. Component 2 can't consume the
>> messages at the rate they're produced, so the size of queue A slowly
>> grows into the thousands.
>>
>> The problem is that, once the size of queue A gets large enough, the
>> broker runs out of memory and everything will screech to a halt (ie
>> messages will be processed at a rate of 1 per minute or something). At
>
>> around 256mb of broker memory for example, it reaches 100% usage when
>> queue A contains around 16.000 messages. The number of messages that
>> queue A can contain before activemq becomes unusably slow grows and
>> shrinks according to the amount of memory the broker has.
>>
>> What I don't get is why the broker slowly uses up all its memory for a
>
>> queue that persists its messages to the database (this actually
>> happens, the database does fill up). I've read a number of times in
>> the documentation and mailing list that AMQ is capable of dealing with
>
>> queues that contain millions of messages, but in my setup it grinds to
>
>> a halt at a number far below that.
>>
>> Can anyone tell me what I'm doing wrong?
>>
>> Below is the setup used for the broker. Queues are defined in Spring
>> using <amq:queue> and producers and consumers use the Spring JMS
>> template and JMS Listeners respectively. I'm using AMQ 5.1 with
>> Postgresql for persistence and Spring 2.5.3 to configure everything.
>>
>> Broker config:
>> <broker id="broker" useJmx="true" brokerName="${brokerA.name}"
>> xmlns="http://activemq.apache.org/schema/core"
>> dataDirectory="${activemq.data}">
>>
>> <persistenceAdapter>
>> <jdbcPersistenceAdapter dataDirectory="${activemq.data}/jdbc"
>> dataSource="#postgres-ds" createTablesOnStartup="true" />
>> </persistenceAdapter>
>>
>> <systemUsage>
>> <systemUsage>
>> <memoryUsage>
>> <memoryUsage limit="128mb" />
>> </memoryUsage>
>> </systemUsage>
>> </systemUsage>
>>
>> <transportConnectors>
>> <transportConnector uri="tcp://localhost:61616" />
>> </transportConnectors>
>> </broker>
>>
>> Template and queue/topic config:
>>
>> <beans>
>> <amq:connectionFactory brokerURL="${broker.jmsUrl}" id="jmsFactory"
>> dispatchAsync="true" />
>>
>> <bean id="scf"
>> class="org.springframework.jms.connection.SingleConnectionFactory"
>> p:targetConnectionFactory-ref="jmsFactory" />
>>
>> <bean class="org.springframework.jms.core.JmsTemplate"
>> id="jmsTemplate" p:receiveTimeout="${jms.receivetimeout}"
>> p:connectionFactory-ref="scf" />
>>
>> <bean class="org.springframework.jms.core.JmsTemplate"
>> id="commandTemplate" p:receiveTimeout="${jms.receivetimeout}"
>> p:explicitQosEnabled="true"
>> p:priority="4" p:deliveryMode="2" p:timeToLive="120000"
>> p:connectionFactory-ref="scf" p:defaultDestination-ref="commandTopic"
>> />
>>
>> <amq:queue id="A" physicalName="queue.A" />
>> <amq:queue id="B" physicalName="queue.B" />
>> <amq:queue id="C" physicalName="queue.C" />
>>
>> <amq:topic id="commandTopic" physicalName="command.topic" />
>> </beans>
>>
>> Thanks in advance,
>> Maarten
>>
>>
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/Broker-hitting-max-memory-with-large-persistent-qu
> eue-tp17014805s2354p17020505.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
>
--
View this message in context: http://www.nabble.com/Broker-hitting-max-memory-with-large-persistent-queue-tp17014805s2354p17052388.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
RE: Broker hitting max memory with large persistent queue
Posted by Maarten Dirkse <Ma...@filterworks.com>.
Allright, the sleuthing continues. I did a heap dump of the broker vm
when the broker was using 100% of its memory and everything had come to
a standstill. I openend the heapdum in visualvm (see pic), but I have to
confess it doesn't mean that much to me. There seem to be an awful lot
of ActiveMQTextMesssages hanging around, and ObjectMessages as well. Not
sure what they're all doing in memory, as I'd rather see then on disk
(in the case of queues) or discarded (in the case of topics).
Anyone have any ideas why my broker is maxing out?
Regards,
Maarten
-----Original Message-----
From: Maarten Dirkse [mailto:Maarten.Dirkse@filterworks.com]
Sent: 2 mei 2008 16:45
To: users@activemq.apache.org
Subject: RE: Broker hitting max memory with large persistent queue
Hi Joe,
Using the Spring listeners, we've actually got about 10-20 threads on
each stage of the pipeline, all consuming messages. The problem is that
we'd like to be able to deal with a situation where there are
potentially millions of messages waiting in to be processed without
having to concede ever more ram to the broker (or ever more threads to
the consumers). New messages should just get in line to be processed,
whether that's 100 milliseconds or 100 minutes from the time they
arrive. I thought persistence would do this for us, as it writes the
messages to disk instead of keeping them in ram, but the broker is still
running out of ram.
It seems as though, for some reason, it's keeping something in memory
for every message in a persistent queue, and once it hits 100% memory
usage it's game over.
Regards,
Maarten
-----Original Message-----
From: ttmdev [mailto:joe.fernandez@ttmsolutions.com]
Sent: 2 mei 2008 16:22
To: users@activemq.apache.org
Subject: RE: Broker hitting max memory with large persistent queue
What about creating clusters of consumers for each of your processing
stages to better-handle the load spikes? The messages will get
load-balanced across the consumers in a cluster.
Joe
Get a free ActiveMQ user guide at www.ttmsolutions.com
Maarten Dirkse-2 wrote:
>
> An update:
> I've now tried it with the standard amqpersistence config:
>
> <persistenceAdapter>
> <amqPersistenceAdapter directory="/var/filterworks/amq/store"
> maxFileLength="32mb"/>
> </persistenceAdapter>
>
> And it still results in the same problem. When I give the broker 128mb
> of memory, everything runs fine until it has processed about 126,000
> messages. Then the memory usage hits 100%, and everything slows to a
> crawl.
>
> Anyone have any ideas as to what's going on?
>
> Regards,
> Maarten
>
> -----Original Message-----
> From: Maarten Dirkse [mailto:Maarten.Dirkse@filterworks.com]
> Sent: 2 mei 2008 9:02
> To: users@activemq.apache.org
> Subject: Broker hitting max memory with large persistent queue
>
> Hi,
> I've spent quite a bit of time trying to find a solution to this
> problem, but nothing that I've come across in the documentation or the
> mailing list seems to help, so here goes:
>
> I've got ActiveMQ set up to provide messaging between a chain of four
> components, connected by three queues like so:
> Component 1 --> queue A --> component 2 --> queue B --> component 3
> --> queue C --> component 4
>
> Component 1 gets things going by messages and putting them on queue A.
> Component 2 then processes these, and, for every message received,
> puts a message on queue B. Component three then does the same thing as
> 2, generating an equal amount of message for component 4. Beside the
> queue pipeline, components also communicate with each other using a
"command"
> topic which produces 2 messages for every queue message that goes
> through the pipeline.
> All queues are (pure JDBC) persistent, topics are non-persistent.
>
> This setup works very well for low volumes of messages. At a certain
> point though, the first component has to start producing messages at
> rates of several thousand per minute. Component 2 can't consume the
> messages at the rate they're produced, so the size of queue A slowly
> grows into the thousands.
>
> The problem is that, once the size of queue A gets large enough, the
> broker runs out of memory and everything will screech to a halt (ie
> messages will be processed at a rate of 1 per minute or something). At
> around 256mb of broker memory for example, it reaches 100% usage when
> queue A contains around 16.000 messages. The number of messages that
> queue A can contain before activemq becomes unusably slow grows and
> shrinks according to the amount of memory the broker has.
>
> What I don't get is why the broker slowly uses up all its memory for a
> queue that persists its messages to the database (this actually
> happens, the database does fill up). I've read a number of times in
> the documentation and mailing list that AMQ is capable of dealing with
> queues that contain millions of messages, but in my setup it grinds to
> a halt at a number far below that.
>
> Can anyone tell me what I'm doing wrong?
>
> Below is the setup used for the broker. Queues are defined in Spring
> using <amq:queue> and producers and consumers use the Spring JMS
> template and JMS Listeners respectively. I'm using AMQ 5.1 with
> Postgresql for persistence and Spring 2.5.3 to configure everything.
>
> Broker config:
> <broker id="broker" useJmx="true" brokerName="${brokerA.name}"
> xmlns="http://activemq.apache.org/schema/core"
> dataDirectory="${activemq.data}">
>
> <persistenceAdapter>
> <jdbcPersistenceAdapter dataDirectory="${activemq.data}/jdbc"
> dataSource="#postgres-ds" createTablesOnStartup="true" />
> </persistenceAdapter>
>
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="128mb" />
> </memoryUsage>
> </systemUsage>
> </systemUsage>
>
> <transportConnectors>
> <transportConnector uri="tcp://localhost:61616" />
> </transportConnectors>
> </broker>
>
> Template and queue/topic config:
>
> <beans>
> <amq:connectionFactory brokerURL="${broker.jmsUrl}" id="jmsFactory"
> dispatchAsync="true" />
>
> <bean id="scf"
> class="org.springframework.jms.connection.SingleConnectionFactory"
> p:targetConnectionFactory-ref="jmsFactory" />
>
> <bean class="org.springframework.jms.core.JmsTemplate"
> id="jmsTemplate" p:receiveTimeout="${jms.receivetimeout}"
> p:connectionFactory-ref="scf" />
>
> <bean class="org.springframework.jms.core.JmsTemplate"
> id="commandTemplate" p:receiveTimeout="${jms.receivetimeout}"
> p:explicitQosEnabled="true"
> p:priority="4" p:deliveryMode="2" p:timeToLive="120000"
> p:connectionFactory-ref="scf" p:defaultDestination-ref="commandTopic"
> />
>
> <amq:queue id="A" physicalName="queue.A" />
> <amq:queue id="B" physicalName="queue.B" />
> <amq:queue id="C" physicalName="queue.C" />
>
> <amq:topic id="commandTopic" physicalName="command.topic" />
> </beans>
>
> Thanks in advance,
> Maarten
>
>
>
>
--
View this message in context:
http://www.nabble.com/Broker-hitting-max-memory-with-large-persistent-qu
eue-tp17014805s2354p17020505.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
RE: Broker hitting max memory with large persistent queue
Posted by Maarten Dirkse <Ma...@filterworks.com>.
Hi Joe,
Using the Spring listeners, we've actually got about 10-20 threads on
each stage of the pipeline, all consuming messages. The problem is that
we'd like to be able to deal with a situation where there are
potentially millions of messages waiting in to be processed without
having to concede ever more ram to the broker (or ever more threads to
the consumers). New messages should just get in line to be processed,
whether that's 100 milliseconds or 100 minutes from the time they
arrive. I thought persistence would do this for us, as it writes the
messages to disk instead of keeping them in ram, but the broker is still
running out of ram.
It seems as though, for some reason, it's keeping something in memory
for every message in a persistent queue, and once it hits 100% memory
usage it's game over.
Regards,
Maarten
-----Original Message-----
From: ttmdev [mailto:joe.fernandez@ttmsolutions.com]
Sent: 2 mei 2008 16:22
To: users@activemq.apache.org
Subject: RE: Broker hitting max memory with large persistent queue
What about creating clusters of consumers for each of your processing
stages to better-handle the load spikes? The messages will get
load-balanced across the consumers in a cluster.
Joe
Get a free ActiveMQ user guide at www.ttmsolutions.com
Maarten Dirkse-2 wrote:
>
> An update:
> I've now tried it with the standard amqpersistence config:
>
> <persistenceAdapter>
> <amqPersistenceAdapter directory="/var/filterworks/amq/store"
> maxFileLength="32mb"/>
> </persistenceAdapter>
>
> And it still results in the same problem. When I give the broker 128mb
> of memory, everything runs fine until it has processed about 126,000
> messages. Then the memory usage hits 100%, and everything slows to a
> crawl.
>
> Anyone have any ideas as to what's going on?
>
> Regards,
> Maarten
>
> -----Original Message-----
> From: Maarten Dirkse [mailto:Maarten.Dirkse@filterworks.com]
> Sent: 2 mei 2008 9:02
> To: users@activemq.apache.org
> Subject: Broker hitting max memory with large persistent queue
>
> Hi,
> I've spent quite a bit of time trying to find a solution to this
> problem, but nothing that I've come across in the documentation or the
> mailing list seems to help, so here goes:
>
> I've got ActiveMQ set up to provide messaging between a chain of four
> components, connected by three queues like so:
> Component 1 --> queue A --> component 2 --> queue B --> component 3
> --> queue C --> component 4
>
> Component 1 gets things going by messages and putting them on queue A.
> Component 2 then processes these, and, for every message received,
> puts a message on queue B. Component three then does the same thing as
> 2, generating an equal amount of message for component 4. Beside the
> queue pipeline, components also communicate with each other using a
"command"
> topic which produces 2 messages for every queue message that goes
> through the pipeline.
> All queues are (pure JDBC) persistent, topics are non-persistent.
>
> This setup works very well for low volumes of messages. At a certain
> point though, the first component has to start producing messages at
> rates of several thousand per minute. Component 2 can't consume the
> messages at the rate they're produced, so the size of queue A slowly
> grows into the thousands.
>
> The problem is that, once the size of queue A gets large enough, the
> broker runs out of memory and everything will screech to a halt (ie
> messages will be processed at a rate of 1 per minute or something). At
> around 256mb of broker memory for example, it reaches 100% usage when
> queue A contains around 16.000 messages. The number of messages that
> queue A can contain before activemq becomes unusably slow grows and
> shrinks according to the amount of memory the broker has.
>
> What I don't get is why the broker slowly uses up all its memory for a
> queue that persists its messages to the database (this actually
> happens, the database does fill up). I've read a number of times in
> the documentation and mailing list that AMQ is capable of dealing with
> queues that contain millions of messages, but in my setup it grinds to
> a halt at a number far below that.
>
> Can anyone tell me what I'm doing wrong?
>
> Below is the setup used for the broker. Queues are defined in Spring
> using <amq:queue> and producers and consumers use the Spring JMS
> template and JMS Listeners respectively. I'm using AMQ 5.1 with
> Postgresql for persistence and Spring 2.5.3 to configure everything.
>
> Broker config:
> <broker id="broker" useJmx="true" brokerName="${brokerA.name}"
> xmlns="http://activemq.apache.org/schema/core"
> dataDirectory="${activemq.data}">
>
> <persistenceAdapter>
> <jdbcPersistenceAdapter dataDirectory="${activemq.data}/jdbc"
> dataSource="#postgres-ds" createTablesOnStartup="true" />
> </persistenceAdapter>
>
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="128mb" />
> </memoryUsage>
> </systemUsage>
> </systemUsage>
>
> <transportConnectors>
> <transportConnector uri="tcp://localhost:61616" />
> </transportConnectors>
> </broker>
>
> Template and queue/topic config:
>
> <beans>
> <amq:connectionFactory brokerURL="${broker.jmsUrl}" id="jmsFactory"
> dispatchAsync="true" />
>
> <bean id="scf"
> class="org.springframework.jms.connection.SingleConnectionFactory"
> p:targetConnectionFactory-ref="jmsFactory" />
>
> <bean class="org.springframework.jms.core.JmsTemplate"
> id="jmsTemplate" p:receiveTimeout="${jms.receivetimeout}"
> p:connectionFactory-ref="scf" />
>
> <bean class="org.springframework.jms.core.JmsTemplate"
> id="commandTemplate" p:receiveTimeout="${jms.receivetimeout}"
> p:explicitQosEnabled="true"
> p:priority="4" p:deliveryMode="2" p:timeToLive="120000"
> p:connectionFactory-ref="scf" p:defaultDestination-ref="commandTopic"
> />
>
> <amq:queue id="A" physicalName="queue.A" />
> <amq:queue id="B" physicalName="queue.B" />
> <amq:queue id="C" physicalName="queue.C" />
>
> <amq:topic id="commandTopic" physicalName="command.topic" />
> </beans>
>
> Thanks in advance,
> Maarten
>
>
>
>
--
View this message in context:
http://www.nabble.com/Broker-hitting-max-memory-with-large-persistent-qu
eue-tp17014805s2354p17020505.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
RE: Broker hitting max memory with large persistent queue
Posted by ttmdev <jo...@ttmsolutions.com>.
What about creating clusters of consumers for each of your processing stages
to better-handle the load spikes? The messages will get load-balanced across
the consumers in a cluster.
Joe
Get a free ActiveMQ user guide at www.ttmsolutions.com
Maarten Dirkse-2 wrote:
>
> An update:
> I've now tried it with the standard amqpersistence config:
>
> <persistenceAdapter>
> <amqPersistenceAdapter directory="/var/filterworks/amq/store"
> maxFileLength="32mb"/>
> </persistenceAdapter>
>
> And it still results in the same problem. When I give the broker 128mb
> of memory, everything runs fine until it has processed about 126,000
> messages. Then the memory usage hits 100%, and everything slows to a
> crawl.
>
> Anyone have any ideas as to what's going on?
>
> Regards,
> Maarten
>
> -----Original Message-----
> From: Maarten Dirkse [mailto:Maarten.Dirkse@filterworks.com]
> Sent: 2 mei 2008 9:02
> To: users@activemq.apache.org
> Subject: Broker hitting max memory with large persistent queue
>
> Hi,
> I've spent quite a bit of time trying to find a solution to this
> problem, but nothing that I've come across in the documentation or the
> mailing list seems to help, so here goes:
>
> I've got ActiveMQ set up to provide messaging between a chain of four
> components, connected by three queues like so:
> Component 1 --> queue A --> component 2 --> queue B --> component 3 -->
> queue C --> component 4
>
> Component 1 gets things going by messages and putting them on queue A.
> Component 2 then processes these, and, for every message received, puts
> a message on queue B. Component three then does the same thing as 2,
> generating an equal amount of message for component 4. Beside the queue
> pipeline, components also communicate with each other using a "command"
> topic which produces 2 messages for every queue message that goes
> through the pipeline.
> All queues are (pure JDBC) persistent, topics are non-persistent.
>
> This setup works very well for low volumes of messages. At a certain
> point though, the first component has to start producing messages at
> rates of several thousand per minute. Component 2 can't consume the
> messages at the rate they're produced, so the size of queue A slowly
> grows into the thousands.
>
> The problem is that, once the size of queue A gets large enough, the
> broker runs out of memory and everything will screech to a halt (ie
> messages will be processed at a rate of 1 per minute or something). At
> around 256mb of broker memory for example, it reaches 100% usage when
> queue A contains around 16.000 messages. The number of messages that
> queue A can contain before activemq becomes unusably slow grows and
> shrinks according to the amount of memory the broker has.
>
> What I don't get is why the broker slowly uses up all its memory for a
> queue that persists its messages to the database (this actually happens,
> the database does fill up). I've read a number of times in the
> documentation and mailing list that AMQ is capable of dealing with
> queues that contain millions of messages, but in my setup it grinds to a
> halt at a number far below that.
>
> Can anyone tell me what I'm doing wrong?
>
> Below is the setup used for the broker. Queues are defined in Spring
> using <amq:queue> and producers and consumers use the Spring JMS
> template and JMS Listeners respectively. I'm using AMQ 5.1 with
> Postgresql for persistence and Spring 2.5.3 to configure everything.
>
> Broker config:
> <broker id="broker" useJmx="true" brokerName="${brokerA.name}"
> xmlns="http://activemq.apache.org/schema/core"
> dataDirectory="${activemq.data}">
>
> <persistenceAdapter>
> <jdbcPersistenceAdapter dataDirectory="${activemq.data}/jdbc"
> dataSource="#postgres-ds" createTablesOnStartup="true" />
> </persistenceAdapter>
>
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="128mb" />
> </memoryUsage>
> </systemUsage>
> </systemUsage>
>
> <transportConnectors>
> <transportConnector uri="tcp://localhost:61616" />
> </transportConnectors>
> </broker>
>
> Template and queue/topic config:
>
> <beans>
> <amq:connectionFactory brokerURL="${broker.jmsUrl}" id="jmsFactory"
> dispatchAsync="true" />
>
> <bean id="scf"
> class="org.springframework.jms.connection.SingleConnectionFactory"
> p:targetConnectionFactory-ref="jmsFactory" />
>
> <bean class="org.springframework.jms.core.JmsTemplate"
> id="jmsTemplate" p:receiveTimeout="${jms.receivetimeout}"
> p:connectionFactory-ref="scf" />
>
> <bean class="org.springframework.jms.core.JmsTemplate"
> id="commandTemplate" p:receiveTimeout="${jms.receivetimeout}"
> p:explicitQosEnabled="true"
> p:priority="4" p:deliveryMode="2" p:timeToLive="120000"
> p:connectionFactory-ref="scf" p:defaultDestination-ref="commandTopic" />
>
> <amq:queue id="A" physicalName="queue.A" />
> <amq:queue id="B" physicalName="queue.B" />
> <amq:queue id="C" physicalName="queue.C" />
>
> <amq:topic id="commandTopic" physicalName="command.topic" /> </beans>
>
> Thanks in advance,
> Maarten
>
>
>
>
--
View this message in context: http://www.nabble.com/Broker-hitting-max-memory-with-large-persistent-queue-tp17014805s2354p17020505.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
RE: Broker hitting max memory with large persistent queue
Posted by Maarten Dirkse <Ma...@filterworks.com>.
An update:
I've now tried it with the standard amqpersistence config:
<persistenceAdapter>
<amqPersistenceAdapter directory="/var/filterworks/amq/store"
maxFileLength="32mb"/>
</persistenceAdapter>
And it still results in the same problem. When I give the broker 128mb
of memory, everything runs fine until it has processed about 126,000
messages. Then the memory usage hits 100%, and everything slows to a
crawl.
Anyone have any ideas as to what's going on?
Regards,
Maarten
-----Original Message-----
From: Maarten Dirkse [mailto:Maarten.Dirkse@filterworks.com]
Sent: 2 mei 2008 9:02
To: users@activemq.apache.org
Subject: Broker hitting max memory with large persistent queue
Hi,
I've spent quite a bit of time trying to find a solution to this
problem, but nothing that I've come across in the documentation or the
mailing list seems to help, so here goes:
I've got ActiveMQ set up to provide messaging between a chain of four
components, connected by three queues like so:
Component 1 --> queue A --> component 2 --> queue B --> component 3 -->
queue C --> component 4
Component 1 gets things going by messages and putting them on queue A.
Component 2 then processes these, and, for every message received, puts
a message on queue B. Component three then does the same thing as 2,
generating an equal amount of message for component 4. Beside the queue
pipeline, components also communicate with each other using a "command"
topic which produces 2 messages for every queue message that goes
through the pipeline.
All queues are (pure JDBC) persistent, topics are non-persistent.
This setup works very well for low volumes of messages. At a certain
point though, the first component has to start producing messages at
rates of several thousand per minute. Component 2 can't consume the
messages at the rate they're produced, so the size of queue A slowly
grows into the thousands.
The problem is that, once the size of queue A gets large enough, the
broker runs out of memory and everything will screech to a halt (ie
messages will be processed at a rate of 1 per minute or something). At
around 256mb of broker memory for example, it reaches 100% usage when
queue A contains around 16.000 messages. The number of messages that
queue A can contain before activemq becomes unusably slow grows and
shrinks according to the amount of memory the broker has.
What I don't get is why the broker slowly uses up all its memory for a
queue that persists its messages to the database (this actually happens,
the database does fill up). I've read a number of times in the
documentation and mailing list that AMQ is capable of dealing with
queues that contain millions of messages, but in my setup it grinds to a
halt at a number far below that.
Can anyone tell me what I'm doing wrong?
Below is the setup used for the broker. Queues are defined in Spring
using <amq:queue> and producers and consumers use the Spring JMS
template and JMS Listeners respectively. I'm using AMQ 5.1 with
Postgresql for persistence and Spring 2.5.3 to configure everything.
Broker config:
<broker id="broker" useJmx="true" brokerName="${brokerA.name}"
xmlns="http://activemq.apache.org/schema/core"
dataDirectory="${activemq.data}">
<persistenceAdapter>
<jdbcPersistenceAdapter dataDirectory="${activemq.data}/jdbc"
dataSource="#postgres-ds" createTablesOnStartup="true" />
</persistenceAdapter>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="128mb" />
</memoryUsage>
</systemUsage>
</systemUsage>
<transportConnectors>
<transportConnector uri="tcp://localhost:61616" />
</transportConnectors>
</broker>
Template and queue/topic config:
<beans>
<amq:connectionFactory brokerURL="${broker.jmsUrl}" id="jmsFactory"
dispatchAsync="true" />
<bean id="scf"
class="org.springframework.jms.connection.SingleConnectionFactory"
p:targetConnectionFactory-ref="jmsFactory" />
<bean class="org.springframework.jms.core.JmsTemplate"
id="jmsTemplate" p:receiveTimeout="${jms.receivetimeout}"
p:connectionFactory-ref="scf" />
<bean class="org.springframework.jms.core.JmsTemplate"
id="commandTemplate" p:receiveTimeout="${jms.receivetimeout}"
p:explicitQosEnabled="true"
p:priority="4" p:deliveryMode="2" p:timeToLive="120000"
p:connectionFactory-ref="scf" p:defaultDestination-ref="commandTopic" />
<amq:queue id="A" physicalName="queue.A" />
<amq:queue id="B" physicalName="queue.B" />
<amq:queue id="C" physicalName="queue.C" />
<amq:topic id="commandTopic" physicalName="command.topic" /> </beans>
Thanks in advance,
Maarten