You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Illtud Daniel <il...@llgc.org.uk> on 2011/03/07 19:19:26 UTC
Large messages, slow consumers and java heap space
I'm using stomp, java clients and camel to move messages through
a 'pipeline' of activemq queues. The messages can be very big
(>120MB) and unfortunately the OS is 32-bit, so I want to keep
memory usage to a minimum (2GB limit). There isn't a huge number
of messages, but I keep running out of heap space. Each queue has
only one producer and one consumer.
I've attempted to read everything I can about producer flow control,
message cursors, prefetch limits, etc. and I still can't get it
to do what I want. I don't care about speed at all, I have plenty
of disk space, and all messages are persistent.
The JVM has the max 2024MB of memory.
bits of activemq.xml:
...
<policyEntry queue=">" producerFlowControl="true"
memoryLimit="150mb" queuePrefetch="1">
<pendingQueuePolicy>
<fileQueueCursor />
</pendingQueuePolicy>
</policyEntry>
...
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="1024 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="25 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="5 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
I've tried it with and without producerFlowControl. I get stuff like:
2011-03-07 18:11:28,004 | ERROR | Failed to page in more queue messages
| org.apache.activemq.broker.region.Queue | Queue:scif_mets_pre_jhove_in
java.lang.OutOfMemoryError: Java heap space
at
org.apache.activemq.openwire.v6.BaseDataStreamMarshaller.looseUnmarshalByteSequence(BaseDataStreamMarshaller.java:639)
and
2011-03-07 18:17:25,011 | WARN | Execution of JMS message listener
failed, and no ErrorHandler has been set. |
org.springframework.jms.listener.DefaultMessageListenerContainer |
DefaultMessageListenerContainer-1
java.lang.OutOfMemoryError: Java heap space
Anybody help me? How can I keep the memory usage to a minimum, at
the expense of performance, disk space, anything else?
Am I just expecting too much because my messages are too big?
--
Illtud Daniel illtud.daniel@llgc.org.uk
Prif Swyddog Technegol Chief Technical Officer
Llyfrgell Genedlaethol Cymru National Library of Wales
Re: Large messages, slow consumers and java heap space
Posted by Illtud Daniel <il...@llgc.org.uk>.
OK,
I fear that what's happening is that producer flow control
is kicking in, and that camel is taking messages off queues,
but it can't put them on queues, so camel is holding all these
messages in memory, which will eventually fill up. I don't
see why producer flow control would be applied, since everything
should go to disk, and there shouldn't be a memory problem for
activemq. I don't get anything in the activemq log about
producer flow control being applied, which I'm sure used to
happen.
So, I try another approach:
<policyEntry queue=">" producerFlowControl="false"
memoryLimit="250mb" queuePrefetch="1">
</policyEntry>
...so activemq should now be using the default store based
cursor. The memoryLimit should be big enough for any message
(remember they get to about 150MB tops) that I'm passing.
I don't have producer flow control, so I'm relying on my
disk storage not to fill up.
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="4 gb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="150 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="10 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
I'm limiting the broker to 4GB, and the JVM has 9GB, so I'm
hoping that camel has plenty to play with.
This plays out well - the heap usage climbs steadily to 4GB
over a couple of hours, but I can garbage collect and recover
over 3GB, so I'm going to leave it be now and see whether
that will run happily 24x7.
One thing I didn't look at was maxPageSize & lazyDispatch,
but I think it was camel that was taking all the heap as
it was being producer-blocked, but still consuming. I'll ask
about that on the camel list.
Thanks to all who replied.
--
Illtud Daniel illtud.daniel@llgc.org.uk
Prif Swyddog Technegol Chief Technical Officer
Llyfrgell Genedlaethol Cymru National Library of Wales
Re: Large messages, slow consumers and java heap space
Posted by Claus Ibsen <cl...@gmail.com>.
On Tue, Mar 8, 2011 at 10:52 AM, Illtud Daniel
<il...@llgc.org.uk> wrote:
> On 08/03/11 09:26, Claus Ibsen wrote:
>
>> Have you looked at some of the guides at FuseSource about the broker?
>> http://fusesource.com/products/enterprise-activemq/#documentation
>
> No, I'll take a look, thanks.
>
>> And there is a ActiveMQ in Action book to be published this month.
>> Maybe it has some details.
>
> That's the MEAP book I have at the moment.
>
Ah you wrote Camel MEAP book in your previous mail.
I think there is a JIRA ticket for BlobMessage support in
activemq-camel component.
That may help bridge Camel and AMQ easier. However I don't know the
state of the JIRA.
> --
> Illtud Daniel illtud.daniel@llgc.org.uk
> Prif Swyddog Technegol Chief Technical Officer
> Llyfrgell Genedlaethol Cymru National Library of Wales
>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: Large messages, slow consumers and java heap space
Posted by Illtud Daniel <il...@llgc.org.uk>.
On 08/03/11 09:26, Claus Ibsen wrote:
> Have you looked at some of the guides at FuseSource about the broker?
> http://fusesource.com/products/enterprise-activemq/#documentation
No, I'll take a look, thanks.
> And there is a ActiveMQ in Action book to be published this month.
> Maybe it has some details.
That's the MEAP book I have at the moment.
--
Illtud Daniel illtud.daniel@llgc.org.uk
Prif Swyddog Technegol Chief Technical Officer
Llyfrgell Genedlaethol Cymru National Library of Wales
Re: Large messages, slow consumers and java heap space
Posted by Illtud Daniel <il...@llgc.org.uk>.
On 08/03/11 09:32, Dejan Bosanac wrote:
> I'd suggest using blob messages for messages of that size:
> http://activemq.apache.org/blob-messages.html
Unfortunately, we've already architected the processes around
xml payload messages, so we've got camel picking some info
from the messages with xpath, populating headers as they go,
etc.
I'm moving our message broker to a 64-bit machine and giving
it more heap. I'll report back.
--
Illtud Daniel illtud.daniel@llgc.org.uk
Prif Swyddog Technegol Chief Technical Officer
Llyfrgell Genedlaethol Cymru National Library of Wales
Re: Large messages, slow consumers and java heap space
Posted by Dejan Bosanac <de...@nighttale.net>.
Hi,
I'd suggest using blob messages for messages of that size:
http://activemq.apache.org/blob-messages.html
<http://activemq.apache.org/blob-messages.html>
Regards
--
Dejan Bosanac
-----------------
FuseSource - The experts in open source integration and messaging.
Email: dejanb@fusesource.com
Web: http://fusesource.com
Twitter: http://twitter.com/dejanb
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net
On Tue, Mar 8, 2011 at 10:26 AM, Claus Ibsen <cl...@gmail.com> wrote:
> On Tue, Mar 8, 2011 at 10:15 AM, Illtud Daniel
> <il...@llgc.org.uk> wrote:
> > On 07/03/11 18:19, Illtud Daniel wrote:
> >
> >> I've attempted to read everything I can about producer flow control,
> >> message cursors, prefetch limits, etc. and I still can't get it
> >> to do what I want. I don't care about speed at all, I have plenty
> >> of disk space, and all messages are persistent.
> >
> > Anyboody? I've read the java.lang.OutOfMemory FAQ, I have the
> > camel book MEAP, I can't figure it out. Is it just that the
> > messages are too large and 2GB of heap won't ever be enough?
> >
> > Does anybody else have experience of large message sizes?
> >
>
> Have you looked at some of the guides at FuseSource about the broker?
> http://fusesource.com/products/enterprise-activemq/#documentation
>
> And there is a ActiveMQ in Action book to be published this month.
> Maybe it has some details.
>
>
>
> > --
> > Illtud Daniel illtud.daniel@llgc.org.uk
> > Prif Swyddog Technegol Chief Technical Officer
> > Llyfrgell Genedlaethol Cymru National Library of Wales
> >
>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: cibsen@fusesource.com
> Web: http://fusesource.com
> Twitter: davsclaus
> Blog: http://davsclaus.blogspot.com/
> Author of Camel in Action: http://www.manning.com/ibsen/
>
Re: Large messages, slow consumers and java heap space
Posted by Claus Ibsen <cl...@gmail.com>.
On Tue, Mar 8, 2011 at 10:15 AM, Illtud Daniel
<il...@llgc.org.uk> wrote:
> On 07/03/11 18:19, Illtud Daniel wrote:
>
>> I've attempted to read everything I can about producer flow control,
>> message cursors, prefetch limits, etc. and I still can't get it
>> to do what I want. I don't care about speed at all, I have plenty
>> of disk space, and all messages are persistent.
>
> Anyboody? I've read the java.lang.OutOfMemory FAQ, I have the
> camel book MEAP, I can't figure it out. Is it just that the
> messages are too large and 2GB of heap won't ever be enough?
>
> Does anybody else have experience of large message sizes?
>
Have you looked at some of the guides at FuseSource about the broker?
http://fusesource.com/products/enterprise-activemq/#documentation
And there is a ActiveMQ in Action book to be published this month.
Maybe it has some details.
> --
> Illtud Daniel illtud.daniel@llgc.org.uk
> Prif Swyddog Technegol Chief Technical Officer
> Llyfrgell Genedlaethol Cymru National Library of Wales
>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: Large messages, slow consumers and java heap space
Posted by Illtud Daniel <il...@llgc.org.uk>.
On 07/03/11 18:19, Illtud Daniel wrote:
> I've attempted to read everything I can about producer flow control,
> message cursors, prefetch limits, etc. and I still can't get it
> to do what I want. I don't care about speed at all, I have plenty
> of disk space, and all messages are persistent.
Anyboody? I've read the java.lang.OutOfMemory FAQ, I have the
camel book MEAP, I can't figure it out. Is it just that the
messages are too large and 2GB of heap won't ever be enough?
Does anybody else have experience of large message sizes?
--
Illtud Daniel illtud.daniel@llgc.org.uk
Prif Swyddog Technegol Chief Technical Officer
Llyfrgell Genedlaethol Cymru National Library of Wales