You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by dcheckoway <dc...@gmail.com> on 2010/04/02 13:30:34 UTC
Help understanding memory usage/limits
I could use some help understanding memory usage & limits in activemq. 5.3.1
is what I'm running, fwiw.
1. Out of the box, <systemUsage> in activemq.xml is commented out. I'm
passing ACTIVEMQ_OPTS="-Xmx2048M ...". What is the default behavior in that
case? Does that mean activemq will take advantage of the 2gb heap by
default, or do I need to configure <memoryUsage> explicitly? If it needs to
be explicit, can I safely set <memoryUsage limit="2 gb"/> ? If not, what's
a recommended setting given -Xmx2048M?
2. On <policyEntry>, what exactly does "memoryLimit" do? I have
producerFlowControl="false", and I wasn't sure if memoryLimit only applies
when producer flow control is enabled, or what?
3. What happens if you set the <policyEntry> elements' memoryLimit to the
same size as your heap and/or <memoryUsage limit>?
I'd like to understand the interaction of all of these factors. Any info
you can provide would be MUCH appreciated!!!
Thanks,
Dan
--
View this message in context: http://old.nabble.com/Help-understanding-memory-usage-limits-tp28118112p28118112.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Help understanding memory usage/limits
Posted by Richard Holt <ri...@btopenworld.com>.
ok, so hypothetically if we had 1 queue. then there should be no out of
memory errors using that configuration?
also this causes us issues anyway as we have a solution that has an unbound
potential number of queues on the system. so we cannot judge the appropriate
value to set for the destination policy.
maybe it makes more sense for the memory allocated to the destination policy
to share the memory pool amongst the queues allocated?
we are testing these configurations as we go and are seeing some OOM issues
with the configuration on 5.3.1. this does seem to be related to the number
of producers, is there also some kind of cost to having producers connect to
the system and is it a known cost? so we can come up with some kind of best
guess?
thanks for all your help and advice
Gary Tully wrote:
>
> 70% is the magic number, there is a cursorHighWaterMark destination policy
> that defaults to 70% of system usage. your config can go to 80%
>
> On 7 April 2010 12:15, Richard Holt <ri...@btopenworld.com> wrote:
>
>>
>> Hi Gary,
>>
>> It appears i am really not understanding this so could you clarify?
>>
>> You wrote:
>>
>> "Which ever destination tips it over the limit will have its memory usage
>> flushed to disk because it is
>> configured with a fileQueueCursor."
>>
>> and then wrote:
>>
>> "The salient point being that paging to disk is dependent on systemUsage
>> configuration, not individual/per destination memory usage."
>>
>> To me, this says that once the combined destinations usage exceeds the
>> systemusage/memoryusage of 20mb then the destination which did this is
>> flushed.
>>
>> However, given that i have 16 * 1mb queues how can this ever occur?
>> Surely
>> this means my maximum memory is 16mb?
>>
>> So, flushing must be occuring when destinatons exceed the 1mb value
>> irrespective of the systemusage/memoryusage setting...
>>
>> Or does it?
>>
>>
>>
>> Gary Tully wrote:
>> >
>> > richard, you have configured a 1mb limit for every queue because you
>> use
>> > the
>> > ">". So the broker will keep up to 1mb of space in memory for each
>> queue,
>> > up
>> > to 16MB for 16 queues.
>> >
>> > The paging (via fileQueueCursor) kicks in for each destination only
>> when
>> > 70%
>> > of the shared system limit (20mb) is reached. Which ever destination
>> tips
>> > it
>> > over the limit will have its memory usage flushed to disk because it is
>> > configured with a fileQueueCursor. System usage will reduce by 1Mb and
>> the
>> > rest of the destinations will continue to use memory up to their limit.
>> >
>> > The salient point being that paging to disk is dependent on systemUsage
>> > configuration, not individual/per destination memory usage.
>> >
>> >
>> > On 7 April 2010 11:22, Richard Holt <ri...@btopenworld.com>
>> wrote:
>> >
>> >>
>> >> Sorry about the thread hijack - however if i have something like this
>> >>
>> >> <destinationPolicy>
>> >> <policyMap>
>> >> <policyEntries>
>> >> <policyEntry queue=">" memoryLimit="1mb"
>> >> producerFlowControl="false">
>> >> <pendingQueuePolicy>
>> >> <fileQueueCursor/>
>> >> </pendingQueuePolicy>
>> >> </policyEntry>
>> >> </policyEntries>
>> >> </policyMap>
>> >> </destinationPolicy>
>> >>
>> >> <systemUsage>
>> >> <systemUsage>
>> >> <memoryUsage>
>> >> <memoryUsage limit="20mb"/>
>> >> </memoryUsage>
>> >> <storeUsage>
>> >> <storeUsage limit="100mb"/>
>> >> </storeUsage>
>> >> <tempUsage>
>> >> <tempUsage limit="100mb"/>
>> >> </tempUsage>
>> >> </systemUsage>
>> >> </systemUsage>
>> >>
>> >> and an Xmx of 32mb
>> >>
>> >> Once it reaches a limit of 1mb of messages per queue it will start to
>> >> page
>> >> out to disk.
>> >>
>> >> And the activemq system reserves 20mb of memory for itself?
>> >>
>> >> Is the 1mb then across all queues. So if i have 16 queues i in fact
>> need
>> >> 16mb. Because then the 20mb + 16 * 1mb will exceed my xmx value?
>> >>
>> >> Sorry if i am being obtuse but like the poster i am trying to see how
>> the
>> >> 3
>> >> memory settings relate to each other.
>> >>
>> >> Coincidentally i have tested the above on 5.3.1 and it works really
>> well.
>> >> If
>> >> i use 5.3 it breaks which i think is due to this
>> >> https://issues.apache.org/activemq/browse/AMQ-2610 and this
>> >>
>> >> --
>> >> View this message in context:
>> >>
>> http://old.nabble.com/Help-understanding-memory-usage-limits-tp28118112p28163080.html
>> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >>
>> >>
>> >
>> >
>> > --
>> > http://blog.garytully.com
>> >
>> > Open Source Integration
>> > http://fusesource.com
>> >
>> >
>>
>> --
>> View this message in context:
>> http://old.nabble.com/Help-understanding-memory-usage-limits-tp28118112p28163528.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> http://blog.garytully.com
>
> Open Source Integration
> http://fusesource.com
>
>
--
View this message in context: http://old.nabble.com/Help-understanding-memory-usage-limits-tp28118112p28164291.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Help understanding memory usage/limits
Posted by Gary Tully <ga...@gmail.com>.
70% is the magic number, there is a cursorHighWaterMark destination policy
that defaults to 70% of system usage. your config can go to 80%
On 7 April 2010 12:15, Richard Holt <ri...@btopenworld.com> wrote:
>
> Hi Gary,
>
> It appears i am really not understanding this so could you clarify?
>
> You wrote:
>
> "Which ever destination tips it over the limit will have its memory usage
> flushed to disk because it is
> configured with a fileQueueCursor."
>
> and then wrote:
>
> "The salient point being that paging to disk is dependent on systemUsage
> configuration, not individual/per destination memory usage."
>
> To me, this says that once the combined destinations usage exceeds the
> systemusage/memoryusage of 20mb then the destination which did this is
> flushed.
>
> However, given that i have 16 * 1mb queues how can this ever occur? Surely
> this means my maximum memory is 16mb?
>
> So, flushing must be occuring when destinatons exceed the 1mb value
> irrespective of the systemusage/memoryusage setting...
>
> Or does it?
>
>
>
> Gary Tully wrote:
> >
> > richard, you have configured a 1mb limit for every queue because you use
> > the
> > ">". So the broker will keep up to 1mb of space in memory for each queue,
> > up
> > to 16MB for 16 queues.
> >
> > The paging (via fileQueueCursor) kicks in for each destination only when
> > 70%
> > of the shared system limit (20mb) is reached. Which ever destination tips
> > it
> > over the limit will have its memory usage flushed to disk because it is
> > configured with a fileQueueCursor. System usage will reduce by 1Mb and
> the
> > rest of the destinations will continue to use memory up to their limit.
> >
> > The salient point being that paging to disk is dependent on systemUsage
> > configuration, not individual/per destination memory usage.
> >
> >
> > On 7 April 2010 11:22, Richard Holt <ri...@btopenworld.com>
> wrote:
> >
> >>
> >> Sorry about the thread hijack - however if i have something like this
> >>
> >> <destinationPolicy>
> >> <policyMap>
> >> <policyEntries>
> >> <policyEntry queue=">" memoryLimit="1mb"
> >> producerFlowControl="false">
> >> <pendingQueuePolicy>
> >> <fileQueueCursor/>
> >> </pendingQueuePolicy>
> >> </policyEntry>
> >> </policyEntries>
> >> </policyMap>
> >> </destinationPolicy>
> >>
> >> <systemUsage>
> >> <systemUsage>
> >> <memoryUsage>
> >> <memoryUsage limit="20mb"/>
> >> </memoryUsage>
> >> <storeUsage>
> >> <storeUsage limit="100mb"/>
> >> </storeUsage>
> >> <tempUsage>
> >> <tempUsage limit="100mb"/>
> >> </tempUsage>
> >> </systemUsage>
> >> </systemUsage>
> >>
> >> and an Xmx of 32mb
> >>
> >> Once it reaches a limit of 1mb of messages per queue it will start to
> >> page
> >> out to disk.
> >>
> >> And the activemq system reserves 20mb of memory for itself?
> >>
> >> Is the 1mb then across all queues. So if i have 16 queues i in fact need
> >> 16mb. Because then the 20mb + 16 * 1mb will exceed my xmx value?
> >>
> >> Sorry if i am being obtuse but like the poster i am trying to see how
> the
> >> 3
> >> memory settings relate to each other.
> >>
> >> Coincidentally i have tested the above on 5.3.1 and it works really
> well.
> >> If
> >> i use 5.3 it breaks which i think is due to this
> >> https://issues.apache.org/activemq/browse/AMQ-2610 and this
> >>
> >> --
> >> View this message in context:
> >>
> http://old.nabble.com/Help-understanding-memory-usage-limits-tp28118112p28163080.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
> > --
> > http://blog.garytully.com
> >
> > Open Source Integration
> > http://fusesource.com
> >
> >
>
> --
> View this message in context:
> http://old.nabble.com/Help-understanding-memory-usage-limits-tp28118112p28163528.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
http://blog.garytully.com
Open Source Integration
http://fusesource.com
Re: Help understanding memory usage/limits
Posted by Richard Holt <ri...@btopenworld.com>.
Hi Gary,
It appears i am really not understanding this so could you clarify?
You wrote:
"Which ever destination tips it over the limit will have its memory usage
flushed to disk because it is
configured with a fileQueueCursor."
and then wrote:
"The salient point being that paging to disk is dependent on systemUsage
configuration, not individual/per destination memory usage."
To me, this says that once the combined destinations usage exceeds the
systemusage/memoryusage of 20mb then the destination which did this is
flushed.
However, given that i have 16 * 1mb queues how can this ever occur? Surely
this means my maximum memory is 16mb?
So, flushing must be occuring when destinatons exceed the 1mb value
irrespective of the systemusage/memoryusage setting...
Or does it?
Gary Tully wrote:
>
> richard, you have configured a 1mb limit for every queue because you use
> the
> ">". So the broker will keep up to 1mb of space in memory for each queue,
> up
> to 16MB for 16 queues.
>
> The paging (via fileQueueCursor) kicks in for each destination only when
> 70%
> of the shared system limit (20mb) is reached. Which ever destination tips
> it
> over the limit will have its memory usage flushed to disk because it is
> configured with a fileQueueCursor. System usage will reduce by 1Mb and the
> rest of the destinations will continue to use memory up to their limit.
>
> The salient point being that paging to disk is dependent on systemUsage
> configuration, not individual/per destination memory usage.
>
>
> On 7 April 2010 11:22, Richard Holt <ri...@btopenworld.com> wrote:
>
>>
>> Sorry about the thread hijack - however if i have something like this
>>
>> <destinationPolicy>
>> <policyMap>
>> <policyEntries>
>> <policyEntry queue=">" memoryLimit="1mb"
>> producerFlowControl="false">
>> <pendingQueuePolicy>
>> <fileQueueCursor/>
>> </pendingQueuePolicy>
>> </policyEntry>
>> </policyEntries>
>> </policyMap>
>> </destinationPolicy>
>>
>> <systemUsage>
>> <systemUsage>
>> <memoryUsage>
>> <memoryUsage limit="20mb"/>
>> </memoryUsage>
>> <storeUsage>
>> <storeUsage limit="100mb"/>
>> </storeUsage>
>> <tempUsage>
>> <tempUsage limit="100mb"/>
>> </tempUsage>
>> </systemUsage>
>> </systemUsage>
>>
>> and an Xmx of 32mb
>>
>> Once it reaches a limit of 1mb of messages per queue it will start to
>> page
>> out to disk.
>>
>> And the activemq system reserves 20mb of memory for itself?
>>
>> Is the 1mb then across all queues. So if i have 16 queues i in fact need
>> 16mb. Because then the 20mb + 16 * 1mb will exceed my xmx value?
>>
>> Sorry if i am being obtuse but like the poster i am trying to see how the
>> 3
>> memory settings relate to each other.
>>
>> Coincidentally i have tested the above on 5.3.1 and it works really well.
>> If
>> i use 5.3 it breaks which i think is due to this
>> https://issues.apache.org/activemq/browse/AMQ-2610 and this
>>
>> --
>> View this message in context:
>> http://old.nabble.com/Help-understanding-memory-usage-limits-tp28118112p28163080.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> http://blog.garytully.com
>
> Open Source Integration
> http://fusesource.com
>
>
--
View this message in context: http://old.nabble.com/Help-understanding-memory-usage-limits-tp28118112p28163528.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Help understanding memory usage/limits
Posted by Gary Tully <ga...@gmail.com>.
richard, you have configured a 1mb limit for every queue because you use the
">". So the broker will keep up to 1mb of space in memory for each queue, up
to 16MB for 16 queues.
The paging (via fileQueueCursor) kicks in for each destination only when 70%
of the shared system limit (20mb) is reached. Which ever destination tips it
over the limit will have its memory usage flushed to disk because it is
configured with a fileQueueCursor. System usage will reduce by 1Mb and the
rest of the destinations will continue to use memory up to their limit.
The salient point being that paging to disk is dependent on systemUsage
configuration, not individual/per destination memory usage.
On 7 April 2010 11:22, Richard Holt <ri...@btopenworld.com> wrote:
>
> Sorry about the thread hijack - however if i have something like this
>
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry queue=">" memoryLimit="1mb"
> producerFlowControl="false">
> <pendingQueuePolicy>
> <fileQueueCursor/>
> </pendingQueuePolicy>
> </policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
>
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="20mb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="100mb"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="100mb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
>
> and an Xmx of 32mb
>
> Once it reaches a limit of 1mb of messages per queue it will start to page
> out to disk.
>
> And the activemq system reserves 20mb of memory for itself?
>
> Is the 1mb then across all queues. So if i have 16 queues i in fact need
> 16mb. Because then the 20mb + 16 * 1mb will exceed my xmx value?
>
> Sorry if i am being obtuse but like the poster i am trying to see how the 3
> memory settings relate to each other.
>
> Coincidentally i have tested the above on 5.3.1 and it works really well.
> If
> i use 5.3 it breaks which i think is due to this
> https://issues.apache.org/activemq/browse/AMQ-2610 and this
>
> --
> View this message in context:
> http://old.nabble.com/Help-understanding-memory-usage-limits-tp28118112p28163080.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
http://blog.garytully.com
Open Source Integration
http://fusesource.com
Re: Help understanding memory usage/limits
Posted by Richard Holt <ri...@btopenworld.com>.
Sorry about the thread hijack - however if i have something like this
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry queue=">" memoryLimit="1mb"
producerFlowControl="false">
<pendingQueuePolicy>
<fileQueueCursor/>
</pendingQueuePolicy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="20mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="100mb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="100mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
and an Xmx of 32mb
Once it reaches a limit of 1mb of messages per queue it will start to page
out to disk.
And the activemq system reserves 20mb of memory for itself?
Is the 1mb then across all queues. So if i have 16 queues i in fact need
16mb. Because then the 20mb + 16 * 1mb will exceed my xmx value?
Sorry if i am being obtuse but like the poster i am trying to see how the 3
memory settings relate to each other.
Coincidentally i have tested the above on 5.3.1 and it works really well. If
i use 5.3 it breaks which i think is due to this
https://issues.apache.org/activemq/browse/AMQ-2610 and this
--
View this message in context: http://old.nabble.com/Help-understanding-memory-usage-limits-tp28118112p28163080.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Help understanding memory usage/limits
Posted by Gary Tully <ga...@gmail.com>.
Some answers in line:
On 2 April 2010 17:47, dcheckoway <dc...@gmail.com> wrote:
>
> Thanks Gary. So lemme toss some actual config numbers out there and please
> tell me if this makes any sense...or if I'm shooting myself in the foot
> somehow:
>
> Destinations:
> - 1 queue with very heavy use, say 50 million messages per day
> - 1 queue with very light use, say up to 500 messages per day
> - 2 topics with very light use, say up to 500 messages per day
>
> ACTIVEMQ_OPTS="-Xmx3072M ..." (so 3gb heap)
>
> My attempt to account for pretty light topic use:
> <policyEntry topic=">" producerFlowControl="false"
> memoryLimit="64mb">
>
> My attempt to account for heavier queue use:
> <policyEntry queue=">" producerFlowControl="false"
> memoryLimit="1024mb">
>
> And then I'll uncomment the <systemUsage> section and do this...you had
> said
> 70% of heap, so if heap is 3gb, let's just say 2gb...
>
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="2048 mb"/>
> </memoryUsage>
> </systemUsage>
> </systemUsage>
>
> What do you make of all of this? Does it look like I have a clue? Cuz I
> don't feel like I do... :-)
> I'm guessing message size may be more critical to sizing the memoryLimit
> parameters, but I still don't fully understand what memoryLimit *IS*
> exactly.
Correct, the message size is important. The memory limit is really just a
number that is the running total of the accumulated messages sizes that are
held in memory at a given time. The broker keeps as many pending
messages as it can in memory and will block (the destination or producer) if
the limit is reached or can page messages to disk to free up memory, if a
file or store pending message cursor is used. see: message
cursors<http://activemq.apache.org/message-cursors.html> for
more detail on the cursor strategies.
For a destination with large queue sizes, to prevent flow control or
blocking, configuring a message cursor that can page to disk when system
usage is reached is often a good idea. In this way, the queue depth is only
limited by disk usage and producers are not inhibited.
> Is that a per-queue and/or per-topic limit of how much memory can
> be allocated to any given destination of the respective type (queue or
> topic)?
>
> Is there a way to allocate a specific memoryLimit to individual specific
> queues & topics? That would probably be smarter in my case if such a thing
> is available.
>
Yes, they are per
destination<http://activemq.apache.org/per-destination-policies.html>,
note the ">" that you are using in the policy entry, this is the
wildcard<http://activemq.apache.org/wildcards.html>that matches all
destinations of a given type. you could reference a single
queue or topic name there to make it destination specific.
> Sorry to blast more questions at you...I'd just like to have a better
> understanding.
>
> Thanks!
> Dan
>
>
> Gary Tully wrote:
> >
> > On 2 April 2010 12:30, dcheckoway <dc...@gmail.com> wrote:
> >> be explicit, can I safely set <memoryUsage limit="2 gb"/> ? If not,
> >> what's
> >> a recommended setting given -Xmx2048M?
> >>
> >> No, there are some hardcoded defaults for systemUsage think default for
> > memory is 64MB.
> > You need to take into account the expected number of destinations and
> > connections and producers/consumers as these will require memory.
> > Something like 70% of available heap is a reasonable initial gestimate.
> >
> >
> >> 2. On <policyEntry>, what exactly does "memoryLimit" do? I have
> >> producerFlowControl="false", and I wasn't sure if memoryLimit only
> >> applies
> >> when producer flow control is enabled, or what?
> >>
> >
> > That limits the amount of memory consumed by messages on that
> destination.
> > When that limit is reached, enqueues block unless systemUsage limits are
> > reached and message are spooled to disk. With producerFlowControl, the
> > producer is blocked rather than the destination in the broker.
> >
> >>
> >> 3. What happens if you set the <policyEntry> elements' memoryLimit to
> the
> >> same size as your heap and/or <memoryUsage limit>?
> >>
> >
> > that destination can block all other destinations by consuming all of the
> > memory resources.
> >
> >
> >> I'd like to understand the interaction of all of these factors. Any
> info
> >> you can provide would be MUCH appreciated!!!
> >>
> >> Thanks,
> >> Dan
> >> --
> >> View this message in context:
> >>
> http://old.nabble.com/Help-understanding-memory-usage-limits-tp28118112p28118112.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
> > --
> > http://blog.garytully.com
> >
> > Open Source Integration
> > http://fusesource.com
> >
> >
>
> --
> View this message in context:
> http://old.nabble.com/Help-understanding-memory-usage-limits-tp28118112p28120846.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
http://blog.garytully.com
Open Source Integration
http://fusesource.com
Re: Help understanding memory usage/limits
Posted by Richard Holt <ri...@btopenworld.com>.
i am also interested in the answers to this please if anyone can help, the
documentation surrounding this is a bit spartan...
dcheckoway wrote:
>
> Thanks Gary. So lemme toss some actual config numbers out there and
> please tell me if this makes any sense...or if I'm shooting myself in the
> foot somehow:
>
> Destinations:
> - 1 queue with very heavy use, say 50 million messages per day
> - 1 queue with very light use, say up to 500 messages per day
> - 2 topics with very light use, say up to 500 messages per day
>
> ACTIVEMQ_OPTS="-Xmx3072M ..." (so 3gb heap)
>
> My attempt to account for pretty light topic use:
> <policyEntry topic=">" producerFlowControl="false"
> memoryLimit="64mb">
>
> My attempt to account for heavier queue use:
> <policyEntry queue=">" producerFlowControl="false"
> memoryLimit="1024mb">
>
> And then I'll uncomment the <systemUsage> section and do this...you had
> said 70% of heap, so if heap is 3gb, let's just say 2gb...
>
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="2048 mb"/>
> </memoryUsage>
> </systemUsage>
> </systemUsage>
>
> What do you make of all of this? Does it look like I have a clue? Cuz I
> don't feel like I do... :-)
>
> I'm guessing message size may be more critical to sizing the memoryLimit
> parameters, but I still don't fully understand what memoryLimit *IS*
> exactly. Is that a per-queue and/or per-topic limit of how much memory
> can be allocated to any given destination of the respective type (queue or
> topic)?
>
> Is there a way to allocate a specific memoryLimit to individual specific
> queues & topics? That would probably be smarter in my case if such a
> thing is available.
>
> Sorry to blast more questions at you...I'd just like to have a better
> understanding.
>
> Thanks!
> Dan
>
>
> Gary Tully wrote:
>>
>> On 2 April 2010 12:30, dcheckoway <dc...@gmail.com> wrote:
>>> be explicit, can I safely set <memoryUsage limit="2 gb"/> ? If not,
>>> what's
>>> a recommended setting given -Xmx2048M?
>>>
>>> No, there are some hardcoded defaults for systemUsage think default for
>> memory is 64MB.
>> You need to take into account the expected number of destinations and
>> connections and producers/consumers as these will require memory.
>> Something like 70% of available heap is a reasonable initial gestimate.
>>
>>
>>> 2. On <policyEntry>, what exactly does "memoryLimit" do? I have
>>> producerFlowControl="false", and I wasn't sure if memoryLimit only
>>> applies
>>> when producer flow control is enabled, or what?
>>>
>>
>> That limits the amount of memory consumed by messages on that
>> destination.
>> When that limit is reached, enqueues block unless systemUsage limits are
>> reached and message are spooled to disk. With producerFlowControl, the
>> producer is blocked rather than the destination in the broker.
>>
>>>
>>> 3. What happens if you set the <policyEntry> elements' memoryLimit to
>>> the
>>> same size as your heap and/or <memoryUsage limit>?
>>>
>>
>> that destination can block all other destinations by consuming all of the
>> memory resources.
>>
>>
>>> I'd like to understand the interaction of all of these factors. Any
>>> info
>>> you can provide would be MUCH appreciated!!!
>>>
>>> Thanks,
>>> Dan
>>> --
>>> View this message in context:
>>> http://old.nabble.com/Help-understanding-memory-usage-limits-tp28118112p28118112.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>> --
>> http://blog.garytully.com
>>
>> Open Source Integration
>> http://fusesource.com
>>
>>
>
>
--
View this message in context: http://old.nabble.com/Help-understanding-memory-usage-limits-tp28118112p28161952.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Help understanding memory usage/limits
Posted by dcheckoway <dc...@gmail.com>.
Thanks Gary. So lemme toss some actual config numbers out there and please
tell me if this makes any sense...or if I'm shooting myself in the foot
somehow:
Destinations:
- 1 queue with very heavy use, say 50 million messages per day
- 1 queue with very light use, say up to 500 messages per day
- 2 topics with very light use, say up to 500 messages per day
ACTIVEMQ_OPTS="-Xmx3072M ..." (so 3gb heap)
My attempt to account for pretty light topic use:
<policyEntry topic=">" producerFlowControl="false"
memoryLimit="64mb">
My attempt to account for heavier queue use:
<policyEntry queue=">" producerFlowControl="false"
memoryLimit="1024mb">
And then I'll uncomment the <systemUsage> section and do this...you had said
70% of heap, so if heap is 3gb, let's just say 2gb...
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="2048 mb"/>
</memoryUsage>
</systemUsage>
</systemUsage>
What do you make of all of this? Does it look like I have a clue? Cuz I
don't feel like I do... :-)
I'm guessing message size may be more critical to sizing the memoryLimit
parameters, but I still don't fully understand what memoryLimit *IS*
exactly. Is that a per-queue and/or per-topic limit of how much memory can
be allocated to any given destination of the respective type (queue or
topic)?
Is there a way to allocate a specific memoryLimit to individual specific
queues & topics? That would probably be smarter in my case if such a thing
is available.
Sorry to blast more questions at you...I'd just like to have a better
understanding.
Thanks!
Dan
Gary Tully wrote:
>
> On 2 April 2010 12:30, dcheckoway <dc...@gmail.com> wrote:
>> be explicit, can I safely set <memoryUsage limit="2 gb"/> ? If not,
>> what's
>> a recommended setting given -Xmx2048M?
>>
>> No, there are some hardcoded defaults for systemUsage think default for
> memory is 64MB.
> You need to take into account the expected number of destinations and
> connections and producers/consumers as these will require memory.
> Something like 70% of available heap is a reasonable initial gestimate.
>
>
>> 2. On <policyEntry>, what exactly does "memoryLimit" do? I have
>> producerFlowControl="false", and I wasn't sure if memoryLimit only
>> applies
>> when producer flow control is enabled, or what?
>>
>
> That limits the amount of memory consumed by messages on that destination.
> When that limit is reached, enqueues block unless systemUsage limits are
> reached and message are spooled to disk. With producerFlowControl, the
> producer is blocked rather than the destination in the broker.
>
>>
>> 3. What happens if you set the <policyEntry> elements' memoryLimit to the
>> same size as your heap and/or <memoryUsage limit>?
>>
>
> that destination can block all other destinations by consuming all of the
> memory resources.
>
>
>> I'd like to understand the interaction of all of these factors. Any info
>> you can provide would be MUCH appreciated!!!
>>
>> Thanks,
>> Dan
>> --
>> View this message in context:
>> http://old.nabble.com/Help-understanding-memory-usage-limits-tp28118112p28118112.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> http://blog.garytully.com
>
> Open Source Integration
> http://fusesource.com
>
>
--
View this message in context: http://old.nabble.com/Help-understanding-memory-usage-limits-tp28118112p28120846.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Help understanding memory usage/limits
Posted by Gary Tully <ga...@gmail.com>.
On 2 April 2010 12:30, dcheckoway <dc...@gmail.com> wrote:
>
> I could use some help understanding memory usage & limits in activemq.
> 5.3.1
> is what I'm running, fwiw.
>
> 1. Out of the box, <systemUsage> in activemq.xml is commented out. I'm
> passing ACTIVEMQ_OPTS="-Xmx2048M ...". What is the default behavior in
> that
> case? Does that mean activemq will take advantage of the 2gb heap by
> default, or do I need to configure <memoryUsage> explicitly? If it needs
> to
> be explicit, can I safely set <memoryUsage limit="2 gb"/> ? If not, what's
> a recommended setting given -Xmx2048M?
>
> No, there are some hardcoded defaults for systemUsage think default for
memory is 64MB.
You need to take into account the expected number of destinations and
connections and producers/consumers as these will require memory.
Something like 70% of available heap is a reasonable initial gestimate.
> 2. On <policyEntry>, what exactly does "memoryLimit" do? I have
> producerFlowControl="false", and I wasn't sure if memoryLimit only applies
> when producer flow control is enabled, or what?
>
That limits the amount of memory consumed by messages on that destination.
When that limit is reached, enqueues block unless systemUsage limits are
reached and message are spooled to disk. With producerFlowControl, the
producer is blocked rather than the destination in the broker.
>
> 3. What happens if you set the <policyEntry> elements' memoryLimit to the
> same size as your heap and/or <memoryUsage limit>?
>
that destination can block all other destinations by consuming all of the
memory resources.
> I'd like to understand the interaction of all of these factors. Any info
> you can provide would be MUCH appreciated!!!
>
> Thanks,
> Dan
> --
> View this message in context:
> http://old.nabble.com/Help-understanding-memory-usage-limits-tp28118112p28118112.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
http://blog.garytully.com
Open Source Integration
http://fusesource.com