You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by aliu <al...@endeca.com> on 2008/07/15 00:47:37 UTC
systemUsage configuration
Hi, I am a new user of the ActiveMQ and I was trying to figure out how to
tweak different memory settings.
I read on the forum that storeUsage controls the maximum size of the
AMQMessageStore, and memoryUsage is the maximum amount of memory the broker
will use.
However, when I ran a test of sending 50,000 messages with 1k for each
message, and I've used the following config:
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="30 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="1 gb" name="foo"/>
</storeUsage>
<tempUsage>
<tempUsage limit="1 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
My producer would block after sending about 30mb worth of data, even though
I have specified the storeUsage to be big enough to contain all my messages.
I thought what it would be is to have 30mb worth of data in memory, then
it'll spill over once it exceeds the limit.
Can someone please tell me in detail how these settings work together? I am
at a loss here.
I am using 5.1, persistent messaging with transactions.
Thanks in advance.
Audrey
--
View this message in context: http://www.nabble.com/systemUsage-configuration-tp18454843p18454843.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: systemUsage configuration
Posted by agrace <ap...@hotmail.com>.
aliu wrote:
>
> Hi, just to clarify. In my last post, when I mentioned 'all the pending
> messages are in memory', I actually mean uncommitted messages within a
> transaction. Again, can someone confirm if this statement (all uncommitted
> messages are kept in memory) is true?
>
I am also new to ActuveMQ, but most of my investigations have led me to
believe the same thing. I am also trying to send large uncommitted
transactions and I am running into the same problem. Any insight would be
great.
Thanks!
--
View this message in context: http://www.nabble.com/systemUsage-configuration-tp18454843p18538077.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: systemUsage configuration
Posted by aliu <al...@endeca.com>.
Hi, just to clarify. In my last post, when I mentioned 'all the pending
messages are in memory', I actually mean uncommitted messages within a
transaction. Again, can someone confirm if this statement (all uncommitted
messages are kept in memory) is true?
aliu wrote:
>
> The reason I've been trying to figure out these memory limit
> configurations is because we have the following use case:
>
> I want to be able to send a large amount of data (>2g) to the queue
> without any consumer consuming it for some time, but it seems like active
> mq actually keeps all the pending messages in memory, I just want to
> confirm this observation since this would put a lot of constraint on our
> memory requirement.
>
> I are using persistent messaging with transactions, with memoryUsage set
> to 1g, when sending over 50mb of data, I get an OutOfMemory error in the
> middle (around 40mb). But when I tried to do a commit with every 3mb of
> data, there was no errors.
>
> I just want to confirm whether active mq actually keeps all the pending
> messages in memory.
>
> Thanks,
> Audrey
>
>
>
> ammulder wrote:
>>
>> Are you validating the XML just to make sure it's not a syntax error?
>>
>> I get stuff onto disk with a configuration like this (and a producer
>> sending 1 MB messages):
>>
>> <destinationPolicy>
>> <policyMap>
>> <policyEntries>
>> <policyEntry queue=">" memoryLimit="50mb"/>
>> <policyEntry topic=">" memoryLimit="50mb">
>> ...
>> <systemUsage>
>> <systemUsage>
>> <memoryUsage>
>> <memoryUsage limit="20 mb"/>
>> </memoryUsage>
>> ...
>>
>>
>> Thanks,
>> Aaron
>>
>> On Tue, Jul 15, 2008 at 10:50 AM, aliu <al...@endeca.com> wrote:
>>>
>>> Hi,
>>> Thanks for the quick reply. If I don't want to keep everything in
>>> memory,
>>> and I want to be able to set how much should be stored in memory and if
>>> there are more stuff coming, it'd be spilled out to disk, what
>>> configuration
>>> settings can I use?
>>>
>>> I actually tried setting the destination limits high and the system
>>> limit
>>> low, with the following config:
>>> <policyEntry queue=">" memoryLimit="1 gb"/>
>>> ...
>>> <memoryUsage>
>>> <memoryUsage limit="30 mb"/>
>>> </memoryUsage>
>>>
>>> When I'm trying to send 50mb worth of data, my producer blocks after
>>> sending
>>> about 30mb of data. And I also tried with destination limits low and
>>> system
>>> limit high, my producer also blocks after sending the lower limit of the
>>> two
>>> settings.
>>>
>>> Just wondering if anyone can give me some advice on what I can set on
>>> how
>>> much to be allowed in memory?
>>>
>>> Thanks in advance.
>>> Audrey
>>>
>>> ammulder wrote:
>>>>
>>>> I am by no means an authority on this, but I'm not sure the store
>>>> usage or temp usage settings are actually used for anything.
>>>>
>>>> I believe you want your memory usage setting to be high enough to hold
>>>> all the traffic you expect to be in memory at once. I think this is
>>>> e.g. non-persistent messages that have not yet been delivered (regular
>>>> in-flight messages, queues that are backed up, topics with durable
>>>> subscribers who are away, etc.). There are separate settings that can
>>>> be applied to the memory limit for each destination (or wildcard
>>>> groups of queues/topics, etc.).
>>>>
>>>> If you set the destination limits high and the system limit low, you
>>>> can cause ActiveMQ to start using disk space (as "swap" I guess), but
>>>> I hear that's prone to deadlocks and etc. -- plus it doesn't seem
>>>> bound by the store/temp usage settings.
>>>>
>>>> If you are up against memory limits (due to slow/disconnected
>>>> consumers or whatever), there are various additional considerations
>>>> and configuration options you may want to use.
>>>>
>>>> Thanks,
>>>> Aaron
>>>>
>>>> On Mon, Jul 14, 2008 at 6:47 PM, aliu <al...@endeca.com> wrote:
>>>>>
>>>>> Hi, I am a new user of the ActiveMQ and I was trying to figure out how
>>>>> to
>>>>> tweak different memory settings.
>>>>> I read on the forum that storeUsage controls the maximum size of the
>>>>> AMQMessageStore, and memoryUsage is the maximum amount of memory the
>>>>> broker
>>>>> will use.
>>>>>
>>>>> However, when I ran a test of sending 50,000 messages with 1k for each
>>>>> message, and I've used the following config:
>>>>> <systemUsage>
>>>>> <systemUsage>
>>>>> <memoryUsage>
>>>>> <memoryUsage limit="30 mb"/>
>>>>> </memoryUsage>
>>>>> <storeUsage>
>>>>> <storeUsage limit="1 gb" name="foo"/>
>>>>> </storeUsage>
>>>>> <tempUsage>
>>>>> <tempUsage limit="1 mb"/>
>>>>> </tempUsage>
>>>>> </systemUsage>
>>>>> </systemUsage>
>>>>>
>>>>> My producer would block after sending about 30mb worth of data, even
>>>>> though
>>>>> I have specified the storeUsage to be big enough to contain all my
>>>>> messages.
>>>>> I thought what it would be is to have 30mb worth of data in memory,
>>>>> then
>>>>> it'll spill over once it exceeds the limit.
>>>>>
>>>>> Can someone please tell me in detail how these settings work together?
>>>>> I
>>>>> am
>>>>> at a loss here.
>>>>>
>>>>> I am using 5.1, persistent messaging with transactions.
>>>>>
>>>>> Thanks in advance.
>>>>> Audrey
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/systemUsage-configuration-tp18454843p18454843.html
>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/systemUsage-configuration-tp18454843p18467144.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>>
>>>
>>
>>
>
>
--
View this message in context: http://www.nabble.com/systemUsage-configuration-tp18454843p18490842.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: systemUsage configuration
Posted by aliu <al...@endeca.com>.
The reason I've been trying to figure out these memory limit configurations
is because we have the following use case:
I want to be able to send a large amount of data (>2g) to the queue without
any consumer consuming it for some time, but it seems like active mq
actually keeps all the pending messages in memory, I just want to confirm
this observation since this would put a lot of constraint on our memory
requirement.
I are using persistent messaging with transactions, with memoryUsage set to
1g, when sending over 50mb of data, I get an OutOfMemory error in the middle
(around 40mb). But when I tried to do a commit with every 3mb of data, there
was no errors.
I just want to confirm whether active mq actually keeps all the pending
messages in memory.
Thanks,
Audrey
ammulder wrote:
>
> Are you validating the XML just to make sure it's not a syntax error?
>
> I get stuff onto disk with a configuration like this (and a producer
> sending 1 MB messages):
>
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry queue=">" memoryLimit="50mb"/>
> <policyEntry topic=">" memoryLimit="50mb">
> ...
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="20 mb"/>
> </memoryUsage>
> ...
>
>
> Thanks,
> Aaron
>
> On Tue, Jul 15, 2008 at 10:50 AM, aliu <al...@endeca.com> wrote:
>>
>> Hi,
>> Thanks for the quick reply. If I don't want to keep everything in memory,
>> and I want to be able to set how much should be stored in memory and if
>> there are more stuff coming, it'd be spilled out to disk, what
>> configuration
>> settings can I use?
>>
>> I actually tried setting the destination limits high and the system limit
>> low, with the following config:
>> <policyEntry queue=">" memoryLimit="1 gb"/>
>> ...
>> <memoryUsage>
>> <memoryUsage limit="30 mb"/>
>> </memoryUsage>
>>
>> When I'm trying to send 50mb worth of data, my producer blocks after
>> sending
>> about 30mb of data. And I also tried with destination limits low and
>> system
>> limit high, my producer also blocks after sending the lower limit of the
>> two
>> settings.
>>
>> Just wondering if anyone can give me some advice on what I can set on how
>> much to be allowed in memory?
>>
>> Thanks in advance.
>> Audrey
>>
>> ammulder wrote:
>>>
>>> I am by no means an authority on this, but I'm not sure the store
>>> usage or temp usage settings are actually used for anything.
>>>
>>> I believe you want your memory usage setting to be high enough to hold
>>> all the traffic you expect to be in memory at once. I think this is
>>> e.g. non-persistent messages that have not yet been delivered (regular
>>> in-flight messages, queues that are backed up, topics with durable
>>> subscribers who are away, etc.). There are separate settings that can
>>> be applied to the memory limit for each destination (or wildcard
>>> groups of queues/topics, etc.).
>>>
>>> If you set the destination limits high and the system limit low, you
>>> can cause ActiveMQ to start using disk space (as "swap" I guess), but
>>> I hear that's prone to deadlocks and etc. -- plus it doesn't seem
>>> bound by the store/temp usage settings.
>>>
>>> If you are up against memory limits (due to slow/disconnected
>>> consumers or whatever), there are various additional considerations
>>> and configuration options you may want to use.
>>>
>>> Thanks,
>>> Aaron
>>>
>>> On Mon, Jul 14, 2008 at 6:47 PM, aliu <al...@endeca.com> wrote:
>>>>
>>>> Hi, I am a new user of the ActiveMQ and I was trying to figure out how
>>>> to
>>>> tweak different memory settings.
>>>> I read on the forum that storeUsage controls the maximum size of the
>>>> AMQMessageStore, and memoryUsage is the maximum amount of memory the
>>>> broker
>>>> will use.
>>>>
>>>> However, when I ran a test of sending 50,000 messages with 1k for each
>>>> message, and I've used the following config:
>>>> <systemUsage>
>>>> <systemUsage>
>>>> <memoryUsage>
>>>> <memoryUsage limit="30 mb"/>
>>>> </memoryUsage>
>>>> <storeUsage>
>>>> <storeUsage limit="1 gb" name="foo"/>
>>>> </storeUsage>
>>>> <tempUsage>
>>>> <tempUsage limit="1 mb"/>
>>>> </tempUsage>
>>>> </systemUsage>
>>>> </systemUsage>
>>>>
>>>> My producer would block after sending about 30mb worth of data, even
>>>> though
>>>> I have specified the storeUsage to be big enough to contain all my
>>>> messages.
>>>> I thought what it would be is to have 30mb worth of data in memory,
>>>> then
>>>> it'll spill over once it exceeds the limit.
>>>>
>>>> Can someone please tell me in detail how these settings work together?
>>>> I
>>>> am
>>>> at a loss here.
>>>>
>>>> I am using 5.1, persistent messaging with transactions.
>>>>
>>>> Thanks in advance.
>>>> Audrey
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/systemUsage-configuration-tp18454843p18454843.html
>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>
>>>>
>>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/systemUsage-configuration-tp18454843p18467144.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>>
>
>
--
View this message in context: http://www.nabble.com/systemUsage-configuration-tp18454843p18473128.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: systemUsage configuration
Posted by Aaron Mulder <am...@alumni.princeton.edu>.
Are you validating the XML just to make sure it's not a syntax error?
I get stuff onto disk with a configuration like this (and a producer
sending 1 MB messages):
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry queue=">" memoryLimit="50mb"/>
<policyEntry topic=">" memoryLimit="50mb">
...
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="20 mb"/>
</memoryUsage>
...
Thanks,
Aaron
On Tue, Jul 15, 2008 at 10:50 AM, aliu <al...@endeca.com> wrote:
>
> Hi,
> Thanks for the quick reply. If I don't want to keep everything in memory,
> and I want to be able to set how much should be stored in memory and if
> there are more stuff coming, it'd be spilled out to disk, what configuration
> settings can I use?
>
> I actually tried setting the destination limits high and the system limit
> low, with the following config:
> <policyEntry queue=">" memoryLimit="1 gb"/>
> ...
> <memoryUsage>
> <memoryUsage limit="30 mb"/>
> </memoryUsage>
>
> When I'm trying to send 50mb worth of data, my producer blocks after sending
> about 30mb of data. And I also tried with destination limits low and system
> limit high, my producer also blocks after sending the lower limit of the two
> settings.
>
> Just wondering if anyone can give me some advice on what I can set on how
> much to be allowed in memory?
>
> Thanks in advance.
> Audrey
>
> ammulder wrote:
>>
>> I am by no means an authority on this, but I'm not sure the store
>> usage or temp usage settings are actually used for anything.
>>
>> I believe you want your memory usage setting to be high enough to hold
>> all the traffic you expect to be in memory at once. I think this is
>> e.g. non-persistent messages that have not yet been delivered (regular
>> in-flight messages, queues that are backed up, topics with durable
>> subscribers who are away, etc.). There are separate settings that can
>> be applied to the memory limit for each destination (or wildcard
>> groups of queues/topics, etc.).
>>
>> If you set the destination limits high and the system limit low, you
>> can cause ActiveMQ to start using disk space (as "swap" I guess), but
>> I hear that's prone to deadlocks and etc. -- plus it doesn't seem
>> bound by the store/temp usage settings.
>>
>> If you are up against memory limits (due to slow/disconnected
>> consumers or whatever), there are various additional considerations
>> and configuration options you may want to use.
>>
>> Thanks,
>> Aaron
>>
>> On Mon, Jul 14, 2008 at 6:47 PM, aliu <al...@endeca.com> wrote:
>>>
>>> Hi, I am a new user of the ActiveMQ and I was trying to figure out how to
>>> tweak different memory settings.
>>> I read on the forum that storeUsage controls the maximum size of the
>>> AMQMessageStore, and memoryUsage is the maximum amount of memory the
>>> broker
>>> will use.
>>>
>>> However, when I ran a test of sending 50,000 messages with 1k for each
>>> message, and I've used the following config:
>>> <systemUsage>
>>> <systemUsage>
>>> <memoryUsage>
>>> <memoryUsage limit="30 mb"/>
>>> </memoryUsage>
>>> <storeUsage>
>>> <storeUsage limit="1 gb" name="foo"/>
>>> </storeUsage>
>>> <tempUsage>
>>> <tempUsage limit="1 mb"/>
>>> </tempUsage>
>>> </systemUsage>
>>> </systemUsage>
>>>
>>> My producer would block after sending about 30mb worth of data, even
>>> though
>>> I have specified the storeUsage to be big enough to contain all my
>>> messages.
>>> I thought what it would be is to have 30mb worth of data in memory, then
>>> it'll spill over once it exceeds the limit.
>>>
>>> Can someone please tell me in detail how these settings work together? I
>>> am
>>> at a loss here.
>>>
>>> I am using 5.1, persistent messaging with transactions.
>>>
>>> Thanks in advance.
>>> Audrey
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/systemUsage-configuration-tp18454843p18454843.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>>
>>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/systemUsage-configuration-tp18454843p18467144.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
>
Re: systemUsage configuration
Posted by aliu <al...@endeca.com>.
Hi,
Thanks for the quick reply. If I don't want to keep everything in memory,
and I want to be able to set how much should be stored in memory and if
there are more stuff coming, it'd be spilled out to disk, what configuration
settings can I use?
I actually tried setting the destination limits high and the system limit
low, with the following config:
<policyEntry queue=">" memoryLimit="1 gb"/>
...
<memoryUsage>
<memoryUsage limit="30 mb"/>
</memoryUsage>
When I'm trying to send 50mb worth of data, my producer blocks after sending
about 30mb of data. And I also tried with destination limits low and system
limit high, my producer also blocks after sending the lower limit of the two
settings.
Just wondering if anyone can give me some advice on what I can set on how
much to be allowed in memory?
Thanks in advance.
Audrey
ammulder wrote:
>
> I am by no means an authority on this, but I'm not sure the store
> usage or temp usage settings are actually used for anything.
>
> I believe you want your memory usage setting to be high enough to hold
> all the traffic you expect to be in memory at once. I think this is
> e.g. non-persistent messages that have not yet been delivered (regular
> in-flight messages, queues that are backed up, topics with durable
> subscribers who are away, etc.). There are separate settings that can
> be applied to the memory limit for each destination (or wildcard
> groups of queues/topics, etc.).
>
> If you set the destination limits high and the system limit low, you
> can cause ActiveMQ to start using disk space (as "swap" I guess), but
> I hear that's prone to deadlocks and etc. -- plus it doesn't seem
> bound by the store/temp usage settings.
>
> If you are up against memory limits (due to slow/disconnected
> consumers or whatever), there are various additional considerations
> and configuration options you may want to use.
>
> Thanks,
> Aaron
>
> On Mon, Jul 14, 2008 at 6:47 PM, aliu <al...@endeca.com> wrote:
>>
>> Hi, I am a new user of the ActiveMQ and I was trying to figure out how to
>> tweak different memory settings.
>> I read on the forum that storeUsage controls the maximum size of the
>> AMQMessageStore, and memoryUsage is the maximum amount of memory the
>> broker
>> will use.
>>
>> However, when I ran a test of sending 50,000 messages with 1k for each
>> message, and I've used the following config:
>> <systemUsage>
>> <systemUsage>
>> <memoryUsage>
>> <memoryUsage limit="30 mb"/>
>> </memoryUsage>
>> <storeUsage>
>> <storeUsage limit="1 gb" name="foo"/>
>> </storeUsage>
>> <tempUsage>
>> <tempUsage limit="1 mb"/>
>> </tempUsage>
>> </systemUsage>
>> </systemUsage>
>>
>> My producer would block after sending about 30mb worth of data, even
>> though
>> I have specified the storeUsage to be big enough to contain all my
>> messages.
>> I thought what it would be is to have 30mb worth of data in memory, then
>> it'll spill over once it exceeds the limit.
>>
>> Can someone please tell me in detail how these settings work together? I
>> am
>> at a loss here.
>>
>> I am using 5.1, persistent messaging with transactions.
>>
>> Thanks in advance.
>> Audrey
>>
>> --
>> View this message in context:
>> http://www.nabble.com/systemUsage-configuration-tp18454843p18454843.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>>
>
>
--
View this message in context: http://www.nabble.com/systemUsage-configuration-tp18454843p18467144.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: systemUsage configuration
Posted by Aaron Mulder <am...@alumni.princeton.edu>.
I am by no means an authority on this, but I'm not sure the store
usage or temp usage settings are actually used for anything.
I believe you want your memory usage setting to be high enough to hold
all the traffic you expect to be in memory at once. I think this is
e.g. non-persistent messages that have not yet been delivered (regular
in-flight messages, queues that are backed up, topics with durable
subscribers who are away, etc.). There are separate settings that can
be applied to the memory limit for each destination (or wildcard
groups of queues/topics, etc.).
If you set the destination limits high and the system limit low, you
can cause ActiveMQ to start using disk space (as "swap" I guess), but
I hear that's prone to deadlocks and etc. -- plus it doesn't seem
bound by the store/temp usage settings.
If you are up against memory limits (due to slow/disconnected
consumers or whatever), there are various additional considerations
and configuration options you may want to use.
Thanks,
Aaron
On Mon, Jul 14, 2008 at 6:47 PM, aliu <al...@endeca.com> wrote:
>
> Hi, I am a new user of the ActiveMQ and I was trying to figure out how to
> tweak different memory settings.
> I read on the forum that storeUsage controls the maximum size of the
> AMQMessageStore, and memoryUsage is the maximum amount of memory the broker
> will use.
>
> However, when I ran a test of sending 50,000 messages with 1k for each
> message, and I've used the following config:
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="30 mb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="1 gb" name="foo"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="1 mb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
>
> My producer would block after sending about 30mb worth of data, even though
> I have specified the storeUsage to be big enough to contain all my messages.
> I thought what it would be is to have 30mb worth of data in memory, then
> it'll spill over once it exceeds the limit.
>
> Can someone please tell me in detail how these settings work together? I am
> at a loss here.
>
> I am using 5.1, persistent messaging with transactions.
>
> Thanks in advance.
> Audrey
>
> --
> View this message in context: http://www.nabble.com/systemUsage-configuration-tp18454843p18454843.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
>