You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by xawiers <xa...@gmail.com> on 2013/01/02 19:43:25 UTC
Re: Reply:Re: Message priority
Anyway. Tested with 5.7.0 and 5.5.1:
if define vmcursor and lower memoryUsage (limit 64mb)
then memory fills up adding more and more messages and then producer is
stopped.
is here any setting to have not big memory usage and small bach size to
fetch messages from persistence DB (I use mysql) and preserve priorities?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Message-priority-tp4040260p4661227.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Message priority
Posted by metatech <me...@gmail.com>.
gtully wrote
> policyEntry queue="xxx" useCache="false"
> expireMessagesPeriod="0"queuePrefetch="1"
Gary,Even tough we configured our queue using all these options, we still
have problems when a JDBC persistence adapter is used : 1. sometimes low
priority messages are consumed although there are thousands of high priority
messages in the queue.2. sometimes high priority message consumption is
"frozen" until broker is restarted.We created a JIRA ticket (AMQ-4489) with
unit test drivers which reproduce these problems.Could you have a look
please ?Thanks in advance,metatech
--
View this message in context: http://activemq.2283324.n4.nabble.com/Message-priority-tp4040260p4666385.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Reply:Re: Reply:Re: Message priority
Posted by SuoNayi <su...@163.com>.
Hi, do you get it work?
You need comment out the vmcursor in the queue policy and the whole policy for the queue
looks like this,
<policyEntry queue="YOUR_prioritized_QUEUE_NAME" producerFlowControl="false"
prioritizedMessages="true" useCache="false" queuePrefetch="1" expireMessagesPeriod="0"/>
I have tested it and it does work but performance penalty becomes arrestive.
At 2013-01-07 22:07:11,"Gary Tully" <ga...@gmail.com> wrote:
>your important message is in the store, but not in the current memory
>snapshot.
>Unless we keep all messages in memory (vmcursor), we need to carefully
>configure the store cursors use of memory.
>
>There are two problems here.
>
>The first is caching messages on send by the store cursor.
>If there is a large batch of low priority messages that cause the cache to
>be exhausted, followed by some high priority messages, the high priority
>messages will only be read and dispatched when the cache is depleted.
>Currently it is not possible for a cached low priority message in memory to
>be replaced by a high priority message.
>The workaround for this is to disable the cursor cache.
><policyEntry queue="xxx" useCache="false" ..>
>
>The second is paging into memory from the store, to ensure true priority
>order this must occur only for dispatch to consumers. By default there is a
>periodic message scan to check for priority. This must be disabled,
>otherwise a consumer will get a view of the store order as seen by the last
>scan rather than at the time it subscribes.
>
>The workaround for this is to disable async expiry processing:
><policyEntry queue="xxx" useCache="false" expireMessagesPeriod="0" >
>
>Lastly, if high priory messages are a must have for each consumer, using
>prefetch=1 is necessary, such that every consumer dispatch goes to the
>store to find the current highest priority message. Otherwise a batch of
>priority ordered messages will be in memory ready to dispatch, irrespective
>of new deliveries.
>
><policyEntry queue="xxx" useCache="false" expireMessagesPeriod="0"
>queuePrefetch="1" >
>
>It is a trade off between taking a snapshot and caching or getting the
>latest up-to-date information w.r.t to producers.
>
>For active durable topic subscribers we support immediate priority
>dispatch, where we bypass the cache if we have a higher priority message
>and dispatch it immediately to all subscribers.
>We may need to investigate a similar arrangement for queues.
>
>
>On 7 January 2013 07:40, xawiers <xa...@gmail.com> wrote:
>
>> According to SuoNayi, amq will respect the message priority order if
>> vmcursor
>> is used.
>> If I will use default cursor - I will not get message priority. And this is
>> bad. Message priority is very important to me.
>> I have tried with storage cursor also:
>> One producer, one consumer
>>
>> added 10 messages with priority 3
>> added 3 messages with priority 5
>> added 100000 messages with priority 3
>> added 3 messages with priority 5
>>
>> consumed 3 messages (all priority 5), consuming 4th message I got message
>> with priority 3. Where is my important message ?
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/Message-priority-tp4040260p4661442.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>
>--
>http://redhat.com
>http://blog.garytully.com
Re: Reply:Re: Message priority
Posted by Gary Tully <ga...@gmail.com>.
your important message is in the store, but not in the current memory
snapshot.
Unless we keep all messages in memory (vmcursor), we need to carefully
configure the store cursors use of memory.
There are two problems here.
The first is caching messages on send by the store cursor.
If there is a large batch of low priority messages that cause the cache to
be exhausted, followed by some high priority messages, the high priority
messages will only be read and dispatched when the cache is depleted.
Currently it is not possible for a cached low priority message in memory to
be replaced by a high priority message.
The workaround for this is to disable the cursor cache.
<policyEntry queue="xxx" useCache="false" ..>
The second is paging into memory from the store, to ensure true priority
order this must occur only for dispatch to consumers. By default there is a
periodic message scan to check for priority. This must be disabled,
otherwise a consumer will get a view of the store order as seen by the last
scan rather than at the time it subscribes.
The workaround for this is to disable async expiry processing:
<policyEntry queue="xxx" useCache="false" expireMessagesPeriod="0" >
Lastly, if high priory messages are a must have for each consumer, using
prefetch=1 is necessary, such that every consumer dispatch goes to the
store to find the current highest priority message. Otherwise a batch of
priority ordered messages will be in memory ready to dispatch, irrespective
of new deliveries.
<policyEntry queue="xxx" useCache="false" expireMessagesPeriod="0"
queuePrefetch="1" >
It is a trade off between taking a snapshot and caching or getting the
latest up-to-date information w.r.t to producers.
For active durable topic subscribers we support immediate priority
dispatch, where we bypass the cache if we have a higher priority message
and dispatch it immediately to all subscribers.
We may need to investigate a similar arrangement for queues.
On 7 January 2013 07:40, xawiers <xa...@gmail.com> wrote:
> According to SuoNayi, amq will respect the message priority order if
> vmcursor
> is used.
> If I will use default cursor - I will not get message priority. And this is
> bad. Message priority is very important to me.
> I have tried with storage cursor also:
> One producer, one consumer
>
> added 10 messages with priority 3
> added 3 messages with priority 5
> added 100000 messages with priority 3
> added 3 messages with priority 5
>
> consumed 3 messages (all priority 5), consuming 4th message I got message
> with priority 3. Where is my important message ?
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Message-priority-tp4040260p4661442.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
http://redhat.com
http://blog.garytully.com
Re: Reply:Re: Message priority
Posted by xawiers <xa...@gmail.com>.
On Mon, Jan 7, 2013 at 9:40 AM, xawiers [via ActiveMQ]
<ml...@n4.nabble.com> wrote:
> According to SuoNayi, amq will respect the message priority order if
> vmcursor is used.
> If I will use default cursor - I will not get message priority. And this is
> bad. Message priority is very important to me.
> I have tried with storage cursor also:
> One producer, one consumer
>
> added 10 messages with priority 3
> added 3 messages with priority 5
> added 100000 messages with priority 3
> added 3 messages with priority 5
>
> consumed 3 messages (all priority 5), consuming 4th message I got message
> with priority 3. Where is my important message ?
>
> ________________________________
> If you reply to this email, your message will be added to the discussion
> below:
> http://activemq.2283324.n4.nabble.com/Message-priority-tp4040260p4661442.html
> To unsubscribe from Message priority, click here.
> NAML
--
View this message in context: http://activemq.2283324.n4.nabble.com/Message-priority-tp4040260p4661443.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Reply:Re: Reply:Re: Message priority
Posted by xawiers <xa...@gmail.com>.
My Config does not use cache and set prefetch size to zero (found on other
forums as recommendations)
I don't use topics, so don't play attention to it.
I also searched how to minimize batch message size to keep memory cursor
small, have not found working results. In my case receiving importand
message is critical and it can't wait till next batch query.
amq could check if produced message has higher priority than in cursor's
message list (let's say last message, because it should be kept in priority
order) and if it's true - add it and remove last in queue (which will be
with lower priority). This is just my thoughts.
So what settings could be best ?
Add memory limit on queue?
How to reduce cursor's waiting message amount size?
Performance is not very much important at this point. Consumer is slower
than amq.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Message-priority-tp4040260p4661453.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Reply:Re: Reply:Re: Message priority
Posted by SuoNayi <su...@163.com>.
Well,let's explore the message priority mechanism inside the broker.
The message sent from a producer will be persistent in the message store at first,
then it will be cached in the message cursor if the memory for the cursor is available.
That cursor is basically a priority list which supports sorting messages by priority.
When messages in the message cursor are all consumed by consumers the broker will
read the next batch messages by message priority from the store.
So if the space of the message cursor is very small or disabling the cache capability
of the message cursor the cursor will not be able to cache messages and the broker
will always read the next batch messages by message priority from the store.
In this case the broker behaves as your issuing the statement as the following in the database,
SELECT * FROM MESSAGE_TABLE ORDER BY priority.
Of course this approach pays some price on peformance penalty or you can consider
using different queues to deliver different priority messages.
At 2013-01-07 15:40:13,xawiers <xa...@gmail.com> wrote:
>According to SuoNayi, amq will respect the message priority order if vmcursor
>is used.
>If I will use default cursor - I will not get message priority. And this is
>bad. Message priority is very important to me.
>I have tried with storage cursor also:
>One producer, one consumer
>
>added 10 messages with priority 3
>added 3 messages with priority 5
>added 100000 messages with priority 3
>added 3 messages with priority 5
>
>consumed 3 messages (all priority 5), consuming 4th message I got message
>with priority 3. Where is my important message ?
>
>
>
>--
>View this message in context: http://activemq.2283324.n4.nabble.com/Message-priority-tp4040260p4661442.html
>Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Reply:Re: Message priority
Posted by xawiers <xa...@gmail.com>.
According to SuoNayi, amq will respect the message priority order if vmcursor
is used.
If I will use default cursor - I will not get message priority. And this is
bad. Message priority is very important to me.
I have tried with storage cursor also:
One producer, one consumer
added 10 messages with priority 3
added 3 messages with priority 5
added 100000 messages with priority 3
added 3 messages with priority 5
consumed 3 messages (all priority 5), consuming 4th message I got message
with priority 3. Where is my important message ?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Message-priority-tp4040260p4661442.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Reply:Re: Message priority
Posted by Gary Tully <ga...@gmail.com>.
use the default (store) cursor. It will just buffer messages up to 70% of
the memory limit so there is no need to block.
On 2 January 2013 18:43, xawiers <xa...@gmail.com> wrote:
> Anyway. Tested with 5.7.0 and 5.5.1:
> if define vmcursor and lower memoryUsage (limit 64mb)
> then memory fills up adding more and more messages and then producer is
> stopped.
>
> is here any setting to have not big memory usage and small bach size to
> fetch messages from persistence DB (I use mysql) and preserve priorities?
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Message-priority-tp4040260p4661227.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
http://redhat.com
http://blog.garytully.com