You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Hervé BARRAULT <he...@gmail.com> on 2012/02/13 14:37:42 UTC
Producer flow control
Hi,
i have looked to the documentation of the producer flow control
functionality (http://activemq.apache.org/producer-flow-control.html).
I see that it is based on the storage size but in my case, i think not
being able to use the size.
I have a queue where i put messages two kind of messages : Big 10MB and
Small 1kB.
As Big messages are rare (compared to small ones), i would limit the Memory
to 100MB (10 messages).
But for small messages i would limit the number of small messages to 5000
(so about 5MB) . We have noted that with JDBC persistence, the number of
pending messages in the broker (not by queue) is important for
"performances".
As i would keep the order of Big and Small messages, i put them in the same
queue.
Is there a way to define a strategy for the flow control which is 100MB and
5k messages ?
Regards
Hervé
Re: Producer flow control
Posted by Torsten Mielke <to...@fusesource.com>.
Hi,
You could use the vmQueueCursor as well.
In both cases (assuming producerFlowControl=true) the cursors will only dispatch msg from memory and not from the store.
Regards,
Torsten Mielke
On Feb 15, 2012, at 3:48 PM, Hervé BARRAULT wrote:
> Hi,
> If i set cursorMemoryHighWaterMark to 100, is it a good idea to use
> vmCursor instead of File Based Cursor (the default one) ?
>
> Regards
> Hervé
>
> On Wed, Feb 15, 2012 at 10:50 AM, Torsten Mielke <to...@fusesource.com>wrote:
>
>> Hello,
>>
>>> So when using persistence, i can't rely on producerFlowControl to
>>> limit the Flow.
>>
>>
>> Yes you can. However when using persistent queue messages,
>> producerFlowControl does not kick-in that early. Which is actually a good
>> thing in general.
>>
>> All this is configurable as well. You can specify
>> cursorMemoryHighWaterMark="100" on your destination policy configuration.
>> With that the cursor will take up to 100% of the configured queue memory
>> limit. When that limit is reached, producer flow control will also kick in
>> (as now 100% of the memory are in use).
>>
>>
>>> What is the best way to add a mechanism to constrain the queue size
>>> (handling slow consumer when using persistence) ?
>>
>> ProducerFlowControl and perhaps in your case also setting
>> cursorMemoryHighWaterMark="100" on the destination policy.
>>
>>
>> Best Regards,
>>
>> Torsten Mielke
>> torsten@fusesource.com
>> tmielke@blogspot.com
>>
>>
>> On Feb 14, 2012, at 2:20 PM, Hervé BARRAULT wrote:
>>
>>> Hi,
>>> thanks for the answer.
>>>
>>> So when using persistence, i can't rely on producerFlowControl to
>>> limit the Flow.
>>>
>>> If i have a consumer which is able to process only 10 messages by
>>> seconds and my queue is filled with 36000 messages (due to faster
>>> producers), i know that it will take 1 hour to process all my
>>> messages. I would reject new messages as my processing time shall be
>>> less than one hour.
>>>
>>> I would also keep the order of incoming messages (small and big) so i
>>> can't easily manage two queues without adding a complex
>>> synchronization mechanism.
>>>
>>> I see http://activemq.apache.org/slow-consumer-handling.html for
>>> non-persistent messages but it should not apply for persistent ones.
>>> (Prefetch is configurable for persistence :
>>>
>> http://activemq.apache.org/slow-consumer-handling.html#SlowConsumerHandling-UsingthePrefetchPolicytoconfigurethelimit
>>> but nothing for Pending Message Limit Strategy).
>>>
>>> What is the best way to add a mechanism to constrain the queue size
>>> (handling slow consumer when using persistence) ?
>>>
>>> Modifying the cursor, producer flow control, another thing ?
>>>
>>> Regards
>>>
>>> Hervé
>>>
>>>
>>> On 2/14/12, Torsten Mielke <to...@fusesource.com> wrote:
>>>> Hello Herve,
>>>>
>>>> You cannot configure producer flow control based on the various message
>>>> sizes.
>>>> However you could consider using different queues for these types of
>> msgs
>>>> and configure
>>>> the limits for each queue differently. That way your small msgs won't be
>>>> blocked just because of a small amount of large msgs on the same queue
>>>> (assuming different producers for both kinds of msgs).
>>>>
>>>> Also, when sending persistent queue messages, you rarely see producer
>> flow
>>>> control happening when using the default store cursor. That is because
>> the
>>>> store cursor is configured to hold messages in memory up to only 70% of
>> the
>>>> configured destination limit. If this 70% limit is reached, it won't
>> keep
>>>> new msgs in memory but reload them from the store (kahadb) later (when
>> its
>>>> time to dispatch the msg). Remember a persistent msg is first written
>> to the
>>>> store when received by the broker.
>>>> Producer flow control however kicks in when 100% of the available
>> memory for
>>>> each destination are used. This limit however you generally don't reach.
>>>> Things are different for non-persistent msgs. The default file cursor
>> keeps
>>>> msgs in memory until 100% of the configured destination limit are
>> reached.
>>>> Thereafter it swaps msgs to temp storage (which is different from
>> kahadb)
>>>> and producer flow control would kick in.
>>>>
>>>> Hope this helps.
>>>>
>>>>
>>>> Torsten Mielke
>>>> torsten@fusesource.com
>>>> tmielke@blogspot.com
>>>>
>>>>
>>>> On Feb 13, 2012, at 2:37 PM, Hervé BARRAULT wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> i have looked to the documentation of the producer flow control
>>>>> functionality (http://activemq.apache.org/producer-flow-control.html).
>>>>>
>>>>> I see that it is based on the storage size but in my case, i think not
>>>>> being able to use the size.
>>>>> I have a queue where i put messages two kind of messages : Big 10MB and
>>>>> Small 1kB.
>>>>>
>>>>> As Big messages are rare (compared to small ones), i would limit the
>>>>> Memory
>>>>> to 100MB (10 messages).
>>>>> But for small messages i would limit the number of small messages to
>> 5000
>>>>> (so about 5MB) . We have noted that with JDBC persistence, the number
>> of
>>>>> pending messages in the broker (not by queue) is important for
>>>>> "performances".
>>>>>
>>>>> As i would keep the order of Big and Small messages, i put them in the
>>>>> same
>>>>> queue.
>>>>>
>>>>> Is there a way to define a strategy for the flow control which is 100MB
>>>>> and
>>>>> 5k messages ?
>>>>>
>>>>>
>>>>> Regards
>>>>> Hervé
Re: Producer flow control
Posted by Hervé BARRAULT <he...@gmail.com>.
Hi,
If i set cursorMemoryHighWaterMark to 100, is it a good idea to use
vmCursor instead of File Based Cursor (the default one) ?
Regards
Hervé
On Wed, Feb 15, 2012 at 10:50 AM, Torsten Mielke <to...@fusesource.com>wrote:
> Hello,
>
> > So when using persistence, i can't rely on producerFlowControl to
> > limit the Flow.
>
>
> Yes you can. However when using persistent queue messages,
> producerFlowControl does not kick-in that early. Which is actually a good
> thing in general.
>
> All this is configurable as well. You can specify
> cursorMemoryHighWaterMark="100" on your destination policy configuration.
> With that the cursor will take up to 100% of the configured queue memory
> limit. When that limit is reached, producer flow control will also kick in
> (as now 100% of the memory are in use).
>
>
> > What is the best way to add a mechanism to constrain the queue size
> > (handling slow consumer when using persistence) ?
>
> ProducerFlowControl and perhaps in your case also setting
> cursorMemoryHighWaterMark="100" on the destination policy.
>
>
> Best Regards,
>
> Torsten Mielke
> torsten@fusesource.com
> tmielke@blogspot.com
>
>
> On Feb 14, 2012, at 2:20 PM, Hervé BARRAULT wrote:
>
> > Hi,
> > thanks for the answer.
> >
> > So when using persistence, i can't rely on producerFlowControl to
> > limit the Flow.
> >
> > If i have a consumer which is able to process only 10 messages by
> > seconds and my queue is filled with 36000 messages (due to faster
> > producers), i know that it will take 1 hour to process all my
> > messages. I would reject new messages as my processing time shall be
> > less than one hour.
> >
> > I would also keep the order of incoming messages (small and big) so i
> > can't easily manage two queues without adding a complex
> > synchronization mechanism.
> >
> > I see http://activemq.apache.org/slow-consumer-handling.html for
> > non-persistent messages but it should not apply for persistent ones.
> > (Prefetch is configurable for persistence :
> >
> http://activemq.apache.org/slow-consumer-handling.html#SlowConsumerHandling-UsingthePrefetchPolicytoconfigurethelimit
> > but nothing for Pending Message Limit Strategy).
> >
> > What is the best way to add a mechanism to constrain the queue size
> > (handling slow consumer when using persistence) ?
> >
> > Modifying the cursor, producer flow control, another thing ?
> >
> > Regards
> >
> > Hervé
> >
> >
> > On 2/14/12, Torsten Mielke <to...@fusesource.com> wrote:
> >> Hello Herve,
> >>
> >> You cannot configure producer flow control based on the various message
> >> sizes.
> >> However you could consider using different queues for these types of
> msgs
> >> and configure
> >> the limits for each queue differently. That way your small msgs won't be
> >> blocked just because of a small amount of large msgs on the same queue
> >> (assuming different producers for both kinds of msgs).
> >>
> >> Also, when sending persistent queue messages, you rarely see producer
> flow
> >> control happening when using the default store cursor. That is because
> the
> >> store cursor is configured to hold messages in memory up to only 70% of
> the
> >> configured destination limit. If this 70% limit is reached, it won't
> keep
> >> new msgs in memory but reload them from the store (kahadb) later (when
> its
> >> time to dispatch the msg). Remember a persistent msg is first written
> to the
> >> store when received by the broker.
> >> Producer flow control however kicks in when 100% of the available
> memory for
> >> each destination are used. This limit however you generally don't reach.
> >> Things are different for non-persistent msgs. The default file cursor
> keeps
> >> msgs in memory until 100% of the configured destination limit are
> reached.
> >> Thereafter it swaps msgs to temp storage (which is different from
> kahadb)
> >> and producer flow control would kick in.
> >>
> >> Hope this helps.
> >>
> >>
> >> Torsten Mielke
> >> torsten@fusesource.com
> >> tmielke@blogspot.com
> >>
> >>
> >> On Feb 13, 2012, at 2:37 PM, Hervé BARRAULT wrote:
> >>
> >>> Hi,
> >>>
> >>> i have looked to the documentation of the producer flow control
> >>> functionality (http://activemq.apache.org/producer-flow-control.html).
> >>>
> >>> I see that it is based on the storage size but in my case, i think not
> >>> being able to use the size.
> >>> I have a queue where i put messages two kind of messages : Big 10MB and
> >>> Small 1kB.
> >>>
> >>> As Big messages are rare (compared to small ones), i would limit the
> >>> Memory
> >>> to 100MB (10 messages).
> >>> But for small messages i would limit the number of small messages to
> 5000
> >>> (so about 5MB) . We have noted that with JDBC persistence, the number
> of
> >>> pending messages in the broker (not by queue) is important for
> >>> "performances".
> >>>
> >>> As i would keep the order of Big and Small messages, i put them in the
> >>> same
> >>> queue.
> >>>
> >>> Is there a way to define a strategy for the flow control which is 100MB
> >>> and
> >>> 5k messages ?
> >>>
> >>>
> >>> Regards
> >>> Hervé
>
>
>
>
>
Re: Producer flow control
Posted by Torsten Mielke <to...@fusesource.com>.
Hello,
> So when using persistence, i can't rely on producerFlowControl to
> limit the Flow.
Yes you can. However when using persistent queue messages, producerFlowControl does not kick-in that early. Which is actually a good thing in general.
All this is configurable as well. You can specify cursorMemoryHighWaterMark="100" on your destination policy configuration. With that the cursor will take up to 100% of the configured queue memory limit. When that limit is reached, producer flow control will also kick in (as now 100% of the memory are in use).
> What is the best way to add a mechanism to constrain the queue size
> (handling slow consumer when using persistence) ?
ProducerFlowControl and perhaps in your case also setting cursorMemoryHighWaterMark="100" on the destination policy.
Best Regards,
Torsten Mielke
torsten@fusesource.com
tmielke@blogspot.com
On Feb 14, 2012, at 2:20 PM, Hervé BARRAULT wrote:
> Hi,
> thanks for the answer.
>
> So when using persistence, i can't rely on producerFlowControl to
> limit the Flow.
>
> If i have a consumer which is able to process only 10 messages by
> seconds and my queue is filled with 36000 messages (due to faster
> producers), i know that it will take 1 hour to process all my
> messages. I would reject new messages as my processing time shall be
> less than one hour.
>
> I would also keep the order of incoming messages (small and big) so i
> can't easily manage two queues without adding a complex
> synchronization mechanism.
>
> I see http://activemq.apache.org/slow-consumer-handling.html for
> non-persistent messages but it should not apply for persistent ones.
> (Prefetch is configurable for persistence :
> http://activemq.apache.org/slow-consumer-handling.html#SlowConsumerHandling-UsingthePrefetchPolicytoconfigurethelimit
> but nothing for Pending Message Limit Strategy).
>
> What is the best way to add a mechanism to constrain the queue size
> (handling slow consumer when using persistence) ?
>
> Modifying the cursor, producer flow control, another thing ?
>
> Regards
>
> Hervé
>
>
> On 2/14/12, Torsten Mielke <to...@fusesource.com> wrote:
>> Hello Herve,
>>
>> You cannot configure producer flow control based on the various message
>> sizes.
>> However you could consider using different queues for these types of msgs
>> and configure
>> the limits for each queue differently. That way your small msgs won't be
>> blocked just because of a small amount of large msgs on the same queue
>> (assuming different producers for both kinds of msgs).
>>
>> Also, when sending persistent queue messages, you rarely see producer flow
>> control happening when using the default store cursor. That is because the
>> store cursor is configured to hold messages in memory up to only 70% of the
>> configured destination limit. If this 70% limit is reached, it won't keep
>> new msgs in memory but reload them from the store (kahadb) later (when its
>> time to dispatch the msg). Remember a persistent msg is first written to the
>> store when received by the broker.
>> Producer flow control however kicks in when 100% of the available memory for
>> each destination are used. This limit however you generally don't reach.
>> Things are different for non-persistent msgs. The default file cursor keeps
>> msgs in memory until 100% of the configured destination limit are reached.
>> Thereafter it swaps msgs to temp storage (which is different from kahadb)
>> and producer flow control would kick in.
>>
>> Hope this helps.
>>
>>
>> Torsten Mielke
>> torsten@fusesource.com
>> tmielke@blogspot.com
>>
>>
>> On Feb 13, 2012, at 2:37 PM, Hervé BARRAULT wrote:
>>
>>> Hi,
>>>
>>> i have looked to the documentation of the producer flow control
>>> functionality (http://activemq.apache.org/producer-flow-control.html).
>>>
>>> I see that it is based on the storage size but in my case, i think not
>>> being able to use the size.
>>> I have a queue where i put messages two kind of messages : Big 10MB and
>>> Small 1kB.
>>>
>>> As Big messages are rare (compared to small ones), i would limit the
>>> Memory
>>> to 100MB (10 messages).
>>> But for small messages i would limit the number of small messages to 5000
>>> (so about 5MB) . We have noted that with JDBC persistence, the number of
>>> pending messages in the broker (not by queue) is important for
>>> "performances".
>>>
>>> As i would keep the order of Big and Small messages, i put them in the
>>> same
>>> queue.
>>>
>>> Is there a way to define a strategy for the flow control which is 100MB
>>> and
>>> 5k messages ?
>>>
>>>
>>> Regards
>>> Hervé
Re: Producer flow control
Posted by Hervé BARRAULT <he...@gmail.com>.
Hi,
thanks for the answer.
So when using persistence, i can't rely on producerFlowControl to
limit the Flow.
If i have a consumer which is able to process only 10 messages by
seconds and my queue is filled with 36000 messages (due to faster
producers), i know that it will take 1 hour to process all my
messages. I would reject new messages as my processing time shall be
less than one hour.
I would also keep the order of incoming messages (small and big) so i
can't easily manage two queues without adding a complex
synchronization mechanism.
I see http://activemq.apache.org/slow-consumer-handling.html for
non-persistent messages but it should not apply for persistent ones.
(Prefetch is configurable for persistence :
http://activemq.apache.org/slow-consumer-handling.html#SlowConsumerHandling-UsingthePrefetchPolicytoconfigurethelimit
but nothing for Pending Message Limit Strategy).
What is the best way to add a mechanism to constrain the queue size
(handling slow consumer when using persistence) ?
Modifying the cursor, producer flow control, another thing ?
Regards
Hervé
On 2/14/12, Torsten Mielke <to...@fusesource.com> wrote:
> Hello Herve,
>
> You cannot configure producer flow control based on the various message
> sizes.
> However you could consider using different queues for these types of msgs
> and configure
> the limits for each queue differently. That way your small msgs won't be
> blocked just because of a small amount of large msgs on the same queue
> (assuming different producers for both kinds of msgs).
>
> Also, when sending persistent queue messages, you rarely see producer flow
> control happening when using the default store cursor. That is because the
> store cursor is configured to hold messages in memory up to only 70% of the
> configured destination limit. If this 70% limit is reached, it won't keep
> new msgs in memory but reload them from the store (kahadb) later (when its
> time to dispatch the msg). Remember a persistent msg is first written to the
> store when received by the broker.
> Producer flow control however kicks in when 100% of the available memory for
> each destination are used. This limit however you generally don't reach.
> Things are different for non-persistent msgs. The default file cursor keeps
> msgs in memory until 100% of the configured destination limit are reached.
> Thereafter it swaps msgs to temp storage (which is different from kahadb)
> and producer flow control would kick in.
>
> Hope this helps.
>
>
> Torsten Mielke
> torsten@fusesource.com
> tmielke@blogspot.com
>
>
> On Feb 13, 2012, at 2:37 PM, Hervé BARRAULT wrote:
>
>> Hi,
>>
>> i have looked to the documentation of the producer flow control
>> functionality (http://activemq.apache.org/producer-flow-control.html).
>>
>> I see that it is based on the storage size but in my case, i think not
>> being able to use the size.
>> I have a queue where i put messages two kind of messages : Big 10MB and
>> Small 1kB.
>>
>> As Big messages are rare (compared to small ones), i would limit the
>> Memory
>> to 100MB (10 messages).
>> But for small messages i would limit the number of small messages to 5000
>> (so about 5MB) . We have noted that with JDBC persistence, the number of
>> pending messages in the broker (not by queue) is important for
>> "performances".
>>
>> As i would keep the order of Big and Small messages, i put them in the
>> same
>> queue.
>>
>> Is there a way to define a strategy for the flow control which is 100MB
>> and
>> 5k messages ?
>>
>>
>> Regards
>> Hervé
>
>
>
>
Re: Producer flow control
Posted by Torsten Mielke <to...@fusesource.com>.
Hello Herve,
You cannot configure producer flow control based on the various message sizes.
However you could consider using different queues for these types of msgs and configure
the limits for each queue differently. That way your small msgs won't be blocked just because of a small amount of large msgs on the same queue (assuming different producers for both kinds of msgs).
Also, when sending persistent queue messages, you rarely see producer flow control happening when using the default store cursor. That is because the store cursor is configured to hold messages in memory up to only 70% of the configured destination limit. If this 70% limit is reached, it won't keep new msgs in memory but reload them from the store (kahadb) later (when its time to dispatch the msg). Remember a persistent msg is first written to the store when received by the broker.
Producer flow control however kicks in when 100% of the available memory for each destination are used. This limit however you generally don't reach.
Things are different for non-persistent msgs. The default file cursor keeps msgs in memory until 100% of the configured destination limit are reached. Thereafter it swaps msgs to temp storage (which is different from kahadb) and producer flow control would kick in.
Hope this helps.
Torsten Mielke
torsten@fusesource.com
tmielke@blogspot.com
On Feb 13, 2012, at 2:37 PM, Hervé BARRAULT wrote:
> Hi,
>
> i have looked to the documentation of the producer flow control
> functionality (http://activemq.apache.org/producer-flow-control.html).
>
> I see that it is based on the storage size but in my case, i think not
> being able to use the size.
> I have a queue where i put messages two kind of messages : Big 10MB and
> Small 1kB.
>
> As Big messages are rare (compared to small ones), i would limit the Memory
> to 100MB (10 messages).
> But for small messages i would limit the number of small messages to 5000
> (so about 5MB) . We have noted that with JDBC persistence, the number of
> pending messages in the broker (not by queue) is important for
> "performances".
>
> As i would keep the order of Big and Small messages, i put them in the same
> queue.
>
> Is there a way to define a strategy for the flow control which is 100MB and
> 5k messages ?
>
>
> Regards
> Hervé