You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Daniel Navrotsky <da...@driivz.com> on 2022/01/05 22:58:44 UTC
Mismatch in documentation regarding producerFlowControl behavior
Hello,
It looks like there is a mismatch in the documentation. From piece 1 (below) I understand that messages would be shunted into the temporary file even if producerFlowControl=true, but from piece 2 it sounds like it only would happen if producerFlowControl=false
Piece 1
https://activemq.apache.org/producer-flow-control.html
Note that, since the introduction of the new file cursor in ActiveMQ 5.x, non-persisted messages are shunted into the temporary file store to reduce the amount of memory used for non-persistent messaging. As a result, you may find that a queue’s memoryLimit is never reached, as the cursor doesn’t use very much memory. If you really do want to keep all your non-persistent messages in memory, and stop producers when the limit is reached, you should configure the <vmQueueCursor>
Piece 2
https://activemq.apache.org/per-destination-policies
producerFlowControl - If true the broker will throttle (flow-control) the producer. Throttling is achieved either by withholding the producer’s ACK or by raising a javax.jms.ResourceAllocationException (that’s propagated back to the client) when local resources, e.g., memory and/or storage, have been exhausted. If false excess messages will be written to the message store to prevent memory exhaustion. However, when the message store reaches capacity the producer will be throttled until resources are freed.
Best regards,
Daniel
Re: Mismatch in documentation regarding producerFlowControl behavior
Posted by Tim Bain <tb...@alumni.duke.edu>.
Correct. Or when memory is exhausted and out-of-memory exceptions are
thrown and the broker starts behaving unpredictably.
The broker can't accept an infinite number of messages, so eventually it'll
throttle the producer, just in ways that are ungraceful and might be
destructive to the broker process and its ability to serve other clients.
PFC ensures that the ungraceful scenarios are avoided.
Tim
On Sun, Jan 9, 2022, 6:27 AM Daniel Navrotsky <da...@driivz.com>
wrote:
> Thanks Tim.
> So you mean that when producerFlowControl=false, the configured recourse
> limits will be ignored and producers throttling will take place in not
> graceful way when the amount of free space on the filesystem volume on
> which the data file lives is exhausted. Is that right?
>
>
> Best regards,
> Daniel
>
> -----Original Message-----
> From: Tim Bain <tb...@alumni.duke.edu>
> Sent: Friday, January 7, 2022 5:52 AM
> To: ActiveMQ Users <us...@activemq.apache.org>
> Subject: Re: Mismatch in documentation regarding producerFlowControl
> behavior
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
> The sentence is fully accurate if we interpret "capacity" in that sentence
> as "the amount of free space on the filesystem volume on which the data
> file lives" rather than "the value you put in the config file." Under that
> definition, a producer that produces beyond the broker's capacity will
> indeed be throttled by the errors or hanging connections (whatever the
> actual behavior of the broker is under those conditions), but it won't be
> the graceful throttling you get from PFC.
>
> But I definitely think that the paragraph in question could be written
> more unambiguously to avoid the types of confusion that Daniel has pointed
> out.
> Just because I can identify a semi-reasonable definition that makes the
> sentence accurate doesn't mean we shouldn't make it better!
>
> Tim
>
> Tim
>
> On Thu, Jan 6, 2022, 3:51 PM Matt Pavlovich <ma...@gmail.com> wrote:
>
> > Interesting, ok that last part of the statement may be outdated. I
> > honestly haven’t tested with producerFlowControl off in a while to
> > know off the top of my head.
> >
> > I’d be a straight-forward test.. set storeUsageLimit to something small..
> > turn off flow control and publish messages to see what happens next.
> >
> > -Matt Pavlovich
> >
> > > On Jan 6, 2022, at 4:32 PM, Daniel Navrotsky <
> > daniel.navrotsky@driivz.com> wrote:
> > >
> > > Thanks Matt,
> > >
> > >
> > >
> > > You wrote "With it off, producers will simply get errors when
> > > resources
> > are exhausted".
> > >
> > > But in the description to producerFlowControl param on
> > https://activemq.apache.org/per-destination-policies when the value is
> > off it says "However, when the message store reaches capacity the
> > producer will be throttled until resources are freed".
> > >
> > >
> > >
> > > Thanks,
> > >
> > > Daniel
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Matt Pavlovich <ma...@gmail.com>
> > > Sent: Thursday, January 6, 2022 11:17 PM
> > > To: users@activemq.apache.org
> > > Subject: Re: Mismatch in documentation regarding producerFlowControl
> > behavior
> > >
> > >
> > >
> > > CAUTION: This email originated from outside of the organization. Do
> > > not
> > click links or open attachments unless you recognize the sender and
> > know the content is safe.
> > >
> > >
> > >
> > >
> > >
> > > Turning off producer flow control just means the broker won’t pause
> > > fast
> > producers to allow for consumers to catch up— it is a ’self-healing’
> > type feature. With it off, producers will simply get errors when
> > resources are exhausted.
> > >
> > >
> > >
> > > The feature helps prevent producer-side errors in the scenario where
> > > a
> > few micro pauses would allow things to settle down without any errors
> > being thrown at all.
> > >
> > >
> > >
> > > -Matt Pavlovich
> > >
> > >
> > >
> > >> On Jan 6, 2022, at 12:36 PM, Daniel Navrotsky <
> > daniel.navrotsky@driivz.com<ma...@driivz.com>> wrote:
> > >
> > >>
> > >
> > >> Thank you Matt for the answer.
> > >
> > >>
> > >
> > >> I am trying to understand the difference between
> > producerFlowControl=true or false. I understand that the type of the
> > cursor defines the resources. When resources are exhausted the flow
> > control will take place no matter if producerFlowControl=true or
> > false. Can you please bring an example of cursor and message type
> > where the behavior will be different depending on producerFlowControl
> param?
> > >
> > >>
> > >
> > >> Thanks,
> > >
> > >>
> > >
> > >> Best regards,
> > >
> > >> Daniel
> > >
> > >>
> > >
> > >> -----Original Message-----
> > >
> > >> From: Matt Pavlovich
> > >> <ma...@gmail.com>>
> > >
> > >> Sent: Thursday, January 6, 2022 7:42 PM
> > >
> > >> To: users@activemq.apache.org<ma...@activemq.apache.org>
> > >
> > >> Subject: Re: Mismatch in documentation regarding
> > >> producerFlowControl
> > behavior
> > >
> > >>
> > >
> > >> CAUTION: This email originated from outside of the organization. Do
> > >> not
> > click links or open attachments unless you recognize the sender and
> > know the content is safe.
> > >
> > >>
> > >
> > >>
> > >
> > >> Daniel-
> > >
> > >>
> > >
> > >> The documentation looks correct. The “triggers” for producer flow
> > control vary based on the configurations and message type (persistent
> > or non-persistent). What is different in the various configuration
> > noted below are what what constitutes “resources” for non-persistent
> messages.
> > >
> > >>
> > >
> > >> For non-persistent messages:
> > >
> > >>
> > >
> > >> a. Default cursor will flow control when memoryLimit and/or
> > tempUsageLimit reached
> > >
> > >> -or-
> > >
> > >> b. vmQueueCurosr will flow control on memoryLimit only (since temp
> > >> and
> > store usage are not used)
> > >
> > >>
> > >
> > >> Hope this helps.
> > >
> > >>
> > >
> > >> -Matt Pavlovich
> > >
> > >>
> > >
> > >>> On Jan 5, 2022, at 4:58 PM, Daniel Navrotsky <
> > daniel.navrotsky@driivz.com<ma...@driivz.com>> wrote:
> > >
> > >>>
> > >
> > >>> Hello,
> > >
> > >>>
> > >
> > >>> It looks like there is a mismatch in the documentation. From piece
> > >>> 1
> > (below) I understand that messages would be shunted into the temporary
> > file even if producerFlowControl=true, but from piece 2 it sounds like
> > it only would happen if producerFlowControl=false
> > >
> > >>>
> > >
> > >>>
> > >
> > >>> Piece 1
> > >
> > >>> https://activemq.apache.org/producer-flow-control.html
> > >
> > >>> Note that, since the introduction of the new file cursor in
> > >>> ActiveMQ
> > 5.x, non-persisted messages are shunted into the temporary file store
> > to reduce the amount of memory used for non-persistent messaging. As a
> > result, you may find that a queue’s memoryLimit is never reached, as
> > the cursor doesn’t use very much memory. If you really do want to keep
> > all your non-persistent messages in memory, and stop producers when
> > the limit is reached, you should configure the <vmQueueCursor>
> > >
> > >>>
> > >
> > >>> Piece 2
> > >
> > >>> https://activemq.apache.org/per-destination-policies
> > >
> > >>> producerFlowControl - If true the broker will throttle
> > >>> (flow-control)
> > the producer. Throttling is achieved either by withholding the
> > producer’s ACK or by raising a javax.jms.ResourceAllocationException
> > (that’s propagated back to the client) when local resources, e.g.,
> > memory and/or storage, have been exhausted. If false excess messages
> > will be written to the message store to prevent memory exhaustion.
> > However, when the message store reaches capacity the producer will be
> > throttled until resources are freed.
> > >
> > >>>
> > >
> > >>>
> > >
> > >>> Best regards,
> > >
> > >>> Daniel
> > >
> > >>
> > >
> > >
> >
> >
>
RE: Mismatch in documentation regarding producerFlowControl behavior
Posted by Daniel Navrotsky <da...@driivz.com>.
Thanks Tim.
So you mean that when producerFlowControl=false, the configured recourse limits will be ignored and producers throttling will take place in not graceful way when the amount of free space on the filesystem volume on which the data file lives is exhausted. Is that right?
Best regards,
Daniel
-----Original Message-----
From: Tim Bain <tb...@alumni.duke.edu>
Sent: Friday, January 7, 2022 5:52 AM
To: ActiveMQ Users <us...@activemq.apache.org>
Subject: Re: Mismatch in documentation regarding producerFlowControl behavior
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
The sentence is fully accurate if we interpret "capacity" in that sentence as "the amount of free space on the filesystem volume on which the data file lives" rather than "the value you put in the config file." Under that definition, a producer that produces beyond the broker's capacity will indeed be throttled by the errors or hanging connections (whatever the actual behavior of the broker is under those conditions), but it won't be the graceful throttling you get from PFC.
But I definitely think that the paragraph in question could be written more unambiguously to avoid the types of confusion that Daniel has pointed out.
Just because I can identify a semi-reasonable definition that makes the sentence accurate doesn't mean we shouldn't make it better!
Tim
Tim
On Thu, Jan 6, 2022, 3:51 PM Matt Pavlovich <ma...@gmail.com> wrote:
> Interesting, ok that last part of the statement may be outdated. I
> honestly haven’t tested with producerFlowControl off in a while to
> know off the top of my head.
>
> I’d be a straight-forward test.. set storeUsageLimit to something small..
> turn off flow control and publish messages to see what happens next.
>
> -Matt Pavlovich
>
> > On Jan 6, 2022, at 4:32 PM, Daniel Navrotsky <
> daniel.navrotsky@driivz.com> wrote:
> >
> > Thanks Matt,
> >
> >
> >
> > You wrote "With it off, producers will simply get errors when
> > resources
> are exhausted".
> >
> > But in the description to producerFlowControl param on
> https://activemq.apache.org/per-destination-policies when the value is
> off it says "However, when the message store reaches capacity the
> producer will be throttled until resources are freed".
> >
> >
> >
> > Thanks,
> >
> > Daniel
> >
> >
> >
> > -----Original Message-----
> > From: Matt Pavlovich <ma...@gmail.com>
> > Sent: Thursday, January 6, 2022 11:17 PM
> > To: users@activemq.apache.org
> > Subject: Re: Mismatch in documentation regarding producerFlowControl
> behavior
> >
> >
> >
> > CAUTION: This email originated from outside of the organization. Do
> > not
> click links or open attachments unless you recognize the sender and
> know the content is safe.
> >
> >
> >
> >
> >
> > Turning off producer flow control just means the broker won’t pause
> > fast
> producers to allow for consumers to catch up— it is a ’self-healing’
> type feature. With it off, producers will simply get errors when
> resources are exhausted.
> >
> >
> >
> > The feature helps prevent producer-side errors in the scenario where
> > a
> few micro pauses would allow things to settle down without any errors
> being thrown at all.
> >
> >
> >
> > -Matt Pavlovich
> >
> >
> >
> >> On Jan 6, 2022, at 12:36 PM, Daniel Navrotsky <
> daniel.navrotsky@driivz.com<ma...@driivz.com>> wrote:
> >
> >>
> >
> >> Thank you Matt for the answer.
> >
> >>
> >
> >> I am trying to understand the difference between
> producerFlowControl=true or false. I understand that the type of the
> cursor defines the resources. When resources are exhausted the flow
> control will take place no matter if producerFlowControl=true or
> false. Can you please bring an example of cursor and message type
> where the behavior will be different depending on producerFlowControl param?
> >
> >>
> >
> >> Thanks,
> >
> >>
> >
> >> Best regards,
> >
> >> Daniel
> >
> >>
> >
> >> -----Original Message-----
> >
> >> From: Matt Pavlovich
> >> <ma...@gmail.com>>
> >
> >> Sent: Thursday, January 6, 2022 7:42 PM
> >
> >> To: users@activemq.apache.org<ma...@activemq.apache.org>
> >
> >> Subject: Re: Mismatch in documentation regarding
> >> producerFlowControl
> behavior
> >
> >>
> >
> >> CAUTION: This email originated from outside of the organization. Do
> >> not
> click links or open attachments unless you recognize the sender and
> know the content is safe.
> >
> >>
> >
> >>
> >
> >> Daniel-
> >
> >>
> >
> >> The documentation looks correct. The “triggers” for producer flow
> control vary based on the configurations and message type (persistent
> or non-persistent). What is different in the various configuration
> noted below are what what constitutes “resources” for non-persistent messages.
> >
> >>
> >
> >> For non-persistent messages:
> >
> >>
> >
> >> a. Default cursor will flow control when memoryLimit and/or
> tempUsageLimit reached
> >
> >> -or-
> >
> >> b. vmQueueCurosr will flow control on memoryLimit only (since temp
> >> and
> store usage are not used)
> >
> >>
> >
> >> Hope this helps.
> >
> >>
> >
> >> -Matt Pavlovich
> >
> >>
> >
> >>> On Jan 5, 2022, at 4:58 PM, Daniel Navrotsky <
> daniel.navrotsky@driivz.com<ma...@driivz.com>> wrote:
> >
> >>>
> >
> >>> Hello,
> >
> >>>
> >
> >>> It looks like there is a mismatch in the documentation. From piece
> >>> 1
> (below) I understand that messages would be shunted into the temporary
> file even if producerFlowControl=true, but from piece 2 it sounds like
> it only would happen if producerFlowControl=false
> >
> >>>
> >
> >>>
> >
> >>> Piece 1
> >
> >>> https://activemq.apache.org/producer-flow-control.html
> >
> >>> Note that, since the introduction of the new file cursor in
> >>> ActiveMQ
> 5.x, non-persisted messages are shunted into the temporary file store
> to reduce the amount of memory used for non-persistent messaging. As a
> result, you may find that a queue’s memoryLimit is never reached, as
> the cursor doesn’t use very much memory. If you really do want to keep
> all your non-persistent messages in memory, and stop producers when
> the limit is reached, you should configure the <vmQueueCursor>
> >
> >>>
> >
> >>> Piece 2
> >
> >>> https://activemq.apache.org/per-destination-policies
> >
> >>> producerFlowControl - If true the broker will throttle
> >>> (flow-control)
> the producer. Throttling is achieved either by withholding the
> producer’s ACK or by raising a javax.jms.ResourceAllocationException
> (that’s propagated back to the client) when local resources, e.g.,
> memory and/or storage, have been exhausted. If false excess messages
> will be written to the message store to prevent memory exhaustion.
> However, when the message store reaches capacity the producer will be
> throttled until resources are freed.
> >
> >>>
> >
> >>>
> >
> >>> Best regards,
> >
> >>> Daniel
> >
> >>
> >
> >
>
>
Re: Mismatch in documentation regarding producerFlowControl behavior
Posted by Tim Bain <tb...@alumni.duke.edu>.
The sentence is fully accurate if we interpret "capacity" in that sentence
as "the amount of free space on the filesystem volume on which the data
file lives" rather than "the value you put in the config file." Under that
definition, a producer that produces beyond the broker's capacity will
indeed be throttled by the errors or hanging connections (whatever the
actual behavior of the broker is under those conditions), but it won't be
the graceful throttling you get from PFC.
But I definitely think that the paragraph in question could be written more
unambiguously to avoid the types of confusion that Daniel has pointed out.
Just because I can identify a semi-reasonable definition that makes the
sentence accurate doesn't mean we shouldn't make it better!
Tim
Tim
On Thu, Jan 6, 2022, 3:51 PM Matt Pavlovich <ma...@gmail.com> wrote:
> Interesting, ok that last part of the statement may be outdated. I
> honestly haven’t tested with producerFlowControl off in a while to know off
> the top of my head.
>
> I’d be a straight-forward test.. set storeUsageLimit to something small..
> turn off flow control and publish messages to see what happens next.
>
> -Matt Pavlovich
>
> > On Jan 6, 2022, at 4:32 PM, Daniel Navrotsky <
> daniel.navrotsky@driivz.com> wrote:
> >
> > Thanks Matt,
> >
> >
> >
> > You wrote "With it off, producers will simply get errors when resources
> are exhausted".
> >
> > But in the description to producerFlowControl param on
> https://activemq.apache.org/per-destination-policies when the value is
> off it says "However, when the message store reaches capacity the producer
> will be throttled until resources are freed".
> >
> >
> >
> > Thanks,
> >
> > Daniel
> >
> >
> >
> > -----Original Message-----
> > From: Matt Pavlovich <ma...@gmail.com>
> > Sent: Thursday, January 6, 2022 11:17 PM
> > To: users@activemq.apache.org
> > Subject: Re: Mismatch in documentation regarding producerFlowControl
> behavior
> >
> >
> >
> > CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
> >
> >
> >
> >
> >
> > Turning off producer flow control just means the broker won’t pause fast
> producers to allow for consumers to catch up— it is a ’self-healing’ type
> feature. With it off, producers will simply get errors when resources are
> exhausted.
> >
> >
> >
> > The feature helps prevent producer-side errors in the scenario where a
> few micro pauses would allow things to settle down without any errors being
> thrown at all.
> >
> >
> >
> > -Matt Pavlovich
> >
> >
> >
> >> On Jan 6, 2022, at 12:36 PM, Daniel Navrotsky <
> daniel.navrotsky@driivz.com<ma...@driivz.com>> wrote:
> >
> >>
> >
> >> Thank you Matt for the answer.
> >
> >>
> >
> >> I am trying to understand the difference between
> producerFlowControl=true or false. I understand that the type of the cursor
> defines the resources. When resources are exhausted the flow control will
> take place no matter if producerFlowControl=true or false. Can you please
> bring an example of cursor and message type where the behavior will be
> different depending on producerFlowControl param?
> >
> >>
> >
> >> Thanks,
> >
> >>
> >
> >> Best regards,
> >
> >> Daniel
> >
> >>
> >
> >> -----Original Message-----
> >
> >> From: Matt Pavlovich <ma...@gmail.com>>
> >
> >> Sent: Thursday, January 6, 2022 7:42 PM
> >
> >> To: users@activemq.apache.org<ma...@activemq.apache.org>
> >
> >> Subject: Re: Mismatch in documentation regarding producerFlowControl
> behavior
> >
> >>
> >
> >> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
> >
> >>
> >
> >>
> >
> >> Daniel-
> >
> >>
> >
> >> The documentation looks correct. The “triggers” for producer flow
> control vary based on the configurations and message type (persistent or
> non-persistent). What is different in the various configuration noted below
> are what what constitutes “resources” for non-persistent messages.
> >
> >>
> >
> >> For non-persistent messages:
> >
> >>
> >
> >> a. Default cursor will flow control when memoryLimit and/or
> tempUsageLimit reached
> >
> >> -or-
> >
> >> b. vmQueueCurosr will flow control on memoryLimit only (since temp and
> store usage are not used)
> >
> >>
> >
> >> Hope this helps.
> >
> >>
> >
> >> -Matt Pavlovich
> >
> >>
> >
> >>> On Jan 5, 2022, at 4:58 PM, Daniel Navrotsky <
> daniel.navrotsky@driivz.com<ma...@driivz.com>> wrote:
> >
> >>>
> >
> >>> Hello,
> >
> >>>
> >
> >>> It looks like there is a mismatch in the documentation. From piece 1
> (below) I understand that messages would be shunted into the temporary file
> even if producerFlowControl=true, but from piece 2 it sounds like it only
> would happen if producerFlowControl=false
> >
> >>>
> >
> >>>
> >
> >>> Piece 1
> >
> >>> https://activemq.apache.org/producer-flow-control.html
> >
> >>> Note that, since the introduction of the new file cursor in ActiveMQ
> 5.x, non-persisted messages are shunted into the temporary file store to
> reduce the amount of memory used for non-persistent messaging. As a result,
> you may find that a queue’s memoryLimit is never reached, as the cursor
> doesn’t use very much memory. If you really do want to keep all your
> non-persistent messages in memory, and stop producers when the limit is
> reached, you should configure the <vmQueueCursor>
> >
> >>>
> >
> >>> Piece 2
> >
> >>> https://activemq.apache.org/per-destination-policies
> >
> >>> producerFlowControl - If true the broker will throttle (flow-control)
> the producer. Throttling is achieved either by withholding the producer’s
> ACK or by raising a javax.jms.ResourceAllocationException (that’s
> propagated back to the client) when local resources, e.g., memory and/or
> storage, have been exhausted. If false excess messages will be written to
> the message store to prevent memory exhaustion. However, when the message
> store reaches capacity the producer will be throttled until resources are
> freed.
> >
> >>>
> >
> >>>
> >
> >>> Best regards,
> >
> >>> Daniel
> >
> >>
> >
> >
>
>
Re: Mismatch in documentation regarding producerFlowControl behavior
Posted by Matt Pavlovich <ma...@gmail.com>.
Interesting, ok that last part of the statement may be outdated. I honestly haven’t tested with producerFlowControl off in a while to know off the top of my head.
I’d be a straight-forward test.. set storeUsageLimit to something small.. turn off flow control and publish messages to see what happens next.
-Matt Pavlovich
> On Jan 6, 2022, at 4:32 PM, Daniel Navrotsky <da...@driivz.com> wrote:
>
> Thanks Matt,
>
>
>
> You wrote "With it off, producers will simply get errors when resources are exhausted".
>
> But in the description to producerFlowControl param on https://activemq.apache.org/per-destination-policies when the value is off it says "However, when the message store reaches capacity the producer will be throttled until resources are freed".
>
>
>
> Thanks,
>
> Daniel
>
>
>
> -----Original Message-----
> From: Matt Pavlovich <ma...@gmail.com>
> Sent: Thursday, January 6, 2022 11:17 PM
> To: users@activemq.apache.org
> Subject: Re: Mismatch in documentation regarding producerFlowControl behavior
>
>
>
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
>
>
>
>
>
> Turning off producer flow control just means the broker won’t pause fast producers to allow for consumers to catch up— it is a ’self-healing’ type feature. With it off, producers will simply get errors when resources are exhausted.
>
>
>
> The feature helps prevent producer-side errors in the scenario where a few micro pauses would allow things to settle down without any errors being thrown at all.
>
>
>
> -Matt Pavlovich
>
>
>
>> On Jan 6, 2022, at 12:36 PM, Daniel Navrotsky <da...@driivz.com>> wrote:
>
>>
>
>> Thank you Matt for the answer.
>
>>
>
>> I am trying to understand the difference between producerFlowControl=true or false. I understand that the type of the cursor defines the resources. When resources are exhausted the flow control will take place no matter if producerFlowControl=true or false. Can you please bring an example of cursor and message type where the behavior will be different depending on producerFlowControl param?
>
>>
>
>> Thanks,
>
>>
>
>> Best regards,
>
>> Daniel
>
>>
>
>> -----Original Message-----
>
>> From: Matt Pavlovich <ma...@gmail.com>>
>
>> Sent: Thursday, January 6, 2022 7:42 PM
>
>> To: users@activemq.apache.org<ma...@activemq.apache.org>
>
>> Subject: Re: Mismatch in documentation regarding producerFlowControl behavior
>
>>
>
>> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
>
>>
>
>>
>
>> Daniel-
>
>>
>
>> The documentation looks correct. The “triggers” for producer flow control vary based on the configurations and message type (persistent or non-persistent). What is different in the various configuration noted below are what what constitutes “resources” for non-persistent messages.
>
>>
>
>> For non-persistent messages:
>
>>
>
>> a. Default cursor will flow control when memoryLimit and/or tempUsageLimit reached
>
>> -or-
>
>> b. vmQueueCurosr will flow control on memoryLimit only (since temp and store usage are not used)
>
>>
>
>> Hope this helps.
>
>>
>
>> -Matt Pavlovich
>
>>
>
>>> On Jan 5, 2022, at 4:58 PM, Daniel Navrotsky <da...@driivz.com>> wrote:
>
>>>
>
>>> Hello,
>
>>>
>
>>> It looks like there is a mismatch in the documentation. From piece 1 (below) I understand that messages would be shunted into the temporary file even if producerFlowControl=true, but from piece 2 it sounds like it only would happen if producerFlowControl=false
>
>>>
>
>>>
>
>>> Piece 1
>
>>> https://activemq.apache.org/producer-flow-control.html
>
>>> Note that, since the introduction of the new file cursor in ActiveMQ 5.x, non-persisted messages are shunted into the temporary file store to reduce the amount of memory used for non-persistent messaging. As a result, you may find that a queue’s memoryLimit is never reached, as the cursor doesn’t use very much memory. If you really do want to keep all your non-persistent messages in memory, and stop producers when the limit is reached, you should configure the <vmQueueCursor>
>
>>>
>
>>> Piece 2
>
>>> https://activemq.apache.org/per-destination-policies
>
>>> producerFlowControl - If true the broker will throttle (flow-control) the producer. Throttling is achieved either by withholding the producer’s ACK or by raising a javax.jms.ResourceAllocationException (that’s propagated back to the client) when local resources, e.g., memory and/or storage, have been exhausted. If false excess messages will be written to the message store to prevent memory exhaustion. However, when the message store reaches capacity the producer will be throttled until resources are freed.
>
>>>
>
>>>
>
>>> Best regards,
>
>>> Daniel
>
>>
>
>
RE: Mismatch in documentation regarding producerFlowControl behavior
Posted by Daniel Navrotsky <da...@driivz.com>.
Thanks Matt,
You wrote "With it off, producers will simply get errors when resources are exhausted".
But in the description to producerFlowControl param on https://activemq.apache.org/per-destination-policies when the value is off it says "However, when the message store reaches capacity the producer will be throttled until resources are freed".
Thanks,
Daniel
-----Original Message-----
From: Matt Pavlovich <ma...@gmail.com>
Sent: Thursday, January 6, 2022 11:17 PM
To: users@activemq.apache.org
Subject: Re: Mismatch in documentation regarding producerFlowControl behavior
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Turning off producer flow control just means the broker won’t pause fast producers to allow for consumers to catch up— it is a ’self-healing’ type feature. With it off, producers will simply get errors when resources are exhausted.
The feature helps prevent producer-side errors in the scenario where a few micro pauses would allow things to settle down without any errors being thrown at all.
-Matt Pavlovich
> On Jan 6, 2022, at 12:36 PM, Daniel Navrotsky <da...@driivz.com>> wrote:
>
> Thank you Matt for the answer.
>
> I am trying to understand the difference between producerFlowControl=true or false. I understand that the type of the cursor defines the resources. When resources are exhausted the flow control will take place no matter if producerFlowControl=true or false. Can you please bring an example of cursor and message type where the behavior will be different depending on producerFlowControl param?
>
> Thanks,
>
> Best regards,
> Daniel
>
> -----Original Message-----
> From: Matt Pavlovich <ma...@gmail.com>>
> Sent: Thursday, January 6, 2022 7:42 PM
> To: users@activemq.apache.org<ma...@activemq.apache.org>
> Subject: Re: Mismatch in documentation regarding producerFlowControl behavior
>
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
>
>
> Daniel-
>
> The documentation looks correct. The “triggers” for producer flow control vary based on the configurations and message type (persistent or non-persistent). What is different in the various configuration noted below are what what constitutes “resources” for non-persistent messages.
>
> For non-persistent messages:
>
> a. Default cursor will flow control when memoryLimit and/or tempUsageLimit reached
> -or-
> b. vmQueueCurosr will flow control on memoryLimit only (since temp and store usage are not used)
>
> Hope this helps.
>
> -Matt Pavlovich
>
>> On Jan 5, 2022, at 4:58 PM, Daniel Navrotsky <da...@driivz.com>> wrote:
>>
>> Hello,
>>
>> It looks like there is a mismatch in the documentation. From piece 1 (below) I understand that messages would be shunted into the temporary file even if producerFlowControl=true, but from piece 2 it sounds like it only would happen if producerFlowControl=false
>>
>>
>> Piece 1
>> https://activemq.apache.org/producer-flow-control.html
>> Note that, since the introduction of the new file cursor in ActiveMQ 5.x, non-persisted messages are shunted into the temporary file store to reduce the amount of memory used for non-persistent messaging. As a result, you may find that a queue’s memoryLimit is never reached, as the cursor doesn’t use very much memory. If you really do want to keep all your non-persistent messages in memory, and stop producers when the limit is reached, you should configure the <vmQueueCursor>
>>
>> Piece 2
>> https://activemq.apache.org/per-destination-policies
>> producerFlowControl - If true the broker will throttle (flow-control) the producer. Throttling is achieved either by withholding the producer’s ACK or by raising a javax.jms.ResourceAllocationException (that’s propagated back to the client) when local resources, e.g., memory and/or storage, have been exhausted. If false excess messages will be written to the message store to prevent memory exhaustion. However, when the message store reaches capacity the producer will be throttled until resources are freed.
>>
>>
>> Best regards,
>> Daniel
>
Re: Mismatch in documentation regarding producerFlowControl behavior
Posted by Matt Pavlovich <ma...@gmail.com>.
Turning off producer flow control just means the broker won’t pause fast producers to allow for consumers to catch up— it is a ’self-healing’ type feature. With it off, producers will simply get errors when resources are exhausted.
The feature helps prevent producer-side errors in the scenario where a few micro pauses would allow things to settle down without any errors being thrown at all.
-Matt Pavlovich
> On Jan 6, 2022, at 12:36 PM, Daniel Navrotsky <da...@driivz.com> wrote:
>
> Thank you Matt for the answer.
>
> I am trying to understand the difference between producerFlowControl=true or false. I understand that the type of the cursor defines the resources. When resources are exhausted the flow control will take place no matter if producerFlowControl=true or false. Can you please bring an example of cursor and message type where the behavior will be different depending on producerFlowControl param?
>
> Thanks,
>
> Best regards,
> Daniel
>
> -----Original Message-----
> From: Matt Pavlovich <ma...@gmail.com>
> Sent: Thursday, January 6, 2022 7:42 PM
> To: users@activemq.apache.org
> Subject: Re: Mismatch in documentation regarding producerFlowControl behavior
>
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
>
>
> Daniel-
>
> The documentation looks correct. The “triggers” for producer flow control vary based on the configurations and message type (persistent or non-persistent). What is different in the various configuration noted below are what what constitutes “resources” for non-persistent messages.
>
> For non-persistent messages:
>
> a. Default cursor will flow control when memoryLimit and/or tempUsageLimit reached
> -or-
> b. vmQueueCurosr will flow control on memoryLimit only (since temp and store usage are not used)
>
> Hope this helps.
>
> -Matt Pavlovich
>
>> On Jan 5, 2022, at 4:58 PM, Daniel Navrotsky <da...@driivz.com> wrote:
>>
>> Hello,
>>
>> It looks like there is a mismatch in the documentation. From piece 1 (below) I understand that messages would be shunted into the temporary file even if producerFlowControl=true, but from piece 2 it sounds like it only would happen if producerFlowControl=false
>>
>>
>> Piece 1
>> https://activemq.apache.org/producer-flow-control.html
>> Note that, since the introduction of the new file cursor in ActiveMQ 5.x, non-persisted messages are shunted into the temporary file store to reduce the amount of memory used for non-persistent messaging. As a result, you may find that a queue’s memoryLimit is never reached, as the cursor doesn’t use very much memory. If you really do want to keep all your non-persistent messages in memory, and stop producers when the limit is reached, you should configure the <vmQueueCursor>
>>
>> Piece 2
>> https://activemq.apache.org/per-destination-policies
>> producerFlowControl - If true the broker will throttle (flow-control) the producer. Throttling is achieved either by withholding the producer’s ACK or by raising a javax.jms.ResourceAllocationException (that’s propagated back to the client) when local resources, e.g., memory and/or storage, have been exhausted. If false excess messages will be written to the message store to prevent memory exhaustion. However, when the message store reaches capacity the producer will be throttled until resources are freed.
>>
>>
>> Best regards,
>> Daniel
>
RE: Mismatch in documentation regarding producerFlowControl behavior
Posted by Daniel Navrotsky <da...@driivz.com>.
Thank you Matt for the answer.
I am trying to understand the difference between producerFlowControl=true or false. I understand that the type of the cursor defines the resources. When resources are exhausted the flow control will take place no matter if producerFlowControl=true or false. Can you please bring an example of cursor and message type where the behavior will be different depending on producerFlowControl param?
Thanks,
Best regards,
Daniel
-----Original Message-----
From: Matt Pavlovich <ma...@gmail.com>
Sent: Thursday, January 6, 2022 7:42 PM
To: users@activemq.apache.org
Subject: Re: Mismatch in documentation regarding producerFlowControl behavior
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Daniel-
The documentation looks correct. The “triggers” for producer flow control vary based on the configurations and message type (persistent or non-persistent). What is different in the various configuration noted below are what what constitutes “resources” for non-persistent messages.
For non-persistent messages:
a. Default cursor will flow control when memoryLimit and/or tempUsageLimit reached
-or-
b. vmQueueCurosr will flow control on memoryLimit only (since temp and store usage are not used)
Hope this helps.
-Matt Pavlovich
> On Jan 5, 2022, at 4:58 PM, Daniel Navrotsky <da...@driivz.com> wrote:
>
> Hello,
>
> It looks like there is a mismatch in the documentation. From piece 1 (below) I understand that messages would be shunted into the temporary file even if producerFlowControl=true, but from piece 2 it sounds like it only would happen if producerFlowControl=false
>
>
> Piece 1
> https://activemq.apache.org/producer-flow-control.html
> Note that, since the introduction of the new file cursor in ActiveMQ 5.x, non-persisted messages are shunted into the temporary file store to reduce the amount of memory used for non-persistent messaging. As a result, you may find that a queue’s memoryLimit is never reached, as the cursor doesn’t use very much memory. If you really do want to keep all your non-persistent messages in memory, and stop producers when the limit is reached, you should configure the <vmQueueCursor>
>
> Piece 2
> https://activemq.apache.org/per-destination-policies
> producerFlowControl - If true the broker will throttle (flow-control) the producer. Throttling is achieved either by withholding the producer’s ACK or by raising a javax.jms.ResourceAllocationException (that’s propagated back to the client) when local resources, e.g., memory and/or storage, have been exhausted. If false excess messages will be written to the message store to prevent memory exhaustion. However, when the message store reaches capacity the producer will be throttled until resources are freed.
>
>
> Best regards,
> Daniel
Re: Mismatch in documentation regarding producerFlowControl behavior
Posted by Matt Pavlovich <ma...@gmail.com>.
Daniel-
The documentation looks correct. The “triggers” for producer flow control vary based on the configurations and message type (persistent or non-persistent). What is different in the various configuration noted below are what what constitutes “resources” for non-persistent messages.
For non-persistent messages:
a. Default cursor will flow control when memoryLimit and/or tempUsageLimit reached
-or-
b. vmQueueCurosr will flow control on memoryLimit only (since temp and store usage are not used)
Hope this helps.
-Matt Pavlovich
> On Jan 5, 2022, at 4:58 PM, Daniel Navrotsky <da...@driivz.com> wrote:
>
> Hello,
>
> It looks like there is a mismatch in the documentation. From piece 1 (below) I understand that messages would be shunted into the temporary file even if producerFlowControl=true, but from piece 2 it sounds like it only would happen if producerFlowControl=false
>
>
> Piece 1
> https://activemq.apache.org/producer-flow-control.html
> Note that, since the introduction of the new file cursor in ActiveMQ 5.x, non-persisted messages are shunted into the temporary file store to reduce the amount of memory used for non-persistent messaging. As a result, you may find that a queue’s memoryLimit is never reached, as the cursor doesn’t use very much memory. If you really do want to keep all your non-persistent messages in memory, and stop producers when the limit is reached, you should configure the <vmQueueCursor>
>
> Piece 2
> https://activemq.apache.org/per-destination-policies
> producerFlowControl - If true the broker will throttle (flow-control) the producer. Throttling is achieved either by withholding the producer’s ACK or by raising a javax.jms.ResourceAllocationException (that’s propagated back to the client) when local resources, e.g., memory and/or storage, have been exhausted. If false excess messages will be written to the message store to prevent memory exhaustion. However, when the message store reaches capacity the producer will be throttled until resources are freed.
>
>
> Best regards,
> Daniel