You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Lorenz Quack <qu...@gmail.com> on 2017/06/13 09:21:26 UTC
[DISCUSSION] Queue Reject Policy Behaviour
Hello all,
QPID-7815 [1] proposes the addition of a Queue Overflow Reject Policy
to the Qpid broker-j (aka Qpid Broker for Java) component.
Queue's allow to define overflow limits (in term of number of messages
and/or cumulative size of the messages). If the limit is breached the
overflow policy determines the behaviour. There are three ways the
limits can be breached.
1) A new message arrives at the queue pushing it over the limit.
2) An operator lowers the limit so that existing messages are in
breach of the limit.
3) An operator changes the policy. For example from a No-op policy
to the reject policy under discussion.
The behaviour of the proposed policy in case 1) is fairly straight
forward and I think uncontroversial. This discussion thread is to
hash out the expected behaviour in case of operator intervention,
i.e. case 2) and 3).
I see two possible behaviours:
A) The policy silently deletes messages that are in breach of the
policy.
B) The policy ignores messages that are already on the queue and
only applies to messages that newly arrive.
What would people expect the behaviour to be?
Please discuss.
Kind regards,
Lorenz
[1] https://issues.apache.org/jira/browse/QPID-7815
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: [DISCUSSION] Queue Reject Policy Behaviour
Posted by Lorenz Quack <qu...@gmail.com>.
Thanks everyone for the feedback.
I think people mostly agree that behaviour B) is the least surprising.
Also, it aligns with the cpp broker's behaviour.
The idea of making the behaviour configurable was rejected because it
added too much complexity for too little gain.
I will comment on QPID-7815 to reference this discussion.
Kind regards,
Lorenz
On Tue, 2017-06-13 at 10:21 +0100, Lorenz Quack wrote:
> Hello all,
>
> QPID-7815 [1] proposes the addition of a Queue Overflow Reject Policy
> to the Qpid broker-j (aka Qpid Broker for Java) component.
>
> Queue's allow to define overflow limits (in term of number of messages
> and/or cumulative size of the messages). If the limit is breached the
> overflow policy determines the behaviour. There are three ways the
> limits can be breached.
>
> 1) A new message arrives at the queue pushing it over the limit.
>
> 2) An operator lowers the limit so that existing messages are in
> breach of the limit.
>
> 3) An operator changes the policy. For example from a No-op policy
> to the reject policy under discussion.
>
> The behaviour of the proposed policy in case 1) is fairly straight
> forward and I think uncontroversial. This discussion thread is to
> hash out the expected behaviour in case of operator intervention,
> i.e. case 2) and 3).
>
> I see two possible behaviours:
>
> A) The policy silently deletes messages that are in breach of the
> policy.
>
> B) The policy ignores messages that are already on the queue and
> only applies to messages that newly arrive.
>
>
> What would people expect the behaviour to be?
> Please discuss.
>
>
> Kind regards,
> Lorenz
>
>
> [1] https://issues.apache.org/jira/browse/QPID-7815
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: [DISCUSSION] Queue Reject Policy Behaviour
Posted by Lorenz Quack <qu...@gmail.com>.
Hi Robbie,
That is an excellent question. I haven't checked but I fear it will
apply the policy to the existing messages. Thanks for bringing this up.
Kind regards,
Lorenz
On Tue, 2017-06-13 at 17:19 +0100, Robbie Gemmell wrote:
> One last thought: if the queue is overfull with persistent messages
> following the reconfiguration, and the broker is restarted, what
> happens to the queue during recovery?
>
> On 13 June 2017 at 17:18, Robbie Gemmell <ro...@gmail.com> wrote:
> >
> > (This time, sending with actual content :P)
> >
> > I'm reading the below as meaning that you would vote for B, rather
> > than A as you ended with. If so, I would agree.
> >
> > I think Adel is correct that different people will have different
> > answers, but I expect the people who sent the message would find it
> > unexpected that it got dropped after successfully being sent,
> > particularly if they are aware of existing use of a reject policy and
> > that send failure would typically relate to that. That said, I also
> > expect it to be rare for people to reduce the limit on a live system
> > with messages people care about.
> >
> > I don't think I would make it configurable, its extra complexity for
> > whats likely to be limited or no usage. If the queue remains overfull,
> > as you said the operator should be able to delete messages that might
> > have been dropped if needed. To Adel's point, the operator would need
> > to do pretty much the same the broker itself would; by making some
> > aribtrary decision such as the first or last <x> messages on the
> > queue.
> >
> > Robbie
> >
> > On 13 June 2017 at 10:22, Lorenz Quack <qu...@gmail.com> wrote:
> > >
> > > Hello,
> > >
> > > I personally would expect behaviour B).
> > >
> > > As an application I would expect a REJECT policy I either accept or
> > > reject a message I send. Silently "rejecting" it retrospectively
> > > seems counter-intuitive to me.
> > >
> > > On the other hand, behaviour A) would bring it more in line with some of
> > > the other overflow policies that immediately react to a change in the
> > > limits.
> > > That being said I would still vote for behaviour A).
> > >
> > > Kind regards,
> > > Lorenz
> > >
> > >
> > > On Tue, 2017-06-13 at 10:21 +0100, Lorenz Quack wrote:
> > > >
> > > > Hello all,
> > > >
> > > > QPID-7815 [1] proposes the addition of a Queue Overflow Reject Policy
> > > > to the Qpid broker-j (aka Qpid Broker for Java) component.
> > > >
> > > > Queue's allow to define overflow limits (in term of number of messages
> > > > and/or cumulative size of the messages). If the limit is breached the
> > > > overflow policy determines the behaviour. There are three ways the
> > > > limits can be breached.
> > > >
> > > > 1) A new message arrives at the queue pushing it over the limit.
> > > >
> > > > 2) An operator lowers the limit so that existing messages are in
> > > > breach of the limit.
> > > >
> > > > 3) An operator changes the policy. For example from a No-op policy
> > > > to the reject policy under discussion.
> > > >
> > > > The behaviour of the proposed policy in case 1) is fairly straight
> > > > forward and I think uncontroversial. This discussion thread is to
> > > > hash out the expected behaviour in case of operator intervention,
> > > > i.e. case 2) and 3).
> > > >
> > > > I see two possible behaviours:
> > > >
> > > > A) The policy silently deletes messages that are in breach of the
> > > > policy.
> > > >
> > > > B) The policy ignores messages that are already on the queue and
> > > > only applies to messages that newly arrive.
> > > >
> > > >
> > > > What would people expect the behaviour to be?
> > > > Please discuss.
> > > >
> > > >
> > > > Kind regards,
> > > > Lorenz
> > > >
> > > >
> > > > [1] https://issues.apache.org/jira/browse/QPID-7815
> > > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> > > For additional commands, e-mail: users-help@qpid.apache.org
> > >
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: [DISCUSSION] Queue Reject Policy Behaviour
Posted by Robbie Gemmell <ro...@gmail.com>.
One last thought: if the queue is overfull with persistent messages
following the reconfiguration, and the broker is restarted, what
happens to the queue during recovery?
On 13 June 2017 at 17:18, Robbie Gemmell <ro...@gmail.com> wrote:
> (This time, sending with actual content :P)
>
> I'm reading the below as meaning that you would vote for B, rather
> than A as you ended with. If so, I would agree.
>
> I think Adel is correct that different people will have different
> answers, but I expect the people who sent the message would find it
> unexpected that it got dropped after successfully being sent,
> particularly if they are aware of existing use of a reject policy and
> that send failure would typically relate to that. That said, I also
> expect it to be rare for people to reduce the limit on a live system
> with messages people care about.
>
> I don't think I would make it configurable, its extra complexity for
> whats likely to be limited or no usage. If the queue remains overfull,
> as you said the operator should be able to delete messages that might
> have been dropped if needed. To Adel's point, the operator would need
> to do pretty much the same the broker itself would; by making some
> aribtrary decision such as the first or last <x> messages on the
> queue.
>
> Robbie
>
> On 13 June 2017 at 10:22, Lorenz Quack <qu...@gmail.com> wrote:
>> Hello,
>>
>> I personally would expect behaviour B).
>>
>> As an application I would expect a REJECT policy I either accept or
>> reject a message I send. Silently "rejecting" it retrospectively
>> seems counter-intuitive to me.
>>
>> On the other hand, behaviour A) would bring it more in line with some of
>> the other overflow policies that immediately react to a change in the
>> limits.
>> That being said I would still vote for behaviour A).
>>
>> Kind regards,
>> Lorenz
>>
>>
>> On Tue, 2017-06-13 at 10:21 +0100, Lorenz Quack wrote:
>>> Hello all,
>>>
>>> QPID-7815 [1] proposes the addition of a Queue Overflow Reject Policy
>>> to the Qpid broker-j (aka Qpid Broker for Java) component.
>>>
>>> Queue's allow to define overflow limits (in term of number of messages
>>> and/or cumulative size of the messages). If the limit is breached the
>>> overflow policy determines the behaviour. There are three ways the
>>> limits can be breached.
>>>
>>> 1) A new message arrives at the queue pushing it over the limit.
>>>
>>> 2) An operator lowers the limit so that existing messages are in
>>> breach of the limit.
>>>
>>> 3) An operator changes the policy. For example from a No-op policy
>>> to the reject policy under discussion.
>>>
>>> The behaviour of the proposed policy in case 1) is fairly straight
>>> forward and I think uncontroversial. This discussion thread is to
>>> hash out the expected behaviour in case of operator intervention,
>>> i.e. case 2) and 3).
>>>
>>> I see two possible behaviours:
>>>
>>> A) The policy silently deletes messages that are in breach of the
>>> policy.
>>>
>>> B) The policy ignores messages that are already on the queue and
>>> only applies to messages that newly arrive.
>>>
>>>
>>> What would people expect the behaviour to be?
>>> Please discuss.
>>>
>>>
>>> Kind regards,
>>> Lorenz
>>>
>>>
>>> [1] https://issues.apache.org/jira/browse/QPID-7815
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> For additional commands, e-mail: users-help@qpid.apache.org
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: [DISCUSSION] Queue Reject Policy Behaviour
Posted by Robbie Gemmell <ro...@gmail.com>.
(This time, sending with actual content :P)
I'm reading the below as meaning that you would vote for B, rather
than A as you ended with. If so, I would agree.
I think Adel is correct that different people will have different
answers, but I expect the people who sent the message would find it
unexpected that it got dropped after successfully being sent,
particularly if they are aware of existing use of a reject policy and
that send failure would typically relate to that. That said, I also
expect it to be rare for people to reduce the limit on a live system
with messages people care about.
I don't think I would make it configurable, its extra complexity for
whats likely to be limited or no usage. If the queue remains overfull,
as you said the operator should be able to delete messages that might
have been dropped if needed. To Adel's point, the operator would need
to do pretty much the same the broker itself would; by making some
aribtrary decision such as the first or last <x> messages on the
queue.
Robbie
On 13 June 2017 at 10:22, Lorenz Quack <qu...@gmail.com> wrote:
> Hello,
>
> I personally would expect behaviour B).
>
> As an application I would expect a REJECT policy I either accept or
> reject a message I send. Silently "rejecting" it retrospectively
> seems counter-intuitive to me.
>
> On the other hand, behaviour A) would bring it more in line with some of
> the other overflow policies that immediately react to a change in the
> limits.
> That being said I would still vote for behaviour A).
>
> Kind regards,
> Lorenz
>
>
> On Tue, 2017-06-13 at 10:21 +0100, Lorenz Quack wrote:
>> Hello all,
>>
>> QPID-7815 [1] proposes the addition of a Queue Overflow Reject Policy
>> to the Qpid broker-j (aka Qpid Broker for Java) component.
>>
>> Queue's allow to define overflow limits (in term of number of messages
>> and/or cumulative size of the messages). If the limit is breached the
>> overflow policy determines the behaviour. There are three ways the
>> limits can be breached.
>>
>> 1) A new message arrives at the queue pushing it over the limit.
>>
>> 2) An operator lowers the limit so that existing messages are in
>> breach of the limit.
>>
>> 3) An operator changes the policy. For example from a No-op policy
>> to the reject policy under discussion.
>>
>> The behaviour of the proposed policy in case 1) is fairly straight
>> forward and I think uncontroversial. This discussion thread is to
>> hash out the expected behaviour in case of operator intervention,
>> i.e. case 2) and 3).
>>
>> I see two possible behaviours:
>>
>> A) The policy silently deletes messages that are in breach of the
>> policy.
>>
>> B) The policy ignores messages that are already on the queue and
>> only applies to messages that newly arrive.
>>
>>
>> What would people expect the behaviour to be?
>> Please discuss.
>>
>>
>> Kind regards,
>> Lorenz
>>
>>
>> [1] https://issues.apache.org/jira/browse/QPID-7815
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: [DISCUSSION] Queue Reject Policy Behaviour
Posted by Adel Boutros <Ad...@live.com>.
Hello Lorenz,
"A) If an operator wishes this behavior but we actually implement B)"
How will the operator know which messages to delete?
"C)"
I could imagine a client who has an "End of Day" process which is triggered at some point of the day. One of its tasks could be to read piled up messages and consume them.
PS: I cannot say I have a specific problem with the behavior you are proposing but as you asked for discussions, I tried to create one 😝
Adel
________________________________
From: Lorenz Quack <qu...@gmail.com>
Sent: Tuesday, June 13, 2017 2:37:29 PM
To: users@qpid.apache.org; lquack@apache.org
Subject: Re: [DISCUSSION] Queue Reject Policy Behaviour
Hello Adel,
Thanks for your feedback.
You are right that the desired behaviour depends on the context of
your application. However, I want to avoid adding configuration
options where it is not necessary. IMHO, the broker-j already suffers
from an abundance of configuration options and I do not want to
exacerbate the problem.
So is the configuration you suggest necessary/useful? Let me examine
the three options you provided:
A) If an operator wishes this behaviour but we actually implement B)
then the operator could still manually/explicitly delete messages
from the queue until it is within the new limits.
B) This is still my preferred behaviour.
C) I have a hard time coming up with an example where this would be
useful or desirable. As you pointed out in the beginning of your
email an application typically regards its messages as critical
(no message loss acceptable) or as dispensable (message loss
acceptable). Choosing to discard messages but using a timeout
period seems to indicate conflicting goals. If the messages
themselves become interesting TTL should be used instead.
Do you have a use case in mind where this would be useful?
Note that I do not expect this situation to come about very often.
Typically a queue is configured and then left alone. Changing the
overflow policy and/or the overflow limits on a live queue should be
very rare.
Kind regards,
Lorenz
On Tue, 2017-06-13 at 10:15 +0000, Adel Boutros wrote:
> Hello Lorenz,
>
>
> In my opinion, it depends on the context and there is no general rule for this.
>
> For example, if your queue has some sensitive data and your producers considers the messages available. So they will never send them back.
>
> If you delete the messages which overflow the queue, you will lose this important information (which could be Bank accounts for example).
>
>
> If on the other hand, your system is capable of overcoming message loss by resending the messages then you have no problem in silently deleting them.
>
>
> So instead of forcing a behavior, I think the choice should be left to the operator when defining the policy (per queue would be even better):
>
> A) He would explicitly choose to silently delete all overflowing messages immediately
>
> B) He would explicitly choose to ignore current overflowing messages
>
> C) He would explicitly define a period after which current overflowing messages would be deleted (A would be a specific implementation of C)
>
>
> Regards,
>
> Adel
>
> ________________________________
> From: Lorenz Quack <qu...@gmail.com>
> Sent: Tuesday, June 13, 2017 11:22:31 AM
> To: Qpid Users
> Subject: Re: [DISCUSSION] Queue Reject Policy Behaviour
>
> Hello,
>
> I personally would expect behaviour B).
>
> As an application I would expect a REJECT policy I either accept or
> reject a message I send. Silently "rejecting" it retrospectively
> seems counter-intuitive to me.
>
> On the other hand, behaviour A) would bring it more in line with some of
> the other overflow policies that immediately react to a change in the
> limits.
> That being said I would still vote for behaviour A).
>
> Kind regards,
> Lorenz
>
>
> On Tue, 2017-06-13 at 10:21 +0100, Lorenz Quack wrote:
> >
> > Hello all,
> >
> > QPID-7815 [1] proposes the addition of a Queue Overflow Reject Policy
> > to the Qpid broker-j (aka Qpid Broker for Java) component.
> >
> > Queue's allow to define overflow limits (in term of number of messages
> > and/or cumulative size of the messages). If the limit is breached the
> > overflow policy determines the behaviour. There are three ways the
> > limits can be breached.
> >
> > 1) A new message arrives at the queue pushing it over the limit.
> >
> > 2) An operator lowers the limit so that existing messages are in
> > breach of the limit.
> >
> > 3) An operator changes the policy. For example from a No-op policy
> > to the reject policy under discussion.
> >
> > The behaviour of the proposed policy in case 1) is fairly straight
> > forward and I think uncontroversial. This discussion thread is to
> > hash out the expected behaviour in case of operator intervention,
> > i.e. case 2) and 3).
> >
> > I see two possible behaviours:
> >
> > A) The policy silently deletes messages that are in breach of the
> > policy.
> >
> > B) The policy ignores messages that are already on the queue and
> > only applies to messages that newly arrive.
> >
> >
> > What would people expect the behaviour to be?
> > Please discuss.
> >
> >
> > Kind regards,
> > Lorenz
> >
> >
> > [1] https://issues.apache.org/jira/browse/QPID-7815
> >
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: [DISCUSSION] Queue Reject Policy Behaviour
Posted by Lorenz Quack <qu...@gmail.com>.
Hello Adel,
Thanks for your feedback.
You are right that the desired behaviour depends on the context of
your application. However, I want to avoid adding configuration
options where it is not necessary. IMHO, the broker-j already suffers
from an abundance of configuration options and I do not want to
exacerbate the problem.
So is the configuration you suggest necessary/useful? Let me examine
the three options you provided:
A) If an operator wishes this behaviour but we actually implement B)
then the operator could still manually/explicitly delete messages
from the queue until it is within the new limits.
B) This is still my preferred behaviour.
C) I have a hard time coming up with an example where this would be
useful or desirable. As you pointed out in the beginning of your
email an application typically regards its messages as critical
(no message loss acceptable) or as dispensable (message loss
acceptable). Choosing to discard messages but using a timeout
period seems to indicate conflicting goals. If the messages
themselves become interesting TTL should be used instead.
Do you have a use case in mind where this would be useful?
Note that I do not expect this situation to come about very often.
Typically a queue is configured and then left alone. Changing the
overflow policy and/or the overflow limits on a live queue should be
very rare.
Kind regards,
Lorenz
On Tue, 2017-06-13 at 10:15 +0000, Adel Boutros wrote:
> Hello Lorenz,
>
>
> In my opinion, it depends on the context and there is no general rule for this.
>
> For example, if your queue has some sensitive data and your producers considers the messages available. So they will never send them back.
>
> If you delete the messages which overflow the queue, you will lose this important information (which could be Bank accounts for example).
>
>
> If on the other hand, your system is capable of overcoming message loss by resending the messages then you have no problem in silently deleting them.
>
>
> So instead of forcing a behavior, I think the choice should be left to the operator when defining the policy (per queue would be even better):
>
> A) He would explicitly choose to silently delete all overflowing messages immediately
>
> B) He would explicitly choose to ignore current overflowing messages
>
> C) He would explicitly define a period after which current overflowing messages would be deleted (A would be a specific implementation of C)
>
>
> Regards,
>
> Adel
>
> ________________________________
> From: Lorenz Quack <qu...@gmail.com>
> Sent: Tuesday, June 13, 2017 11:22:31 AM
> To: Qpid Users
> Subject: Re: [DISCUSSION] Queue Reject Policy Behaviour
>
> Hello,
>
> I personally would expect behaviour B).
>
> As an application I would expect a REJECT policy I either accept or
> reject a message I send. Silently "rejecting" it retrospectively
> seems counter-intuitive to me.
>
> On the other hand, behaviour A) would bring it more in line with some of
> the other overflow policies that immediately react to a change in the
> limits.
> That being said I would still vote for behaviour A).
>
> Kind regards,
> Lorenz
>
>
> On Tue, 2017-06-13 at 10:21 +0100, Lorenz Quack wrote:
> >
> > Hello all,
> >
> > QPID-7815 [1] proposes the addition of a Queue Overflow Reject Policy
> > to the Qpid broker-j (aka Qpid Broker for Java) component.
> >
> > Queue's allow to define overflow limits (in term of number of messages
> > and/or cumulative size of the messages). If the limit is breached the
> > overflow policy determines the behaviour. There are three ways the
> > limits can be breached.
> >
> > 1) A new message arrives at the queue pushing it over the limit.
> >
> > 2) An operator lowers the limit so that existing messages are in
> > breach of the limit.
> >
> > 3) An operator changes the policy. For example from a No-op policy
> > to the reject policy under discussion.
> >
> > The behaviour of the proposed policy in case 1) is fairly straight
> > forward and I think uncontroversial. This discussion thread is to
> > hash out the expected behaviour in case of operator intervention,
> > i.e. case 2) and 3).
> >
> > I see two possible behaviours:
> >
> > A) The policy silently deletes messages that are in breach of the
> > policy.
> >
> > B) The policy ignores messages that are already on the queue and
> > only applies to messages that newly arrive.
> >
> >
> > What would people expect the behaviour to be?
> > Please discuss.
> >
> >
> > Kind regards,
> > Lorenz
> >
> >
> > [1] https://issues.apache.org/jira/browse/QPID-7815
> >
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: [DISCUSSION] Queue Reject Policy Behaviour
Posted by Adel Boutros <Ad...@live.com>.
Hello Lorenz,
In my opinion, it depends on the context and there is no general rule for this.
For example, if your queue has some sensitive data and your producers considers the messages available. So they will never send them back.
If you delete the messages which overflow the queue, you will lose this important information (which could be Bank accounts for example).
If on the other hand, your system is capable of overcoming message loss by resending the messages then you have no problem in silently deleting them.
So instead of forcing a behavior, I think the choice should be left to the operator when defining the policy (per queue would be even better):
A) He would explicitly choose to silently delete all overflowing messages immediately
B) He would explicitly choose to ignore current overflowing messages
C) He would explicitly define a period after which current overflowing messages would be deleted (A would be a specific implementation of C)
Regards,
Adel
________________________________
From: Lorenz Quack <qu...@gmail.com>
Sent: Tuesday, June 13, 2017 11:22:31 AM
To: Qpid Users
Subject: Re: [DISCUSSION] Queue Reject Policy Behaviour
Hello,
I personally would expect behaviour B).
As an application I would expect a REJECT policy I either accept or
reject a message I send. Silently "rejecting" it retrospectively
seems counter-intuitive to me.
On the other hand, behaviour A) would bring it more in line with some of
the other overflow policies that immediately react to a change in the
limits.
That being said I would still vote for behaviour A).
Kind regards,
Lorenz
On Tue, 2017-06-13 at 10:21 +0100, Lorenz Quack wrote:
> Hello all,
>
> QPID-7815 [1] proposes the addition of a Queue Overflow Reject Policy
> to the Qpid broker-j (aka Qpid Broker for Java) component.
>
> Queue's allow to define overflow limits (in term of number of messages
> and/or cumulative size of the messages). If the limit is breached the
> overflow policy determines the behaviour. There are three ways the
> limits can be breached.
>
> 1) A new message arrives at the queue pushing it over the limit.
>
> 2) An operator lowers the limit so that existing messages are in
> breach of the limit.
>
> 3) An operator changes the policy. For example from a No-op policy
> to the reject policy under discussion.
>
> The behaviour of the proposed policy in case 1) is fairly straight
> forward and I think uncontroversial. This discussion thread is to
> hash out the expected behaviour in case of operator intervention,
> i.e. case 2) and 3).
>
> I see two possible behaviours:
>
> A) The policy silently deletes messages that are in breach of the
> policy.
>
> B) The policy ignores messages that are already on the queue and
> only applies to messages that newly arrive.
>
>
> What would people expect the behaviour to be?
> Please discuss.
>
>
> Kind regards,
> Lorenz
>
>
> [1] https://issues.apache.org/jira/browse/QPID-7815
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: [DISCUSSION] Queue Reject Policy Behaviour
Posted by Robbie Gemmell <ro...@gmail.com>.
On 13 June 2017 at 10:22, Lorenz Quack <qu...@gmail.com> wrote:
> Hello,
>
> I personally would expect behaviour B).
>
> As an application I would expect a REJECT policy I either accept or
> reject a message I send. Silently "rejecting" it retrospectively
> seems counter-intuitive to me.
>
> On the other hand, behaviour A) would bring it more in line with some of
> the other overflow policies that immediately react to a change in the
> limits.
> That being said I would still vote for behaviour A).
>
> Kind regards,
> Lorenz
>
>
> On Tue, 2017-06-13 at 10:21 +0100, Lorenz Quack wrote:
>> Hello all,
>>
>> QPID-7815 [1] proposes the addition of a Queue Overflow Reject Policy
>> to the Qpid broker-j (aka Qpid Broker for Java) component.
>>
>> Queue's allow to define overflow limits (in term of number of messages
>> and/or cumulative size of the messages). If the limit is breached the
>> overflow policy determines the behaviour. There are three ways the
>> limits can be breached.
>>
>> 1) A new message arrives at the queue pushing it over the limit.
>>
>> 2) An operator lowers the limit so that existing messages are in
>> breach of the limit.
>>
>> 3) An operator changes the policy. For example from a No-op policy
>> to the reject policy under discussion.
>>
>> The behaviour of the proposed policy in case 1) is fairly straight
>> forward and I think uncontroversial. This discussion thread is to
>> hash out the expected behaviour in case of operator intervention,
>> i.e. case 2) and 3).
>>
>> I see two possible behaviours:
>>
>> A) The policy silently deletes messages that are in breach of the
>> policy.
>>
>> B) The policy ignores messages that are already on the queue and
>> only applies to messages that newly arrive.
>>
>>
>> What would people expect the behaviour to be?
>> Please discuss.
>>
>>
>> Kind regards,
>> Lorenz
>>
>>
>> [1] https://issues.apache.org/jira/browse/QPID-7815
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: [DISCUSSION] Queue Reject Policy Behaviour
Posted by Lorenz Quack <qu...@gmail.com>.
Hello,
I personally would expect behaviour B).
As an application I would expect a REJECT policy I either accept or
reject a message I send. Silently "rejecting" it retrospectively
seems counter-intuitive to me.
On the other hand, behaviour A) would bring it more in line with some of
the other overflow policies that immediately react to a change in the
limits.
That being said I would still vote for behaviour A).
Kind regards,
Lorenz
On Tue, 2017-06-13 at 10:21 +0100, Lorenz Quack wrote:
> Hello all,
>
> QPID-7815 [1] proposes the addition of a Queue Overflow Reject Policy
> to the Qpid broker-j (aka Qpid Broker for Java) component.
>
> Queue's allow to define overflow limits (in term of number of messages
> and/or cumulative size of the messages). If the limit is breached the
> overflow policy determines the behaviour. There are three ways the
> limits can be breached.
>
> 1) A new message arrives at the queue pushing it over the limit.
>
> 2) An operator lowers the limit so that existing messages are in
> breach of the limit.
>
> 3) An operator changes the policy. For example from a No-op policy
> to the reject policy under discussion.
>
> The behaviour of the proposed policy in case 1) is fairly straight
> forward and I think uncontroversial. This discussion thread is to
> hash out the expected behaviour in case of operator intervention,
> i.e. case 2) and 3).
>
> I see two possible behaviours:
>
> A) The policy silently deletes messages that are in breach of the
> policy.
>
> B) The policy ignores messages that are already on the queue and
> only applies to messages that newly arrive.
>
>
> What would people expect the behaviour to be?
> Please discuss.
>
>
> Kind regards,
> Lorenz
>
>
> [1] https://issues.apache.org/jira/browse/QPID-7815
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: [DISCUSSION] Queue Reject Policy Behaviour
Posted by Chuck Rolke <cr...@redhat.com>.
In the Qpid C++ broker you *can* reload the policy on-the-fly. C++ broker
policy allows existing connections or other quotas that are in excess of the
new policy limits to persist. As old connections or queues are disposed
eventually the user goes within the new policy limits and at that point new
items may be created. This behavior is Lorenz' original type B.
An administrator is free to delete queues or close connections as he wishes.
Having the policy itself delete queues or close connections would be
indeterminate. That's why we stuck with strategy B. That makes the most
sense to me for similar decisions.
-Chuck
----- Original Message -----
> From: "Jakub Scholz" <ja...@scholz.cz>
> To: users@qpid.apache.org
> Sent: Tuesday, June 13, 2017 2:52:33 PM
> Subject: Re: [DISCUSSION] Queue Reject Policy Behaviour
>
> This is lot easier in C++ broker where you cannot change the policy or the
> limits once you create the queue :-). But as a user I would have
> appreciated the possibility to change the limits on the fly many times, so
> I'm not suggesting this is the way to go.
>
> From your possibilities B) would be what I would expect. I assume that it
> is quite easy to change the limit and purge some messages in two steps if
> the operator wants to do that.
>
> I can also imagine another possibility, where you don't allow the operator
> to set the limit below the actual queue size. I.e.:
> - user can freely increase the limit
> - If the queue has limit 1000 messages and 100 actual messages inside, user
> can change the limit to 101 or 100. But not to 99.
> This would ensure that the limit is always kept and leave the task to
> decide what to delete or not to delete on the operator. But I guess this
> might be also non-trivial to implement so that it doesn't run into any race
> conditions etc.
>
> Regards
> Jakub
>
>
> On Tue, Jun 13, 2017 at 11:21 AM, Lorenz Quack <qu...@gmail.com>
> wrote:
>
> > Hello all,
> >
> > QPID-7815 [1] proposes the addition of a Queue Overflow Reject Policy
> > to the Qpid broker-j (aka Qpid Broker for Java) component.
> >
> > Queue's allow to define overflow limits (in term of number of messages
> > and/or cumulative size of the messages). If the limit is breached the
> > overflow policy determines the behaviour. There are three ways the
> > limits can be breached.
> >
> > 1) A new message arrives at the queue pushing it over the limit.
> >
> > 2) An operator lowers the limit so that existing messages are in
> > breach of the limit.
> >
> > 3) An operator changes the policy. For example from a No-op policy
> > to the reject policy under discussion.
> >
> > The behaviour of the proposed policy in case 1) is fairly straight
> > forward and I think uncontroversial. This discussion thread is to
> > hash out the expected behaviour in case of operator intervention,
> > i.e. case 2) and 3).
> >
> > I see two possible behaviours:
> >
> > A) The policy silently deletes messages that are in breach of the
> > policy.
> >
> > B) The policy ignores messages that are already on the queue and
> > only applies to messages that newly arrive.
> >
> >
> > What would people expect the behaviour to be?
> > Please discuss.
> >
> >
> > Kind regards,
> > Lorenz
> >
> >
> > [1] https://issues.apache.org/jira/browse/QPID-7815
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> > For additional commands, e-mail: users-help@qpid.apache.org
> >
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: [DISCUSSION] Queue Reject Policy Behaviour
Posted by Jakub Scholz <ja...@scholz.cz>.
This is lot easier in C++ broker where you cannot change the policy or the
limits once you create the queue :-). But as a user I would have
appreciated the possibility to change the limits on the fly many times, so
I'm not suggesting this is the way to go.
From your possibilities B) would be what I would expect. I assume that it
is quite easy to change the limit and purge some messages in two steps if
the operator wants to do that.
I can also imagine another possibility, where you don't allow the operator
to set the limit below the actual queue size. I.e.:
- user can freely increase the limit
- If the queue has limit 1000 messages and 100 actual messages inside, user
can change the limit to 101 or 100. But not to 99.
This would ensure that the limit is always kept and leave the task to
decide what to delete or not to delete on the operator. But I guess this
might be also non-trivial to implement so that it doesn't run into any race
conditions etc.
Regards
Jakub
On Tue, Jun 13, 2017 at 11:21 AM, Lorenz Quack <qu...@gmail.com>
wrote:
> Hello all,
>
> QPID-7815 [1] proposes the addition of a Queue Overflow Reject Policy
> to the Qpid broker-j (aka Qpid Broker for Java) component.
>
> Queue's allow to define overflow limits (in term of number of messages
> and/or cumulative size of the messages). If the limit is breached the
> overflow policy determines the behaviour. There are three ways the
> limits can be breached.
>
> 1) A new message arrives at the queue pushing it over the limit.
>
> 2) An operator lowers the limit so that existing messages are in
> breach of the limit.
>
> 3) An operator changes the policy. For example from a No-op policy
> to the reject policy under discussion.
>
> The behaviour of the proposed policy in case 1) is fairly straight
> forward and I think uncontroversial. This discussion thread is to
> hash out the expected behaviour in case of operator intervention,
> i.e. case 2) and 3).
>
> I see two possible behaviours:
>
> A) The policy silently deletes messages that are in breach of the
> policy.
>
> B) The policy ignores messages that are already on the queue and
> only applies to messages that newly arrive.
>
>
> What would people expect the behaviour to be?
> Please discuss.
>
>
> Kind regards,
> Lorenz
>
>
> [1] https://issues.apache.org/jira/browse/QPID-7815
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: [DISCUSSION] Queue Reject Policy Behaviour
Posted by Lorenz Quack <qu...@gmail.com>.
Hi Paul,
There is some logging in this area indicating that the broker did
encounter the overflow condition but it is not configurable and
not as extensive as you suggest.
The reason is that logging (most commonly to disk) can be quite
expensive and we want to minimize this on the critical (message
delivery) path.
Kind regards,
Lorenz
On Tue, 2017-06-13 at 15:45 +0000, Flores, Paul A. wrote:
> Hi Lorenz,
>
> I am not one usually to "pipe in" to discussion so I am be a bit off topic. I did read Adel's discussion replies which I found interesting.
>
> I am not all that familiar with the configuration details so please forgive my ignorance.
>
> Is it possible to opt to have a specific message placed into a log (perhaps queue specific) to indicate any or all of the following?:
> * queue name
> * time
> * that the message queue limit was exceeded and
> ** that the queue was configured to either
> - push an older message off of "the top" of the queue
> or
> - ignore/ not place a new message onto the queue
> or
> - push a message with a specific characteristics such as importance
> * Impacted Message details to include:
> ** sender information
> ** timestamp when message was originally received
> ** message characteristic used in determination action
> Just an idea.
>
> Paul
>
> From: Lorenz Quack <qu...@gmail.com>
> Sent: Tuesday, June 13, 2017 3:21 AM
> To: Qpid Users
> Subject: [DISCUSSION] Queue Reject Policy Behaviour
>
> Hello all,
>
> QPID-7815 [1] proposes the addition of a Queue Overflow Reject Policy
> to the Qpid broker-j (aka Qpid Broker for Java) component.
>
> Queue's allow to define overflow limits (in term of number of messages
> and/or cumulative size of the messages). If the limit is breached the
> overflow policy determines the behaviour. There are three ways the
> limits can be breached.
>
> 1) A new message arrives at the queue pushing it over the limit.
>
> 2) An operator lowers the limit so that existing messages are in
> breach of the limit.
>
> 3) An operator changes the policy. For example from a No-op policy
> to the reject policy under discussion.
>
> The behaviour of the proposed policy in case 1) is fairly straight
> forward and I think uncontroversial. This discussion thread is to
> hash out the expected behaviour in case of operator intervention,
> i.e. case 2) and 3).
>
> I see two possible behaviours:
>
> A) The policy silently deletes messages that are in breach of the
> policy.
>
> B) The policy ignores messages that are already on the queue and
> only applies to messages that newly arrive.
>
>
> What would people expect the behaviour to be?
> Please discuss.
>
>
> Kind regards,
> Lorenz
>
>
> [1] https://issues.apache.org/jira/browse/QPID-7815
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
>
> This communication (including any attachments) may contain information that is proprietary, confidential or exempt from disclosure. If you are not the intended recipient, please note that further
> dissemination, distribution, use or copying of this communication is strictly prohibited. Anyone who received this message in error should notify the sender immediately by telephone or by return
> email and delete it from his or her computer.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: [DISCUSSION] Queue Reject Policy Behaviour
Posted by "Flores, Paul A." <PA...@SAIC.COM>.
Hi Lorenz,
I am not one usually to "pipe in" to discussion so I am be a bit off topic. I did read Adel's discussion replies which I found interesting.
I am not all that familiar with the configuration details so please forgive my ignorance.
Is it possible to opt to have a specific message placed into a log (perhaps queue specific) to indicate any or all of the following?:
* queue name
* time
* that the message queue limit was exceeded and
** that the queue was configured to either
- push an older message off of "the top" of the queue
or
- ignore/ not place a new message onto the queue
or
- push a message with a specific characteristics such as importance
* Impacted Message details to include:
** sender information
** timestamp when message was originally received
** message characteristic used in determination action
Just an idea.
Paul
________________________________
From: Lorenz Quack <qu...@gmail.com>
Sent: Tuesday, June 13, 2017 3:21 AM
To: Qpid Users
Subject: [DISCUSSION] Queue Reject Policy Behaviour
Hello all,
QPID-7815 [1] proposes the addition of a Queue Overflow Reject Policy
to the Qpid broker-j (aka Qpid Broker for Java) component.
Queue's allow to define overflow limits (in term of number of messages
and/or cumulative size of the messages). If the limit is breached the
overflow policy determines the behaviour. There are three ways the
limits can be breached.
1) A new message arrives at the queue pushing it over the limit.
2) An operator lowers the limit so that existing messages are in
breach of the limit.
3) An operator changes the policy. For example from a No-op policy
to the reject policy under discussion.
The behaviour of the proposed policy in case 1) is fairly straight
forward and I think uncontroversial. This discussion thread is to
hash out the expected behaviour in case of operator intervention,
i.e. case 2) and 3).
I see two possible behaviours:
A) The policy silently deletes messages that are in breach of the
policy.
B) The policy ignores messages that are already on the queue and
only applies to messages that newly arrive.
What would people expect the behaviour to be?
Please discuss.
Kind regards,
Lorenz
[1] https://issues.apache.org/jira/browse/QPID-7815
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
________________________________
This communication (including any attachments) may contain information that is proprietary, confidential or exempt from disclosure. If you are not the intended recipient, please note that further dissemination, distribution, use or copying of this communication is strictly prohibited. Anyone who received this message in error should notify the sender immediately by telephone or by return email and delete it from his or her computer.