You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by Alan Conway <ac...@redhat.com> on 2009/12/16 15:56:10 UTC
Sender flow control via resource-limit-exceeded
I'm looking at some code that does sender flow control as follows:
- set a queue size limit
- sender catches 'resource-limit-exceeded' exception, sleeps for a bit,
creates a new session & re-tries
Is there a better way to flow control senders to bound broker resources? I can't
think of one as the broker & protocol currently stand.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: Sender flow control via resource-limit-exceeded
Posted by Alan Conway <ac...@redhat.com>.
On 12/17/2009 08:13 AM, Rafael Schloming wrote:
> Andrew Stitcher wrote:
>> On Wed, 2009-12-16 at 09:56 -0500, Alan Conway wrote:
>>> I'm looking at some code that does sender flow control as follows:
>>> - set a queue size limit
>>> - sender catches 'resource-limit-exceeded' exception, sleeps for a
>>> bit, creates a new session & re-tries
>>
>> The essential problem with this approach is that the queue size is not
>> directly caused by any single producer. So it's entirely possible in
>> general for a producer that is producing at a low rate to be entirely
>> killed by another producer sending messages into the same queue.
>>
>> The only way that this overall scheme can work fairly is if there is a
>> 1-1 producer-queue mapping. If that is the case then it should work.
>
> Why wouldn't we use flow control as specified by the protocol? It
> doesn't help you figure out *who* to flow control, but once you've made
> that decision, it provides a much better way to actually flow control an
> individual producer.
>
Agreed, I'm talking about what a user can do with the code as it stands today.
We definitely need to implement some producer flow control.
I'm still not sure how this would work. As you point out its tricky to give any
exact guarantees since you have no idea which queues a client may publish to (at
least in 0.10) You can bound some notion of "total messages in the system", but
you need to figure out
a) what should that bound be? It probably varies with number of queues,
producers/consumers and queue limits but I don't know what the relationship
would be.
b) how to divide up the credit so fast producers with slow consumers can't hog
the show.
I don't think it's possible with the protocol flow control to enforce a depth
limit on individual queues which is unfortunate. So retry after
resource-limit-exceeded is still probably necessary for a robust producer.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: Sender flow control via resource-limit-exceeded
Posted by Rafael Schloming <ra...@redhat.com>.
Andrew Stitcher wrote:
> On Wed, 2009-12-16 at 09:56 -0500, Alan Conway wrote:
>> I'm looking at some code that does sender flow control as follows:
>> - set a queue size limit
>> - sender catches 'resource-limit-exceeded' exception, sleeps for a bit,
>> creates a new session & re-tries
>
> The essential problem with this approach is that the queue size is not
> directly caused by any single producer. So it's entirely possible in
> general for a producer that is producing at a low rate to be entirely
> killed by another producer sending messages into the same queue.
>
> The only way that this overall scheme can work fairly is if there is a
> 1-1 producer-queue mapping. If that is the case then it should work.
Why wouldn't we use flow control as specified by the protocol? It
doesn't help you figure out *who* to flow control, but once you've made
that decision, it provides a much better way to actually flow control an
individual producer.
--Rafael
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: Sender flow control via resource-limit-exceeded
Posted by Andrew Stitcher <as...@redhat.com>.
On Wed, 2009-12-16 at 09:56 -0500, Alan Conway wrote:
> I'm looking at some code that does sender flow control as follows:
> - set a queue size limit
> - sender catches 'resource-limit-exceeded' exception, sleeps for a bit,
> creates a new session & re-tries
The essential problem with this approach is that the queue size is not
directly caused by any single producer. So it's entirely possible in
general for a producer that is producing at a low rate to be entirely
killed by another producer sending messages into the same queue.
The only way that this overall scheme can work fairly is if there is a
1-1 producer-queue mapping. If that is the case then it should work.
Andrew
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org