You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by Robert Greig <ro...@gmail.com> on 2008/11/10 23:22:02 UTC
Slow consumer handling
Hi all,
Following a thread on qpid-user, I have created
https://issues.apache.org/jira/browse/QPID-1447 to list the two issues
identified. This is specifically for the Java broker; I think the C++
broker is better but still needs some enhancement to support
disconnecting slow consumers (typically on private queues) but I'll
let someone more familiar with that broker raise any Jiras for that.
The MINA issue is analogous to the "fast producer" issue we have known
about for a long time. Martin, I thought this had been fixed ages ago
but I can't find the Jira. Do you know the status of that issue?
Thanks,
Robert
Re: Slow consumer handling
Posted by Aidan Skinner <ai...@apache.org>.
On Tue, Nov 11, 2008 at 10:59 PM, Robert Godfrey
<ro...@gmail.com> wrote:
> 2008/11/11 Aidan Skinner <ai...@apache.org>:
>> On Tue, Nov 11, 2008 at 10:44 PM, Carl Trieloff <cc...@redhat.com> wrote:
>>
>>> I think MINA is being tossed as we work to 0-10 in Java. Agree that the slow
>>> connection
>>> problem on private queues is un-related.
>>
>> This really can't wait to be fixed until we add new protocols to the
>> Java broker. 0-10 makes things a lot easier, in many ways, but the
>> basic problems remain the same and need to be fixed in similar ways.
>>
>
> Once we have M4 out of the door I would like to discuss the work I've
> done for both the broker and 0-8 client allowing them to select
> alternative IO layers. For continuity stability it can use the
> existing MINA IO, but to address some of the issues being brought up
> here it can also be configured to use the Java 0-10 IO, or a similar
> NIO/selector based transport I wrote. While this won't answer all
> our issues it does offer one way of getting out from under some of the
> issues we've seen with MINA... and experimentally showed much improved
> performance for the Java Broker.
The transport layer is only part of the picture, although it's a big
part. Having pluggable ones would be awesome. Fancy attaching the
patch to the Jira? :)
- Aidan
--
Apache Qpid - World Domination through Advanced Message Queueing
http://cwiki.apache.org/qpid
"Nine-tenths of wisdom consists in being wise in time." - Theodore Roosevelt
Re: Slow consumer handling
Posted by Carl Trieloff <cc...@redhat.com>.
>
> My aim would be to get rid of MINA eventually... However there still
> needs to be some work to free the broker of dependence on Mina
> ByteBuffers etc (the I/O layer I've written just wraps the java.nio
> buffers in Mina buffers before passing them to the rest of the
> broker). I see that as a separate, but desirable, change.
>
>
As we are discussion, thought it would be good to throw what I have been
looking to
add on C++ broker into the mix. I want to create a policy that can be
set per connection
where you can specify a time derivative for messages, either in
messages/sec or bytes/sec
in a policy. This policy then would 'rate limit' a connection by only
issuing credits to that
limit for message transfer per rate interval. This would protect from a
producer that has
gone mad, by limiting the max bytes/messages any connection can pump
into the broker.
it would also be possible to connect this to the queue depth policies we
already have, but the
linkage from queue to connection is not always clear.
Carl.
Re: Slow consumer handling
Posted by Robert Godfrey <ro...@gmail.com>.
2008/11/11 Robert Greig <ro...@gmail.com>:
> 2008/11/11 Robert Godfrey <ro...@gmail.com>:
>
>> Once we have M4 out of the door I would like to discuss the work I've
>> done for both the broker and 0-8 client allowing them to select
>> alternative IO layers.
>
> Having a pluggable IO layer would definitely be a great step forward.
> It may give us an option in future to offer different I/O layers tuned
> for, say, maximum throughput or predictable latency (these are just
> examples, I'm not saying these are tradeoffs people would actually
> need to make).
Yes - that was the general idea. I've not actually tried to implement
any other I/O layers yet, but the interface is such that it should be
possible.
>
>> For continuity stability it can use the
>> existing MINA IO, but to address some of the issues being brought up
>> here it can also be configured to use the Java 0-10 IO, or a similar
>> NIO/selector based transport I wrote. While this won't answer all
>> our issues it does offer one way of getting out from under some of the
>> issues we've seen with MINA... and experimentally showed much improved
>> performance for the Java Broker.
>
> With some major refactoring being done anyway, does keeping MINA make
> sense? Or is it more a case of known problems versus unknown problems?
My aim would be to get rid of MINA eventually... However there still
needs to be some work to free the broker of dependence on Mina
ByteBuffers etc (the I/O layer I've written just wraps the java.nio
buffers in Mina buffers before passing them to the rest of the
broker). I see that as a separate, but desirable, change.
-- Rob
Re: Slow consumer handling
Posted by Robert Greig <ro...@gmail.com>.
2008/11/11 Robert Godfrey <ro...@gmail.com>:
> Once we have M4 out of the door I would like to discuss the work I've
> done for both the broker and 0-8 client allowing them to select
> alternative IO layers.
Having a pluggable IO layer would definitely be a great step forward.
It may give us an option in future to offer different I/O layers tuned
for, say, maximum throughput or predictable latency (these are just
examples, I'm not saying these are tradeoffs people would actually
need to make).
> For continuity stability it can use the
> existing MINA IO, but to address some of the issues being brought up
> here it can also be configured to use the Java 0-10 IO, or a similar
> NIO/selector based transport I wrote. While this won't answer all
> our issues it does offer one way of getting out from under some of the
> issues we've seen with MINA... and experimentally showed much improved
> performance for the Java Broker.
With some major refactoring being done anyway, does keeping MINA make
sense? Or is it more a case of known problems versus unknown problems?
RG
Re: Slow consumer handling
Posted by Robert Godfrey <ro...@gmail.com>.
2008/11/11 Aidan Skinner <ai...@apache.org>:
> On Tue, Nov 11, 2008 at 10:44 PM, Carl Trieloff <cc...@redhat.com> wrote:
>
>> I think MINA is being tossed as we work to 0-10 in Java. Agree that the slow
>> connection
>> problem on private queues is un-related.
>
> This really can't wait to be fixed until we add new protocols to the
> Java broker. 0-10 makes things a lot easier, in many ways, but the
> basic problems remain the same and need to be fixed in similar ways.
>
Once we have M4 out of the door I would like to discuss the work I've
done for both the broker and 0-8 client allowing them to select
alternative IO layers. For continuity stability it can use the
existing MINA IO, but to address some of the issues being brought up
here it can also be configured to use the Java 0-10 IO, or a similar
NIO/selector based transport I wrote. While this won't answer all
our issues it does offer one way of getting out from under some of the
issues we've seen with MINA... and experimentally showed much improved
performance for the Java Broker.
-- Rob
Re: Slow consumer handling
Posted by Carl Trieloff <cc...@redhat.com>.
Aidan Skinner wrote:
> On Tue, Nov 11, 2008 at 10:44 PM, Carl Trieloff <cc...@redhat.com> wrote:
>
>
>> I think MINA is being tossed as we work to 0-10 in Java. Agree that the slow
>> connection
>> problem on private queues is un-related.
>>
>
> This really can't wait to be fixed until we add new protocols to the
> Java broker. 0-10 makes things a lot easier, in many ways, but the
> basic problems remain the same and need to be fixed in similar ways.
>
>
I am not saying wait -- just wanted to ask the question
Carl.
Re: Slow consumer handling
Posted by Aidan Skinner <ai...@apache.org>.
On Tue, Nov 11, 2008 at 10:44 PM, Carl Trieloff <cc...@redhat.com> wrote:
> I think MINA is being tossed as we work to 0-10 in Java. Agree that the slow
> connection
> problem on private queues is un-related.
This really can't wait to be fixed until we add new protocols to the
Java broker. 0-10 makes things a lot easier, in many ways, but the
basic problems remain the same and need to be fixed in similar ways.
- Aidan
--
Apache Qpid - World Domination through Advanced Message Queueing
http://cwiki.apache.org/qpid
"Nine-tenths of wisdom consists in being wise in time." - Theodore Roosevelt
Re: Slow consumer handling
Posted by Carl Trieloff <cc...@redhat.com>.
Robert Greig wrote:
> 2008/11/11 Carl Trieloff <cc...@redhat.com>:
>
>
>> I have to think it would be a lot easier to do this with credit on 0-10. I
>> assume the extra
>> pain trying to do it on 0-9 is merited by it been needed for a deployment.
>> If not this would
>> be a lot easier to do using 0-10 - I have been looking at extending our flow
>> control capabilities
>> on the C++ broker, and it is a lot easier given you can revoke credit, so
>> you don't have to try
>> push back on threads which get ugly fast. In 0-10 is comes down to
>> determining policies on
>> which to do it (harder part) and then revoking or limiting credit based on
>> the policy.
>>
>
> I think that is certainly true for some of the issues but not for all
> of them, like the problem with MINA event queues, and dealing with
> slow consumers of private queues by closing the connection.
>
> RG
>
I think MINA is being tossed as we work to 0-10 in Java. Agree that the
slow connection
problem on private queues is un-related.
Carl.
Re: Slow consumer handling
Posted by Robert Greig <ro...@gmail.com>.
2008/11/11 Carl Trieloff <cc...@redhat.com>:
> I have to think it would be a lot easier to do this with credit on 0-10. I
> assume the extra
> pain trying to do it on 0-9 is merited by it been needed for a deployment.
> If not this would
> be a lot easier to do using 0-10 - I have been looking at extending our flow
> control capabilities
> on the C++ broker, and it is a lot easier given you can revoke credit, so
> you don't have to try
> push back on threads which get ugly fast. In 0-10 is comes down to
> determining policies on
> which to do it (harder part) and then revoking or limiting credit based on
> the policy.
I think that is certainly true for some of the issues but not for all
of them, like the problem with MINA event queues, and dealing with
slow consumers of private queues by closing the connection.
RG
Re: Slow consumer handling
Posted by Carl Trieloff <cc...@redhat.com>.
Aidan Skinner wrote:
> On Tue, Nov 11, 2008 at 8:24 PM, Robert Greig <ro...@gmail.com> wrote:
>
>
>> 2008/11/10 Aidan Skinner <ai...@apache.org>:
>>
>>
>>>> The MINA issue is analogous to the "fast producer" issue we have known
>>>> about for a long time. Martin, I thought this had been fixed ages ago
>>>> but I can't find the Jira. Do you know the status of that issue?
>>>>
>>> This is basically what I want to address with the changes discussed at
>>> http://cwiki.apache.org/confluence/display/qpid/Broker+job+queue+limits
>>>
>> Ah, ok I hadn't been aware of this. Was there an existing Jira for
>> this that I have now duplicated?
>>
>
> I think there are several that address different aspects of the
> fundamental problem. You can run across it in different ways, one bit
> feels like a rope, one bit like a tree etc. Jira's a bit of a state
> right now though I can't find any of them, so I wouldn't worry about
> it. ;)
>
>
>>> It won't make M4 sadly.
>>>
>> Yes, it's not a two minute fix. Possibly merits a release when it is
>> done since I imagine that a number of users will want this, and we
>> will look bad if people are doing bake offs.
>>
>
> Yeah, I'm hoping to put together a more concrete plan of attack in the
> near future for this and maybe get it out this year. We'll have to see
> how big and invasive a change it is though, I have a feeling it's
> going to slippery.
I have to think it would be a lot easier to do this with credit on 0-10.
I assume the extra
pain trying to do it on 0-9 is merited by it been needed for a
deployment. If not this would
be a lot easier to do using 0-10 - I have been looking at extending our
flow control capabilities
on the C++ broker, and it is a lot easier given you can revoke credit,
so you don't have to try
push back on threads which get ugly fast. In 0-10 is comes down to
determining policies on
which to do it (harder part) and then revoking or limiting credit based
on the policy.
Carl.
Re: Slow consumer handling
Posted by Aidan Skinner <ai...@apache.org>.
On Tue, Nov 11, 2008 at 8:24 PM, Robert Greig <ro...@gmail.com> wrote:
> 2008/11/10 Aidan Skinner <ai...@apache.org>:
>
>>> The MINA issue is analogous to the "fast producer" issue we have known
>>> about for a long time. Martin, I thought this had been fixed ages ago
>>> but I can't find the Jira. Do you know the status of that issue?
>>
>> This is basically what I want to address with the changes discussed at
>> http://cwiki.apache.org/confluence/display/qpid/Broker+job+queue+limits
>
> Ah, ok I hadn't been aware of this. Was there an existing Jira for
> this that I have now duplicated?
I think there are several that address different aspects of the
fundamental problem. You can run across it in different ways, one bit
feels like a rope, one bit like a tree etc. Jira's a bit of a state
right now though I can't find any of them, so I wouldn't worry about
it. ;)
>> It won't make M4 sadly.
>
> Yes, it's not a two minute fix. Possibly merits a release when it is
> done since I imagine that a number of users will want this, and we
> will look bad if people are doing bake offs.
Yeah, I'm hoping to put together a more concrete plan of attack in the
near future for this and maybe get it out this year. We'll have to see
how big and invasive a change it is though, I have a feeling it's
going to slippery.
- Aidan
--
Apache Qpid - World Domination through Advanced Message Queueing
http://cwiki.apache.org/qpid
"Nine-tenths of wisdom consists in being wise in time." - Theodore Roosevelt
Re: Slow consumer handling
Posted by Robert Greig <ro...@gmail.com>.
2008/11/10 Aidan Skinner <ai...@apache.org>:
>> The MINA issue is analogous to the "fast producer" issue we have known
>> about for a long time. Martin, I thought this had been fixed ages ago
>> but I can't find the Jira. Do you know the status of that issue?
>
> This is basically what I want to address with the changes discussed at
> http://cwiki.apache.org/confluence/display/qpid/Broker+job+queue+limits
Ah, ok I hadn't been aware of this. Was there an existing Jira for
this that I have now duplicated?
> It won't make M4 sadly.
Yes, it's not a two minute fix. Possibly merits a release when it is
done since I imagine that a number of users will want this, and we
will look bad if people are doing bake offs.
RG
Re: Slow consumer handling
Posted by Aidan Skinner <ai...@apache.org>.
On Mon, Nov 10, 2008 at 10:22 PM, Robert Greig <ro...@gmail.com> wrote:
> The MINA issue is analogous to the "fast producer" issue we have known
> about for a long time. Martin, I thought this had been fixed ages ago
> but I can't find the Jira. Do you know the status of that issue?
This is basically what I want to address with the changes discussed at
http://cwiki.apache.org/confluence/display/qpid/Broker+job+queue+limits
It won't make M4 sadly.
- Aidan
--
Apache Qpid - World Domination through Advanced Message Queueing
http://cwiki.apache.org/qpid
"Nine-tenths of wisdom consists in being wise in time." - Theodore Roosevelt