You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Pavel Moravec <pm...@redhat.com> on 2012/02/03 12:51:53 UTC
maxprefecth / capacity concerns
Hello qpid users,
I am somehow confused about difference between capacity and maxprefetch. Both define maximum number of messages that can be sent without receiving a response to the first msg, but maxprefetch is applicable to consumer part of the communication only.
My questions:
1) "maxprefetch" is applicable rather to Java client only while capacity rather to C++, python and .NET clients.
2) How to setup capacity in Java client (to limit producers to qpid)? I tried using address string "small-queue;{link:{capacity:'5'}}" but without an effect. I spotted non-JMS method setProducerCapacity from org.apache.qpid.client.messaging.address package (within class Link) but I am unable to use it via Connection, Session or MessageProducer objects.
3) Why Programming in Apache Qpid talks about prefetch having impact to performance, while I can't find a method to setting it in C++ client?
4) The same doc says prefetch is by default 1, while https://cwiki.apache.org/qpid/use-priority-queues.html states it has default value 5000 - what is correct then?
5) Are there some reasons to have those two features separated? And with different default values (capacity has 50)?
Thanks in advance for your feedback.
Kind regards,
Pavel
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: maxprefecth / capacity concerns
Posted by Rajith Attapattu <ra...@gmail.com>.
Sorry I was late to the discussion.
I was sick the whole of last week and this week I had to deal with a
few issues and was late in catching up on email.
Both max-prefetch and capacity are only relevant on the consumer side
and they mean the same thing.
If capacity is specified in the address string it overrides the
max-prefetch set for the connection.
Currently max-prefetch and capacity works at the session level, not
really at the destination level.
All though we set message-credits to max-prefetch (or capacity), if we
have more than one consumer on the session then completions are sent
based on a metric calculated at the session level, rather at a
destination level. See QPID-3612 and QPID-3613
There is currently work to ensure we implement capacity properly on a
per destination basis for the consumer.
There is an outstanding JIRA for producer side capacity and also to
align it with the flow control work done on the C++ broker.
I believe max-prefetch was retained more for historical reasons.
Going forward we should probably just use capacity to avoid confusion.
Regards,
Rajith
On Wed, Feb 15, 2012 at 4:45 PM, Gordon Sim <gs...@redhat.com> wrote:
> On 02/15/2012 09:23 PM, Pavel Moravec wrote:
>>
>> While in Java client without ability to setup capacity, Java client
>> can send to the queue any further number of messages, simply ignoring
>> it hasn't received ACKs. Only when the client would try to close the
>> session or connection, it is blocked (in AUTO ACK mode) until ACKs
>> are received. Applying prefetch has no impact.
>>
>> So the only way (I found so far) is adding session.sync() after
>> sending a bunch of messages. What is not ideal not only from
>> performance point of view.
>>
>> I know there are more important issues with Java client, but is the
>> request of adding capacity to Java client reasonable?
>
>
> Yes, I think that would be a very reasonable request.
>
> There is a qpid.session.command_limit option that restricts the number of
> outstanding commands that can be held. That is similar to the capacity but
> (a) is session level rather then producer level and (b) needs to include
> other AMQP 0-10 commands, not just message transfers.
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: maxprefecth / capacity concerns
Posted by Gordon Sim <gs...@redhat.com>.
On 02/15/2012 09:23 PM, Pavel Moravec wrote:
> While in Java client without ability to setup capacity, Java client
> can send to the queue any further number of messages, simply ignoring
> it hasn't received ACKs. Only when the client would try to close the
> session or connection, it is blocked (in AUTO ACK mode) until ACKs
> are received. Applying prefetch has no impact.
>
> So the only way (I found so far) is adding session.sync() after
> sending a bunch of messages. What is not ideal not only from
> performance point of view.
>
> I know there are more important issues with Java client, but is the
> request of adding capacity to Java client reasonable?
Yes, I think that would be a very reasonable request.
There is a qpid.session.command_limit option that restricts the number
of outstanding commands that can be held. That is similar to the
capacity but (a) is session level rather then producer level and (b)
needs to include other AMQP 0-10 commands, not just message transfers.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: maxprefecth / capacity concerns
Posted by Pavel Moravec <pm...@redhat.com>.
Hi Gordon,
the "default prefetch is 1" is my conclusion from Section 2.6: "By default, a receiver requests the next message from the server in response to each fetch call, resulting in messages being sent to the receiver one at a time."
Anyway, that is rather an explanation of prefetch, whereas default values are later on provided (500 for Java client).
I agree that prefetch and capacity is something else (though having something common in consumer side).
What I am missing in Java client (and what in fact triggered writing of my original email) is ability to automatically apply/follow producer flow control in Java client. Using the plain programming skeleton from Spout.java (or spout.cpp), we are "just" sending messages with no session synchronization (like session.sync() ). In C++, when I setup capacity to e.g. 50, then:
- assume the broker applies producer flow control by reaching a queue its 80% capacity
- so it stops ACKing messages from the producer but further messages are still put to the queue
- so then C++ send method is blocked by C++ client library after sending 50 further messages
While in Java client without ability to setup capacity, Java client can send to the queue any further number of messages, simply ignoring it hasn't received ACKs. Only when the client would try to close the session or connection, it is blocked (in AUTO ACK mode) until ACKs are received. Applying prefetch has no impact.
So the only way (I found so far) is adding session.sync() after sending a bunch of messages. What is not ideal not only from performance point of view.
I know there are more important issues with Java client, but is the request of adding capacity to Java client reasonable?
Kind regards,
Pavel
----- Original Message -----
> From: "Gordon Sim" <gs...@redhat.com>
> To: users@qpid.apache.org
> Sent: Friday, February 10, 2012 8:47:33 PM
> Subject: Re: maxprefecth / capacity concerns
>
> On 02/03/2012 11:51 AM, Pavel Moravec wrote:
> > Hello qpid users,
> > I am somehow confused about difference between capacity and
> > maxprefetch. Both define maximum number of messages that can be
> > sent without receiving a response to the first msg, but
> > maxprefetch is applicable to consumer part of the communication
> > only.
> >
> > My questions:
> > 1) "maxprefetch" is applicable rather to Java client only while
> > capacity rather to C++, python and .NET clients.
>
> Correct.
>
> > 2) How to setup capacity in Java client (to limit producers to
> > qpid)? I tried using address string
> > "small-queue;{link:{capacity:'5'}}" but without an effect. I
> > spotted non-JMS method setProducerCapacity from
> > org.apache.qpid.client.messaging.address package (within class
> > Link) but I am unable to use it via Connection, Session or
> > MessageProducer objects.
>
> I don't believe you can at present. The closest it comes is that you
> can
> specify a limit to the command 'buffer' which is similiar but not
> quite
> the same thing.
> >
> > 3) Why Programming in Apache Qpid talks about prefetch having
> > impact to performance, while I can't find a method to setting it
> > in C++ client?
>
> The performance tips state: 'Consider prefetching messages for
> receivers
> (see Section 2.6, “Receiver Capacity (Prefetch)”)' where section 2.6
> is
> http://qpid.apache.org/books/trunk/Programming-In-Apache-Qpid/html/prefetch.html
> which - in my view - gives a reasonable description of how prefetch
> relates to receiver capacity. I'd be interested to hear if that is or
> is
> not clear.
>
> Receiver capacity and prefetch are essentially the same thing. The
> receivers capacity determines how many messages it is prepared to
> buffer
> pending fetch() calls.
>
> > 4) The same doc says prefetch is by default 1,
>
> Where is that? The default receiver capacity for python , c++ and
> .NET
> is actually 0 (i.e. no prefetch is enabled).
>
> > while https://cwiki.apache.org/qpid/use-priority-queues.html states
> > it has default value 5000 - what is correct then?
> >
> > 5) Are there some reasons to have those two features separated? And
> > with different default values (capacity has 50)?
>
> I don't think there are two features here, merely a difference in how
> the JMS client works and how the messaging API implementations work.
> Clearly we want to move towards more commonality but JMS is still
> quite
> a different beast.
>
> Hope this helps...
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: maxprefecth / capacity concerns
Posted by Gordon Sim <gs...@redhat.com>.
On 02/03/2012 11:51 AM, Pavel Moravec wrote:
> Hello qpid users,
> I am somehow confused about difference between capacity and maxprefetch. Both define maximum number of messages that can be sent without receiving a response to the first msg, but maxprefetch is applicable to consumer part of the communication only.
>
> My questions:
> 1) "maxprefetch" is applicable rather to Java client only while capacity rather to C++, python and .NET clients.
Correct.
> 2) How to setup capacity in Java client (to limit producers to qpid)? I tried using address string "small-queue;{link:{capacity:'5'}}" but without an effect. I spotted non-JMS method setProducerCapacity from org.apache.qpid.client.messaging.address package (within class Link) but I am unable to use it via Connection, Session or MessageProducer objects.
I don't believe you can at present. The closest it comes is that you can
specify a limit to the command 'buffer' which is similiar but not quite
the same thing.
>
> 3) Why Programming in Apache Qpid talks about prefetch having impact to performance, while I can't find a method to setting it in C++ client?
The performance tips state: 'Consider prefetching messages for receivers
(see Section 2.6, “Receiver Capacity (Prefetch)”)' where section 2.6 is
http://qpid.apache.org/books/trunk/Programming-In-Apache-Qpid/html/prefetch.html
which - in my view - gives a reasonable description of how prefetch
relates to receiver capacity. I'd be interested to hear if that is or is
not clear.
Receiver capacity and prefetch are essentially the same thing. The
receivers capacity determines how many messages it is prepared to buffer
pending fetch() calls.
> 4) The same doc says prefetch is by default 1,
Where is that? The default receiver capacity for python , c++ and .NET
is actually 0 (i.e. no prefetch is enabled).
> while https://cwiki.apache.org/qpid/use-priority-queues.html states it has default value 5000 - what is correct then?
>
> 5) Are there some reasons to have those two features separated? And with different default values (capacity has 50)?
I don't think there are two features here, merely a difference in how
the JMS client works and how the messaging API implementations work.
Clearly we want to move towards more commonality but JMS is still quite
a different beast.
Hope this helps...
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: maxprefecth / capacity concerns
Posted by Alan Conway <ac...@redhat.com>.
On Fri, 2012-02-10 at 08:37 +0000, Fraser Adams wrote:
> Hi Pavel,
> Sorry I can't be any more help. Given what I said previously based on my
> casual rummage through the source code my bet is that configuring the
> producer capacity simply isn't supported on the Java client run time. My
> comment below about
> "Hmmm scratch that.. I'm typing this as I'm looking, I've just tried
> "grep -rH getProducerCapacity ." and the only place where that seems to
> be referenced is in Link.java, so I'm now suspecting that the above sets
> the value, but nothing is using it yet" is my best bet on the subject.
>
> Unless a Java client runtime guru can step in and prove us wrong I'd say
> raise a Jira on it.
>
>
> I have to admit that I'm not really clear on how the capacity stuff
> works (on C++ or Java really). I'm fairly sure that the consumer
> capacity (AKA prefetch) maps to an internal queue in the client runtime
> (I think in the old qpid::client API there was a local queue explicitly
> involved in the API, though I'm not overly familiar with that API).
>
> I assume the consumer capacity works by allowing the client runtime to
> pull messages as fast as it can asynchronously to the client application
> in order to minimise round trip network delays that may be caused by a
> more chatty synchronous network mechanism.
>
> I'm not so sure on the producer capacity though.....
Consumer capacity limits prefetch. The client libraries are allowed to
fetch messages in advance of the application asking for them up to the
capacity limit.
Producer capacity limits "in doubt" messages. The client library is
allowed to have up to N messages sent but not yet acknowledged. If the
limit is reached the client library will block until the broker acks
some messages.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: maxprefecth / capacity concerns
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
Hi Pavel,
Sorry I can't be any more help. Given what I said previously based on my
casual rummage through the source code my bet is that configuring the
producer capacity simply isn't supported on the Java client run time. My
comment below about
"Hmmm scratch that.. I'm typing this as I'm looking, I've just tried
"grep -rH getProducerCapacity ." and the only place where that seems to
be referenced is in Link.java, so I'm now suspecting that the above sets
the value, but nothing is using it yet" is my best bet on the subject.
Unless a Java client runtime guru can step in and prove us wrong I'd say
raise a Jira on it.
I have to admit that I'm not really clear on how the capacity stuff
works (on C++ or Java really). I'm fairly sure that the consumer
capacity (AKA prefetch) maps to an internal queue in the client runtime
(I think in the old qpid::client API there was a local queue explicitly
involved in the API, though I'm not overly familiar with that API).
I assume the consumer capacity works by allowing the client runtime to
pull messages as fast as it can asynchronously to the client application
in order to minimise round trip network delays that may be caused by a
more chatty synchronous network mechanism.
I'm not so sure on the producer capacity though.....
Good luck with your digging around.
Cheers
Frase
On 07/02/12 08:26, Pavel Moravec wrote:
> Hi Frase,
> thanks for your comments and hints. I tried various attempts including capacity option in address string and also direct calling of setProducerCapacity but without an impact.
>
> I will debug it somehow more to pinpoint what is wrong and raise JIRA later on, as currently it seems there is no option to apply capacity limit (i.e. setCapacity from C++) in Java client.
>
> Kind regards,
> Pavel
>
>
> ----- Original Message -----
>> From: "Fraser Adams"<fr...@blueyonder.co.uk>
>> To: users@qpid.apache.org
>> Sent: Saturday, February 4, 2012 12:02:04 PM
>> Subject: Re: maxprefecth / capacity concerns
>>
>> Hi Pavel,
>> I *think* that they are the same thing.
>>
>> With C++ I've only used qpid::messaging and that *definitely* has a
>> default consumer capacity of 1 and I "think" a default producer
>> capacity
>> of 50, so if you write a qpid::messaging consumer by default its
>> performance won't be great hence it's almost always a good idea to do
>> a
>> setCapacity() on the consumer session to something sensible.
>>
>> In Java I *think* the default is 500, but I've never had reason to
>> mess
>> around with it. I only picked up on this stuff when I was ahem
>> "surprised" that my Java consumer was faster than my C++ consumer....
>>
>> Re your question 2. according to
>> http://qpid.apache.org/books/0.12/Programming-In-Apache-Qpid/html/ch03s06.html
>> there are JVM arguments too e.g.
>>
>>
>> max_prefetch int 500 Maximum number of messages to credits.
>>
>> This could also be set per connection (see connection paramters) or
>> per
>> destination (see the |capacity| option under link properties in
>> addressing).
>>
>>
>> Though that looks like receiver capacity and it sounds like you're
>> talking about sender capacity?
>>
>> Ex. |-Dmax_prefetch=1000| property specifies the message credits to
>> use.
>>
>> There's a related Connection URL Property
>>
>> |amqp://guest:guest@test/test?max_prefetch='1000'
>> &brokerlist='tcp://localhost:5672'|
>>
>> Again it's called prefetch so I suspect this relates to the consumer
>> capacity in C++ qpid::messaging.
>>
>> also as you've commented there's
>>
>> |my-queue; {create: always, link:{capacity: 10}}
>> |
>> Again this would seem to relate to consumer capacity.
>>
>> Unfortunately I can't see anything obvious in the documentation that
>> looks like it maps to producer capacity.
>>
>>
>> I used your comment on "setProducerCapacity" as a "seed" for a few
>> greps, this is called from AddressHelper in the same package
>>
>>
>> if (((Map) address.getOptions().get(LINK)).get(CAPACITY)
>> instanceof Map)
>> {
>> MapAccessor capacityProps = new MapAccessor(
>> (Map) ((Map) address.getOptions().get(LINK))
>> .get(CAPACITY));
>> link
>> .setConsumerCapacity(capacityProps
>> .getInt(CAPACITY_SOURCE) == null ? 0
>> :
>> capacityProps.getInt(CAPACITY_SOURCE));
>> link
>> .setProducerCapacity(capacityProps
>> .getInt(CAPACITY_TARGET) == null ? 0
>> :
>> capacityProps.getInt(CAPACITY_TARGET));
>> }
>> else
>> {
>> int cap = linkProps.getInt(CAPACITY) == null ? 0 :
>> linkProps
>> .getInt(CAPACITY);
>> link.setConsumerCapacity(cap);
>> link.setProducerCapacity(cap);
>> }
>>
>> In the first block above it's talking about "CAPACITY_SOURCE" and
>> "CAPACITY_TARGET" in the code these are strings "source" and "target"
>> respectively. The first block also seems to be looking for a Map
>> value
>> for the capacity property of link so I wonder......how about trying
>>
>> |my-queue; {create: always, link:{capacity: {source: 50, target:
>> |200}}}|
>>
>> I haven't tried this, and I've just got there by looking at the code,
>> but it looks plausible. I've not come across any documentation that
>> covers this either, but it *might* just help.
>>
>>
>> Hmmm scratch that.. I'm typing this as I'm looking, I've just tried
>> "grep -rH getProducerCapacity ." and the only place where that seems
>> to
>> be referenced is in Link.java, so I'm now suspecting that the above
>> sets
>> the value, but nothing is using it yet....
>>
>>
>> A few more greps seem to suggest that max_prefetch relates to a
>> session
>> but link capacity relates to a destination, but it looks like they do
>> similar things (I think they buffer messages into an internal queue
>> on
>> the client runtime) but so far I've not found anything obvious about
>> producer capacity aside from the red herrings above, so I'm
>> suspecting
>> that there's no equivalent to qpid::messaging producer setCapacity in
>> JMS.
>>
>>
>> Sorry for rambling a bit, I hope that the train of thought is useful
>> and
>> perhaps someone who knows the code a bit better can confirm what I'm
>> seeing.
>>
>> As an aside, the code base could do with a few more comments :-)
>>
>> Let us know how you get on
>> Frase.
>>
>>
>> On 03/02/12 11:51, Pavel Moravec wrote:
>>> Hello qpid users,
>>> I am somehow confused about difference between capacity and
>>> maxprefetch. Both define maximum number of messages that can be
>>> sent without receiving a response to the first msg, but
>>> maxprefetch is applicable to consumer part of the communication
>>> only.
>>>
>>> My questions:
>>> 1) "maxprefetch" is applicable rather to Java client only while
>>> capacity rather to C++, python and .NET clients.
>>>
>>> 2) How to setup capacity in Java client (to limit producers to
>>> qpid)? I tried using address string
>>> "small-queue;{link:{capacity:'5'}}" but without an effect. I
>>> spotted non-JMS method setProducerCapacity from
>>> org.apache.qpid.client.messaging.address package (within class
>>> Link) but I am unable to use it via Connection, Session or
>>> MessageProducer objects.
>>>
>>> 3) Why Programming in Apache Qpid talks about prefetch having
>>> impact to performance, while I can't find a method to setting it
>>> in C++ client?
>>>
>>> 4) The same doc says prefetch is by default 1, while
>>> https://cwiki.apache.org/qpid/use-priority-queues.html states it
>>> has default value 5000 - what is correct then?
>>>
>>> 5) Are there some reasons to have those two features separated? And
>>> with different default values (capacity has 50)?
>>>
>>>
>>> Thanks in advance for your feedback.
>>>
>>> Kind regards,
>>> Pavel
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> Apache Qpid - AMQP Messaging Implementation
>>> Project: http://qpid.apache.org
>>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>>
>>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: maxprefecth / capacity concerns
Posted by Pavel Moravec <pm...@redhat.com>.
Hi Frase,
thanks for your comments and hints. I tried various attempts including capacity option in address string and also direct calling of setProducerCapacity but without an impact.
I will debug it somehow more to pinpoint what is wrong and raise JIRA later on, as currently it seems there is no option to apply capacity limit (i.e. setCapacity from C++) in Java client.
Kind regards,
Pavel
----- Original Message -----
> From: "Fraser Adams" <fr...@blueyonder.co.uk>
> To: users@qpid.apache.org
> Sent: Saturday, February 4, 2012 12:02:04 PM
> Subject: Re: maxprefecth / capacity concerns
>
> Hi Pavel,
> I *think* that they are the same thing.
>
> With C++ I've only used qpid::messaging and that *definitely* has a
> default consumer capacity of 1 and I "think" a default producer
> capacity
> of 50, so if you write a qpid::messaging consumer by default its
> performance won't be great hence it's almost always a good idea to do
> a
> setCapacity() on the consumer session to something sensible.
>
> In Java I *think* the default is 500, but I've never had reason to
> mess
> around with it. I only picked up on this stuff when I was ahem
> "surprised" that my Java consumer was faster than my C++ consumer....
>
> Re your question 2. according to
> http://qpid.apache.org/books/0.12/Programming-In-Apache-Qpid/html/ch03s06.html
> there are JVM arguments too e.g.
>
>
> max_prefetch int 500 Maximum number of messages to credits.
>
> This could also be set per connection (see connection paramters) or
> per
> destination (see the |capacity| option under link properties in
> addressing).
>
>
> Though that looks like receiver capacity and it sounds like you're
> talking about sender capacity?
>
> Ex. |-Dmax_prefetch=1000| property specifies the message credits to
> use.
>
> There's a related Connection URL Property
>
> |amqp://guest:guest@test/test?max_prefetch='1000'
> &brokerlist='tcp://localhost:5672'|
>
> Again it's called prefetch so I suspect this relates to the consumer
> capacity in C++ qpid::messaging.
>
> also as you've commented there's
>
> |my-queue; {create: always, link:{capacity: 10}}
> |
> Again this would seem to relate to consumer capacity.
>
> Unfortunately I can't see anything obvious in the documentation that
> looks like it maps to producer capacity.
>
>
> I used your comment on "setProducerCapacity" as a "seed" for a few
> greps, this is called from AddressHelper in the same package
>
>
> if (((Map) address.getOptions().get(LINK)).get(CAPACITY)
> instanceof Map)
> {
> MapAccessor capacityProps = new MapAccessor(
> (Map) ((Map) address.getOptions().get(LINK))
> .get(CAPACITY));
> link
> .setConsumerCapacity(capacityProps
> .getInt(CAPACITY_SOURCE) == null ? 0
> :
> capacityProps.getInt(CAPACITY_SOURCE));
> link
> .setProducerCapacity(capacityProps
> .getInt(CAPACITY_TARGET) == null ? 0
> :
> capacityProps.getInt(CAPACITY_TARGET));
> }
> else
> {
> int cap = linkProps.getInt(CAPACITY) == null ? 0 :
> linkProps
> .getInt(CAPACITY);
> link.setConsumerCapacity(cap);
> link.setProducerCapacity(cap);
> }
>
> In the first block above it's talking about "CAPACITY_SOURCE" and
> "CAPACITY_TARGET" in the code these are strings "source" and "target"
> respectively. The first block also seems to be looking for a Map
> value
> for the capacity property of link so I wonder......how about trying
>
> |my-queue; {create: always, link:{capacity: {source: 50, target:
> |200}}}|
>
> I haven't tried this, and I've just got there by looking at the code,
> but it looks plausible. I've not come across any documentation that
> covers this either, but it *might* just help.
>
>
> Hmmm scratch that.. I'm typing this as I'm looking, I've just tried
> "grep -rH getProducerCapacity ." and the only place where that seems
> to
> be referenced is in Link.java, so I'm now suspecting that the above
> sets
> the value, but nothing is using it yet....
>
>
> A few more greps seem to suggest that max_prefetch relates to a
> session
> but link capacity relates to a destination, but it looks like they do
> similar things (I think they buffer messages into an internal queue
> on
> the client runtime) but so far I've not found anything obvious about
> producer capacity aside from the red herrings above, so I'm
> suspecting
> that there's no equivalent to qpid::messaging producer setCapacity in
> JMS.
>
>
> Sorry for rambling a bit, I hope that the train of thought is useful
> and
> perhaps someone who knows the code a bit better can confirm what I'm
> seeing.
>
> As an aside, the code base could do with a few more comments :-)
>
> Let us know how you get on
> Frase.
>
>
> On 03/02/12 11:51, Pavel Moravec wrote:
> > Hello qpid users,
> > I am somehow confused about difference between capacity and
> > maxprefetch. Both define maximum number of messages that can be
> > sent without receiving a response to the first msg, but
> > maxprefetch is applicable to consumer part of the communication
> > only.
> >
> > My questions:
> > 1) "maxprefetch" is applicable rather to Java client only while
> > capacity rather to C++, python and .NET clients.
> >
> > 2) How to setup capacity in Java client (to limit producers to
> > qpid)? I tried using address string
> > "small-queue;{link:{capacity:'5'}}" but without an effect. I
> > spotted non-JMS method setProducerCapacity from
> > org.apache.qpid.client.messaging.address package (within class
> > Link) but I am unable to use it via Connection, Session or
> > MessageProducer objects.
> >
> > 3) Why Programming in Apache Qpid talks about prefetch having
> > impact to performance, while I can't find a method to setting it
> > in C++ client?
> >
> > 4) The same doc says prefetch is by default 1, while
> > https://cwiki.apache.org/qpid/use-priority-queues.html states it
> > has default value 5000 - what is correct then?
> >
> > 5) Are there some reasons to have those two features separated? And
> > with different default values (capacity has 50)?
> >
> >
> > Thanks in advance for your feedback.
> >
> > Kind regards,
> > Pavel
> >
> >
> >
> > ---------------------------------------------------------------------
> > Apache Qpid - AMQP Messaging Implementation
> > Project: http://qpid.apache.org
> > Use/Interact: mailto:users-subscribe@qpid.apache.org
> >
>
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: maxprefecth / capacity concerns
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
Hi Pavel,
I *think* that they are the same thing.
With C++ I've only used qpid::messaging and that *definitely* has a
default consumer capacity of 1 and I "think" a default producer capacity
of 50, so if you write a qpid::messaging consumer by default its
performance won't be great hence it's almost always a good idea to do a
setCapacity() on the consumer session to something sensible.
In Java I *think* the default is 500, but I've never had reason to mess
around with it. I only picked up on this stuff when I was ahem
"surprised" that my Java consumer was faster than my C++ consumer....
Re your question 2. according to
http://qpid.apache.org/books/0.12/Programming-In-Apache-Qpid/html/ch03s06.html
there are JVM arguments too e.g.
max_prefetch int 500 Maximum number of messages to credits.
This could also be set per connection (see connection paramters) or per
destination (see the |capacity| option under link properties in addressing).
Though that looks like receiver capacity and it sounds like you're
talking about sender capacity?
Ex. |-Dmax_prefetch=1000| property specifies the message credits to use.
There's a related Connection URL Property
|amqp://guest:guest@test/test?max_prefetch='1000'
&brokerlist='tcp://localhost:5672'|
Again it's called prefetch so I suspect this relates to the consumer
capacity in C++ qpid::messaging.
also as you've commented there's
|my-queue; {create: always, link:{capacity: 10}}
|
Again this would seem to relate to consumer capacity.
Unfortunately I can't see anything obvious in the documentation that
looks like it maps to producer capacity.
I used your comment on "setProducerCapacity" as a "seed" for a few
greps, this is called from AddressHelper in the same package
if (((Map) address.getOptions().get(LINK)).get(CAPACITY)
instanceof Map)
{
MapAccessor capacityProps = new MapAccessor(
(Map) ((Map) address.getOptions().get(LINK))
.get(CAPACITY));
link
.setConsumerCapacity(capacityProps
.getInt(CAPACITY_SOURCE) == null ? 0
: capacityProps.getInt(CAPACITY_SOURCE));
link
.setProducerCapacity(capacityProps
.getInt(CAPACITY_TARGET) == null ? 0
: capacityProps.getInt(CAPACITY_TARGET));
}
else
{
int cap = linkProps.getInt(CAPACITY) == null ? 0 :
linkProps
.getInt(CAPACITY);
link.setConsumerCapacity(cap);
link.setProducerCapacity(cap);
}
In the first block above it's talking about "CAPACITY_SOURCE" and
"CAPACITY_TARGET" in the code these are strings "source" and "target"
respectively. The first block also seems to be looking for a Map value
for the capacity property of link so I wonder......how about trying
|my-queue; {create: always, link:{capacity: {source: 50, target: 200}}}|
I haven't tried this, and I've just got there by looking at the code,
but it looks plausible. I've not come across any documentation that
covers this either, but it *might* just help.
Hmmm scratch that.. I'm typing this as I'm looking, I've just tried
"grep -rH getProducerCapacity ." and the only place where that seems to
be referenced is in Link.java, so I'm now suspecting that the above sets
the value, but nothing is using it yet....
A few more greps seem to suggest that max_prefetch relates to a session
but link capacity relates to a destination, but it looks like they do
similar things (I think they buffer messages into an internal queue on
the client runtime) but so far I've not found anything obvious about
producer capacity aside from the red herrings above, so I'm suspecting
that there's no equivalent to qpid::messaging producer setCapacity in JMS.
Sorry for rambling a bit, I hope that the train of thought is useful and
perhaps someone who knows the code a bit better can confirm what I'm seeing.
As an aside, the code base could do with a few more comments :-)
Let us know how you get on
Frase.
On 03/02/12 11:51, Pavel Moravec wrote:
> Hello qpid users,
> I am somehow confused about difference between capacity and maxprefetch. Both define maximum number of messages that can be sent without receiving a response to the first msg, but maxprefetch is applicable to consumer part of the communication only.
>
> My questions:
> 1) "maxprefetch" is applicable rather to Java client only while capacity rather to C++, python and .NET clients.
>
> 2) How to setup capacity in Java client (to limit producers to qpid)? I tried using address string "small-queue;{link:{capacity:'5'}}" but without an effect. I spotted non-JMS method setProducerCapacity from org.apache.qpid.client.messaging.address package (within class Link) but I am unable to use it via Connection, Session or MessageProducer objects.
>
> 3) Why Programming in Apache Qpid talks about prefetch having impact to performance, while I can't find a method to setting it in C++ client?
>
> 4) The same doc says prefetch is by default 1, while https://cwiki.apache.org/qpid/use-priority-queues.html states it has default value 5000 - what is correct then?
>
> 5) Are there some reasons to have those two features separated? And with different default values (capacity has 50)?
>
>
> Thanks in advance for your feedback.
>
> Kind regards,
> Pavel
>
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>