You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Zhemzhitsky Sergey <Se...@troika.ru> on 2012/09/19 10:06:36 UTC
How does the prefetch/receiver capacity really work?
Hi guys,
I have few questions regarding the receiver capacity.
Consider, there is a broker with a queue that contains 10 messages and receiver with receiver capacity set to 5.
As I understand the broker will push 5 messages to the receiver to fill in its internal queue and internal queue of receiver will contain 5 messages and broker will contain 5 unpushed messages too.
Then I will call Receiver::fetch(Message&,bool) without acknowledging the message.
Does it mean that receiver’s internal queue now has a free slot, so the broker can push another message to it, so that the receiver can be considered to receive 6 messages in total and broker can be considered to have 4 unpushed messages?
Best Regards,
Sergey Zhemzhitsky
_______________________________________________________
The information contained in this message may be privileged and conf idential and protected from disclosure. If you are not the original intended recipient, you are hereby notified that any review, retransmission, dissemination, or other use of, or taking of any action in reliance upon, this information is prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and delete it from your computer. Thank you for your cooperation. Troika Dialog, Russia.
If you need assistance please contact our Contact Center (+7495) 258 0500 or go to www.troika.ru/eng/Contacts/system.wbp
RE: How does the prefetch/receiver capacity really work?
Posted by Zhemzhitsky Sergey <Se...@troika.ru>.
Hello Gordon,
Sorry for the late answer.
>> How soon would you need this?
It's a nice to have feature, but is not critical for the moment.
>> It seems like we could also perhaps allow that set to be bounded. and e.g. have the fetch() call throw a particular exception if a call is made that would exceed the bound.
Completely agree. I suppose throwing an exception if particular number of messages are fetched, but not yet acknowledged is a perfect choice.
I've raise the JIRA: https://issues.apache.org/jira/browse/QPID-4346
Best Regards,
Sergey
-----Original Message-----
From: Gordon Sim [mailto:gsim@redhat.com]
Sent: Thursday, September 20, 2012 10:51 PM
To: users@qpid.apache.org
Subject: Re: How does the prefetch/receiver capacity really work?
On 09/19/2012 04:07 PM, Zhemzhitsky Sergey wrote:
>>> Can you elaborate a little on the use cases for that behaviour?
>
> The usecase is the following: we have a c++ client (messaging api) working as a forwarder to a proprietary system.
> The receiver looks like this:
>
> Message message;
> while(!isStopped())
> {
> If(receiver.fetch(message, duration))
> {
> proprietarySystemAdapter.send(message)
> }
> }
>
> In another thread this proprietary system acknowledges processed messages.
> The problem is that some of messages can take rather long time to be processed and some messages can be processed very fast. What I'd like to achieve is prevent system overloading with a large amount off messages.
Thanks, that makes a lot of sense.
>>> There are two questions in my mind. The first is simply how to allow users to specify they want that behaviour in a neat and simple way. The second is more problematic. ...
>
> Currently the receiver has a number of useful methods, such as
> getAvailable() - number of messages in the internal queue,
> getUnsettled() - number of acknowledged, but not yet confirmed
> messages
>
> I suppose that if there is a method like getUnacknowledged () - number of received (by means of Receiver::fetch(...) or Receiver::get(...)), but not yet acknowledged messages it would be possible to implement such a behavior (receiving the messages only if there are not more than a predefined number of unacknowledged ones exist) manually.
That might be a quick route to solving your issue. It seems like we could also perhaps allow that set to be bounded. and e.g. have the
fetch() call throw a particular exception if a call is made that would exceed the bound.
How soon would you need this? I'm a little short of time at present, but if you create a JIRA you can assign it to me for now and I'll get to it as soon as I can.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For additional commands, e-mail: users-help@qpid.apache.org
_______________________________________________________
The information contained in this message may be privileged and conf idential and protected from disclosure. If you are not the original intended recipient, you are hereby notified that any review, retransmission, dissemination, or other use of, or taking of any action in reliance upon, this information is prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and delete it from your computer. Thank you for your cooperation. Troika Dialog, Russia.
If you need assistance please contact our Contact Center (+7495) 258 0500 or go to www.troika.ru/eng/Contacts/system.wbp
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: How does the prefetch/receiver capacity really work?
Posted by Gordon Sim <gs...@redhat.com>.
On 09/19/2012 04:07 PM, Zhemzhitsky Sergey wrote:
>>> Can you elaborate a little on the use cases for that behaviour?
>
> The usecase is the following: we have a c++ client (messaging api) working as a forwarder to a proprietary system.
> The receiver looks like this:
>
> Message message;
> while(!isStopped())
> {
> If(receiver.fetch(message, duration))
> {
> proprietarySystemAdapter.send(message)
> }
> }
>
> In another thread this proprietary system acknowledges processed messages.
> The problem is that some of messages can take rather long time to be processed and some messages can be processed very fast. What I'd like to achieve is prevent system overloading with a large amount off messages.
Thanks, that makes a lot of sense.
>>> There are two questions in my mind. The first is simply how to allow users to specify they want that behaviour in a neat and simple way. The second is more problematic. ...
>
> Currently the receiver has a number of useful methods, such as getAvailable() - number of messages in the internal queue, getUnsettled() - number of acknowledged, but not yet confirmed messages
>
> I suppose that if there is a method like getUnacknowledged () - number of received (by means of Receiver::fetch(...) or Receiver::get(...)), but not yet acknowledged messages it would be possible to implement such a behavior (receiving the messages only if there are not more than a predefined number of unacknowledged ones exist) manually.
That might be a quick route to solving your issue. It seems like we
could also perhaps allow that set to be bounded. and e.g. have the
fetch() call throw a particular exception if a call is made that would
exceed the bound.
How soon would you need this? I'm a little short of time at present, but
if you create a JIRA you can assign it to me for now and I'll get to it
as soon as I can.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
RE: How does the prefetch/receiver capacity really work?
Posted by Zhemzhitsky Sergey <Se...@troika.ru>.
>> Can you elaborate a little on the use cases for that behaviour?
The usecase is the following: we have a c++ client (messaging api) working as a forwarder to a proprietary system.
The receiver looks like this:
Message message;
while(!isStopped())
{
If(receiver.fetch(message, duration))
{
proprietarySystemAdapter.send(message)
}
}
In another thread this proprietary system acknowledges processed messages.
The problem is that some of messages can take rather long time to be processed and some messages can be processed very fast. What I'd like to achieve is prevent system overloading with a large amount off messages.
>> There are two questions in my mind. The first is simply how to allow users to specify they want that behaviour in a neat and simple way. The second is more problematic. ...
Currently the receiver has a number of useful methods, such as getAvailable() - number of messages in the internal queue, getUnsettled() - number of acknowledged, but not yet confirmed messages
I suppose that if there is a method like getUnacknowledged () - number of received (by means of Receiver::fetch(...) or Receiver::get(...)), but not yet acknowledged messages it would be possible to implement such a behavior (receiving the messages only if there are not more than a predefined number of unacknowledged ones exist) manually.
Best Regards,
Sergey
-----Original Message-----
From: Gordon Sim [mailto:gsim@redhat.com]
Sent: Wednesday, September 19, 2012 2:12 PM
To: users@qpid.apache.org
Subject: Re: How does the prefetch/receiver capacity really work?
On 09/19/2012 09:28 AM, Zhemzhitsky Sergey wrote:
> Hi Gordon,
>
> Thanks for the answer.
>
> I'm wondering whether there is any possibility to move the prefetch window only after a message or a number of messages are acknowledged?
It isn't supported now. Can you elaborate a little on the use cases for that behaviour?
There are two questions in my mind. The first is simply how to allow users to specify they want that behaviour in a neat and simple way. The second is more problematic.
At present for capacity > 1 I use the AMQP 0-10 'window' mode for credit, where the window is moved by session-completed controls. However messages are considered completed as soon as they are received by the messaging library. All that is then needed is to periodically send information about the completed messages.
However, in this scheme, other actions on the session can result in that completion information being sent to the broker (which would move the window forward).
To give a guarantee that the 'prefetch' window moved only when messages are acknowledged (not just fetched), I would need to change to use explicit credits. I haven't thought it through in detail but if that turned out to be a large amount of work, I'd probably argue for dealing with it only on the 1.0 implementation.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For additional commands, e-mail: users-help@qpid.apache.org
_______________________________________________________
The information contained in this message may be privileged and conf idential and protected from disclosure. If you are not the original intended recipient, you are hereby notified that any review, retransmission, dissemination, or other use of, or taking of any action in reliance upon, this information is prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and delete it from your computer. Thank you for your cooperation. Troika Dialog, Russia.
If you need assistance please contact our Contact Center (+7495) 258 0500 or go to www.troika.ru/eng/Contacts/system.wbp
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: How does the prefetch/receiver capacity really work?
Posted by Gordon Sim <gs...@redhat.com>.
On 09/19/2012 09:28 AM, Zhemzhitsky Sergey wrote:
> Hi Gordon,
>
> Thanks for the answer.
>
> I'm wondering whether there is any possibility to move the prefetch window only after a message or a number of messages are acknowledged?
It isn't supported now. Can you elaborate a little on the use cases for
that behaviour?
There are two questions in my mind. The first is simply how to allow
users to specify they want that behaviour in a neat and simple way. The
second is more problematic.
At present for capacity > 1 I use the AMQP 0-10 'window' mode for
credit, where the window is moved by session-completed controls. However
messages are considered completed as soon as they are received by the
messaging library. All that is then needed is to periodically send
information about the completed messages.
However, in this scheme, other actions on the session can result in that
completion information being sent to the broker (which would move the
window forward).
To give a guarantee that the 'prefetch' window moved only when messages
are acknowledged (not just fetched), I would need to change to use
explicit credits. I haven't thought it through in detail but if that
turned out to be a large amount of work, I'd probably argue for dealing
with it only on the 1.0 implementation.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
RE: How does the prefetch/receiver capacity really work?
Posted by Zhemzhitsky Sergey <Se...@troika.ru>.
Hi Gordon,
Thanks for the answer.
I'm wondering whether there is any possibility to move the prefetch window only after a message or a number of messages are acknowledged?
Best Regards,
Sergey
-----Original Message-----
From: Gordon Sim [mailto:gsim@redhat.com]
Sent: Wednesday, September 19, 2012 12:19 PM
To: users@qpid.apache.org
Subject: Re: How does the prefetch/receiver capacity really work?
On 09/19/2012 09:06 AM, Zhemzhitsky Sergey wrote:
> Consider, there is a broker with a queue that contains 10 messages and receiver with receiver capacity set to 5.
>
> As I understand the broker will push 5 messages to the receiver to fill in its internal queue and internal queue of receiver will contain 5 messages and broker will contain 5 unpushed messages too.
> Then I will call Receiver::fetch(Message&,bool) without acknowledging the message.
> Does it mean that receiver’s internal queue now has a free slot, so the broker can push another message to it, so that the receiver can be considered to receive 6 messages in total and broker can be considered to have 4 unpushed messages?
The broker doesn't know about the free slot until the client library tells it. In AMQP 0-10, the c++ client does this via a session-completed, that includes the fetched message and thus moves the prefetch window. However to avoid excessive back and forth traffic, the
c++ client will only send completions when the application has used up
half the prefetch.
So in your example, the client would only move the prefetch window on
the broker after the application fetched 3 messages. (At that point the
broker could send a further 3 messages).
Its quite possible this could be fine tuned a little more, or made
configurable (right now its hardcoded) if there are cases where a
slightly different approach would be preferred.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
_______________________________________________________
The information contained in this message may be privileged and conf idential and protected from disclosure. If you are not the original intended recipient, you are hereby notified that any review, retransmission, dissemination, or other use of, or taking of any action in reliance upon, this information is prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and delete it from your computer. Thank you for your cooperation. Troika Dialog, Russia.
If you need assistance please contact our Contact Center (+7495) 258 0500 or go to www.troika.ru/eng/Contacts/system.wbp
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: How does the prefetch/receiver capacity really work?
Posted by Gordon Sim <gs...@redhat.com>.
On 09/19/2012 09:06 AM, Zhemzhitsky Sergey wrote:
> Consider, there is a broker with a queue that contains 10 messages and receiver with receiver capacity set to 5.
>
> As I understand the broker will push 5 messages to the receiver to fill in its internal queue and internal queue of receiver will contain 5 messages and broker will contain 5 unpushed messages too.
> Then I will call Receiver::fetch(Message&,bool) without acknowledging the message.
> Does it mean that receiver’s internal queue now has a free slot, so the broker can push another message to it, so that the receiver can be considered to receive 6 messages in total and broker can be considered to have 4 unpushed messages?
The broker doesn't know about the free slot until the client library
tells it. In AMQP 0-10, the c++ client does this via a
session-completed, that includes the fetched message and thus moves the
prefetch window. However to avoid excessive back and forth traffic, the
c++ client will only send completions when the application has used up
half the prefetch.
So in your example, the client would only move the prefetch window on
the broker after the application fetched 3 messages. (At that point the
broker could send a further 3 messages).
Its quite possible this could be fine tuned a little more, or made
configurable (right now its hardcoded) if there are cases where a
slightly different approach would be preferred.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org