You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by Alan Conway <ac...@redhat.com> on 2014/05/14 22:33:24 UTC
Proton: Messenger API - how do you know when its done?
I've been playing with Messenger and I've found I need to do this if I
want to be sure it has done what I've asked it to do:
def flush(self):
"""Call work() till there is no work left."""
while self.work(0.01): pass
I've found I need to do this after subscribe() and accept() if I want to
be sure the action has been carried out. (e.g. because I want to check
if the corresponding message has been removed from a broker.)
Is there a better way of doing this? If not should we add something like
this to messenger API?
This would be analogous to the sync() call and sync arguments in the
qpid.messaging API.
Cheers,
Alan.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: Proton: Messenger API - how do you know when its done?
Posted by Alan Conway <ac...@redhat.com>.
On Mon, 2014-05-19 at 11:25 -0400, Rafael Schloming wrote:
> FWIW I tend to think using subscribe in that way is really a bit of an
> anti-pattern because it imposes a strict ordering requirement on the
> startup sequence of the various distributed components in the system. To be
> specific, you always have to start the broker first, the subscriber second,
> and the publisher third. In any real system this is almost never going to
> be possible to guarantee. The order will most likely be completely random,
> and ideally you should get exactly the same result regardless.
>
> I can't speak to Alan's particular case without more detail, but I think in
> general it would be most useful for users if we could define broker/client
> behaviour such that it is robust to randomized startup order.
You hit the nail on the head - my case is exactly testing a messenger
+router+broker scenario which doesn't work unless you set things up in
the right order. I agree we want to fix the behavior so we can be robust
to different start-up orders in the long term.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: Proton: Messenger API - how do you know when its done?
Posted by Rafael Schloming <rh...@alum.mit.edu>.
On Mon, May 19, 2014 at 4:14 PM, Gordon Sim <gs...@redhat.com> wrote:
> On 05/19/2014 06:57 PM, Rafael Schloming wrote:
>
>> That doesn't require the same kind of synchronization, it just requires
>> that the client implementation doesn't internally reorder execution of the
>> given API requests.
>>
>
> If the subscription and the request are sent on the same connection and
> are guaranteed to be written to the wire in the order in which the API
> calls are made, then you are right, that would normally be sufficient
> (unless the broker in use can process different sessions in parallel, in
> which case they would need to be sent on the same session as well).
>
> If the request is for any reason on a different connection, or otherwise
> 'out-of-band', then you don't have that guarantee.
>
Even with multiple connections messenger could still (hypothetically)
guarantee order of completion by synchronizing internally in those cases.
But if we are considering multiple connections and interesting broker
semantics, even the ability to synchronize on the subscribe might not get
you that far, e.g. I suspect the dispatch router can't actually guarantee
that when a subscribe completes the routing information has propagated to
other routers.
>
> To be clear, I'm not arguing for any change. I was just pointing out that
> subscribe() can be tracked through the current message tracker state.
I'm not actually opposed to exposing the subscription status, or defining
that as part of the semantics of messenger.start(). My point is more that
there are a lot of cases where it is not safe in general to depend on it,
and it would be good to provide other capabilities people can use to avoid
this. The common testing scenario where this comes up is something like:
1. test receiver subscribes/creates a queue
2. test receiver tells sender it's ok to go ahead via out of band
communication (e.g. wait/notify on a condition variable or some such thing)
3. test sender then goes ahead
Given that a real distributed system can never do (2), a much better
pattern to follow would be to use the namespace stuff you implemented to
cause the appropriate queue to get auto created when referenced.
--Rafael
Re: Proton: Messenger API - how do you know when its done?
Posted by Gordon Sim <gs...@redhat.com>.
On 05/19/2014 09:14 PM, Gordon Sim wrote:
> I was just pointing out that subscribe() can be tracked through the
> current message tracker state.
I meant *can't* of course, sorry! Second time in this thread I've made
that same typo.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: Proton: Messenger API - how do you know when its done?
Posted by Gordon Sim <gs...@redhat.com>.
On 05/19/2014 06:57 PM, Rafael Schloming wrote:
> That doesn't require the same kind of synchronization, it just requires
> that the client implementation doesn't internally reorder execution of the
> given API requests.
If the subscription and the request are sent on the same connection and
are guaranteed to be written to the wire in the order in which the API
calls are made, then you are right, that would normally be sufficient
(unless the broker in use can process different sessions in parallel, in
which case they would need to be sent on the same session as well).
If the request is for any reason on a different connection, or otherwise
'out-of-band', then you don't have that guarantee.
To be clear, I'm not arguing for any change. I was just pointing out
that subscribe() can be tracked through the current message tracker state.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: Proton: Messenger API - how do you know when its done?
Posted by Rafael Schloming <rh...@alum.mit.edu>.
On Mon, May 19, 2014 at 11:59 AM, Gordon Sim <gs...@redhat.com> wrote:
> On 05/19/2014 04:25 PM, Rafael Schloming wrote:
>
>> FWIW I tend to think using subscribe in that way is really a bit of an
>> anti-pattern because it imposes a strict ordering requirement on the
>> startup sequence of the various distributed components in the system. To
>> be
>> specific, you always have to start the broker first, the subscriber
>> second,
>> and the publisher third.
>>
>
> No, I don't think that is the case. Consider a discovery case, where there
> are many 'agents' up and running, and some component wants to get a list of
> them.
>
> One common solution is to subscribe to some topic on which agents
> advertise themselves, then send a request that every agent re-announce
> themselves. (E.g as in the reservations example for the qpid.messaging
> client).
>
> If you can be sure that the subscription is active before you send your
> request, you may miss announcements. So while you do indeed want to start
> the publisher (of the request) before the subscriber (to the announcement),
> these are in the same process.
That doesn't require the same kind of synchronization, it just requires
that the client implementation doesn't internally reorder execution of the
given API requests. I believe the same reservations example (as well as
various similar reply-to scenarios) work fine with messenger for exactly
this reason (it doesn't reorder what you ask it to do).
--Rafael
Re: Proton: Messenger API - how do you know when its done?
Posted by Gordon Sim <gs...@redhat.com>.
On 05/19/2014 04:25 PM, Rafael Schloming wrote:
> FWIW I tend to think using subscribe in that way is really a bit of an
> anti-pattern because it imposes a strict ordering requirement on the
> startup sequence of the various distributed components in the system. To be
> specific, you always have to start the broker first, the subscriber second,
> and the publisher third.
No, I don't think that is the case. Consider a discovery case, where
there are many 'agents' up and running, and some component wants to get
a list of them.
One common solution is to subscribe to some topic on which agents
advertise themselves, then send a request that every agent re-announce
themselves. (E.g as in the reservations example for the qpid.messaging
client).
If you can be sure that the subscription is active before you send your
request, you may miss announcements. So while you do indeed want to
start the publisher (of the request) before the subscriber (to the
announcement), these are in the same process.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: Proton: Messenger API - how do you know when its done?
Posted by Rafael Schloming <rh...@alum.mit.edu>.
FWIW I tend to think using subscribe in that way is really a bit of an
anti-pattern because it imposes a strict ordering requirement on the
startup sequence of the various distributed components in the system. To be
specific, you always have to start the broker first, the subscriber second,
and the publisher third. In any real system this is almost never going to
be possible to guarantee. The order will most likely be completely random,
and ideally you should get exactly the same result regardless.
I can't speak to Alan's particular case without more detail, but I think in
general it would be most useful for users if we could define broker/client
behaviour such that it is robust to randomized startup order.
--Rafael
On Mon, May 19, 2014 at 5:22 AM, Gordon Sim <gs...@redhat.com> wrote:
> On 05/15/2014 12:48 PM, Rafael Schloming wrote:
>
>> On Wed, May 14, 2014 at 4:33 PM, Alan Conway <ac...@redhat.com> wrote:
>>
>> I've been playing with Messenger and I've found I need to do this if I
>>> want to be sure it has done what I've asked it to do:
>>>
>>> def flush(self):
>>> """Call work() till there is no work left."""
>>> while self.work(0.01): pass
>>>
>>>
>>> I've found I need to do this after subscribe() and accept() if I want to
>>> be sure the action has been carried out. (e.g. because I want to check
>>> if the corresponding message has been removed from a broker.)
>>>
>>> Is there a better way of doing this? If not should we add something like
>>> this to messenger API?
>>>
>>>
>> You are supposed to be able to check the tracker status for your message
>> to
>> figure stuff like this out, although we may not (yet) expose a state that
>> would correspond to the message being dequeued.
>>
>>
>>
>>> This would be analogous to the sync() call and sync arguments in the
>>> qpid.messaging API.
>>>
>>>
>> I don't think we need a sync() analog (at least not for this case) since
>> it
>> seems like what you care about has mostly to do with the message, i.e. did
>> I get the message I want and has it reached the appropriate state.
>>
>
> The subscribe case is not about a specific message. It perhaps comes up
> more in testing than 'real world' applications, but e.g. if you want to
> ensure a subscription is active before 'advertising' reachability through
> that e.g. in reply-to, then it is sometimes useful.
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
> For additional commands, e-mail: dev-help@qpid.apache.org
>
>
Re: Proton: Messenger API - how do you know when its done?
Posted by Gordon Sim <gs...@redhat.com>.
On 05/15/2014 12:48 PM, Rafael Schloming wrote:
> On Wed, May 14, 2014 at 4:33 PM, Alan Conway <ac...@redhat.com> wrote:
>
>> I've been playing with Messenger and I've found I need to do this if I
>> want to be sure it has done what I've asked it to do:
>>
>> def flush(self):
>> """Call work() till there is no work left."""
>> while self.work(0.01): pass
>>
>>
>> I've found I need to do this after subscribe() and accept() if I want to
>> be sure the action has been carried out. (e.g. because I want to check
>> if the corresponding message has been removed from a broker.)
>>
>> Is there a better way of doing this? If not should we add something like
>> this to messenger API?
>>
>
> You are supposed to be able to check the tracker status for your message to
> figure stuff like this out, although we may not (yet) expose a state that
> would correspond to the message being dequeued.
>
>
>>
>> This would be analogous to the sync() call and sync arguments in the
>> qpid.messaging API.
>>
>
> I don't think we need a sync() analog (at least not for this case) since it
> seems like what you care about has mostly to do with the message, i.e. did
> I get the message I want and has it reached the appropriate state.
The subscribe case is not about a specific message. It perhaps comes up
more in testing than 'real world' applications, but e.g. if you want to
ensure a subscription is active before 'advertising' reachability
through that e.g. in reply-to, then it is sometimes useful.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: Proton: Messenger API - how do you know when its done?
Posted by Rafael Schloming <rh...@alum.mit.edu>.
On Wed, May 14, 2014 at 4:33 PM, Alan Conway <ac...@redhat.com> wrote:
> I've been playing with Messenger and I've found I need to do this if I
> want to be sure it has done what I've asked it to do:
>
> def flush(self):
> """Call work() till there is no work left."""
> while self.work(0.01): pass
>
>
> I've found I need to do this after subscribe() and accept() if I want to
> be sure the action has been carried out. (e.g. because I want to check
> if the corresponding message has been removed from a broker.)
>
> Is there a better way of doing this? If not should we add something like
> this to messenger API?
>
You are supposed to be able to check the tracker status for your message to
figure stuff like this out, although we may not (yet) expose a state that
would correspond to the message being dequeued.
>
> This would be analogous to the sync() call and sync arguments in the
> qpid.messaging API.
>
I don't think we need a sync() analog (at least not for this case) since it
seems like what you care about has mostly to do with the message, i.e. did
I get the message I want and has it reached the appropriate state.
--Rafael