You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Frank Quinn <fq...@gmail.com> on 2017/05/03 07:55:40 UTC
Alternatives to pn_messenger (QPID Proton C)
Hi Folks,
I look after the OpenMAMA bridge for Qpid Proton C and we originally built
our API based on the pn_messenger interface. However I see that interface
is now marked for deprecation, so I'm looking for alternatives. I'd
appreciate any feedback on my assessment of the options available which are
listed below.
When I look at the alternative options available in
https://github.com/apache/qpid-proton/tree/master/examples/c, the options
seem to be between:
*Messenger:* Deprecated - so let's assume that's going away and not an
option
*Proactor: *Looks interesting, though no subscription level support and
looks to be experimental
*Reactor: *I think the proactor pattern looks like a better fit for us due
to its asyncronous nature, but this looks to be more stable? (or at least
not marked as experimental?) Again, light on subscription integration
though.
So my initial feeling is to try moving to proactor. My main concern is
stability since it looks like it has already made an interface change since
0.17.0 (which expected while it's marked as experimental):
https://qpid.apache.org/releases/qpid-proton-0.17.0/proton/c/api/group__proactor.html#ga523ea983380a1566b3b1a7606d66422c.
Is this something which is definitely going to be non-experimental soon or
is there chance the whole interface could get canned?
Which leaves the final question of how to solve the issue with addressing.
We have traditionally used one URI per topic to give the subscriber
isolation from other data sources (this is market data - large volumes,
large number of topics). This has proven very slow so maybe it's going
against the grain (it was configurable though in our application).
The new API looks like it would encourage more coarse addressing (e.g.
maybe one URI per exchange?), then defer to the payload to handle
addressing. Does that sound fair or are you supposed to be able to call
pn_proactor_connect many times (is it thread safe also)? I'm curious about
what sort of addressing patterns that others have used specifically for
market data or other use cases with a large number of topics and high
volumes?
Cheers,
Frank
Re: Alternatives to pn_messenger (QPID Proton C)
Posted by Robbie Gemmell <ro...@gmail.com>.
On 3 May 2017 at 17:27, Robbie Gemmell <ro...@gmail.com> wrote:
> On 3 May 2017 at 08:55, Frank Quinn <fq...@gmail.com> wrote:
>> Hi Folks,
>>
>> I look after the OpenMAMA bridge for Qpid Proton C and we originally built
>> our API based on the pn_messenger interface. However I see that interface
>> is now marked for deprecation, so I'm looking for alternatives. I'd
>> appreciate any feedback on my assessment of the options available which are
>> listed below.
>>
>> When I look at the alternative options available in
>> https://github.com/apache/qpid-proton/tree/master/examples/c, the options
>> seem to be between:
>>
>> *Messenger:* Deprecated - so let's assume that's going away and not an
>> option
>> *Proactor: *Looks interesting, though no subscription level support and
>> looks to be experimental
>> *Reactor: *I think the proactor pattern looks like a better fit for us due
>> to its asyncronous nature, but this looks to be more stable? (or at least
>> not marked as experimental?) Again, light on subscription integration
>> though.
>>
>
> The latter is where activity/attention is mostly going currently.
>
Erm, no :)
I originally went to say former, but didn't since the list had
messegner at the top, and then somehow typed latter. Lets just say
*Proactor* :)
> Its probably worth stating that Messenger fits at a slightly higher
> level than the others. At a similar higher level would be the reactive
> C++ API (http://qpid.apache.org/releases/qpid-proton-0.17.0/proton/cpp/api/tutorial.html),
> built atop the C reactor but likely heading to being atop the
> proactor.
>
>>
>> So my initial feeling is to try moving to proactor. My main concern is
>> stability since it looks like it has already made an interface change since
>> 0.17.0 (which expected while it's marked as experimental):
>> https://qpid.apache.org/releases/qpid-proton-0.17.0/proton/c/api/group__proactor.html#ga523ea983380a1566b3b1a7606d66422c.
>> Is this something which is definitely going to be non-experimental soon or
>> is there chance the whole interface could get canned?
>>
>
> It's going to be sticking around and is in the process of being
> utilised in Qpid Dispatch to replace the routers existing IO handling
> via https://issues.apache.org/jira/browse/DISPATCH-390 currently. I'll
> let others who actually work on these bits speak to stability.
>
>> Which leaves the final question of how to solve the issue with addressing.
>> We have traditionally used one URI per topic to give the subscriber
>> isolation from other data sources (this is market data - large volumes,
>> large number of topics). This has proven very slow so maybe it's going
>> against the grain (it was configurable though in our application).
>>
>> The new API looks like it would encourage more coarse addressing (e.g.
>> maybe one URI per exchange?), then defer to the payload to handle
>> addressing. Does that sound fair or are you supposed to be able to call
>> pn_proactor_connect many times (is it thread safe also)? I'm curious about
>> what sort of addressing patterns that others have used specifically for
>> market data or other use cases with a large number of topics and high
>> volumes?
>>
>
> I wouldnt expect the API to encourage a particular form of addressing,
> with you rather using what best suits your needs after taking into
> consideration anything such as particular servers behaviours or use of
> addressing to facilitate handling different sets of data in specific
> ways, either in the servers or application code.
>
> You can ultimately create as many or as few connections as you like,
> and use as many or as few sender/receiver links over them, against as
> many different addresses as suits your needs, save for certain fairly
> large protocol level restrictions and any given servers abilities.
>
> Somewhat related however, one thing the other APIs can expose that I'm
> not sure if Messenger ever did, would be ability to use an 'anonymous
> sender' link to send messages to difference addresses based on the
> message 'to' address field, when supported by the server (such as the
> Qpid brokers, Qpid Dispatch router, or ActiveMQ brokers).
>
> Its also worth saying the address the AMQP message is sent to or is
> received from is independent from the connection host details, even
> though Messenger might have tied the two together somewhat in its
> particular use of URIs.
>
>> Cheers,
>> Frank
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Alternatives to pn_messenger (QPID Proton C)
Posted by Frank Quinn <fq...@gmail.com>.
Yup its all open source:
Middleware:
https://github.com/OpenMAMA/OpenMAMA/tree/master/mama/c_cpp/src/c/bridge/qpid
Payload:
https://github.com/OpenMAMA/OpenMAMA/tree/master/mama/c_cpp/src/c/payload/qpidmsg
Its an API to provide a unified interface into a variety of (mostly market
data) middlewares and payloads, so writing a payload bridge for c++ would
effectively be a rewrite.
Cheers,
Frank
On Wed, 3 May 2017, 22:49 Andrew Stitcher, <as...@apache.org> wrote:
> On Wed, 2017-05-03 at 21:27 +0000, Frank Quinn wrote:
> > Thanks Andrew,
> >
> > Our application distinguishes between transport and payload. We have
> > a
> > *lot* of code around the payload which happily works with
> > pn_message_t
> > which would be extremely painful to port to the C++ API. We then use
> > pn_messenger purely for transport send / recv.
>
> Is it possible to see this code anywhere? I'm curious what sort of
> volume of code you are talking about.
>
> Is code actually using the pn_message_t API extensively? Or is it
> creating a binary message that gets put in the message, or in some
> message properties?
>
> Is it using the pn_data_t API? - If you are using complex AMQP
> datastructures via pn_data_t then that might be complex to recode.
>
> IMO the proton::message API ergonomics are so much easier and better
> than the pn_message_t that I would hardly consider using pn_message_t
> anymore.
>
> However I understand that working code is working code.
>
> If don't want to change the pn_message_t using code then using the
> proactor directly might be the way to go.
>
> Andrew
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: Alternatives to pn_messenger (QPID Proton C)
Posted by Alan Conway <ac...@redhat.com>.
On Wed, 2017-05-03 at 17:49 -0400, Andrew Stitcher wrote:
> On Wed, 2017-05-03 at 21:27 +0000, Frank Quinn wrote:
> > Thanks Andrew,
> >
> > Our application distinguishes between transport and payload. We
> > have
> > a
> > *lot* of code around the payload which happily works with
> > pn_message_t
> > which would be extremely painful to port to the C++ API. We then
> > use
> > pn_messenger purely for transport send / recv.
>
> Is it possible to see this code anywhere? I'm curious what sort of
> volume of code you are talking about.
>
> Is code actually using the pn_message_t API extensively? Or is it
> creating a binary message that gets put in the message, or in some
> message properties?
>
> Is it using the pn_data_t API? - If you are using complex AMQP
> datastructures via pn_data_t then that might be complex to recode.
>
> IMO the proton::message API ergonomics are so much easier and better
> than the pn_message_t that I would hardly consider using pn_message_t
> anymore.
>
> However I understand that working code is working code.
>
> If don't want to change the pn_message_t using code then using the
> proactor directly might be the way to go.
I agree with Andrew that the C++ API is easier to use because of the
features of C++. However if you want to work in C, the proactor API is
(in my *very* biased opinion) not significantly harder to use than
pn_messanger or pn_reactor, and provides much more user control and
implementation flexibility.
The proactor API has not been widely used yet, but is (informally)
finalized and will soon be released. We have just moved a significant C
project (http://qpid.apache.org/components/dispatch-router/index.html)
onto the proactor APIs with a good reduction in code and no obvious
problems (yet). Dispatch was not using pn_messenger so I can't comment
on migrating from messenger, but I will be happy to answer any
questions as I'm keen that the proactor succeed as a replacement for
pn_messanger_t.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Alternatives to pn_messenger (QPID Proton C)
Posted by Andrew Stitcher <as...@apache.org>.
On Wed, 2017-05-03 at 21:27 +0000, Frank Quinn wrote:
> Thanks Andrew,
>
> Our application distinguishes between transport and payload. We have
> a
> *lot* of code around the payload which happily works with
> pn_message_t
> which would be extremely painful to port to the C++ API. We then use
> pn_messenger purely for transport send / recv.
Is it possible to see this code anywhere? I'm curious what sort of
volume of code you are talking about.
Is code actually using the pn_message_t API extensively? Or is it
creating a binary message that gets put in the message, or in some
message properties?
Is it using the pn_data_t API? - If you are using complex AMQP
datastructures via pn_data_t then that might be complex to recode.
IMO the proton::message API ergonomics are so much easier and better
than the pn_message_t that I would hardly consider using pn_message_t
anymore.
However I understand that working code is working code.
If don't want to change the pn_message_t using code then using the
proactor directly might be the way to go.
Andrew
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Alternatives to pn_messenger (QPID Proton C)
Posted by Frank Quinn <fq...@gmail.com>.
Thanks Andrew,
Our application distinguishes between transport and payload. We have a
*lot* of code around the payload which happily works with pn_message_t
which would be extremely painful to port to the C++ API. We then use
pn_messenger purely for transport send / recv.
Cheers,
Frank
On Wed, 3 May 2017, 21:58 Andrew Stitcher, <as...@apache.org> wrote:
> On Wed, 2017-05-03 at 20:48 +0000, Frank Quinn wrote:
> > ...
> > We also make heavy use of pn_message_t which I thought might have
> > been a
> > possible overload for the c++ proton::message constructor but I can't
> > see
> > one unless there are conversion functions somewhere public?
>
> The C++ API is intended to be used by itself, and not in conjunction
> with the C API at the same time.
>
> Having said that proton::message is only a wrapper for pn_message_t.
> But if you are using the API there's no string reason to ever use the C
> pn_message_t API.
>
> What are you doing the uses the pn_message_t API that is separate from
> sending messages, that can't be done solely in C++?
>
> You should be able to everything more easily just using the C++ API. If
> not, then we'll seriously look at what you are doing that can't be
> done, as we do want the API to cover a wide swath.
>
> Andrew
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: Alternatives to pn_messenger (QPID Proton C)
Posted by Andrew Stitcher <as...@apache.org>.
On Wed, 2017-05-03 at 20:48 +0000, Frank Quinn wrote:
> ...
> We also make heavy use of pn_message_t which I thought might have
> been a
> possible overload for the c++ proton::message constructor but I can't
> see
> one unless there are conversion functions somewhere public?
The C++ API is intended to be used by itself, and not in conjunction
with the C API at the same time.
Having said that proton::message is only a wrapper for pn_message_t.
But if you are using the API there's no string reason to ever use the C
pn_message_t API.
What are you doing the uses the pn_message_t API that is separate from
sending messages, that can't be done solely in C++?
You should be able to everything more easily just using the C++ API. If
not, then we'll seriously look at what you are doing that can't be
done, as we do want the API to cover a wide swath.
Andrew
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Alternatives to pn_messenger (QPID Proton C)
Posted by Frank Quinn <fq...@gmail.com>.
Thanks for the quick responses folks! Those comments help a lot.
Honestly for our use case, we're happy to go lower level. In fact that's
usually where we integrate our bridges - the messenger approach was more of
an exception in our case and we actually suffered a little from it at the
start because there was no way to hook into things like connection and
disconnection events and it all felt very synchronous and batchy. Similarly
though we were bitten by a fair few interface changes to pn_messenger in
the earlier days and keeping cross-version compatibility in tact and I'm
just conscious of trying to adopt at this point if the interface is still
being ironed out. Sounds like it might be best to wait until the next
release if we go down the proactor route.
We also make heavy use of pn_message_t which I thought might have been a
possible overload for the c++ proton::message constructor but I can't see
one unless there are conversion functions somewhere public?
Cheers,
Frank
On Wed, 3 May 2017, 20:43 Andrew Stitcher, <as...@apache.org> wrote:
> On Wed, 2017-05-03 at 17:27 +0100, Robbie Gemmell wrote:
> > On 3 May 2017 at 08:55, Frank Quinn <fq...@gmail.com> wrote:
> > > Hi Folks,
> > >
> > > I look after the OpenMAMA bridge for Qpid Proton C and we
> > > originally built
> > > our API based on the pn_messenger interface. However I see that
> > > interface
> > > is now marked for deprecation, so I'm looking for alternatives. I'd
> > > appreciate any feedback on my assessment of the options available
> > > which are
> > > listed below.
> > >
> > > When I look at the alternative options available in
> > > https://github.com/apache/qpid-proton/tree/master/examples/c, the
> > > options
> > > seem to be between:
> > >
> > > *Messenger:* Deprecated - so let's assume that's going away and not
> > > an
> > > option
> > > *Proactor: *Looks interesting, though no subscription level support
> > > and
> > > looks to be experimental
> > > *Reactor: *I think the proactor pattern looks like a better fit for
> > > us due
> > > to its asyncronous nature, but this looks to be more stable? (or at
> > > least
> > > not marked as experimental?) Again, light on subscription
> > > integration
> > > though.
> > >
>
> To emphasize Robbie's points:
>
> * We are actively working on the proactor as a replacement IO layer for
> reactor which will be deprecated in due course.
>
> * The messenger API is high level and not really comparable with the
> proactor/reactor APIs which are significantly lower level.
>
> * The closest API to the same level of abstraction is the C++ binding,
> however it does split apart the concept of network connection address
> from message address, which makes for a different "API feel". It sounds
> that in your case this might fit your requirements better.
>
> The C++ API is mostly stable currently, especially for client purposes.
> But we plan that when we release 0.18 the API will stabilize further
> and will shift on top of the proton proactor IO rather than the reactor
> IO.
>
> HTH
>
> Andrew
>
>
Re: Alternatives to pn_messenger (QPID Proton C)
Posted by Andrew Stitcher <as...@apache.org>.
On Wed, 2017-05-03 at 17:27 +0100, Robbie Gemmell wrote:
> On 3 May 2017 at 08:55, Frank Quinn <fq...@gmail.com> wrote:
> > Hi Folks,
> >
> > I look after the OpenMAMA bridge for Qpid Proton C and we
> > originally built
> > our API based on the pn_messenger interface. However I see that
> > interface
> > is now marked for deprecation, so I'm looking for alternatives. I'd
> > appreciate any feedback on my assessment of the options available
> > which are
> > listed below.
> >
> > When I look at the alternative options available in
> > https://github.com/apache/qpid-proton/tree/master/examples/c, the
> > options
> > seem to be between:
> >
> > *Messenger:* Deprecated - so let's assume that's going away and not
> > an
> > option
> > *Proactor: *Looks interesting, though no subscription level support
> > and
> > looks to be experimental
> > *Reactor: *I think the proactor pattern looks like a better fit for
> > us due
> > to its asyncronous nature, but this looks to be more stable? (or at
> > least
> > not marked as experimental?) Again, light on subscription
> > integration
> > though.
> >
To emphasize Robbie's points:
* We are actively working on the proactor as a replacement IO layer for
reactor which will be deprecated in due course.
* The messenger API is high level and not really comparable with the
proactor/reactor APIs which are significantly lower level.
* The closest API to the same level of abstraction is the C++ binding,
however it does split apart the concept of network connection address
from message address, which makes for a different "API feel". It sounds
that in your case this might fit your requirements better.
The C++ API is mostly stable currently, especially for client purposes.
But we plan that when we release 0.18 the API will stabilize further
and will shift on top of the proton proactor IO rather than the reactor
IO.
HTH
Andrew
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Alternatives to pn_messenger (QPID Proton C)
Posted by Robbie Gemmell <ro...@gmail.com>.
On 3 May 2017 at 08:55, Frank Quinn <fq...@gmail.com> wrote:
> Hi Folks,
>
> I look after the OpenMAMA bridge for Qpid Proton C and we originally built
> our API based on the pn_messenger interface. However I see that interface
> is now marked for deprecation, so I'm looking for alternatives. I'd
> appreciate any feedback on my assessment of the options available which are
> listed below.
>
> When I look at the alternative options available in
> https://github.com/apache/qpid-proton/tree/master/examples/c, the options
> seem to be between:
>
> *Messenger:* Deprecated - so let's assume that's going away and not an
> option
> *Proactor: *Looks interesting, though no subscription level support and
> looks to be experimental
> *Reactor: *I think the proactor pattern looks like a better fit for us due
> to its asyncronous nature, but this looks to be more stable? (or at least
> not marked as experimental?) Again, light on subscription integration
> though.
>
The latter is where activity/attention is mostly going currently.
Its probably worth stating that Messenger fits at a slightly higher
level than the others. At a similar higher level would be the reactive
C++ API (http://qpid.apache.org/releases/qpid-proton-0.17.0/proton/cpp/api/tutorial.html),
built atop the C reactor but likely heading to being atop the
proactor.
>
> So my initial feeling is to try moving to proactor. My main concern is
> stability since it looks like it has already made an interface change since
> 0.17.0 (which expected while it's marked as experimental):
> https://qpid.apache.org/releases/qpid-proton-0.17.0/proton/c/api/group__proactor.html#ga523ea983380a1566b3b1a7606d66422c.
> Is this something which is definitely going to be non-experimental soon or
> is there chance the whole interface could get canned?
>
It's going to be sticking around and is in the process of being
utilised in Qpid Dispatch to replace the routers existing IO handling
via https://issues.apache.org/jira/browse/DISPATCH-390 currently. I'll
let others who actually work on these bits speak to stability.
> Which leaves the final question of how to solve the issue with addressing.
> We have traditionally used one URI per topic to give the subscriber
> isolation from other data sources (this is market data - large volumes,
> large number of topics). This has proven very slow so maybe it's going
> against the grain (it was configurable though in our application).
>
> The new API looks like it would encourage more coarse addressing (e.g.
> maybe one URI per exchange?), then defer to the payload to handle
> addressing. Does that sound fair or are you supposed to be able to call
> pn_proactor_connect many times (is it thread safe also)? I'm curious about
> what sort of addressing patterns that others have used specifically for
> market data or other use cases with a large number of topics and high
> volumes?
>
I wouldnt expect the API to encourage a particular form of addressing,
with you rather using what best suits your needs after taking into
consideration anything such as particular servers behaviours or use of
addressing to facilitate handling different sets of data in specific
ways, either in the servers or application code.
You can ultimately create as many or as few connections as you like,
and use as many or as few sender/receiver links over them, against as
many different addresses as suits your needs, save for certain fairly
large protocol level restrictions and any given servers abilities.
Somewhat related however, one thing the other APIs can expose that I'm
not sure if Messenger ever did, would be ability to use an 'anonymous
sender' link to send messages to difference addresses based on the
message 'to' address field, when supported by the server (such as the
Qpid brokers, Qpid Dispatch router, or ActiveMQ brokers).
Its also worth saying the address the AMQP message is sent to or is
received from is independent from the connection host details, even
though Messenger might have tied the two together somewhat in its
particular use of URIs.
> Cheers,
> Frank
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org