You are viewing a plain text version of this content. The canonical link for it is here.
Posted to proton@qpid.apache.org by Ted Ross <tr...@redhat.com> on 2012/09/06 23:10:59 UTC
Using Proton in Multi-Threaded Environments
In order to make sure that it can be done, I'm attempting to use
Proton-c in a multi-threaded server environment.The basic assumptions
that I'm making are:
1. The root data structure that threads can operate on is
pn_connector_t/pn_connection_t. An application thread cannotuse
pn_driver_t without taking measures to ensure that it is the only
one doing so at a time.
2. As long as no two threads are concurrently operating on the same
instance of pn_connection_t, everything should work properly.
So far, so good. There is one interesting exception to the above.
It will commonly be the case that a thread processing the reception of a
message from an inbound link will need to indicate to a different
outbound link that there is now data to be sent (i.e. activate that link
for write). It is likely that the outbound link will be associated with
a different connection that the thread is not allowed to use because
another thread may be processing it concurrently.
This is not difficult to handle, but it requires that my threading layer
be able to get the pn_connector_t associated with the link. Proton-c
provides the following linkage: pn_link_t => pn_session_t =>
pn_connection_t. There's no way to get the pn_connector_t though.
So, I think I need one of two possible changes:
1. Provide linkage from pn_connection_t to pn_connector_t (may be
architecturally undesirable), or
2. Add a void* context field to pn_connector_t.
Thoughts, Feelings, Emotions?
-Ted
Re: Using Proton in Multi-Threaded Environments
Posted by Rafael Schloming <rh...@alum.mit.edu>.
On Thu, Sep 6, 2012 at 5:10 PM, Ted Ross <tr...@redhat.com> wrote:
> In order to make sure that it can be done, I'm attempting to use Proton-c
> in a multi-threaded server environment.The basic assumptions that I'm
> making are:
>
> 1. The root data structure that threads can operate on is
> pn_connector_t/pn_connection_**t. An application thread cannotuse
> pn_driver_t without taking measures to ensure that it is the only
> one doing so at a time.
> 2. As long as no two threads are concurrently operating on the same
> instance of pn_connection_t, everything should work properly.
>
> So far, so good. There is one interesting exception to the above.
>
> It will commonly be the case that a thread processing the reception of a
> message from an inbound link will need to indicate to a different outbound
> link that there is now data to be sent (i.e. activate that link for write).
> It is likely that the outbound link will be associated with a different
> connection that the thread is not allowed to use because another thread may
> be processing it concurrently.
>
> This is not difficult to handle, but it requires that my threading layer
> be able to get the pn_connector_t associated with the link. Proton-c
> provides the following linkage: pn_link_t => pn_session_t =>
> pn_connection_t. There's no way to get the pn_connector_t though.
>
> So, I think I need one of two possible changes:
>
> 1. Provide linkage from pn_connection_t to pn_connector_t (may be
> architecturally undesirable), or
> 2. Add a void* context field to pn_connector_t.
>
> Thoughts, Feelings, Emotions?
>
Regarding option (1), I think it would definitely be architecturally
undesirable since it assumes we are welded to a single driver rather than
leaving people the option to write their own.
Regarding option (2) I'm assuming you mean pn_connection_t since
pn_connector_t already has a void* context field. I think it makes sense to
have void* context on pretty much all the significant objects. I will very
shortly have reason to want one on pn_message_t to make messenger offer
explicit acking, and I've already encountered a bunch of cases where it
would be nice to have them on pn_link_t, pn_session_t, and/or
pn_connection_t. I suspect option (2) is just more generally useful since
you could use it not only to implement backpointers to pn_connector_t, but
also to any arbitrary application state you might care to associated with a
connection.
Why don't we try option (2) and see how it works out for you?
--Rafael
Re: Using Proton in Multi-Threaded Environments
Posted by William Henry <wh...@redhat.com>.
----- Original Message -----
> On 09/06/2012 05:44 PM, William Henry wrote:
> >
> > ----- Original Message -----
> >> In order to make sure that it can be done, I'm attempting to use
> >> Proton-c in a multi-threaded server environment.The basic
> >> assumptions
> >> that I'm making are:
> >>
> >> 1. The root data structure that threads can operate on is
> >> pn_connector_t/pn_connection_t. An application thread
> >> cannotuse
> >> pn_driver_t without taking measures to ensure that it is the
> >> only
> >> one doing so at a time.
> >> 2. As long as no two threads are concurrently operating on the
> >> same
> >> instance of pn_connection_t, everything should work properly.
> >>
> >> So far, so good. There is one interesting exception to the above.
> >>
> >> It will commonly be the case that a thread processing the
> >> reception
> >> of a
> >> message from an inbound link will need to indicate to a different
> >> outbound link that there is now data to be sent (i.e. activate
> >> that
> >> link
> >> for write). It is likely that the outbound link will be
> >> associated
> >> with
> >> a different connection that the thread is not allowed to use
> >> because
> >> another thread may be processing it concurrently.
> > FYI, I had to deal with this with the OpenMAMA integration work.
> > The publisher example program also has a reply queue and so
> > subscribes for incoming messages. To be safe I had to use two
> > pn_messenger_t instances, one for the outbound and one for the
> > inbound. It was a bit ugly but because I am listening for replies
> > on another thread while publishing on the main thread I wasn't
> > sure what else I could do until Proton handled multiple threads.
> >
> >> This is not difficult to handle, but it requires that my threading
> >> layer
> >> be able to get the pn_connector_t associated with the link.
> >> Proton-c
> >> provides the following linkage: pn_link_t => pn_session_t =>
> >> pn_connection_t. There's no way to get the pn_connector_t though.
> >>
> >> So, I think I need one of two possible changes:
> >>
> >> 1. Provide linkage from pn_connection_t to pn_connector_t (may
> >> be
> >> architecturally undesirable), or
> >> 2. Add a void* context field to pn_connector_t.
> >>
> >> Thoughts, Feelings, Emotions?
> > Whichever way you go I'm presuming that much of this will get
> > hidden underneath the pn_messenger_t interface.(?)
>
> To be clear, I'm looking at the underpinnings of messaging
> intermediaries (brokers, routers, proxies, firewalls, toolkits, etc.)
> rather than endpoints. I don't think that pn_messenger_t is relevant
> in
> such environments. However, there's no reason why a multi-threaded
> client or server couldn't use the same patterns. There won't be any
> benefit unless the endpoint is dealing with multiple connections
> though.
>
Ack.
William
> >
> > William
> >
> >> -Ted
> >>
> >>
>
>
Re: Using Proton in Multi-Threaded Environments
Posted by Ted Ross <tr...@redhat.com>.
On 09/06/2012 05:44 PM, William Henry wrote:
>
> ----- Original Message -----
>> In order to make sure that it can be done, I'm attempting to use
>> Proton-c in a multi-threaded server environment.The basic assumptions
>> that I'm making are:
>>
>> 1. The root data structure that threads can operate on is
>> pn_connector_t/pn_connection_t. An application thread cannotuse
>> pn_driver_t without taking measures to ensure that it is the only
>> one doing so at a time.
>> 2. As long as no two threads are concurrently operating on the same
>> instance of pn_connection_t, everything should work properly.
>>
>> So far, so good. There is one interesting exception to the above.
>>
>> It will commonly be the case that a thread processing the reception
>> of a
>> message from an inbound link will need to indicate to a different
>> outbound link that there is now data to be sent (i.e. activate that
>> link
>> for write). It is likely that the outbound link will be associated
>> with
>> a different connection that the thread is not allowed to use because
>> another thread may be processing it concurrently.
> FYI, I had to deal with this with the OpenMAMA integration work. The publisher example program also has a reply queue and so subscribes for incoming messages. To be safe I had to use two pn_messenger_t instances, one for the outbound and one for the inbound. It was a bit ugly but because I am listening for replies on another thread while publishing on the main thread I wasn't sure what else I could do until Proton handled multiple threads.
>
>> This is not difficult to handle, but it requires that my threading
>> layer
>> be able to get the pn_connector_t associated with the link. Proton-c
>> provides the following linkage: pn_link_t => pn_session_t =>
>> pn_connection_t. There's no way to get the pn_connector_t though.
>>
>> So, I think I need one of two possible changes:
>>
>> 1. Provide linkage from pn_connection_t to pn_connector_t (may be
>> architecturally undesirable), or
>> 2. Add a void* context field to pn_connector_t.
>>
>> Thoughts, Feelings, Emotions?
> Whichever way you go I'm presuming that much of this will get hidden underneath the pn_messenger_t interface.(?)
To be clear, I'm looking at the underpinnings of messaging
intermediaries (brokers, routers, proxies, firewalls, toolkits, etc.)
rather than endpoints. I don't think that pn_messenger_t is relevant in
such environments. However, there's no reason why a multi-threaded
client or server couldn't use the same patterns. There won't be any
benefit unless the endpoint is dealing with multiple connections though.
>
> William
>
>> -Ted
>>
>>
Re: Using Proton in Multi-Threaded Environments
Posted by William Henry <wh...@redhat.com>.
----- Original Message -----
> In order to make sure that it can be done, I'm attempting to use
> Proton-c in a multi-threaded server environment.The basic assumptions
> that I'm making are:
>
> 1. The root data structure that threads can operate on is
> pn_connector_t/pn_connection_t. An application thread cannotuse
> pn_driver_t without taking measures to ensure that it is the only
> one doing so at a time.
> 2. As long as no two threads are concurrently operating on the same
> instance of pn_connection_t, everything should work properly.
>
> So far, so good. There is one interesting exception to the above.
>
> It will commonly be the case that a thread processing the reception
> of a
> message from an inbound link will need to indicate to a different
> outbound link that there is now data to be sent (i.e. activate that
> link
> for write). It is likely that the outbound link will be associated
> with
> a different connection that the thread is not allowed to use because
> another thread may be processing it concurrently.
FYI, I had to deal with this with the OpenMAMA integration work. The publisher example program also has a reply queue and so subscribes for incoming messages. To be safe I had to use two pn_messenger_t instances, one for the outbound and one for the inbound. It was a bit ugly but because I am listening for replies on another thread while publishing on the main thread I wasn't sure what else I could do until Proton handled multiple threads.
>
> This is not difficult to handle, but it requires that my threading
> layer
> be able to get the pn_connector_t associated with the link. Proton-c
> provides the following linkage: pn_link_t => pn_session_t =>
> pn_connection_t. There's no way to get the pn_connector_t though.
>
> So, I think I need one of two possible changes:
>
> 1. Provide linkage from pn_connection_t to pn_connector_t (may be
> architecturally undesirable), or
> 2. Add a void* context field to pn_connector_t.
>
> Thoughts, Feelings, Emotions?
Whichever way you go I'm presuming that much of this will get hidden underneath the pn_messenger_t interface.(?)
William
>
> -Ted
>
>