You are viewing a plain text version of this content. The canonical link for it is here.
Posted to proton@qpid.apache.org by "Kritikos, Alex" <Al...@softwareag.com> on 2015/06/10 11:34:14 UTC
[Resending] - Proton-J engine and thread safety
[Resending as it ended up in the wrong thread]
Hi all,
is the proton-j engine meant to be thread safe?
We have been experiencing some sporadic issues where under load, the engine sends callbacks to registered handlers with null as the event. We do not have a standalone repro case yet but just wondered what other people’s experience is as well as what are the recommendations around thread safety.
Thanks,
Alex Kritikos
Software AG
This communication contains information which is confidential and may also be privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s), please note that any distribution, copying, or use of this communication or the information in it, is strictly prohibited. If you have received this communication in error please notify us by e-mail and then delete the e-mail and any copies of it.
Software AG (UK) Limited Registered in England & Wales 1310740 - http://www.softwareag.com/uk
Re: [Resending] - Proton-J engine and thread safety
Posted by aconway <ac...@redhat.com>.
On Wed, 2015-06-10 at 14:18 +0000, Kritikos, Alex wrote:
> Hi Alan,
>
> thanks for your response. We also use an engine per connection
> however there are different read and write threads interacting with
> it and the issues only occur under load.
> I guess we should try to create a repro case.
You need to serialize the read and write threads, the engine is not
safe for concurrent use at all. My blabbering about read and write
concurrency may have misled you.
You could simply mutex-lock the engine in your read/write threads.
Depending on what else you are doing beware contention and deadlocks.
The C++ (and I think Java) brokers handle this in the poller: we take
the FD out of the poller on a read or write event, do the relevant
proton work, then put it back to get further events. That way you can't
have concurrent read/write on an engine.
>
> Thanks,
>
> Alex Kritikos
> Software AG
> On 10 Jun 2015, at 16:50, aconway <ac...@redhat.com> wrote:
>
> > On Wed, 2015-06-10 at 09:34 +0000, Kritikos, Alex wrote:
> > > [Resending as it ended up in the wrong thread]
> > >
> > > Hi all,
> > >
> > > is the proton-j engine meant to be thread safe?
> >
> > The C engine is definitely NOT meant to be thread safe. Unless you
> > have
> > found an explicit written declaration that the java engine is
> > supposed
> > to be AND a bunch of code to back that up I wouldn't rely on it.
> >
> > The way we use proton in the C++ broker and in the upcoming Go
> > binding
> > is to create an engine per connection and serialize the action on
> > each
> > connection. In principle you can read and write from the OS
> > connection
> > concurrently but it's debatable how much you gain, you are likely
> > just
> > moving OS buffers into app buffers which is not a big win.
> >
> > The inbound and outbound protocol state *for a single connection*
> > is
> > pretty closely tied together. Proton is probably taking the right
> > approach by assuming both are handled in a single concurrency
> > context.
> >
> > The engine state for separate connections is *completely
> > independent*
> > so it's safe to run engines for separate connections in separte
> > contexts.
> >
> > The recent reactor extensions to proton are interesting but not
> > thread
> > friendly. They force the protocol handling for multiple connections
> > into a single thread context, which is great for single threaded
> > apps
> > but IMO the wrong way to go for concurrent apps.
> >
> > The go binding uses channels to pump data from connection
> > read/write
> > goroutines to a proton engine event loop goroutine per connection.
> > The
> > C++ broker predates the reactor and does it's own polling with
> > read/write activity on an FD handled dispatched sequentially to
> > worker
> > threads so the proton engine for a connection is never used
> > concurrently.
> >
> > There may be something interesting we can do at the proton layer to
> > help with this pattern or it may be better to leave concurrency
> > above
> > the binding to be handled by the languages own concurrency tools, I
> > am
> > not sure yet.
> >
> >
> > > We have been experiencing some sporadic issues where under load,
> > > the
> > > engine sends callbacks to registered handlers with null as the
> > > event.
> > > We do not have a standalone repro case yet but just wondered what
> > > other people’s experience is as well as what are the
> > > recommendations
> > > around thread safety.
> > >
> > > Thanks,
> > >
> > > Alex Kritikos
> > > Software AG
> > > This communication contains information which is confidential and
> > > may
> > > also be privileged. It is for the exclusive use of the intended
> > > recipient(s). If you are not the intended recipient(s), please
> > > note
> > > that any distribution, copying, or use of this communication or
> > > the
> > > information in it, is strictly prohibited. If you have received
> > > this
> > > communication in error please notify us by e-mail and then delete
> > > the
> > > e-mail and any copies of it.
> > > Software AG (UK) Limited Registered in England & Wales 1310740 -
> > > http://www.softwareag.com/uk
> > >
>
> This communication contains information which is confidential and may
> also be privileged. It is for the exclusive use of the intended
> recipient(s). If you are not the intended recipient(s), please note
> that any distribution, copying, or use of this communication or the
> information in it, is strictly prohibited. If you have received this
> communication in error please notify us by e-mail and then delete the
> e-mail and any copies of it.
> Software AG (UK) Limited Registered in England & Wales 1310740 -
> http://www.softwareag.com/uk
>
Re: [Resending] - Proton-J engine and thread safety
Posted by Timothy Bish <ta...@gmail.com>.
On 06/10/2015 10:18 AM, Kritikos, Alex wrote:
> Hi Alan,
>
> thanks for your response. We also use an engine per connection however there are different read and write threads interacting with it and the issues only occur under load.
> I guess we should try to create a repro case.
>
> Thanks,
>
> Alex Kritikos
> Software AG
> On 10 Jun 2015, at 16:50, aconway <ac...@redhat.com> wrote:
To my knowledge there is not expectation the Proton-J is any more thread
safe than Proton-C. In both the new QPid JMS client and in the ActiveMQ
broker that uses Proton-J we serialize access to the engine for that
reason.
>> On Wed, 2015-06-10 at 09:34 +0000, Kritikos, Alex wrote:
>>> [Resending as it ended up in the wrong thread]
>>>
>>> Hi all,
>>>
>>> is the proton-j engine meant to be thread safe?
>> The C engine is definitely NOT meant to be thread safe. Unless you have
>> found an explicit written declaration that the java engine is supposed
>> to be AND a bunch of code to back that up I wouldn't rely on it.
>>
>> The way we use proton in the C++ broker and in the upcoming Go binding
>> is to create an engine per connection and serialize the action on each
>> connection. In principle you can read and write from the OS connection
>> concurrently but it's debatable how much you gain, you are likely just
>> moving OS buffers into app buffers which is not a big win.
>>
>> The inbound and outbound protocol state *for a single connection* is
>> pretty closely tied together. Proton is probably taking the right
>> approach by assuming both are handled in a single concurrency context.
>>
>> The engine state for separate connections is *completely independent*
>> so it's safe to run engines for separate connections in separte
>> contexts.
>>
>> The recent reactor extensions to proton are interesting but not thread
>> friendly. They force the protocol handling for multiple connections
>> into a single thread context, which is great for single threaded apps
>> but IMO the wrong way to go for concurrent apps.
>>
>> The go binding uses channels to pump data from connection read/write
>> goroutines to a proton engine event loop goroutine per connection. The
>> C++ broker predates the reactor and does it's own polling with
>> read/write activity on an FD handled dispatched sequentially to worker
>> threads so the proton engine for a connection is never used
>> concurrently.
>>
>> There may be something interesting we can do at the proton layer to
>> help with this pattern or it may be better to leave concurrency above
>> the binding to be handled by the languages own concurrency tools, I am
>> not sure yet.
>>
>>
>>> We have been experiencing some sporadic issues where under load, the
>>> engine sends callbacks to registered handlers with null as the event.
>>> We do not have a standalone repro case yet but just wondered what
>>> other people’s experience is as well as what are the recommendations
>>> around thread safety.
>>>
>>> Thanks,
>>>
>>> Alex Kritikos
>>> Software AG
>>> This communication contains information which is confidential and may
>>> also be privileged. It is for the exclusive use of the intended
>>> recipient(s). If you are not the intended recipient(s), please note
>>> that any distribution, copying, or use of this communication or the
>>> information in it, is strictly prohibited. If you have received this
>>> communication in error please notify us by e-mail and then delete the
>>> e-mail and any copies of it.
>>> Software AG (UK) Limited Registered in England & Wales 1310740 -
>>> http://www.softwareag.com/uk
>>>
> This communication contains information which is confidential and may also be privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s), please note that any distribution, copying, or use of this communication or the information in it, is strictly prohibited. If you have received this communication in error please notify us by e-mail and then delete the e-mail and any copies of it.
> Software AG (UK) Limited Registered in England & Wales 1310740 - http://www.softwareag.com/uk
>
>
--
Tim Bish
Sr Software Engineer | RedHat Inc.
tim.bish@redhat.com | www.redhat.com
twitter: @tabish121
blog: http://timbish.blogspot.com/
Re: [Resending] - Proton-J engine and thread safety
Posted by "Kritikos, Alex" <Al...@softwareag.com>.
Hi Alan,
thanks for your response. We also use an engine per connection however there are different read and write threads interacting with it and the issues only occur under load.
I guess we should try to create a repro case.
Thanks,
Alex Kritikos
Software AG
On 10 Jun 2015, at 16:50, aconway <ac...@redhat.com> wrote:
> On Wed, 2015-06-10 at 09:34 +0000, Kritikos, Alex wrote:
>> [Resending as it ended up in the wrong thread]
>>
>> Hi all,
>>
>> is the proton-j engine meant to be thread safe?
>
> The C engine is definitely NOT meant to be thread safe. Unless you have
> found an explicit written declaration that the java engine is supposed
> to be AND a bunch of code to back that up I wouldn't rely on it.
>
> The way we use proton in the C++ broker and in the upcoming Go binding
> is to create an engine per connection and serialize the action on each
> connection. In principle you can read and write from the OS connection
> concurrently but it's debatable how much you gain, you are likely just
> moving OS buffers into app buffers which is not a big win.
>
> The inbound and outbound protocol state *for a single connection* is
> pretty closely tied together. Proton is probably taking the right
> approach by assuming both are handled in a single concurrency context.
>
> The engine state for separate connections is *completely independent*
> so it's safe to run engines for separate connections in separte
> contexts.
>
> The recent reactor extensions to proton are interesting but not thread
> friendly. They force the protocol handling for multiple connections
> into a single thread context, which is great for single threaded apps
> but IMO the wrong way to go for concurrent apps.
>
> The go binding uses channels to pump data from connection read/write
> goroutines to a proton engine event loop goroutine per connection. The
> C++ broker predates the reactor and does it's own polling with
> read/write activity on an FD handled dispatched sequentially to worker
> threads so the proton engine for a connection is never used
> concurrently.
>
> There may be something interesting we can do at the proton layer to
> help with this pattern or it may be better to leave concurrency above
> the binding to be handled by the languages own concurrency tools, I am
> not sure yet.
>
>
>> We have been experiencing some sporadic issues where under load, the
>> engine sends callbacks to registered handlers with null as the event.
>> We do not have a standalone repro case yet but just wondered what
>> other people’s experience is as well as what are the recommendations
>> around thread safety.
>>
>> Thanks,
>>
>> Alex Kritikos
>> Software AG
>> This communication contains information which is confidential and may
>> also be privileged. It is for the exclusive use of the intended
>> recipient(s). If you are not the intended recipient(s), please note
>> that any distribution, copying, or use of this communication or the
>> information in it, is strictly prohibited. If you have received this
>> communication in error please notify us by e-mail and then delete the
>> e-mail and any copies of it.
>> Software AG (UK) Limited Registered in England & Wales 1310740 -
>> http://www.softwareag.com/uk
>>
This communication contains information which is confidential and may also be privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s), please note that any distribution, copying, or use of this communication or the information in it, is strictly prohibited. If you have received this communication in error please notify us by e-mail and then delete the e-mail and any copies of it.
Software AG (UK) Limited Registered in England & Wales 1310740 - http://www.softwareag.com/uk
Re: [Resending] - Proton-J engine and thread safety
Posted by aconway <ac...@redhat.com>.
On Wed, 2015-06-10 at 09:34 +0000, Kritikos, Alex wrote:
> [Resending as it ended up in the wrong thread]
>
> Hi all,
>
> is the proton-j engine meant to be thread safe?
The C engine is definitely NOT meant to be thread safe. Unless you have
found an explicit written declaration that the java engine is supposed
to be AND a bunch of code to back that up I wouldn't rely on it.
The way we use proton in the C++ broker and in the upcoming Go binding
is to create an engine per connection and serialize the action on each
connection. In principle you can read and write from the OS connection
concurrently but it's debatable how much you gain, you are likely just
moving OS buffers into app buffers which is not a big win.
The inbound and outbound protocol state *for a single connection* is
pretty closely tied together. Proton is probably taking the right
approach by assuming both are handled in a single concurrency context.
The engine state for separate connections is *completely independent*
so it's safe to run engines for separate connections in separte
contexts.
The recent reactor extensions to proton are interesting but not thread
friendly. They force the protocol handling for multiple connections
into a single thread context, which is great for single threaded apps
but IMO the wrong way to go for concurrent apps.
The go binding uses channels to pump data from connection read/write
goroutines to a proton engine event loop goroutine per connection. The
C++ broker predates the reactor and does it's own polling with
read/write activity on an FD handled dispatched sequentially to worker
threads so the proton engine for a connection is never used
concurrently.
There may be something interesting we can do at the proton layer to
help with this pattern or it may be better to leave concurrency above
the binding to be handled by the languages own concurrency tools, I am
not sure yet.
> We have been experiencing some sporadic issues where under load, the
> engine sends callbacks to registered handlers with null as the event.
> We do not have a standalone repro case yet but just wondered what
> other people’s experience is as well as what are the recommendations
> around thread safety.
>
> Thanks,
>
> Alex Kritikos
> Software AG
> This communication contains information which is confidential and may
> also be privileged. It is for the exclusive use of the intended
> recipient(s). If you are not the intended recipient(s), please note
> that any distribution, copying, or use of this communication or the
> information in it, is strictly prohibited. If you have received this
> communication in error please notify us by e-mail and then delete the
> e-mail and any copies of it.
> Software AG (UK) Limited Registered in England & Wales 1310740 -
> http://www.softwareag.com/uk
>