You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cxf.apache.org by Jason Pell <ja...@pellcorp.com> on 2013/11/06 23:22:51 UTC
cxf deadlock with spring 3.2?
Hi,
I am getting intermittent dead lock on startup of my application. The two
threads that are locking are accessing:
java.lang.Thread.State: BLOCKED (on object monitor)
INFO | jvm 1 | 2013/11/07 08:50:30 | at
org.apache.cxf.bus.extension.ExtensionManagerImpl.getExtension(ExtensionManagerImpl.java:257)
INFO | jvm 1 | 2013/11/07 08:50:30 | - waiting to lock
<0x00000007ee31c6b0> (a org.apache.cxf.bus.extension.ExtensionManagerImpl)
And...
java.lang.Thread.State: BLOCKED (on object monitor)
INFO | jvm 1 | 2013/11/07 08:50:29 | at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:181)
We did not notice this with spring 3.1.2
We also don't get this all the time. None of our developers get it on
their Linux Mint based machines, but we do on Centos 6.3.
But the JVMs are exactly the same - both 64bit and actually the same tar
ball.
Re: cxf deadlock with spring 3.2?
Posted by Daniel Kulp <dk...@apache.org>.
Committed some fixes for this to the 2.7.x branch. Would be great if you could try the snapshots. Since it’s not easily reproduced, I’m not really sure it’s fixed. If you see it again, definitely get new stack traces to me.
Dan
On Nov 8, 2013, at 4:46 AM, Jason Pell <ja...@pellcorp.com> wrote:
> We use the maven wait plugin to wait for the ?wsdl url to return a 200 code
> for the web services we need to use in soapui for integration builds.
>
> We are now investigating other ways of working around this.
>
> But i am relieved you replied as i am a little lost and reproducing the
> thread dead lock is not easy.
>
> It only happens on jenkins and not every time either.
>
> Its only started happening recently. We uograded our build box to have more
> ram and processors so maybe thats part of it.
> On 08/11/2013 6:43 PM, "Daniel Kulp" <dk...@apache.org> wrote:
>
>>
>> The irony of this was that I first looked at this yesterday while sitting
>> in a session at WJAX entitled “Concurrency: The Good, The Bad, The Ugly”.
>> :-(
>>
>> The GOOD news is this won’t occur with 3.0/trunk. The QueryHandlers are
>> gone and thus the one lookup wouldn’t happen.
>>
>>
>> It seems like your setup starts processing requests before all the
>> endpoints are fully published. That’s not a problem per say, but it is
>> causing this.
>>
>> The JettyHTTPDestination likely could grab the QueryHandlerRegistry and
>> stuff during it’s constructor to pre-cache the stuff it knows it’s going to
>> need. That may end up working around this. That’s won’t fix the same
>> issue with the ServletController though. Need to think about it a bit
>> more.
>>
>> My gut feeling is the synchronized blocks in ExtensionManagerImpl are too
>> coarse. I’ll try and look at this a bit on the plane back to the US.
>>
>>
>> Dan
>>
>>
>> On Nov 6, 2013, at 11:32 PM, Jason Pell <ja...@pellcorp.com> wrote:
>>
>>> Hi,
>>>
>>> I am continuing to try and understand the thread dump myself, but here it
>>> is in case anyone feels like giving me some pointers :-)
>>>
>>> http://pastebin.com/p75BtvPk
>>>
>>>
>>> On Thu, Nov 7, 2013 at 9:22 AM, Jason Pell <ja...@pellcorp.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am getting intermittent dead lock on startup of my application. The
>> two
>>>> threads that are locking are accessing:
>>>>
>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>> INFO | jvm 1 | 2013/11/07 08:50:30 | at
>>>>
>> org.apache.cxf.bus.extension.ExtensionManagerImpl.getExtension(ExtensionManagerImpl.java:257)
>>>> INFO | jvm 1 | 2013/11/07 08:50:30 | - waiting to lock
>>>> <0x00000007ee31c6b0> (a
>> org.apache.cxf.bus.extension.ExtensionManagerImpl)
>>>>
>>>> And...
>>>>
>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>> INFO | jvm 1 | 2013/11/07 08:50:29 | at
>>>>
>> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:181)
>>>>
>>>>
>>>> We did not notice this with spring 3.1.2
>>>>
>>>> We also don't get this all the time. None of our developers get it on
>>>> their Linux Mint based machines, but we do on Centos 6.3.
>>>>
>>>> But the JVMs are exactly the same - both 64bit and actually the same tar
>>>> ball.
>>>>
>>>>
>>>>
>>
>> --
>> Daniel Kulp
>> dkulp@apache.org - http://dankulp.com/blog
>> Talend Community Coder - http://coders.talend.com
>>
>>
--
Daniel Kulp
dkulp@apache.org - http://dankulp.com/blog
Talend Community Coder - http://coders.talend.com
Re: cxf deadlock with spring 3.2?
Posted by Jason Pell <ja...@pellcorp.com>.
We use the maven wait plugin to wait for the ?wsdl url to return a 200 code
for the web services we need to use in soapui for integration builds.
We are now investigating other ways of working around this.
But i am relieved you replied as i am a little lost and reproducing the
thread dead lock is not easy.
It only happens on jenkins and not every time either.
Its only started happening recently. We uograded our build box to have more
ram and processors so maybe thats part of it.
On 08/11/2013 6:43 PM, "Daniel Kulp" <dk...@apache.org> wrote:
>
> The irony of this was that I first looked at this yesterday while sitting
> in a session at WJAX entitled “Concurrency: The Good, The Bad, The Ugly”.
> :-(
>
> The GOOD news is this won’t occur with 3.0/trunk. The QueryHandlers are
> gone and thus the one lookup wouldn’t happen.
>
>
> It seems like your setup starts processing requests before all the
> endpoints are fully published. That’s not a problem per say, but it is
> causing this.
>
> The JettyHTTPDestination likely could grab the QueryHandlerRegistry and
> stuff during it’s constructor to pre-cache the stuff it knows it’s going to
> need. That may end up working around this. That’s won’t fix the same
> issue with the ServletController though. Need to think about it a bit
> more.
>
> My gut feeling is the synchronized blocks in ExtensionManagerImpl are too
> coarse. I’ll try and look at this a bit on the plane back to the US.
>
>
> Dan
>
>
> On Nov 6, 2013, at 11:32 PM, Jason Pell <ja...@pellcorp.com> wrote:
>
> > Hi,
> >
> > I am continuing to try and understand the thread dump myself, but here it
> > is in case anyone feels like giving me some pointers :-)
> >
> > http://pastebin.com/p75BtvPk
> >
> >
> > On Thu, Nov 7, 2013 at 9:22 AM, Jason Pell <ja...@pellcorp.com> wrote:
> >
> >> Hi,
> >>
> >> I am getting intermittent dead lock on startup of my application. The
> two
> >> threads that are locking are accessing:
> >>
> >> java.lang.Thread.State: BLOCKED (on object monitor)
> >> INFO | jvm 1 | 2013/11/07 08:50:30 | at
> >>
> org.apache.cxf.bus.extension.ExtensionManagerImpl.getExtension(ExtensionManagerImpl.java:257)
> >> INFO | jvm 1 | 2013/11/07 08:50:30 | - waiting to lock
> >> <0x00000007ee31c6b0> (a
> org.apache.cxf.bus.extension.ExtensionManagerImpl)
> >>
> >> And...
> >>
> >> java.lang.Thread.State: BLOCKED (on object monitor)
> >> INFO | jvm 1 | 2013/11/07 08:50:29 | at
> >>
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:181)
> >>
> >>
> >> We did not notice this with spring 3.1.2
> >>
> >> We also don't get this all the time. None of our developers get it on
> >> their Linux Mint based machines, but we do on Centos 6.3.
> >>
> >> But the JVMs are exactly the same - both 64bit and actually the same tar
> >> ball.
> >>
> >>
> >>
>
> --
> Daniel Kulp
> dkulp@apache.org - http://dankulp.com/blog
> Talend Community Coder - http://coders.talend.com
>
>
Re: cxf deadlock with spring 3.2?
Posted by Daniel Kulp <dk...@apache.org>.
The irony of this was that I first looked at this yesterday while sitting in a session at WJAX entitled “Concurrency: The Good, The Bad, The Ugly”. :-(
The GOOD news is this won’t occur with 3.0/trunk. The QueryHandlers are gone and thus the one lookup wouldn’t happen.
It seems like your setup starts processing requests before all the endpoints are fully published. That’s not a problem per say, but it is causing this.
The JettyHTTPDestination likely could grab the QueryHandlerRegistry and stuff during it’s constructor to pre-cache the stuff it knows it’s going to need. That may end up working around this. That’s won’t fix the same issue with the ServletController though. Need to think about it a bit more.
My gut feeling is the synchronized blocks in ExtensionManagerImpl are too coarse. I’ll try and look at this a bit on the plane back to the US.
Dan
On Nov 6, 2013, at 11:32 PM, Jason Pell <ja...@pellcorp.com> wrote:
> Hi,
>
> I am continuing to try and understand the thread dump myself, but here it
> is in case anyone feels like giving me some pointers :-)
>
> http://pastebin.com/p75BtvPk
>
>
> On Thu, Nov 7, 2013 at 9:22 AM, Jason Pell <ja...@pellcorp.com> wrote:
>
>> Hi,
>>
>> I am getting intermittent dead lock on startup of my application. The two
>> threads that are locking are accessing:
>>
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> INFO | jvm 1 | 2013/11/07 08:50:30 | at
>> org.apache.cxf.bus.extension.ExtensionManagerImpl.getExtension(ExtensionManagerImpl.java:257)
>> INFO | jvm 1 | 2013/11/07 08:50:30 | - waiting to lock
>> <0x00000007ee31c6b0> (a org.apache.cxf.bus.extension.ExtensionManagerImpl)
>>
>> And...
>>
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> INFO | jvm 1 | 2013/11/07 08:50:29 | at
>> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:181)
>>
>>
>> We did not notice this with spring 3.1.2
>>
>> We also don't get this all the time. None of our developers get it on
>> their Linux Mint based machines, but we do on Centos 6.3.
>>
>> But the JVMs are exactly the same - both 64bit and actually the same tar
>> ball.
>>
>>
>>
--
Daniel Kulp
dkulp@apache.org - http://dankulp.com/blog
Talend Community Coder - http://coders.talend.com
Re: cxf deadlock with spring 3.2?
Posted by Jason Pell <ja...@pellcorp.com>.
So the spring launcher and jaxws endpoint are both trying to access the
spring beans map at same time. I have not found a solution so far.
Sent from my Android phone
On 07/11/2013 9:32 AM, "Jason Pell" <ja...@pellcorp.com> wrote:
> Hi,
>
> I am continuing to try and understand the thread dump myself, but here it
> is in case anyone feels like giving me some pointers :-)
>
> http://pastebin.com/p75BtvPk
>
>
> On Thu, Nov 7, 2013 at 9:22 AM, Jason Pell <ja...@pellcorp.com> wrote:
>
>> Hi,
>>
>> I am getting intermittent dead lock on startup of my application. The
>> two threads that are locking are accessing:
>>
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> INFO | jvm 1 | 2013/11/07 08:50:30 | at
>> org.apache.cxf.bus.extension.ExtensionManagerImpl.getExtension(ExtensionManagerImpl.java:257)
>> INFO | jvm 1 | 2013/11/07 08:50:30 | - waiting to lock
>> <0x00000007ee31c6b0> (a org.apache.cxf.bus.extension.ExtensionManagerImpl)
>>
>> And...
>>
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> INFO | jvm 1 | 2013/11/07 08:50:29 | at
>> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:181)
>>
>>
>> We did not notice this with spring 3.1.2
>>
>> We also don't get this all the time. None of our developers get it on
>> their Linux Mint based machines, but we do on Centos 6.3.
>>
>> But the JVMs are exactly the same - both 64bit and actually the same tar
>> ball.
>>
>>
>>
>
Re: cxf deadlock with spring 3.2?
Posted by Jason Pell <ja...@pellcorp.com>.
Hi,
I am continuing to try and understand the thread dump myself, but here it
is in case anyone feels like giving me some pointers :-)
http://pastebin.com/p75BtvPk
On Thu, Nov 7, 2013 at 9:22 AM, Jason Pell <ja...@pellcorp.com> wrote:
> Hi,
>
> I am getting intermittent dead lock on startup of my application. The two
> threads that are locking are accessing:
>
> java.lang.Thread.State: BLOCKED (on object monitor)
> INFO | jvm 1 | 2013/11/07 08:50:30 | at
> org.apache.cxf.bus.extension.ExtensionManagerImpl.getExtension(ExtensionManagerImpl.java:257)
> INFO | jvm 1 | 2013/11/07 08:50:30 | - waiting to lock
> <0x00000007ee31c6b0> (a org.apache.cxf.bus.extension.ExtensionManagerImpl)
>
> And...
>
> java.lang.Thread.State: BLOCKED (on object monitor)
> INFO | jvm 1 | 2013/11/07 08:50:29 | at
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:181)
>
>
> We did not notice this with spring 3.1.2
>
> We also don't get this all the time. None of our developers get it on
> their Linux Mint based machines, but we do on Centos 6.3.
>
> But the JVMs are exactly the same - both 64bit and actually the same tar
> ball.
>
>
>