You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cxf.apache.org by xuhb <xu...@tongtech.com> on 2011/08/19 07:34:41 UTC
A mysteriously deadlock of CXF OnewayProcessorInterceptor
Hi:
Recently when I am checking/testing CXF , there is a mysteriously deadlock of CXF Oneway Process; Normally CXF engine will invoke the one way bussiness logical asynchronized ,, so the servlet handle will finished and return back to servlet engine immediately;
But sometime, I noticed that the servlet handle(JettyHTTPHandler) at server side doesn't return back to servlet engine(Jetty) immediately , it will waiting until the asynchrouse business logical finished;
After dig source of CXF, I find it 's relate to OnewayProcessorInterceptor;But until now I can only show when will the deadlock occurs, but I still can not explain why;
Following is details:
OnewayProcessInterceptor.handleMessage{
synchronized (chain) {
message.getExchange().get(Bus.class).getExtension(WorkQueueManager.class)
.getAutomaticWorkQueue().execute(new Runnable() {
public void run() {
synchronized (chain) {
System.out.println("--notify all");
chain.notifyAll();
}
chain.resume(); //if chain.resume is called before chain.wait finished , the dead lock will occurs; It seems as chain.resume is synchronized, so it will relock on chain object, so the chain.wait() will deadlocked (... I feel confused for this, because jdk doesn't say so...) ;After chain.resume finished, locking on chain is released, deadlock of chain.wait() is also released; but I don't think this is problem of CXF , maybe it's jdk's problem ?? I feels confused;
}
});
System.out.println("--wait begin");
chain.wait(20);
System.out.println("--wait end");
}
}
syncrhonized PhaseInterceptorChain.resume(){
System.out.println("--api chain resume");
...
}
if the execute sequence as following, every thing is ok. there is no dead lock;
chain.wait enter
chian.notify invoked
chain.wait return;
chain.resume(); //resume also synchronzed on chain object;
if the execute sequence as following , dead lock will occurs:
chain.wait enter
chain.notify
chain.resume// ..now waiting on chain will blocked until chain.resume finished(release sync on chain)
chain.wait return;
following dump on console indicate the above sequence:
No DeadLock dump :
--wait begin
--notify all
--wait end
--api chain resume
product service begin Fri Aug 19 12:10:28 CST 2011 //a lone time(10 seconds) one way business logical begin
product service end Fri Aug 19 12:10:38 CST 2011 .//a lone time(10 seconds) one way business logical end;
DeadLock Dump:
--wait begin
--notify all
--api chain resume
product service begin Fri Aug 19 12:10:40 CST 2011
product service end Fri Aug 19 12:10:50 CST 2011
--wait end
Until now I am not sure if problem is CXF's or JDK's, or something which I don't know cause such a deadlock;
I also wrote a simple program to simulate the execute sequnce which causeddead lock in CXF, but the simple program never dead lock;
I tried CXF 2.3.3 version && Jetty transport && (JDK1.5_22 || JDK 1.6_17) && Windows XP system;
I also post this question as a JIRA issue:
Re: A mysteriously deadlock of CXF OnewayProcessorInterceptor(A question about CXF chain interceptor mechanism)
Posted by Daniel Kulp <dk...@apache.org>.
On Monday, August 22, 2011 1:00:06 PM xuhb wrote:
> Hi, Dan
> Unfortunately, the fixed for CXF-3750 may cause another question. which I
> am not sure if it's a issuem of CXF or things the user should take care;
> Details I have comments in CXF-3750;
OK. This should now be fixed.
Dan
>
>
>
> ----- Original Message -----
> From: "xuhb" <xu...@tongtech.com>
> To: <us...@cxf.apache.org>
> Sent: Monday, August 22, 2011 11:06 AM
> Subject: Re: A mysteriously deadlock of CXF OnewayProcessorInterceptor
>
>
>
> > Thanks Dan:
> >
> > I tried, it's ok now;
> > thanks to your rapid fix;
> >
> > ----- Original Message -----
> > From: "Daniel Kulp" <dk...@apache.org>
> > To: <us...@cxf.apache.org>
> > Cc: "xuhb" <xu...@tongtech.com>
> > Sent: Saturday, August 20, 2011 1:21 AM
> > Subject: Re: A mysteriously deadlock of CXF OnewayProcessorInterceptor
> >
> >
> >
> >>
> >> I just committed a fix to trunk (pulling back to 2.4.x now as well).
> >> When
you get a chance, can you give it a try to see if it fixes
> >> your issue? Threading things like these are hard to really test.
> >>
> >> Dan
> >>
> >>
> >> On Friday, August 19, 2011 2:32:03 PM xuhb wrote:
> >>
> >>> Maybe it is becuase before chain.wait() return, the sync of chain
> >>> will be
re-locked, so it will block untill chain.resume()
> >>> finished;
> >>
> >>
> >>
> >>> ----- Original Message -----
> >>> From: "xuhb" <xu...@tongtech.com>
> >>> To: <us...@cxf.apache.org>
> >>> Sent: Friday, August 19, 2011 1:39 PM
> >>> Subject: Re: A mysteriously deadlock of CXF
> >>> OnewayProcessorInterceptor
> >>>
> >>>
> >>>
> >>>
> >>> > Sorry, I foget post the issue link:
> >>> >
> >>> > https://issues.apache.org/jira/browse/CXF-3750
> >>> >
> >>> > ----- Original Message -----
> >>> > From: "xuhb" <xu...@tongtech.com>
> >>> > To: <us...@cxf.apache.org>
> >>> > Sent: Friday, August 19, 2011 1:34 PM
> >>> > Subject: A mysteriously deadlock of CXF
> >>> > OnewayProcessorInterceptor
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >> Hi:
> >>> >>
> >>> >>
> >>> >> Recently when I am checking/testing CXF , there is a
> >>> >> mysteriously deadlock of CXF Oneway Process; Normally
> >>> >> CXF
> >>> >> engine will invoke the one way bussiness logical
> >>> >> asynchronized
> >>> >> ,, so the servlet handle will finished and return back
> >>> >> to
> >>> >> servlet engine immediately;
> >>>>
> >>>>
> >>>>
> >>> >>
> >>> >>
> >>> >>
> >>> >> But sometime, I noticed that the servlet
> >>> >> handle(JettyHTTPHandler) at server side doesn't return
> >>> >> back to
> >>> >> servlet engine(Jetty) immediately , it will waiting
> >>> >> until the
> >>> >> asynchrouse business logical finished;
> >>
> >> After dig source of
> >>
> >>> >> CXF, I find it 's relate to
> >>> >> OnewayProcessorInterceptor;But
> >>> >> until now I can only show when will the deadlock
> >>> >> occurs, but I
> >>> >> still can not explain why;>>
> >>> >>
> >>> >>
> >>> >> Following is details:
> >>> >> OnewayProcessInterceptor.handleMessage{
> >>> >> synchronized (chain) {
> >>> >> message.getExchange().get(Bus.class).getExtension(WorkQueueMan
> >>> >> ager.cla
ss)
> >>
> >> .getAutomaticWorkQueue().execute(new Runnable() {
> >>
> >>> >>
> >>> >>
> >>> >> public void run() {
> >>> >> synchronized (chain) {
> >>> >>
> >>> >>
> >>> >> System.out.println("--notify all");
> >>> >> chain.notifyAll();
> >>> >>
> >>> >>
> >>> >> }
> >>> >>
> >>> >> chain.resume(); //if chain.resume is called before
> >>> >> chain.wait
> >>> >> finished , the dead lock will occurs; It seems as
> >>> >> chain.resume is synchronized, so it will relock on
> >>> >> chain
> >>> >> object, so the chain.wait() will deadlocked (... I feel
> >>> >> confused for this, because jdk doesn't say so...)
> >>> >> ;After
> >>> >> chain.resume finished, locking on chain is released,
> >>> >> deadlock
> >>> >> of chain.wait() is also released; but I don't think
> >>> >> this is
> >>> >> problem of CXF , maybe it's jdk's problem ?? I feels
> >>> >> confused;
> >>>>
> >>>>
> >>>>
> >>> >> }
> >>> >>
> >>> >>
> >>> >> });
> >>> >>
> >>> >>
> >>> >> System.out.println("--wait begin");
> >>> >> chain.wait(20);
> >>> >> System.out.println("--wait end");
> >>> >>
> >>> >>
> >>> >> }
> >>> >> }
> >>> >> syncrhonized PhaseInterceptorChain.resume(){
> >>> >>
> >>> >>
> >>> >> System.out.println("--api chain resume");
> >>> >>
> >>> >>
> >>> >> ...
> >>> >>
> >>> >>
> >>> >> }
> >>> >>
> >>> >> if the execute sequence as following, every thing is ok.
> >>> >> there is no
dead lock;
> >>>>
> >>>>
> >>>>
> >>> >> chain.wait enter
> >>> >> chian.notify invoked
> >>> >> chain.wait return;
> >>> >> chain.resume(); //resume also synchronzed on chain
> >>> >> object;
> >>> >>
> >>> >>
> >>> >>
> >>> >> if the execute sequence as following , dead lock will occurs:
> >>> >>
> >>> >>
> >>> >> chain.wait enter
> >>> >> chain.notify
> >>> >> chain.resume// ..now waiting on chain will blocked
> >>> >> until
> >>> >> chain.resume finished(release sync on chain)
> >>
> >> chain.wait
> >>
> >>> >> return;
> >>> >>
> >>> >>
> >>> >>
> >>> >> following dump on console indicate the above sequence:
> >>> >>
> >>> >> No DeadLock dump :
> >>> >> --wait begin
> >>> >> --notify all
> >>> >> --wait end
> >>> >> --api chain resume
> >>> >> product service begin Fri Aug 19 12:10:28 CST 2011 //a lone
> >>> >> time(10
> >>> >> seconds) one way business logical begin
> >>
> >> product service end Fri Aug
> >>
> >>> >> 19 12:10:38 CST 2011 .//a lone time(10 seconds) one way
> >>> >> business
> >>> >> logical end;
> >>> >>
> >>> >> DeadLock Dump:
> >>> >> --wait begin
> >>> >> --notify all
> >>> >> --api chain resume
> >>> >> product service begin Fri Aug 19 12:10:40 CST 2011
> >>> >> product service end Fri Aug 19 12:10:50 CST 2011
> >>> >> --wait end
> >>> >>
> >>> >>
> >>> >> Until now I am not sure if problem is CXF's or JDK's, or
> >>> >> something
> >>> >> which I don't know cause such a deadlock;
> >>
> >> I also wrote a simple
> >>
> >>> >> program to simulate the execute sequnce which causeddead lock
> >>> >> in
> >>> >> CXF, but the simple program never dead lock;
> >>> >> I tried CXF 2.3.3 version && Jetty transport && (JDK1.5_22 ||
> >>> >> JDK
> >>> >> 1.6_17) && Windows XP system;
> >>
> >>
> >>
> >>> >> I also post this question as a JIRA issue:
> >>
> >>
> >> --
> >> Daniel Kulp
> >> dkulp@apache.org
> >> http://dankulp.com/blog
> >> Talend - http://www.talend.com
--
Daniel Kulp
dkulp@apache.org
http://dankulp.com/blog
Talend - http://www.talend.com
A question about CXF chain interceptor mechanism
Posted by xuhb <xu...@tongtech.com>.
Hi:
CXF Chain Interceptor is simple and good mechanism for protocol interpret;
But sometimes it seem "Chain Interceptor " mechanism is not very suitable for things such as system resource release; To illustrate this, following is a sample:
1) Recently I am using CXF to expose an existing system as a webservice;
When asking for data from the existing system , it will generate a temporary file and return it as a result; That's to say: when I expose it as a webservice, the temporary file must be deleted after the webservice response is sent over(no mattacher successfully or fault);
At first, I think the it's simple and easy by using a interceptor to delete the temporary file; and configure it in Out & FaultOut Chains;
But actually, things is not some simple. Because what I really want is some mechanism which is like "try{interceptor-chains}finally {do-some-system-resource-release}" ; Chain Interceptors may be aborted by some internal failure (and it should be so because it's designed for protocol interpret) . So if I use interceptor to delete the temporary file, sometimes the temporary file cannot be deleted (for example: network break down while sending soap-response, the Out Chain will be aborted, and temporary file cannot be deleted);
So I wonderring if CXF support mechanism suitable for such a usage beside just using interceptors ?
2) Maybe following is another sample:
Recently while I looking into CXF's source, it seems some interceptor of CXF also do something which like system-resource release. If so , I guess, sometimes this will also cause issues.
For example:
OutgoingChainInterceptor.handleMessage(){
if (null != bin && null != bin.getOperationInfo() && bin.getOperationInfo().isOneWay()) {
closeInput(message);
return;
}
}
OutgoingChainInterceptor will release system resources relate to input stream for one-way operation, but it's not always be guaranteed invoked;
For example if the one-way business logical throw a runtime-exception , it will breaked the interceptor chain, and OutgoingChainInterceptor will not be called. So the input stream's underly system resource may not be released; (I have show a real problem relate to OutgoingChainInterceptor in another issue CXF-3750's comments).
Re: A mysteriously deadlock of CXF OnewayProcessorInterceptor(A question about CXF chain interceptor mechanism)
Posted by xuhb <xu...@tongtech.com>.
Hi, Dan
Unfortunately, the fixed for CXF-3750 may cause another question. which I am not sure if it's a issuem of CXF or things the user should take care;
Details I have comments in CXF-3750;
----- Original Message -----
From: "xuhb" <xu...@tongtech.com>
To: <us...@cxf.apache.org>
Sent: Monday, August 22, 2011 11:06 AM
Subject: Re: A mysteriously deadlock of CXF OnewayProcessorInterceptor
> Thanks Dan:
>
> I tried, it's ok now;
> thanks to your rapid fix;
>
> ----- Original Message -----
> From: "Daniel Kulp" <dk...@apache.org>
> To: <us...@cxf.apache.org>
> Cc: "xuhb" <xu...@tongtech.com>
> Sent: Saturday, August 20, 2011 1:21 AM
> Subject: Re: A mysteriously deadlock of CXF OnewayProcessorInterceptor
>
>
>>
>> I just committed a fix to trunk (pulling back to 2.4.x now as well). When
>> you get a chance, can you give it a try to see if it fixes your issue?
>> Threading things like these are hard to really test.
>>
>> Dan
>>
>>
>> On Friday, August 19, 2011 2:32:03 PM xuhb wrote:
>>> Maybe it is becuase before chain.wait() return, the sync of chain will be
>>> re-locked, so it will block untill chain.resume() finished;
>>
>>> ----- Original Message -----
>>> From: "xuhb" <xu...@tongtech.com>
>>> To: <us...@cxf.apache.org>
>>> Sent: Friday, August 19, 2011 1:39 PM
>>> Subject: Re: A mysteriously deadlock of CXF OnewayProcessorInterceptor
>>>
>>>
>>>
>>> > Sorry, I foget post the issue link:
>>> >
>>> > https://issues.apache.org/jira/browse/CXF-3750
>>> >
>>> > ----- Original Message -----
>>> > From: "xuhb" <xu...@tongtech.com>
>>> > To: <us...@cxf.apache.org>
>>> > Sent: Friday, August 19, 2011 1:34 PM
>>> > Subject: A mysteriously deadlock of CXF OnewayProcessorInterceptor
>>> >
>>> >
>>> >
>>> >> Hi:
>>> >>
>>> >> Recently when I am checking/testing CXF , there is a
>>> >> mysteriously deadlock of CXF Oneway Process; Normally CXF
>>> >> engine will invoke the one way bussiness logical asynchronized
>>> >> ,, so the servlet handle will finished and return back to
>>> >> servlet engine immediately;
>>>>
>>> >>
>>> >>
>>> >> But sometime, I noticed that the servlet
>>> >> handle(JettyHTTPHandler) at server side doesn't return back to
>>> >> servlet engine(Jetty) immediately , it will waiting until the
>>> >> asynchrouse business logical finished;
>> After dig source of
>>> >> CXF, I find it 's relate to OnewayProcessorInterceptor;But
>>> >> until now I can only show when will the deadlock occurs, but I
>>> >> still can not explain why;>>
>>> >>
>>> >> Following is details:
>>> >> OnewayProcessInterceptor.handleMessage{
>>> >> synchronized (chain) {
>>> >> message.getExchange().get(Bus.class).getExtension(WorkQueueManager.cla
>>> >> ss)
>> .getAutomaticWorkQueue().execute(new Runnable() {
>>> >>
>>> >> public void run() {
>>> >> synchronized (chain) {
>>> >>
>>> >> System.out.println("--notify all");
>>> >> chain.notifyAll();
>>> >>
>>> >> }
>>> >>
>>> >> chain.resume(); //if chain.resume is called before chain.wait
>>> >> finished , the dead lock will occurs; It seems as
>>> >> chain.resume is synchronized, so it will relock on chain
>>> >> object, so the chain.wait() will deadlocked (... I feel
>>> >> confused for this, because jdk doesn't say so...) ;After
>>> >> chain.resume finished, locking on chain is released, deadlock
>>> >> of chain.wait() is also released; but I don't think this is
>>> >> problem of CXF , maybe it's jdk's problem ?? I feels confused;
>>>>
>>> >> }
>>> >>
>>> >> });
>>> >>
>>> >> System.out.println("--wait begin");
>>> >> chain.wait(20);
>>> >> System.out.println("--wait end");
>>> >>
>>> >> }
>>> >> }
>>> >> syncrhonized PhaseInterceptorChain.resume(){
>>> >>
>>> >> System.out.println("--api chain resume");
>>> >>
>>> >> ...
>>> >>
>>> >> }
>>> >>
>>> >> if the execute sequence as following, every thing is ok. there is no
>>> >> dead lock;
>>>>
>>> >> chain.wait enter
>>> >> chian.notify invoked
>>> >> chain.wait return;
>>> >> chain.resume(); //resume also synchronzed on chain object;
>>> >>
>>> >>
>>> >> if the execute sequence as following , dead lock will occurs:
>>> >>
>>> >> chain.wait enter
>>> >> chain.notify
>>> >> chain.resume// ..now waiting on chain will blocked until
>>> >> chain.resume finished(release sync on chain)
>> chain.wait
>>> >> return;
>>> >>
>>> >>
>>> >> following dump on console indicate the above sequence:
>>> >>
>>> >> No DeadLock dump :
>>> >> --wait begin
>>> >> --notify all
>>> >> --wait end
>>> >> --api chain resume
>>> >> product service begin Fri Aug 19 12:10:28 CST 2011 //a lone time(10
>>> >> seconds) one way business logical begin
>> product service end Fri Aug
>>> >> 19 12:10:38 CST 2011 .//a lone time(10 seconds) one way business
>>> >> logical end;
>>> >>
>>> >> DeadLock Dump:
>>> >> --wait begin
>>> >> --notify all
>>> >> --api chain resume
>>> >> product service begin Fri Aug 19 12:10:40 CST 2011
>>> >> product service end Fri Aug 19 12:10:50 CST 2011
>>> >> --wait end
>>> >>
>>> >>
>>> >> Until now I am not sure if problem is CXF's or JDK's, or something
>>> >> which I don't know cause such a deadlock;
>> I also wrote a simple
>>> >> program to simulate the execute sequnce which causeddead lock in
>>> >> CXF, but the simple program never dead lock;
>>> >> I tried CXF 2.3.3 version && Jetty transport && (JDK1.5_22 || JDK
>>> >> 1.6_17) && Windows XP system;
>>
>>> >> I also post this question as a JIRA issue:
>>
>> --
>> Daniel Kulp
>> dkulp@apache.org
>> http://dankulp.com/blog
>> Talend - http://www.talend.com
>>
Re: A mysteriously deadlock of CXF OnewayProcessorInterceptor
Posted by xuhb <xu...@tongtech.com>.
Thanks Dan:
I tried, it's ok now;
thanks to your rapid fix;
----- Original Message -----
From: "Daniel Kulp" <dk...@apache.org>
To: <us...@cxf.apache.org>
Cc: "xuhb" <xu...@tongtech.com>
Sent: Saturday, August 20, 2011 1:21 AM
Subject: Re: A mysteriously deadlock of CXF OnewayProcessorInterceptor
>
> I just committed a fix to trunk (pulling back to 2.4.x now as well). When
> you get a chance, can you give it a try to see if it fixes your issue?
> Threading things like these are hard to really test.
>
> Dan
>
>
> On Friday, August 19, 2011 2:32:03 PM xuhb wrote:
>> Maybe it is becuase before chain.wait() return, the sync of chain will be
>> re-locked, so it will block untill chain.resume() finished;
>
>> ----- Original Message -----
>> From: "xuhb" <xu...@tongtech.com>
>> To: <us...@cxf.apache.org>
>> Sent: Friday, August 19, 2011 1:39 PM
>> Subject: Re: A mysteriously deadlock of CXF OnewayProcessorInterceptor
>>
>>
>>
>> > Sorry, I foget post the issue link:
>> >
>> > https://issues.apache.org/jira/browse/CXF-3750
>> >
>> > ----- Original Message -----
>> > From: "xuhb" <xu...@tongtech.com>
>> > To: <us...@cxf.apache.org>
>> > Sent: Friday, August 19, 2011 1:34 PM
>> > Subject: A mysteriously deadlock of CXF OnewayProcessorInterceptor
>> >
>> >
>> >
>> >> Hi:
>> >>
>> >> Recently when I am checking/testing CXF , there is a
>> >> mysteriously deadlock of CXF Oneway Process; Normally CXF
>> >> engine will invoke the one way bussiness logical asynchronized
>> >> ,, so the servlet handle will finished and return back to
>> >> servlet engine immediately;
>>>
>> >>
>> >>
>> >> But sometime, I noticed that the servlet
>> >> handle(JettyHTTPHandler) at server side doesn't return back to
>> >> servlet engine(Jetty) immediately , it will waiting until the
>> >> asynchrouse business logical finished;
> After dig source of
>> >> CXF, I find it 's relate to OnewayProcessorInterceptor;But
>> >> until now I can only show when will the deadlock occurs, but I
>> >> still can not explain why;>>
>> >>
>> >> Following is details:
>> >> OnewayProcessInterceptor.handleMessage{
>> >> synchronized (chain) {
>> >> message.getExchange().get(Bus.class).getExtension(WorkQueueManager.cla
>> >> ss)
> .getAutomaticWorkQueue().execute(new Runnable() {
>> >>
>> >> public void run() {
>> >> synchronized (chain) {
>> >>
>> >> System.out.println("--notify all");
>> >> chain.notifyAll();
>> >>
>> >> }
>> >>
>> >> chain.resume(); //if chain.resume is called before chain.wait
>> >> finished , the dead lock will occurs; It seems as
>> >> chain.resume is synchronized, so it will relock on chain
>> >> object, so the chain.wait() will deadlocked (... I feel
>> >> confused for this, because jdk doesn't say so...) ;After
>> >> chain.resume finished, locking on chain is released, deadlock
>> >> of chain.wait() is also released; but I don't think this is
>> >> problem of CXF , maybe it's jdk's problem ?? I feels confused;
>>>
>> >> }
>> >>
>> >> });
>> >>
>> >> System.out.println("--wait begin");
>> >> chain.wait(20);
>> >> System.out.println("--wait end");
>> >>
>> >> }
>> >> }
>> >> syncrhonized PhaseInterceptorChain.resume(){
>> >>
>> >> System.out.println("--api chain resume");
>> >>
>> >> ...
>> >>
>> >> }
>> >>
>> >> if the execute sequence as following, every thing is ok. there is no
>> >> dead lock;
>>>
>> >> chain.wait enter
>> >> chian.notify invoked
>> >> chain.wait return;
>> >> chain.resume(); //resume also synchronzed on chain object;
>> >>
>> >>
>> >> if the execute sequence as following , dead lock will occurs:
>> >>
>> >> chain.wait enter
>> >> chain.notify
>> >> chain.resume// ..now waiting on chain will blocked until
>> >> chain.resume finished(release sync on chain)
> chain.wait
>> >> return;
>> >>
>> >>
>> >> following dump on console indicate the above sequence:
>> >>
>> >> No DeadLock dump :
>> >> --wait begin
>> >> --notify all
>> >> --wait end
>> >> --api chain resume
>> >> product service begin Fri Aug 19 12:10:28 CST 2011 //a lone time(10
>> >> seconds) one way business logical begin
> product service end Fri Aug
>> >> 19 12:10:38 CST 2011 .//a lone time(10 seconds) one way business
>> >> logical end;
>> >>
>> >> DeadLock Dump:
>> >> --wait begin
>> >> --notify all
>> >> --api chain resume
>> >> product service begin Fri Aug 19 12:10:40 CST 2011
>> >> product service end Fri Aug 19 12:10:50 CST 2011
>> >> --wait end
>> >>
>> >>
>> >> Until now I am not sure if problem is CXF's or JDK's, or something
>> >> which I don't know cause such a deadlock;
> I also wrote a simple
>> >> program to simulate the execute sequnce which causeddead lock in
>> >> CXF, but the simple program never dead lock;
>> >> I tried CXF 2.3.3 version && Jetty transport && (JDK1.5_22 || JDK
>> >> 1.6_17) && Windows XP system;
>
>> >> I also post this question as a JIRA issue:
>
> --
> Daniel Kulp
> dkulp@apache.org
> http://dankulp.com/blog
> Talend - http://www.talend.com
>
Re: A mysteriously deadlock of CXF OnewayProcessorInterceptor
Posted by Daniel Kulp <dk...@apache.org>.
I just committed a fix to trunk (pulling back to 2.4.x now as well). When
you get a chance, can you give it a try to see if it fixes your issue?
Threading things like these are hard to really test.
Dan
On Friday, August 19, 2011 2:32:03 PM xuhb wrote:
> Maybe it is becuase before chain.wait() return, the sync of chain will be
> re-locked, so it will block untill chain.resume() finished;
> ----- Original Message -----
> From: "xuhb" <xu...@tongtech.com>
> To: <us...@cxf.apache.org>
> Sent: Friday, August 19, 2011 1:39 PM
> Subject: Re: A mysteriously deadlock of CXF OnewayProcessorInterceptor
>
>
>
> > Sorry, I foget post the issue link:
> >
> > https://issues.apache.org/jira/browse/CXF-3750
> >
> > ----- Original Message -----
> > From: "xuhb" <xu...@tongtech.com>
> > To: <us...@cxf.apache.org>
> > Sent: Friday, August 19, 2011 1:34 PM
> > Subject: A mysteriously deadlock of CXF OnewayProcessorInterceptor
> >
> >
> >
> >> Hi:
> >>
> >> Recently when I am checking/testing CXF , there is a
> >> mysteriously deadlock of CXF Oneway Process; Normally CXF
> >> engine will invoke the one way bussiness logical asynchronized
> >> ,, so the servlet handle will finished and return back to
> >> servlet engine immediately;
>>
> >>
> >>
> >> But sometime, I noticed that the servlet
> >> handle(JettyHTTPHandler) at server side doesn't return back to
> >> servlet engine(Jetty) immediately , it will waiting until the
> >> asynchrouse business logical finished;
After dig source of
> >> CXF, I find it 's relate to OnewayProcessorInterceptor;But
> >> until now I can only show when will the deadlock occurs, but I
> >> still can not explain why;>>
> >>
> >> Following is details:
> >> OnewayProcessInterceptor.handleMessage{
> >> synchronized (chain) {
> >> message.getExchange().get(Bus.class).getExtension(WorkQueueManager.cla
> >> ss)
.getAutomaticWorkQueue().execute(new Runnable() {
> >>
> >> public void run() {
> >> synchronized (chain) {
> >>
> >> System.out.println("--notify all");
> >> chain.notifyAll();
> >>
> >> }
> >>
> >> chain.resume(); //if chain.resume is called before chain.wait
> >> finished , the dead lock will occurs; It seems as
> >> chain.resume is synchronized, so it will relock on chain
> >> object, so the chain.wait() will deadlocked (... I feel
> >> confused for this, because jdk doesn't say so...) ;After
> >> chain.resume finished, locking on chain is released, deadlock
> >> of chain.wait() is also released; but I don't think this is
> >> problem of CXF , maybe it's jdk's problem ?? I feels confused;
>>
> >> }
> >>
> >> });
> >>
> >> System.out.println("--wait begin");
> >> chain.wait(20);
> >> System.out.println("--wait end");
> >>
> >> }
> >> }
> >> syncrhonized PhaseInterceptorChain.resume(){
> >>
> >> System.out.println("--api chain resume");
> >>
> >> ...
> >>
> >> }
> >>
> >> if the execute sequence as following, every thing is ok. there is no
> >> dead lock;
>>
> >> chain.wait enter
> >> chian.notify invoked
> >> chain.wait return;
> >> chain.resume(); //resume also synchronzed on chain object;
> >>
> >>
> >> if the execute sequence as following , dead lock will occurs:
> >>
> >> chain.wait enter
> >> chain.notify
> >> chain.resume// ..now waiting on chain will blocked until
> >> chain.resume finished(release sync on chain)
chain.wait
> >> return;
> >>
> >>
> >> following dump on console indicate the above sequence:
> >>
> >> No DeadLock dump :
> >> --wait begin
> >> --notify all
> >> --wait end
> >> --api chain resume
> >> product service begin Fri Aug 19 12:10:28 CST 2011 //a lone time(10
> >> seconds) one way business logical begin
product service end Fri Aug
> >> 19 12:10:38 CST 2011 .//a lone time(10 seconds) one way business
> >> logical end;
> >>
> >> DeadLock Dump:
> >> --wait begin
> >> --notify all
> >> --api chain resume
> >> product service begin Fri Aug 19 12:10:40 CST 2011
> >> product service end Fri Aug 19 12:10:50 CST 2011
> >> --wait end
> >>
> >>
> >> Until now I am not sure if problem is CXF's or JDK's, or something
> >> which I don't know cause such a deadlock;
I also wrote a simple
> >> program to simulate the execute sequnce which causeddead lock in
> >> CXF, but the simple program never dead lock;
> >> I tried CXF 2.3.3 version && Jetty transport && (JDK1.5_22 || JDK
> >> 1.6_17) && Windows XP system;
> >> I also post this question as a JIRA issue:
--
Daniel Kulp
dkulp@apache.org
http://dankulp.com/blog
Talend - http://www.talend.com
Re: A mysteriously deadlock of CXF OnewayProcessorInterceptor
Posted by Aki Yoshida <el...@googlemail.com>.
2011/8/24 Daniel Kulp <dk...@apache.org>:
> On Wednesday, August 24, 2011 1:57:59 PM Aki Yoshida wrote:
>> 2011/8/22 Daniel Kulp <dk...@apache.org>:
>> > On Monday, August 22, 2011 10:26:36 AM Aki Yoshida wrote:
>> >> Hi Dan,
>> >> I was confused about the original intention of this wait(20) code. I
>> >> didn't understand why it was important to wait 20msec and let the
>> >> executor thread start instead of not waiting. The comment in the code
>> >> says the intention is to try to keep the ordering of the messages from
>> >> the client. But I had an impression that this code would not be
>> >> helping in this matter. So I initially thought that we could just get
>> >> rid of this wait to avoid the reported blocking situation. If keeping
>> >> the ordering is important, shouldn't we introduce this mechanism
>> >> optionally instead of trying to put this in there implicitly?
>> >
>> > Basically, if a client proxy sends a one-way request, and then
>> > immediately follows it up with a two-way request, we were seeing (even
>> > in our unit tests) where, in most situations, the two-way would get
>> > dispatched into the impl before the one-way. While perfectly valid,
>> > this was kind of confusing to users. Thus we added a very short wait
>> > there to make sure the runnable has been started on a thread which
>> > usually allows it to proceed something closer to in order. If the
>> > queue is full or all threads are busy, the wait will timeout and the
>> > one-way will get there eventually, but in most cases this works fairly
>> > well.
>> >
>> > Basically, it's a "best effort, but no guarantees". If they need to
>> > make sure it's there in order, they would need two ways or use WS-RM.
>>
>> Thanks for the explanation. I am sorry to hear that some people are
>> not happy when a oneway call gets passed by a later invoked req-resp
>> call. I personally think we could get rid of this wait(20) to make a
>> oneway call more efficient.
>
> In MOST cases, it doesn't wait for 20 ms, it's usually on the order of nano-
> seconds. It's only when the queue starts filling with lots of one-ways does
> it ever wait that long. That said, I'd definitely support dropping this down
> to a wait(5) or so. I'm really not sure why 20 was chosen. Even a
> "Thread.yield()" may work, but the wait with the lock is more accurate,
> especially on multi-core machines.
>
>> If someone wants to keep the invocation ordering of oneway and
>> req-resp calls from a single client and while keeping it as it is and
>> simple, could he also use the USE_ORIGINAL_THREAD option of this
>> interceptor or was this option intended to be a workaround for the
>> ordering problem?
>
> Well, that's actually there for things like transactions. If using JMS with
> transactions or even http with Spring transactions, we need to keep on the
> original thread so the transaction is properly associated with the request.
> If we jump threads and the original thread unwinds, the transaction may close.
Hi Dan,
okay, right. Why didn't I realize this option was intended for transactions.
>
> However, with USE_ORIGINAL_THREAD, it does consume the main thread which when
> running in a Sevlet container would consume a thread for http processing. For
> a long running process, that's likely not a good thing. Thus, it's really
> not a "good" option for long running one-ways.
Right. But any one-way service that takes significantly more time than
its following req-resp service, the completion of the service calls
will be out of order anyway. Then, we might ask why we needed to wait.
Butt this current wait thing is not critical and we can keep this wait
thing as it is.
Thanks for explaining to me the background.
Regards, aki
>
> Dan
>
>
>>
>> thanks.
>>
>> regards, aki
>>
>> > Dan
>> >
>> >> Thanks.
>> >> regards, aki
>> >>
>> >> 2011/8/19 Daniel Kulp <dk...@apache.org>:
>> >> > On Friday, August 19, 2011 2:32:03 PM xuhb wrote:
>> >> >> Maybe it is becuase before chain.wait() return, the sync of
>> >> >> chain
>> >> >> will be re-locked, so it will block untill chain.resume()
>> >> >> finished;
>> >> >
>> >> > Thanks for the test case. There definitely is an issue here.
>> >> > At this point, the thread has already synchronized on the chain
>> >> > (due to the doIntercept method on the PhaseInterceptorChain being
>> >> > synchronized). Thus, some of the synchronized blocks in there
>> >> > don't look correct. If I create a simple:
>> >> >
>> >> > final Object lock = new Object();
>> >> >
>> >> > in there, then that problem goes away.
>> >> >
>> >> > That said, I hit another couple of issues with your testcase as
>> >> > well
>> >> > that I'm looking into. :-(
>> >> >
>> >> > Dan
>> >> >
>> >> >> ----- Original Message -----
>> >> >> From: "xuhb" <xu...@tongtech.com>
>> >> >> To: <us...@cxf.apache.org>
>> >> >> Sent: Friday, August 19, 2011 1:39 PM
>> >> >> Subject: Re: A mysteriously deadlock of CXF
>> >> >> OnewayProcessorInterceptor>> >>
>> >> >> > Sorry, I foget post the issue link:
>> >> >> >
>> >> >> > https://issues.apache.org/jira/browse/CXF-3750
>> >> >> >
>> >> >> > ----- Original Message -----
>> >> >> > From: "xuhb" <xu...@tongtech.com>
>> >> >> > To: <us...@cxf.apache.org>
>> >> >> > Sent: Friday, August 19, 2011 1:34 PM
>> >> >> > Subject: A mysteriously deadlock of CXF
>> >> >> > OnewayProcessorInterceptor
>> >> >> >
>> >> >> >> Hi:
>> >> >> >>
>> >> >> >> Recently when I am checking/testing CXF , there is a
>> >> >> >> mysteriously deadlock of CXF Oneway Process; Normally
>> >> >> >> CXF
>> >> >> >> engine will invoke the one way bussiness logical
>> >> >> >> asynchronized
>> >> >> >> ,, so the servlet handle will finished and return back
>> >> >> >> to
>> >> >> >> servlet engine immediately;
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> But sometime, I noticed that the servlet
>> >> >> >> handle(JettyHTTPHandler) at server side doesn't return
>> >> >> >> back
>> >> >> >> to
>> >> >> >> servlet engine(Jetty) immediately , it will waiting
>> >> >> >> until the
>> >> >> >> asynchrouse business logical finished;
>> >> >
>> >> > After dig source of
>> >> >
>> >> >> >> CXF, I find it 's relate to
>> >> >> >> OnewayProcessorInterceptor;But
>> >> >> >> until now I can only show when will the deadlock
>> >> >> >> occurs, but
>> >> >> >> I
>> >> >> >> still can not explain why;>>
>> >> >> >>
>> >> >> >> Following is details:
>> >> >> >> OnewayProcessInterceptor.handleMessage{
>> >> >> >> synchronized (chain) {
>> >> >> >> message.getExchange().get(Bus.class).getExtension(WorkQueu
>> >> >> >> eManag
>> >> >> >> er.cla ss)
>> >> >
>> >> > .getAutomaticWorkQueue().execute(new Runnable() {
>> >> >
>> >> >> >> public void run() {
>> >> >> >> synchronized (chain) {
>> >> >> >>
>> >> >> >> System.out.println("--notify all");
>> >> >> >> chain.notifyAll();
>> >> >> >>
>> >> >> >> }
>> >> >> >>
>> >> >> >> chain.resume(); //if chain.resume is called before
>> >> >> >> chain.wait
>> >> >> >> finished , the dead lock will occurs; It seems as
>> >> >> >> chain.resume is synchronized, so it will relock on
>> >> >> >> chain
>> >> >> >> object, so the chain.wait() will deadlocked (... I feel
>> >> >> >> confused for this, because jdk doesn't say so...)
>> >> >> >> ;After
>> >> >> >> chain.resume finished, locking on chain is released,
>> >> >> >> deadlock
>> >> >> >> of chain.wait() is also released; but I don't think
>> >> >> >> this is
>> >> >> >> problem of CXF , maybe it's jdk's problem ?? I feels
>> >> >> >> confused;
>> >> >> >>
>> >> >> >> }
>> >> >> >>
>> >> >> >> });
>> >> >> >>
>> >> >> >> System.out.println("--wait begin");
>> >> >> >> chain.wait(20);
>> >> >> >> System.out.println("--wait end");
>> >> >> >>
>> >> >> >> }
>> >> >> >> }
>> >> >> >> syncrhonized PhaseInterceptorChain.resume(){
>> >> >> >>
>> >> >> >> System.out.println("--api chain resume");
>> >> >> >>
>> >> >> >> ...
>> >> >> >>
>> >> >> >> }
>> >> >> >>
>> >> >> >> if the execute sequence as following, every thing is ok.
>> >> >> >> there
>> >> >> >> is no dead lock;
>> >> >> >>
>> >> >> >> chain.wait enter
>> >> >> >> chian.notify invoked
>> >> >> >> chain.wait return;
>> >> >> >> chain.resume(); //resume also synchronzed on chain
>> >> >> >> object;
>> >> >> >>
>> >> >> >>
>> >> >> >> if the execute sequence as following , dead lock will
>> >> >> >> occurs:
>> >> >> >>
>> >> >> >> chain.wait enter
>> >> >> >> chain.notify
>> >> >> >> chain.resume// ..now waiting on chain will blocked
>> >> >> >> until
>> >> >> >> chain.resume finished(release sync on chain)
>> >> >
>> >> > chain.wait
>> >> >
>> >> >> >> return;
>> >> >> >>
>> >> >> >>
>> >> >> >> following dump on console indicate the above sequence:
>> >> >> >>
>> >> >> >> No DeadLock dump :
>> >> >> >> --wait begin
>> >> >> >> --notify all
>> >> >> >> --wait end
>> >> >> >> --api chain resume
>> >> >> >> product service begin Fri Aug 19 12:10:28 CST 2011 //a
>> >> >> >> lone
>> >> >> >> time(10
>> >> >> >> seconds) one way business logical begin
>> >> >
>> >> > product service end Fri Aug
>> >> >
>> >> >> >> 19 12:10:38 CST 2011 .//a lone time(10 seconds) one way
>> >> >> >> business
>> >> >> >> logical end;
>> >> >> >>
>> >> >> >> DeadLock Dump:
>> >> >> >> --wait begin
>> >> >> >> --notify all
>> >> >> >> --api chain resume
>> >> >> >> product service begin Fri Aug 19 12:10:40 CST 2011
>> >> >> >> product service end Fri Aug 19 12:10:50 CST 2011
>> >> >> >> --wait end
>> >> >> >>
>> >> >> >>
>> >> >> >> Until now I am not sure if problem is CXF's or JDK's, or
>> >> >> >> something
>> >> >> >> which I don't know cause such a deadlock;
>> >> >
>> >> > I also wrote a simple
>> >> >
>> >> >> >> program to simulate the execute sequnce which causeddead
>> >> >> >> lock
>> >> >> >> in
>> >> >> >> CXF, but the simple program never dead lock;
>> >> >> >> I tried CXF 2.3.3 version && Jetty transport && (JDK1.5_22
>> >> >> >> ||
>> >> >> >> JDK
>> >> >> >> 1.6_17) && Windows XP system;
>> >> >
>> >> >> >> I also post this question as a JIRA issue:
>> >> > --
>> >> > Daniel Kulp
>> >> > dkulp@apache.org
>> >> > http://dankulp.com/blog
>> >> > Talend - http://www.talend.com
>> >
>> > --
>> > Daniel Kulp
>> > dkulp@apache.org
>> > http://dankulp.com/blog
>> > Talend - http://www.talend.com
> --
> Daniel Kulp
> dkulp@apache.org
> http://dankulp.com/blog
> Talend - http://www.talend.com
>
Re: A mysteriously deadlock of CXF OnewayProcessorInterceptor
Posted by Daniel Kulp <dk...@apache.org>.
On Wednesday, August 24, 2011 1:57:59 PM Aki Yoshida wrote:
> 2011/8/22 Daniel Kulp <dk...@apache.org>:
> > On Monday, August 22, 2011 10:26:36 AM Aki Yoshida wrote:
> >> Hi Dan,
> >> I was confused about the original intention of this wait(20) code. I
> >> didn't understand why it was important to wait 20msec and let the
> >> executor thread start instead of not waiting. The comment in the code
> >> says the intention is to try to keep the ordering of the messages from
> >> the client. But I had an impression that this code would not be
> >> helping in this matter. So I initially thought that we could just get
> >> rid of this wait to avoid the reported blocking situation. If keeping
> >> the ordering is important, shouldn't we introduce this mechanism
> >> optionally instead of trying to put this in there implicitly?
> >
> > Basically, if a client proxy sends a one-way request, and then
> > immediately follows it up with a two-way request, we were seeing (even
> > in our unit tests) where, in most situations, the two-way would get
> > dispatched into the impl before the one-way. While perfectly valid,
> > this was kind of confusing to users. Thus we added a very short wait
> > there to make sure the runnable has been started on a thread which
> > usually allows it to proceed something closer to in order. If the
> > queue is full or all threads are busy, the wait will timeout and the
> > one-way will get there eventually, but in most cases this works fairly
> > well.
> >
> > Basically, it's a "best effort, but no guarantees". If they need to
> > make sure it's there in order, they would need two ways or use WS-RM.
>
> Thanks for the explanation. I am sorry to hear that some people are
> not happy when a oneway call gets passed by a later invoked req-resp
> call. I personally think we could get rid of this wait(20) to make a
> oneway call more efficient.
In MOST cases, it doesn't wait for 20 ms, it's usually on the order of nano-
seconds. It's only when the queue starts filling with lots of one-ways does
it ever wait that long. That said, I'd definitely support dropping this down
to a wait(5) or so. I'm really not sure why 20 was chosen. Even a
"Thread.yield()" may work, but the wait with the lock is more accurate,
especially on multi-core machines.
> If someone wants to keep the invocation ordering of oneway and
> req-resp calls from a single client and while keeping it as it is and
> simple, could he also use the USE_ORIGINAL_THREAD option of this
> interceptor or was this option intended to be a workaround for the
> ordering problem?
Well, that's actually there for things like transactions. If using JMS with
transactions or even http with Spring transactions, we need to keep on the
original thread so the transaction is properly associated with the request.
If we jump threads and the original thread unwinds, the transaction may close.
However, with USE_ORIGINAL_THREAD, it does consume the main thread which when
running in a Sevlet container would consume a thread for http processing. For
a long running process, that's likely not a good thing. Thus, it's really
not a "good" option for long running one-ways.
Dan
>
> thanks.
>
> regards, aki
>
> > Dan
> >
> >> Thanks.
> >> regards, aki
> >>
> >> 2011/8/19 Daniel Kulp <dk...@apache.org>:
> >> > On Friday, August 19, 2011 2:32:03 PM xuhb wrote:
> >> >> Maybe it is becuase before chain.wait() return, the sync of
> >> >> chain
> >> >> will be re-locked, so it will block untill chain.resume()
> >> >> finished;
> >> >
> >> > Thanks for the test case. There definitely is an issue here.
> >> > At this point, the thread has already synchronized on the chain
> >> > (due to the doIntercept method on the PhaseInterceptorChain being
> >> > synchronized). Thus, some of the synchronized blocks in there
> >> > don't look correct. If I create a simple:
> >> >
> >> > final Object lock = new Object();
> >> >
> >> > in there, then that problem goes away.
> >> >
> >> > That said, I hit another couple of issues with your testcase as
> >> > well
> >> > that I'm looking into. :-(
> >> >
> >> > Dan
> >> >
> >> >> ----- Original Message -----
> >> >> From: "xuhb" <xu...@tongtech.com>
> >> >> To: <us...@cxf.apache.org>
> >> >> Sent: Friday, August 19, 2011 1:39 PM
> >> >> Subject: Re: A mysteriously deadlock of CXF
> >> >> OnewayProcessorInterceptor>> >>
> >> >> > Sorry, I foget post the issue link:
> >> >> >
> >> >> > https://issues.apache.org/jira/browse/CXF-3750
> >> >> >
> >> >> > ----- Original Message -----
> >> >> > From: "xuhb" <xu...@tongtech.com>
> >> >> > To: <us...@cxf.apache.org>
> >> >> > Sent: Friday, August 19, 2011 1:34 PM
> >> >> > Subject: A mysteriously deadlock of CXF
> >> >> > OnewayProcessorInterceptor
> >> >> >
> >> >> >> Hi:
> >> >> >>
> >> >> >> Recently when I am checking/testing CXF , there is a
> >> >> >> mysteriously deadlock of CXF Oneway Process; Normally
> >> >> >> CXF
> >> >> >> engine will invoke the one way bussiness logical
> >> >> >> asynchronized
> >> >> >> ,, so the servlet handle will finished and return back
> >> >> >> to
> >> >> >> servlet engine immediately;
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> But sometime, I noticed that the servlet
> >> >> >> handle(JettyHTTPHandler) at server side doesn't return
> >> >> >> back
> >> >> >> to
> >> >> >> servlet engine(Jetty) immediately , it will waiting
> >> >> >> until the
> >> >> >> asynchrouse business logical finished;
> >> >
> >> > After dig source of
> >> >
> >> >> >> CXF, I find it 's relate to
> >> >> >> OnewayProcessorInterceptor;But
> >> >> >> until now I can only show when will the deadlock
> >> >> >> occurs, but
> >> >> >> I
> >> >> >> still can not explain why;>>
> >> >> >>
> >> >> >> Following is details:
> >> >> >> OnewayProcessInterceptor.handleMessage{
> >> >> >> synchronized (chain) {
> >> >> >> message.getExchange().get(Bus.class).getExtension(WorkQueu
> >> >> >> eManag
> >> >> >> er.cla ss)
> >> >
> >> > .getAutomaticWorkQueue().execute(new Runnable() {
> >> >
> >> >> >> public void run() {
> >> >> >> synchronized (chain) {
> >> >> >>
> >> >> >> System.out.println("--notify all");
> >> >> >> chain.notifyAll();
> >> >> >>
> >> >> >> }
> >> >> >>
> >> >> >> chain.resume(); //if chain.resume is called before
> >> >> >> chain.wait
> >> >> >> finished , the dead lock will occurs; It seems as
> >> >> >> chain.resume is synchronized, so it will relock on
> >> >> >> chain
> >> >> >> object, so the chain.wait() will deadlocked (... I feel
> >> >> >> confused for this, because jdk doesn't say so...)
> >> >> >> ;After
> >> >> >> chain.resume finished, locking on chain is released,
> >> >> >> deadlock
> >> >> >> of chain.wait() is also released; but I don't think
> >> >> >> this is
> >> >> >> problem of CXF , maybe it's jdk's problem ?? I feels
> >> >> >> confused;
> >> >> >>
> >> >> >> }
> >> >> >>
> >> >> >> });
> >> >> >>
> >> >> >> System.out.println("--wait begin");
> >> >> >> chain.wait(20);
> >> >> >> System.out.println("--wait end");
> >> >> >>
> >> >> >> }
> >> >> >> }
> >> >> >> syncrhonized PhaseInterceptorChain.resume(){
> >> >> >>
> >> >> >> System.out.println("--api chain resume");
> >> >> >>
> >> >> >> ...
> >> >> >>
> >> >> >> }
> >> >> >>
> >> >> >> if the execute sequence as following, every thing is ok.
> >> >> >> there
> >> >> >> is no dead lock;
> >> >> >>
> >> >> >> chain.wait enter
> >> >> >> chian.notify invoked
> >> >> >> chain.wait return;
> >> >> >> chain.resume(); //resume also synchronzed on chain
> >> >> >> object;
> >> >> >>
> >> >> >>
> >> >> >> if the execute sequence as following , dead lock will
> >> >> >> occurs:
> >> >> >>
> >> >> >> chain.wait enter
> >> >> >> chain.notify
> >> >> >> chain.resume// ..now waiting on chain will blocked
> >> >> >> until
> >> >> >> chain.resume finished(release sync on chain)
> >> >
> >> > chain.wait
> >> >
> >> >> >> return;
> >> >> >>
> >> >> >>
> >> >> >> following dump on console indicate the above sequence:
> >> >> >>
> >> >> >> No DeadLock dump :
> >> >> >> --wait begin
> >> >> >> --notify all
> >> >> >> --wait end
> >> >> >> --api chain resume
> >> >> >> product service begin Fri Aug 19 12:10:28 CST 2011 //a
> >> >> >> lone
> >> >> >> time(10
> >> >> >> seconds) one way business logical begin
> >> >
> >> > product service end Fri Aug
> >> >
> >> >> >> 19 12:10:38 CST 2011 .//a lone time(10 seconds) one way
> >> >> >> business
> >> >> >> logical end;
> >> >> >>
> >> >> >> DeadLock Dump:
> >> >> >> --wait begin
> >> >> >> --notify all
> >> >> >> --api chain resume
> >> >> >> product service begin Fri Aug 19 12:10:40 CST 2011
> >> >> >> product service end Fri Aug 19 12:10:50 CST 2011
> >> >> >> --wait end
> >> >> >>
> >> >> >>
> >> >> >> Until now I am not sure if problem is CXF's or JDK's, or
> >> >> >> something
> >> >> >> which I don't know cause such a deadlock;
> >> >
> >> > I also wrote a simple
> >> >
> >> >> >> program to simulate the execute sequnce which causeddead
> >> >> >> lock
> >> >> >> in
> >> >> >> CXF, but the simple program never dead lock;
> >> >> >> I tried CXF 2.3.3 version && Jetty transport && (JDK1.5_22
> >> >> >> ||
> >> >> >> JDK
> >> >> >> 1.6_17) && Windows XP system;
> >> >
> >> >> >> I also post this question as a JIRA issue:
> >> > --
> >> > Daniel Kulp
> >> > dkulp@apache.org
> >> > http://dankulp.com/blog
> >> > Talend - http://www.talend.com
> >
> > --
> > Daniel Kulp
> > dkulp@apache.org
> > http://dankulp.com/blog
> > Talend - http://www.talend.com
--
Daniel Kulp
dkulp@apache.org
http://dankulp.com/blog
Talend - http://www.talend.com
Re: A mysteriously deadlock of CXF OnewayProcessorInterceptor
Posted by Aki Yoshida <el...@googlemail.com>.
2011/8/22 Daniel Kulp <dk...@apache.org>:
> On Monday, August 22, 2011 10:26:36 AM Aki Yoshida wrote:
>> Hi Dan,
>> I was confused about the original intention of this wait(20) code. I
>> didn't understand why it was important to wait 20msec and let the
>> executor thread start instead of not waiting. The comment in the code
>> says the intention is to try to keep the ordering of the messages from
>> the client. But I had an impression that this code would not be
>> helping in this matter. So I initially thought that we could just get
>> rid of this wait to avoid the reported blocking situation. If keeping
>> the ordering is important, shouldn't we introduce this mechanism
>> optionally instead of trying to put this in there implicitly?
>
> Basically, if a client proxy sends a one-way request, and then immediately
> follows it up with a two-way request, we were seeing (even in our unit tests)
> where, in most situations, the two-way would get dispatched into the impl
> before the one-way. While perfectly valid, this was kind of confusing to
> users. Thus we added a very short wait there to make sure the runnable has
> been started on a thread which usually allows it to proceed something closer
> to in order. If the queue is full or all threads are busy, the wait will
> timeout and the one-way will get there eventually, but in most cases this
> works fairly well.
>
> Basically, it's a "best effort, but no guarantees". If they need to make sure
> it's there in order, they would need two ways or use WS-RM.
Thanks for the explanation. I am sorry to hear that some people are
not happy when a oneway call gets passed by a later invoked req-resp
call. I personally think we could get rid of this wait(20) to make a
oneway call more efficient.
If someone wants to keep the invocation ordering of oneway and
req-resp calls from a single client and while keeping it as it is and
simple, could he also use the USE_ORIGINAL_THREAD option of this
interceptor or was this option intended to be a workaround for the
ordering problem?
thanks.
regards, aki
>
> Dan
>
>
>
>>
>> Thanks.
>> regards, aki
>>
>> 2011/8/19 Daniel Kulp <dk...@apache.org>:
>> > On Friday, August 19, 2011 2:32:03 PM xuhb wrote:
>> >> Maybe it is becuase before chain.wait() return, the sync of chain
>> >> will be re-locked, so it will block untill chain.resume() finished;
>> >
>> > Thanks for the test case. There definitely is an issue here. At this
>> > point, the thread has already synchronized on the chain (due to the
>> > doIntercept method on the PhaseInterceptorChain being synchronized).
>> > Thus, some of the synchronized blocks in there don't look correct.
>> > If I create a simple:
>> >
>> > final Object lock = new Object();
>> >
>> > in there, then that problem goes away.
>> >
>> > That said, I hit another couple of issues with your testcase as well
>> > that I'm looking into. :-(
>> >
>> > Dan
>> >
>> >> ----- Original Message -----
>> >> From: "xuhb" <xu...@tongtech.com>
>> >> To: <us...@cxf.apache.org>
>> >> Sent: Friday, August 19, 2011 1:39 PM
>> >> Subject: Re: A mysteriously deadlock of CXF OnewayProcessorInterceptor
>> >>
>> >> > Sorry, I foget post the issue link:
>> >> >
>> >> > https://issues.apache.org/jira/browse/CXF-3750
>> >> >
>> >> > ----- Original Message -----
>> >> > From: "xuhb" <xu...@tongtech.com>
>> >> > To: <us...@cxf.apache.org>
>> >> > Sent: Friday, August 19, 2011 1:34 PM
>> >> > Subject: A mysteriously deadlock of CXF OnewayProcessorInterceptor
>> >> >
>> >> >> Hi:
>> >> >>
>> >> >> Recently when I am checking/testing CXF , there is a
>> >> >> mysteriously deadlock of CXF Oneway Process; Normally CXF
>> >> >> engine will invoke the one way bussiness logical
>> >> >> asynchronized
>> >> >> ,, so the servlet handle will finished and return back to
>> >> >> servlet engine immediately;
>> >> >>
>> >> >>
>> >> >>
>> >> >> But sometime, I noticed that the servlet
>> >> >> handle(JettyHTTPHandler) at server side doesn't return back
>> >> >> to
>> >> >> servlet engine(Jetty) immediately , it will waiting until the
>> >> >> asynchrouse business logical finished;
>> >
>> > After dig source of
>> >
>> >> >> CXF, I find it 's relate to OnewayProcessorInterceptor;But
>> >> >> until now I can only show when will the deadlock occurs, but
>> >> >> I
>> >> >> still can not explain why;>>
>> >> >>
>> >> >> Following is details:
>> >> >> OnewayProcessInterceptor.handleMessage{
>> >> >> synchronized (chain) {
>> >> >> message.getExchange().get(Bus.class).getExtension(WorkQueueManag
>> >> >> er.cla ss)
>> >
>> > .getAutomaticWorkQueue().execute(new Runnable() {
>> >
>> >> >> public void run() {
>> >> >> synchronized (chain) {
>> >> >>
>> >> >> System.out.println("--notify all");
>> >> >> chain.notifyAll();
>> >> >>
>> >> >> }
>> >> >>
>> >> >> chain.resume(); //if chain.resume is called before
>> >> >> chain.wait
>> >> >> finished , the dead lock will occurs; It seems as
>> >> >> chain.resume is synchronized, so it will relock on chain
>> >> >> object, so the chain.wait() will deadlocked (... I feel
>> >> >> confused for this, because jdk doesn't say so...) ;After
>> >> >> chain.resume finished, locking on chain is released,
>> >> >> deadlock
>> >> >> of chain.wait() is also released; but I don't think this is
>> >> >> problem of CXF , maybe it's jdk's problem ?? I feels
>> >> >> confused;
>> >> >>
>> >> >> }
>> >> >>
>> >> >> });
>> >> >>
>> >> >> System.out.println("--wait begin");
>> >> >> chain.wait(20);
>> >> >> System.out.println("--wait end");
>> >> >>
>> >> >> }
>> >> >> }
>> >> >> syncrhonized PhaseInterceptorChain.resume(){
>> >> >>
>> >> >> System.out.println("--api chain resume");
>> >> >>
>> >> >> ...
>> >> >>
>> >> >> }
>> >> >>
>> >> >> if the execute sequence as following, every thing is ok. there
>> >> >> is no dead lock;
>> >> >>
>> >> >> chain.wait enter
>> >> >> chian.notify invoked
>> >> >> chain.wait return;
>> >> >> chain.resume(); //resume also synchronzed on chain object;
>> >> >>
>> >> >>
>> >> >> if the execute sequence as following , dead lock will occurs:
>> >> >>
>> >> >> chain.wait enter
>> >> >> chain.notify
>> >> >> chain.resume// ..now waiting on chain will blocked until
>> >> >> chain.resume finished(release sync on chain)
>> >
>> > chain.wait
>> >
>> >> >> return;
>> >> >>
>> >> >>
>> >> >> following dump on console indicate the above sequence:
>> >> >>
>> >> >> No DeadLock dump :
>> >> >> --wait begin
>> >> >> --notify all
>> >> >> --wait end
>> >> >> --api chain resume
>> >> >> product service begin Fri Aug 19 12:10:28 CST 2011 //a lone
>> >> >> time(10
>> >> >> seconds) one way business logical begin
>> >
>> > product service end Fri Aug
>> >
>> >> >> 19 12:10:38 CST 2011 .//a lone time(10 seconds) one way
>> >> >> business
>> >> >> logical end;
>> >> >>
>> >> >> DeadLock Dump:
>> >> >> --wait begin
>> >> >> --notify all
>> >> >> --api chain resume
>> >> >> product service begin Fri Aug 19 12:10:40 CST 2011
>> >> >> product service end Fri Aug 19 12:10:50 CST 2011
>> >> >> --wait end
>> >> >>
>> >> >>
>> >> >> Until now I am not sure if problem is CXF's or JDK's, or
>> >> >> something
>> >> >> which I don't know cause such a deadlock;
>> >
>> > I also wrote a simple
>> >
>> >> >> program to simulate the execute sequnce which causeddead lock
>> >> >> in
>> >> >> CXF, but the simple program never dead lock;
>> >> >> I tried CXF 2.3.3 version && Jetty transport && (JDK1.5_22 ||
>> >> >> JDK
>> >> >> 1.6_17) && Windows XP system;
>> >
>> >> >> I also post this question as a JIRA issue:
>> > --
>> > Daniel Kulp
>> > dkulp@apache.org
>> > http://dankulp.com/blog
>> > Talend - http://www.talend.com
> --
> Daniel Kulp
> dkulp@apache.org
> http://dankulp.com/blog
> Talend - http://www.talend.com
>
Re: A mysteriously deadlock of CXF OnewayProcessorInterceptor
Posted by Daniel Kulp <dk...@apache.org>.
On Monday, August 22, 2011 10:26:36 AM Aki Yoshida wrote:
> Hi Dan,
> I was confused about the original intention of this wait(20) code. I
> didn't understand why it was important to wait 20msec and let the
> executor thread start instead of not waiting. The comment in the code
> says the intention is to try to keep the ordering of the messages from
> the client. But I had an impression that this code would not be
> helping in this matter. So I initially thought that we could just get
> rid of this wait to avoid the reported blocking situation. If keeping
> the ordering is important, shouldn't we introduce this mechanism
> optionally instead of trying to put this in there implicitly?
Basically, if a client proxy sends a one-way request, and then immediately
follows it up with a two-way request, we were seeing (even in our unit tests)
where, in most situations, the two-way would get dispatched into the impl
before the one-way. While perfectly valid, this was kind of confusing to
users. Thus we added a very short wait there to make sure the runnable has
been started on a thread which usually allows it to proceed something closer
to in order. If the queue is full or all threads are busy, the wait will
timeout and the one-way will get there eventually, but in most cases this
works fairly well.
Basically, it's a "best effort, but no guarantees". If they need to make sure
it's there in order, they would need two ways or use WS-RM.
Dan
>
> Thanks.
> regards, aki
>
> 2011/8/19 Daniel Kulp <dk...@apache.org>:
> > On Friday, August 19, 2011 2:32:03 PM xuhb wrote:
> >> Maybe it is becuase before chain.wait() return, the sync of chain
> >> will be re-locked, so it will block untill chain.resume() finished;
> >
> > Thanks for the test case. There definitely is an issue here. At this
> > point, the thread has already synchronized on the chain (due to the
> > doIntercept method on the PhaseInterceptorChain being synchronized).
> > Thus, some of the synchronized blocks in there don't look correct.
> > If I create a simple:
> >
> > final Object lock = new Object();
> >
> > in there, then that problem goes away.
> >
> > That said, I hit another couple of issues with your testcase as well
> > that I'm looking into. :-(
> >
> > Dan
> >
> >> ----- Original Message -----
> >> From: "xuhb" <xu...@tongtech.com>
> >> To: <us...@cxf.apache.org>
> >> Sent: Friday, August 19, 2011 1:39 PM
> >> Subject: Re: A mysteriously deadlock of CXF OnewayProcessorInterceptor
> >>
> >> > Sorry, I foget post the issue link:
> >> >
> >> > https://issues.apache.org/jira/browse/CXF-3750
> >> >
> >> > ----- Original Message -----
> >> > From: "xuhb" <xu...@tongtech.com>
> >> > To: <us...@cxf.apache.org>
> >> > Sent: Friday, August 19, 2011 1:34 PM
> >> > Subject: A mysteriously deadlock of CXF OnewayProcessorInterceptor
> >> >
> >> >> Hi:
> >> >>
> >> >> Recently when I am checking/testing CXF , there is a
> >> >> mysteriously deadlock of CXF Oneway Process; Normally CXF
> >> >> engine will invoke the one way bussiness logical
> >> >> asynchronized
> >> >> ,, so the servlet handle will finished and return back to
> >> >> servlet engine immediately;
> >> >>
> >> >>
> >> >>
> >> >> But sometime, I noticed that the servlet
> >> >> handle(JettyHTTPHandler) at server side doesn't return back
> >> >> to
> >> >> servlet engine(Jetty) immediately , it will waiting until the
> >> >> asynchrouse business logical finished;
> >
> > After dig source of
> >
> >> >> CXF, I find it 's relate to OnewayProcessorInterceptor;But
> >> >> until now I can only show when will the deadlock occurs, but
> >> >> I
> >> >> still can not explain why;>>
> >> >>
> >> >> Following is details:
> >> >> OnewayProcessInterceptor.handleMessage{
> >> >> synchronized (chain) {
> >> >> message.getExchange().get(Bus.class).getExtension(WorkQueueManag
> >> >> er.cla ss)
> >
> > .getAutomaticWorkQueue().execute(new Runnable() {
> >
> >> >> public void run() {
> >> >> synchronized (chain) {
> >> >>
> >> >> System.out.println("--notify all");
> >> >> chain.notifyAll();
> >> >>
> >> >> }
> >> >>
> >> >> chain.resume(); //if chain.resume is called before
> >> >> chain.wait
> >> >> finished , the dead lock will occurs; It seems as
> >> >> chain.resume is synchronized, so it will relock on chain
> >> >> object, so the chain.wait() will deadlocked (... I feel
> >> >> confused for this, because jdk doesn't say so...) ;After
> >> >> chain.resume finished, locking on chain is released,
> >> >> deadlock
> >> >> of chain.wait() is also released; but I don't think this is
> >> >> problem of CXF , maybe it's jdk's problem ?? I feels
> >> >> confused;
> >> >>
> >> >> }
> >> >>
> >> >> });
> >> >>
> >> >> System.out.println("--wait begin");
> >> >> chain.wait(20);
> >> >> System.out.println("--wait end");
> >> >>
> >> >> }
> >> >> }
> >> >> syncrhonized PhaseInterceptorChain.resume(){
> >> >>
> >> >> System.out.println("--api chain resume");
> >> >>
> >> >> ...
> >> >>
> >> >> }
> >> >>
> >> >> if the execute sequence as following, every thing is ok. there
> >> >> is no dead lock;
> >> >>
> >> >> chain.wait enter
> >> >> chian.notify invoked
> >> >> chain.wait return;
> >> >> chain.resume(); //resume also synchronzed on chain object;
> >> >>
> >> >>
> >> >> if the execute sequence as following , dead lock will occurs:
> >> >>
> >> >> chain.wait enter
> >> >> chain.notify
> >> >> chain.resume// ..now waiting on chain will blocked until
> >> >> chain.resume finished(release sync on chain)
> >
> > chain.wait
> >
> >> >> return;
> >> >>
> >> >>
> >> >> following dump on console indicate the above sequence:
> >> >>
> >> >> No DeadLock dump :
> >> >> --wait begin
> >> >> --notify all
> >> >> --wait end
> >> >> --api chain resume
> >> >> product service begin Fri Aug 19 12:10:28 CST 2011 //a lone
> >> >> time(10
> >> >> seconds) one way business logical begin
> >
> > product service end Fri Aug
> >
> >> >> 19 12:10:38 CST 2011 .//a lone time(10 seconds) one way
> >> >> business
> >> >> logical end;
> >> >>
> >> >> DeadLock Dump:
> >> >> --wait begin
> >> >> --notify all
> >> >> --api chain resume
> >> >> product service begin Fri Aug 19 12:10:40 CST 2011
> >> >> product service end Fri Aug 19 12:10:50 CST 2011
> >> >> --wait end
> >> >>
> >> >>
> >> >> Until now I am not sure if problem is CXF's or JDK's, or
> >> >> something
> >> >> which I don't know cause such a deadlock;
> >
> > I also wrote a simple
> >
> >> >> program to simulate the execute sequnce which causeddead lock
> >> >> in
> >> >> CXF, but the simple program never dead lock;
> >> >> I tried CXF 2.3.3 version && Jetty transport && (JDK1.5_22 ||
> >> >> JDK
> >> >> 1.6_17) && Windows XP system;
> >
> >> >> I also post this question as a JIRA issue:
> > --
> > Daniel Kulp
> > dkulp@apache.org
> > http://dankulp.com/blog
> > Talend - http://www.talend.com
--
Daniel Kulp
dkulp@apache.org
http://dankulp.com/blog
Talend - http://www.talend.com
Re: A mysteriously deadlock of CXF OnewayProcessorInterceptor
Posted by Aki Yoshida <el...@googlemail.com>.
Hi Dan,
I was confused about the original intention of this wait(20) code. I
didn't understand why it was important to wait 20msec and let the
executor thread start instead of not waiting. The comment in the code
says the intention is to try to keep the ordering of the messages from
the client. But I had an impression that this code would not be
helping in this matter. So I initially thought that we could just get
rid of this wait to avoid the reported blocking situation. If keeping
the ordering is important, shouldn't we introduce this mechanism
optionally instead of trying to put this in there implicitly?
Thanks.
regards, aki
2011/8/19 Daniel Kulp <dk...@apache.org>:
> On Friday, August 19, 2011 2:32:03 PM xuhb wrote:
>> Maybe it is becuase before chain.wait() return, the sync of chain will be
>> re-locked, so it will block untill chain.resume() finished;
>
> Thanks for the test case. There definitely is an issue here. At this
> point, the thread has already synchronized on the chain (due to the
> doIntercept method on the PhaseInterceptorChain being synchronized). Thus,
> some of the synchronized blocks in there don't look correct. If I create a
> simple:
>
> final Object lock = new Object();
>
> in there, then that problem goes away.
>
> That said, I hit another couple of issues with your testcase as well that I'm
> looking into. :-(
>
> Dan
>
>
>
>
>> ----- Original Message -----
>> From: "xuhb" <xu...@tongtech.com>
>> To: <us...@cxf.apache.org>
>> Sent: Friday, August 19, 2011 1:39 PM
>> Subject: Re: A mysteriously deadlock of CXF OnewayProcessorInterceptor
>>
>>
>>
>> > Sorry, I foget post the issue link:
>> >
>> > https://issues.apache.org/jira/browse/CXF-3750
>> >
>> > ----- Original Message -----
>> > From: "xuhb" <xu...@tongtech.com>
>> > To: <us...@cxf.apache.org>
>> > Sent: Friday, August 19, 2011 1:34 PM
>> > Subject: A mysteriously deadlock of CXF OnewayProcessorInterceptor
>> >
>> >
>> >
>> >> Hi:
>> >>
>> >> Recently when I am checking/testing CXF , there is a
>> >> mysteriously deadlock of CXF Oneway Process; Normally CXF
>> >> engine will invoke the one way bussiness logical asynchronized
>> >> ,, so the servlet handle will finished and return back to
>> >> servlet engine immediately;
>>>
>> >>
>> >>
>> >> But sometime, I noticed that the servlet
>> >> handle(JettyHTTPHandler) at server side doesn't return back to
>> >> servlet engine(Jetty) immediately , it will waiting until the
>> >> asynchrouse business logical finished;
> After dig source of
>> >> CXF, I find it 's relate to OnewayProcessorInterceptor;But
>> >> until now I can only show when will the deadlock occurs, but I
>> >> still can not explain why;>>
>> >>
>> >> Following is details:
>> >> OnewayProcessInterceptor.handleMessage{
>> >> synchronized (chain) {
>> >> message.getExchange().get(Bus.class).getExtension(WorkQueueManager.cla
>> >> ss)
> .getAutomaticWorkQueue().execute(new Runnable() {
>> >>
>> >> public void run() {
>> >> synchronized (chain) {
>> >>
>> >> System.out.println("--notify all");
>> >> chain.notifyAll();
>> >>
>> >> }
>> >>
>> >> chain.resume(); //if chain.resume is called before chain.wait
>> >> finished , the dead lock will occurs; It seems as
>> >> chain.resume is synchronized, so it will relock on chain
>> >> object, so the chain.wait() will deadlocked (... I feel
>> >> confused for this, because jdk doesn't say so...) ;After
>> >> chain.resume finished, locking on chain is released, deadlock
>> >> of chain.wait() is also released; but I don't think this is
>> >> problem of CXF , maybe it's jdk's problem ?? I feels confused;
>>>
>> >> }
>> >>
>> >> });
>> >>
>> >> System.out.println("--wait begin");
>> >> chain.wait(20);
>> >> System.out.println("--wait end");
>> >>
>> >> }
>> >> }
>> >> syncrhonized PhaseInterceptorChain.resume(){
>> >>
>> >> System.out.println("--api chain resume");
>> >>
>> >> ...
>> >>
>> >> }
>> >>
>> >> if the execute sequence as following, every thing is ok. there is no
>> >> dead lock;
>>>
>> >> chain.wait enter
>> >> chian.notify invoked
>> >> chain.wait return;
>> >> chain.resume(); //resume also synchronzed on chain object;
>> >>
>> >>
>> >> if the execute sequence as following , dead lock will occurs:
>> >>
>> >> chain.wait enter
>> >> chain.notify
>> >> chain.resume// ..now waiting on chain will blocked until
>> >> chain.resume finished(release sync on chain)
> chain.wait
>> >> return;
>> >>
>> >>
>> >> following dump on console indicate the above sequence:
>> >>
>> >> No DeadLock dump :
>> >> --wait begin
>> >> --notify all
>> >> --wait end
>> >> --api chain resume
>> >> product service begin Fri Aug 19 12:10:28 CST 2011 //a lone time(10
>> >> seconds) one way business logical begin
> product service end Fri Aug
>> >> 19 12:10:38 CST 2011 .//a lone time(10 seconds) one way business
>> >> logical end;
>> >>
>> >> DeadLock Dump:
>> >> --wait begin
>> >> --notify all
>> >> --api chain resume
>> >> product service begin Fri Aug 19 12:10:40 CST 2011
>> >> product service end Fri Aug 19 12:10:50 CST 2011
>> >> --wait end
>> >>
>> >>
>> >> Until now I am not sure if problem is CXF's or JDK's, or something
>> >> which I don't know cause such a deadlock;
> I also wrote a simple
>> >> program to simulate the execute sequnce which causeddead lock in
>> >> CXF, but the simple program never dead lock;
>> >> I tried CXF 2.3.3 version && Jetty transport && (JDK1.5_22 || JDK
>> >> 1.6_17) && Windows XP system;
>
>> >> I also post this question as a JIRA issue:
>
> --
> Daniel Kulp
> dkulp@apache.org
> http://dankulp.com/blog
> Talend - http://www.talend.com
>
Re: A mysteriously deadlock of CXF OnewayProcessorInterceptor
Posted by Daniel Kulp <dk...@apache.org>.
On Friday, August 19, 2011 2:32:03 PM xuhb wrote:
> Maybe it is becuase before chain.wait() return, the sync of chain will be
> re-locked, so it will block untill chain.resume() finished;
Thanks for the test case. There definitely is an issue here. At this
point, the thread has already synchronized on the chain (due to the
doIntercept method on the PhaseInterceptorChain being synchronized). Thus,
some of the synchronized blocks in there don't look correct. If I create a
simple:
final Object lock = new Object();
in there, then that problem goes away.
That said, I hit another couple of issues with your testcase as well that I'm
looking into. :-(
Dan
> ----- Original Message -----
> From: "xuhb" <xu...@tongtech.com>
> To: <us...@cxf.apache.org>
> Sent: Friday, August 19, 2011 1:39 PM
> Subject: Re: A mysteriously deadlock of CXF OnewayProcessorInterceptor
>
>
>
> > Sorry, I foget post the issue link:
> >
> > https://issues.apache.org/jira/browse/CXF-3750
> >
> > ----- Original Message -----
> > From: "xuhb" <xu...@tongtech.com>
> > To: <us...@cxf.apache.org>
> > Sent: Friday, August 19, 2011 1:34 PM
> > Subject: A mysteriously deadlock of CXF OnewayProcessorInterceptor
> >
> >
> >
> >> Hi:
> >>
> >> Recently when I am checking/testing CXF , there is a
> >> mysteriously deadlock of CXF Oneway Process; Normally CXF
> >> engine will invoke the one way bussiness logical asynchronized
> >> ,, so the servlet handle will finished and return back to
> >> servlet engine immediately;
>>
> >>
> >>
> >> But sometime, I noticed that the servlet
> >> handle(JettyHTTPHandler) at server side doesn't return back to
> >> servlet engine(Jetty) immediately , it will waiting until the
> >> asynchrouse business logical finished;
After dig source of
> >> CXF, I find it 's relate to OnewayProcessorInterceptor;But
> >> until now I can only show when will the deadlock occurs, but I
> >> still can not explain why;>>
> >>
> >> Following is details:
> >> OnewayProcessInterceptor.handleMessage{
> >> synchronized (chain) {
> >> message.getExchange().get(Bus.class).getExtension(WorkQueueManager.cla
> >> ss)
.getAutomaticWorkQueue().execute(new Runnable() {
> >>
> >> public void run() {
> >> synchronized (chain) {
> >>
> >> System.out.println("--notify all");
> >> chain.notifyAll();
> >>
> >> }
> >>
> >> chain.resume(); //if chain.resume is called before chain.wait
> >> finished , the dead lock will occurs; It seems as
> >> chain.resume is synchronized, so it will relock on chain
> >> object, so the chain.wait() will deadlocked (... I feel
> >> confused for this, because jdk doesn't say so...) ;After
> >> chain.resume finished, locking on chain is released, deadlock
> >> of chain.wait() is also released; but I don't think this is
> >> problem of CXF , maybe it's jdk's problem ?? I feels confused;
>>
> >> }
> >>
> >> });
> >>
> >> System.out.println("--wait begin");
> >> chain.wait(20);
> >> System.out.println("--wait end");
> >>
> >> }
> >> }
> >> syncrhonized PhaseInterceptorChain.resume(){
> >>
> >> System.out.println("--api chain resume");
> >>
> >> ...
> >>
> >> }
> >>
> >> if the execute sequence as following, every thing is ok. there is no
> >> dead lock;
>>
> >> chain.wait enter
> >> chian.notify invoked
> >> chain.wait return;
> >> chain.resume(); //resume also synchronzed on chain object;
> >>
> >>
> >> if the execute sequence as following , dead lock will occurs:
> >>
> >> chain.wait enter
> >> chain.notify
> >> chain.resume// ..now waiting on chain will blocked until
> >> chain.resume finished(release sync on chain)
chain.wait
> >> return;
> >>
> >>
> >> following dump on console indicate the above sequence:
> >>
> >> No DeadLock dump :
> >> --wait begin
> >> --notify all
> >> --wait end
> >> --api chain resume
> >> product service begin Fri Aug 19 12:10:28 CST 2011 //a lone time(10
> >> seconds) one way business logical begin
product service end Fri Aug
> >> 19 12:10:38 CST 2011 .//a lone time(10 seconds) one way business
> >> logical end;
> >>
> >> DeadLock Dump:
> >> --wait begin
> >> --notify all
> >> --api chain resume
> >> product service begin Fri Aug 19 12:10:40 CST 2011
> >> product service end Fri Aug 19 12:10:50 CST 2011
> >> --wait end
> >>
> >>
> >> Until now I am not sure if problem is CXF's or JDK's, or something
> >> which I don't know cause such a deadlock;
I also wrote a simple
> >> program to simulate the execute sequnce which causeddead lock in
> >> CXF, but the simple program never dead lock;
> >> I tried CXF 2.3.3 version && Jetty transport && (JDK1.5_22 || JDK
> >> 1.6_17) && Windows XP system;
> >> I also post this question as a JIRA issue:
--
Daniel Kulp
dkulp@apache.org
http://dankulp.com/blog
Talend - http://www.talend.com
Re: A mysteriously deadlock of CXF OnewayProcessorInterceptor
Posted by Aki Yoshida <el...@googlemail.com>.
Hi,
I think you are right. Although the http 202 response is written back
to the response by the calling thread before this thread going into
the block and therefore the client should not be blocked as it can
read the http 202 response, the calling thread itself may get blocked
(not being able to come out of the wait until the other thread
finishes processing the rest of the chain). I am not sure why this 20
msec wait code is added there. The comment in the code says
//wait a few milliseconds for the background
thread to start processing
//Mostly just to make an attempt at keeping
the ordering of the
//messages coming in from a client. Not
guaranteed though.
maybe, we should try to resolve this elsewhere and not use wait(20) here.
regards, aki
2011/8/19 xuhb <xu...@tongtech.com>:
> Maybe it is becuase before chain.wait() return, the sync of chain will be re-locked, so it will block untill chain.resume() finished;
>
> ----- Original Message -----
> From: "xuhb" <xu...@tongtech.com>
> To: <us...@cxf.apache.org>
> Sent: Friday, August 19, 2011 1:39 PM
> Subject: Re: A mysteriously deadlock of CXF OnewayProcessorInterceptor
>
>
>> Sorry, I foget post the issue link:
>>
>> https://issues.apache.org/jira/browse/CXF-3750
>>
>> ----- Original Message -----
>> From: "xuhb" <xu...@tongtech.com>
>> To: <us...@cxf.apache.org>
>> Sent: Friday, August 19, 2011 1:34 PM
>> Subject: A mysteriously deadlock of CXF OnewayProcessorInterceptor
>>
>>
>>> Hi:
>>> Recently when I am checking/testing CXF , there is a mysteriously deadlock of CXF Oneway Process; Normally CXF engine will invoke the one way bussiness logical asynchronized ,, so the servlet handle will finished and return back to servlet engine immediately;
>>>
>>> But sometime, I noticed that the servlet handle(JettyHTTPHandler) at server side doesn't return back to servlet engine(Jetty) immediately , it will waiting until the asynchrouse business logical finished;
>>> After dig source of CXF, I find it 's relate to OnewayProcessorInterceptor;But until now I can only show when will the deadlock occurs, but I still can not explain why;
>>>
>>> Following is details:
>>> OnewayProcessInterceptor.handleMessage{
>>> synchronized (chain) {
>>> message.getExchange().get(Bus.class).getExtension(WorkQueueManager.class)
>>> .getAutomaticWorkQueue().execute(new Runnable() {
>>> public void run() {
>>> synchronized (chain) {
>>> System.out.println("--notify all");
>>> chain.notifyAll();
>>> }
>>>
>>> chain.resume(); //if chain.resume is called before chain.wait finished , the dead lock will occurs; It seems as chain.resume is synchronized, so it will relock on chain object, so the chain.wait() will deadlocked (... I feel confused for this, because jdk doesn't say so...) ;After chain.resume finished, locking on chain is released, deadlock of chain.wait() is also released; but I don't think this is problem of CXF , maybe it's jdk's problem ?? I feels confused;
>>> }
>>> });
>>> System.out.println("--wait begin");
>>> chain.wait(20);
>>> System.out.println("--wait end");
>>> }
>>> }
>>> syncrhonized PhaseInterceptorChain.resume(){
>>> System.out.println("--api chain resume");
>>> ...
>>> }
>>>
>>> if the execute sequence as following, every thing is ok. there is no dead lock;
>>> chain.wait enter
>>> chian.notify invoked
>>> chain.wait return;
>>> chain.resume(); //resume also synchronzed on chain object;
>>>
>>> if the execute sequence as following , dead lock will occurs:
>>> chain.wait enter
>>> chain.notify
>>> chain.resume// ..now waiting on chain will blocked until chain.resume finished(release sync on chain)
>>> chain.wait return;
>>>
>>> following dump on console indicate the above sequence:
>>>
>>> No DeadLock dump :
>>> --wait begin
>>> --notify all
>>> --wait end
>>> --api chain resume
>>> product service begin Fri Aug 19 12:10:28 CST 2011 //a lone time(10 seconds) one way business logical begin
>>> product service end Fri Aug 19 12:10:38 CST 2011 .//a lone time(10 seconds) one way business logical end;
>>>
>>>
>>> DeadLock Dump:
>>> --wait begin
>>> --notify all
>>> --api chain resume
>>> product service begin Fri Aug 19 12:10:40 CST 2011
>>> product service end Fri Aug 19 12:10:50 CST 2011
>>> --wait end
>>>
>>>
>>> Until now I am not sure if problem is CXF's or JDK's, or something which I don't know cause such a deadlock;
>>> I also wrote a simple program to simulate the execute sequnce which causeddead lock in CXF, but the simple program never dead lock;
>>>
>>> I tried CXF 2.3.3 version && Jetty transport && (JDK1.5_22 || JDK 1.6_17) && Windows XP system;
>>>
>>> I also post this question as a JIRA issue:
>>>
Re: A mysteriously deadlock of CXF OnewayProcessorInterceptor
Posted by xuhb <xu...@tongtech.com>.
Maybe it is becuase before chain.wait() return, the sync of chain will be re-locked, so it will block untill chain.resume() finished;
----- Original Message -----
From: "xuhb" <xu...@tongtech.com>
To: <us...@cxf.apache.org>
Sent: Friday, August 19, 2011 1:39 PM
Subject: Re: A mysteriously deadlock of CXF OnewayProcessorInterceptor
> Sorry, I foget post the issue link:
>
> https://issues.apache.org/jira/browse/CXF-3750
>
> ----- Original Message -----
> From: "xuhb" <xu...@tongtech.com>
> To: <us...@cxf.apache.org>
> Sent: Friday, August 19, 2011 1:34 PM
> Subject: A mysteriously deadlock of CXF OnewayProcessorInterceptor
>
>
>> Hi:
>> Recently when I am checking/testing CXF , there is a mysteriously deadlock of CXF Oneway Process; Normally CXF engine will invoke the one way bussiness logical asynchronized ,, so the servlet handle will finished and return back to servlet engine immediately;
>>
>> But sometime, I noticed that the servlet handle(JettyHTTPHandler) at server side doesn't return back to servlet engine(Jetty) immediately , it will waiting until the asynchrouse business logical finished;
>> After dig source of CXF, I find it 's relate to OnewayProcessorInterceptor;But until now I can only show when will the deadlock occurs, but I still can not explain why;
>>
>> Following is details:
>> OnewayProcessInterceptor.handleMessage{
>> synchronized (chain) {
>> message.getExchange().get(Bus.class).getExtension(WorkQueueManager.class)
>> .getAutomaticWorkQueue().execute(new Runnable() {
>> public void run() {
>> synchronized (chain) {
>> System.out.println("--notify all");
>> chain.notifyAll();
>> }
>>
>> chain.resume(); //if chain.resume is called before chain.wait finished , the dead lock will occurs; It seems as chain.resume is synchronized, so it will relock on chain object, so the chain.wait() will deadlocked (... I feel confused for this, because jdk doesn't say so...) ;After chain.resume finished, locking on chain is released, deadlock of chain.wait() is also released; but I don't think this is problem of CXF , maybe it's jdk's problem ?? I feels confused;
>> }
>> });
>> System.out.println("--wait begin");
>> chain.wait(20);
>> System.out.println("--wait end");
>> }
>> }
>> syncrhonized PhaseInterceptorChain.resume(){
>> System.out.println("--api chain resume");
>> ...
>> }
>>
>> if the execute sequence as following, every thing is ok. there is no dead lock;
>> chain.wait enter
>> chian.notify invoked
>> chain.wait return;
>> chain.resume(); //resume also synchronzed on chain object;
>>
>> if the execute sequence as following , dead lock will occurs:
>> chain.wait enter
>> chain.notify
>> chain.resume// ..now waiting on chain will blocked until chain.resume finished(release sync on chain)
>> chain.wait return;
>>
>> following dump on console indicate the above sequence:
>>
>> No DeadLock dump :
>> --wait begin
>> --notify all
>> --wait end
>> --api chain resume
>> product service begin Fri Aug 19 12:10:28 CST 2011 //a lone time(10 seconds) one way business logical begin
>> product service end Fri Aug 19 12:10:38 CST 2011 .//a lone time(10 seconds) one way business logical end;
>>
>>
>> DeadLock Dump:
>> --wait begin
>> --notify all
>> --api chain resume
>> product service begin Fri Aug 19 12:10:40 CST 2011
>> product service end Fri Aug 19 12:10:50 CST 2011
>> --wait end
>>
>>
>> Until now I am not sure if problem is CXF's or JDK's, or something which I don't know cause such a deadlock;
>> I also wrote a simple program to simulate the execute sequnce which causeddead lock in CXF, but the simple program never dead lock;
>>
>> I tried CXF 2.3.3 version && Jetty transport && (JDK1.5_22 || JDK 1.6_17) && Windows XP system;
>>
>> I also post this question as a JIRA issue:
>>
Re: A mysteriously deadlock of CXF OnewayProcessorInterceptor
Posted by xuhb <xu...@tongtech.com>.
Sorry, I foget post the issue link:
https://issues.apache.org/jira/browse/CXF-3750
----- Original Message -----
From: "xuhb" <xu...@tongtech.com>
To: <us...@cxf.apache.org>
Sent: Friday, August 19, 2011 1:34 PM
Subject: A mysteriously deadlock of CXF OnewayProcessorInterceptor
> Hi:
> Recently when I am checking/testing CXF , there is a mysteriously deadlock of CXF Oneway Process; Normally CXF engine will invoke the one way bussiness logical asynchronized ,, so the servlet handle will finished and return back to servlet engine immediately;
>
> But sometime, I noticed that the servlet handle(JettyHTTPHandler) at server side doesn't return back to servlet engine(Jetty) immediately , it will waiting until the asynchrouse business logical finished;
> After dig source of CXF, I find it 's relate to OnewayProcessorInterceptor;But until now I can only show when will the deadlock occurs, but I still can not explain why;
>
> Following is details:
> OnewayProcessInterceptor.handleMessage{
> synchronized (chain) {
> message.getExchange().get(Bus.class).getExtension(WorkQueueManager.class)
> .getAutomaticWorkQueue().execute(new Runnable() {
> public void run() {
> synchronized (chain) {
> System.out.println("--notify all");
> chain.notifyAll();
> }
>
> chain.resume(); //if chain.resume is called before chain.wait finished , the dead lock will occurs; It seems as chain.resume is synchronized, so it will relock on chain object, so the chain.wait() will deadlocked (... I feel confused for this, because jdk doesn't say so...) ;After chain.resume finished, locking on chain is released, deadlock of chain.wait() is also released; but I don't think this is problem of CXF , maybe it's jdk's problem ?? I feels confused;
> }
> });
> System.out.println("--wait begin");
> chain.wait(20);
> System.out.println("--wait end");
> }
> }
> syncrhonized PhaseInterceptorChain.resume(){
> System.out.println("--api chain resume");
> ...
> }
>
> if the execute sequence as following, every thing is ok. there is no dead lock;
> chain.wait enter
> chian.notify invoked
> chain.wait return;
> chain.resume(); //resume also synchronzed on chain object;
>
> if the execute sequence as following , dead lock will occurs:
> chain.wait enter
> chain.notify
> chain.resume// ..now waiting on chain will blocked until chain.resume finished(release sync on chain)
> chain.wait return;
>
> following dump on console indicate the above sequence:
>
> No DeadLock dump :
> --wait begin
> --notify all
> --wait end
> --api chain resume
> product service begin Fri Aug 19 12:10:28 CST 2011 //a lone time(10 seconds) one way business logical begin
> product service end Fri Aug 19 12:10:38 CST 2011 .//a lone time(10 seconds) one way business logical end;
>
>
> DeadLock Dump:
> --wait begin
> --notify all
> --api chain resume
> product service begin Fri Aug 19 12:10:40 CST 2011
> product service end Fri Aug 19 12:10:50 CST 2011
> --wait end
>
>
> Until now I am not sure if problem is CXF's or JDK's, or something which I don't know cause such a deadlock;
> I also wrote a simple program to simulate the execute sequnce which causeddead lock in CXF, but the simple program never dead lock;
>
> I tried CXF 2.3.3 version && Jetty transport && (JDK1.5_22 || JDK 1.6_17) && Windows XP system;
>
> I also post this question as a JIRA issue:
>