Intermittent STUCK threads in Weblogic Server due to camel application

[cmy <cm...@gmail.com>]

Hello,
I am seeing intermittent stuck threads in Weblogic 11g Application Server in
Production environment. We created a ticket with Weblogic team and they said
that the issue is caused by
org.apache.camel.processor.DelegateAsysncProcessor.process in the Web
Service we developed.
I am unable to replicate the issue in development environment.  Not sure
what is the problem.   
Has anyone faced similar problems?

Here are some of the additional information you may like to know - 

#1. Call Volume - 350/hr
#2. Camel Version 2.10.3
#3. Http Configuration:
	<http-conf:conduit
	
name="{http://xml.comcast.com/commercialcustomer/services}CommercialCustomerServicePort.http-conduit">
		<http-conf:client ReceiveTimeout="20000"
			ConnectionTimeout="20000" />
	</http-conf:conduit>






--
View this message in context: http://camel.465427.n5.nabble.com/Intermittent-STUCK-threads-in-Weblogic-Server-due-to-camel-application-tp5750624.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Intermittent STUCK threads in Weblogic Server due to camel application

[cmy <cm...@gmail.com>]

I am trying to understand the basics here.  

It is possible to have a STUCK thread only when the application does not
return back the thread to Weblogic Server within the configured time (10
minutes by default).  I have configured my Camel application to disconnect
from service I am consuming if it does not respond within 20 seconds.  I
have done this by setting RecieveTimeOut to 20 sec in http-conf:conduit.

Is there anything else I need to look into to make sure the application is
really disconnecting within 20 seconds?



--
View this message in context: http://camel.465427.n5.nabble.com/Intermittent-STUCK-threads-in-Weblogic-Server-due-to-camel-application-tp5750624p5750652.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Intermittent STUCK threads in Weblogic Server due to camel application

["Preben.Asmussen" <pr...@dr.dk>]

Hi

We are running a lot of Camel routes on Weblogic. The only time I have seen
stuck threads in on the commons.net.ftp, and the uses here is not related to
any Camel component.

"[STUCK] ExecuteThread: '11' for queue: 'weblogic.kernel.Default
(self-tuning)'" - Thread t@65134
   java.lang.Thread.State: RUNNABLE
	at jrockit.net.SocketNativeIO.readBytesPinned(Native Method)
	at jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:32)
	at java.net.SocketInputStream.socketRead0(SocketInputStream.java)
	at java.net.SocketInputStream.read(SocketInputStream.java:129)
	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
	at java.io.InputStreamReader.read(InputStreamReader.java:167)
	- locked <2fb4f2d3> (a java.io.InputStreamReader)
	at java.io.BufferedReader.fill(BufferedReader.java:136)
	at java.io.BufferedReader.read(BufferedReader.java:157)
	- locked <2fb4f2d3> (a java.io.InputStreamReader)
	at
org.apache.commons.net.io.CRLFLineReader.readLine(CRLFLineReader.java:58)
	- locked <2fb4f2d3> (a java.io.InputStreamReader)
	at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:310)
	at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:290)
	at org.apache.commons.net.ftp.FTP._connectAction_(FTP.java:396)
	at org.apache.commons.net.ftp.FTPClient._connectAction_(FTPClient.java:796)
	at org.apache.commons.net.SocketClient.connect(SocketClient.java:172)
	at org.apache.commons.net.SocketClient.connect(SocketClient.java:192)


Are you using CXF or http component in your route ?.
Are you using the http configuration as consumer or provider ?

I have used camel-cxf in many cases without any stuck thread issues.

/Preben



--
View this message in context: http://camel.465427.n5.nabble.com/Intermittent-STUCK-threads-in-Weblogic-Server-due-to-camel-application-tp5750624p5750643.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Intermittent STUCK threads in Weblogic Server due to camel application

[cmy <cm...@gmail.com>]

Hi Willem,
Thanks for your suggestion.  We configured the producer endpoints to
synchronous and re-ran the load test.  We ran two versions in parallel - one
with asynchronous (default) and the other with synchronous.  In our load
test that ran about 12 hrs continuously we observed two stuck threads in
weblogic server for asynchronous version and no stuck thread for the
synchronous version.

We are still planning to run more tests to see if the result is consistent. 
But I am interested to know why asynchronous approach is causing stuck
threads in weblogic app server.  Is it a known bug in CAMEL?  Can we expect
the same behavior in other application servers also?



--
View this message in context: http://camel.465427.n5.nabble.com/Intermittent-STUCK-threads-in-Weblogic-Server-due-to-camel-application-tp5750624p5752162.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Intermittent STUCK threads in Weblogic Server due to camel application

[Willem Jiang <wi...@gmail.com>]

Can you try to setup the camel cxf producer endpoint synchronous option to be true?
With help of the this option camel-cxf producer can use the thread in a manageable way.

--  
Willem Jiang

Red Hat, Inc.
Web: http://www.redhat.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.iteye.com (Chinese)
Twitter: willemjiang  
Weibo: 姜宁willem



On April 29, 2014 at 6:04:21 AM, cmy (cmyahya@gmail.com) wrote:
> Does anyone have any thought on this issue?
>  
> Thanks in advance..
>  
>  
>  
> --
> View this message in context: http://camel.465427.n5.nabble.com/Intermittent-STUCK-threads-in-Weblogic-Server-due-to-camel-application-tp5750624p5750662.html  
> Sent from the Camel - Users mailing list archive at Nabble.com.
>  

Re: Intermittent STUCK threads in Weblogic Server due to camel application

[cmy <cm...@gmail.com>]

Does anyone have any thought on this issue?

Thanks in advance..



--
View this message in context: http://camel.465427.n5.nabble.com/Intermittent-STUCK-threads-in-Weblogic-Server-due-to-camel-application-tp5750624p5750662.html
Sent from the Camel - Users mailing list archive at Nabble.com.