Huge amount ActiveMQ threads leads application slowness

[Yuding Zhou <yu...@nokia.com>]

Hi,

The application is running in Servcemix contains as a SA. The application
has two components : DataCollector and DataConverter. 

DataCollector uses fileAlternativeMonitor to monitor directory and copy file
to DataConverter.
DataConverter parse raw file, mostly it is CSV file.
The application doesn't use Servicemix or ActiveMQ, but only boot by
servicemix. Xbean attached.
xbean.xml <http://activemq.2283324.n4.nabble.com/file/n4713535/xbean.xml>  

Symptom:


1.  DataCollector uses fileAlternativeMonitor to monitor directory and copy
file to DataConverter. It basically have 1 thread for whole application.

2.  DataConverter has around 80 threads, around 75 threads are in WAITING,
since no raw data for converting.

3.  [Suspicion] ActiveMQ transportation threads amount is huge, around 60 ~
80, most of them are parking on socketRead0 () method  
ActiveMQ Transport: tcp:///127.0.0.1:56204" - Thread t@279
   java.lang.Thread.State: RUNNABLE
                at java.net.SocketInputStream.socketRead0(Native Method)
                at java.net.SocketInputStream.read(Unknown Source)
                at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
                at
org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:576)
                at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
                at
org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:561)
                at java.io.DataInputStream.readInt(Unknown Source)
                at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
                at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:227)
                at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:219)
                at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:202)
                at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
                - None


4.   [Suspicion] ActiveMQConnection threads amount is huge, about 30~ 40.
All of them are WAITING on 
"ActiveMQConnection[ID:gccs1-52411-1466641430250-2:26] Scheduler" - Thread
t@228
   java.lang.Thread.State: WAITING
                at java.lang.Object.wait(Native Method)
                - waiting on <71db1e> (a java.util.TaskQueue)
                at java.lang.Object.wait(Object.java:485)
                at java.util.TimerThread.mainLoop(Unknown Source)
                at java.util.TimerThread.run(Unknown Source)

   Locked ownable synchronizers:
                - None

Thread dump and analyze attached thread_sort_running.JPG
<http://activemq.2283324.n4.nabble.com/file/n4713535/thread_sort_running.JPG>  
threaddump-1467175636951.JPG
<http://activemq.2283324.n4.nabble.com/file/n4713535/threaddump-1467175636951.JPG>  
threaddump-1467175636951.tdump
<http://activemq.2283324.n4.nabble.com/file/n4713535/threaddump-1467175636951.tdump>  


I'm new to ActiveMQ and servicemix, please let me know what is the suspicion
and any clues available.

Appreciate for your support, thanks!

Regards
Yuding



--
View this message in context: http://activemq.2283324.n4.nabble.com/Huge-amount-ActiveMQ-threads-leads-application-slowness-tp4713535.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.