You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by "Zhou, Yuding (Nokia - CN/Chengdu)" <yu...@nokia.com> on 2016/07/01 04:22:10 UTC

Huge amount of ActiveMQ threads which leads application slowness

Hi,  there

The application is running in Servcemix contains as a SA. The application has two components : DataCollector and DataConverter.

DataCollector uses fileAlternativeMonitor to monitor directory and copy file to DataConverter.
DataConverter parse raw file, mostly it is CSV file.
The application doesn't use Servicemix or ActiveMQ, but only boot by servicemix. Xbean attached.
xbean.xml<http://activemq.2283324.n4.nabble.com/file/n4713535/xbean.xml>


Symptom:

1.  DataCollector uses fileAlternativeMonitor to monitor directory and copy file to DataConverter. It basically have 1 thread for whole application.

2.  DataConverter has around 80 threads, around 75 threads are in WAITING, since no raw data for converting.

3.  [Suspicion] ActiveMQ transportation threads amount is huge, around 60 ~ 80, most of them are parking on socketRead0 () method
ActiveMQ Transport: tcp:///127.0.0.1:56204" - Thread t@279
   java.lang.Thread.State: RUNNABLE
                at java.net.SocketInputStream.socketRead0(Native Method)
                at java.net.SocketInputStream.read(Unknown Source)
                at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
                at org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:576)
                at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
                at org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:561)
                at java.io.DataInputStream.readInt(Unknown Source)
                at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
                at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:227)
                at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:219)
                at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:202)
                at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
                - None


4.   [Suspicion] ActiveMQConnection threads amount is huge, about 30~ 40. All of them are WAITING on
"ActiveMQConnection[ID:gccs1-52411-1466641430250-2:26] Scheduler" - Thread t@228
   java.lang.Thread.State: WAITING
                at java.lang.Object.wait(Native Method)
                - waiting on <71db1e> (a java.util.TaskQueue)
                at java.lang.Object.wait(Object.java:485)
                at java.util.TimerThread.mainLoop(Unknown Source)
                at java.util.TimerThread.run(Unknown Source)

   Locked ownable synchronizers:
                - None

Thread dump and analyze attached
thread_sort_running.JPG<http://activemq.2283324.n4.nabble.com/file/n4713535/thread_sort_running.JPG>
threaddump-1467175636951.JPG<http://activemq.2283324.n4.nabble.com/file/n4713535/threaddump-1467175636951.JPG>
threaddump-1467175636951.tdump<http://activemq.2283324.n4.nabble.com/file/n4713535/threaddump-1467175636951.tdump>


I'm new to ActiveMQ and servicemix, please let me know what is the suspicion and any clues available.

Appreciate for your support, thanks!

Regards
Yuding

Re: Huge amount of ActiveMQ threads which leads application slowness

Posted by Tim Bain <tb...@alumni.duke.edu>.
What evidence do you have that the number of threads is what leads to
application slowness?  You've stated that conclusion as a fact, but modem
JVMs are generally pretty efficient at handling threads, and I wouldn't
expect them to have issues with a few hundred that are doing occasional
I/O.  If it really is a problem, the NIO transport could help, but I'd
suggest you do more investigation before you conclude that threading really
is the issue.

For any performance problem, I'd always start by characterizing which
resource is the bottleneck.  If the CPU is pegged, then your application
code (including libraries) is doing lots of work, or the JVM id doing lots
of work (thread context switches or garbage collection, typically), or some
combination of the two.  At that point you'll want to attach a profiler or
a sampler to the process to see where the various threads are spending
their time, as well as finding out if you're spending an unreasonable
amount of time doing full GCs.  If CPU usage is low, something else is your
bottleneck: possibly memory swapping, possibly disk throughput, possibly
network throughput, possibly slow resources at the other end of a network
connection, and you'll have to dig into that.

Tim
On Jun 30, 2016 10:22 PM, "Zhou, Yuding (Nokia - CN/Chengdu)" <
yuding.zhou@nokia.com> wrote:

Hi,  there

The application is running in Servcemix contains as a SA. The application
has two components : DataCollector and DataConverter.

DataCollector uses fileAlternativeMonitor to monitor directory and copy
file to DataConverter.
DataConverter parse raw file, mostly it is CSV file.
The application doesn't use Servicemix or ActiveMQ, but only boot by
servicemix. Xbean attached.
xbean.xml<http://activemq.2283324.n4.nabble.com/file/n4713535/xbean.xml>


Symptom:

1.  DataCollector uses fileAlternativeMonitor to monitor directory and copy
file to DataConverter. It basically have 1 thread for whole application.

2.  DataConverter has around 80 threads, around 75 threads are in WAITING,
since no raw data for converting.

3.  [Suspicion] ActiveMQ transportation threads amount is huge, around 60 ~
80, most of them are parking on socketRead0 () method
ActiveMQ Transport: tcp:///127.0.0.1:56204" - Thread t@279
   java.lang.Thread.State: RUNNABLE
                at java.net.SocketInputStream.socketRead0(Native Method)
                at java.net.SocketInputStream.read(Unknown Source)
                at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
                at
org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:576)
                at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
                at
org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:561)
                at java.io.DataInputStream.readInt(Unknown Source)
                at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
                at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:227)
                at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:219)
                at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:202)
                at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
                - None


4.   [Suspicion] ActiveMQConnection threads amount is huge, about 30~ 40.
All of them are WAITING on
"ActiveMQConnection[ID:gccs1-52411-1466641430250-2:26] Scheduler" - Thread
t@228
   java.lang.Thread.State: WAITING
                at java.lang.Object.wait(Native Method)
                - waiting on <71db1e> (a java.util.TaskQueue)
                at java.lang.Object.wait(Object.java:485)
                at java.util.TimerThread.mainLoop(Unknown Source)
                at java.util.TimerThread.run(Unknown Source)

   Locked ownable synchronizers:
                - None

Thread dump and analyze attached
thread_sort_running.JPG<
http://activemq.2283324.n4.nabble.com/file/n4713535/thread_sort_running.JPG>
threaddump-1467175636951.JPG<
http://activemq.2283324.n4.nabble.com/file/n4713535/threaddump-1467175636951.JPG
>
threaddump-1467175636951.tdump<
http://activemq.2283324.n4.nabble.com/file/n4713535/threaddump-1467175636951.tdump
>


I'm new to ActiveMQ and servicemix, please let me know what is the
suspicion and any clues available.

Appreciate for your support, thanks!

Regards
Yuding