Tomcat performance problem

[Alex Moskvin <mo...@gmail.com>]

Hi,

I am using Tomcat 7.0.30 and OpenEJB 4.5 to host high loaded app and at
some point found with VisualVM there is a bottleneck when there is a lot of
concurrent requests (about 400-600 req/sec). Requests and responses are
small (usually not larger than 2kb in JSON format) - almost all CPU time is
occupied by TaskQueue.poll() calls and general response latency becomes
very slow.
I tried APR/Nio connectors and different threading setting and that didn't
help anyhow. Tuning (increasing) amount of threads in thread pools didn't
help as well.
Tests were conducted on Intel Core i7 3450, 8Gb RAM, 100Mb network, RHEL
6.3 x64.

How to interpret this relatively low request rate? What I might be doing
wrong? What can be tuned in Tomcat to improve request handling?
Because we expect much larger request rate processing speed on the
mentioned hardware.

Here is a VisualVM top output:

Hot spots - method; self time [%]; self time; self time (cpu)

org.apache.tomcat.util.threads.TaskQueue.poll() 75.508965 57 668 249 ms
(75,5%) 0,000 ms
net.spy.memcached.internal.OperationFuture.get() 14.22508 10 864 079 ms
(14,2%) 6 798 ms
net.spy.memcached.internal.BulkGetFuture.internalGet() 5.6411114 4 308 270
ms (5,6%) 0,000 ms
org.apache.tomcat.util.net.NioEndpoint$Poller.run() 0.71728086 547 806 ms
(0,7%) 530 252 ms
org.quartz.core.QuartzSchedulerThread.run() 0.69727546 532 528 ms (0,7%)
0,000 ms

Regards,
Alex

Re: Tomcat performance problem

[Mark Thomas <ma...@apache.org>]

On 25/11/2012 00:50, Alex Moskvin wrote:
> Hi,
> 
> I am using Tomcat 7.0.30 and OpenEJB 4.5 to host high loaded app and at
> some point found with VisualVM there is a bottleneck when there is a lot of
> concurrent requests (about 400-600 req/sec). Requests and responses are
> small (usually not larger than 2kb in JSON format) - almost all CPU time is
> occupied by TaskQueue.poll() calls and general response latency becomes
> very slow.
> I tried APR/Nio connectors and different threading setting and that didn't
> help anyhow. Tuning (increasing) amount of threads in thread pools didn't
> help as well.
> Tests were conducted on Intel Core i7 3450, 8Gb RAM, 100Mb network, RHEL
> 6.3 x64.
> 
> How to interpret this relatively low request rate? What I might be doing
> wrong? What can be tuned in Tomcat to improve request handling?
> Because we expect much larger request rate processing speed on the
> mentioned hardware.
> 
> Here is a VisualVM top output:
> 
> Hot spots - method; self time [%]; self time; self time (cpu)
> 
> org.apache.tomcat.util.threads.TaskQueue.poll() 75.508965 57 668 249 ms
> (75,5%) 0,000 ms

That indicates idle threads waiting for work to do. Not an issue.

> net.spy.memcached.internal.OperationFuture.get() 14.22508 10 864 079 ms
> (14,2%) 6 798 ms
> net.spy.memcached.internal.BulkGetFuture.internalGet() 5.6411114 4 308 270
> ms (5,6%) 0,000 ms

This is more interesting. That looks like the application is waiting for
memCached to complete a task. I'd look into this first.

> org.apache.tomcat.util.net.NioEndpoint$Poller.run() 0.71728086 547 806 ms
> (0,7%) 530 252 ms

That is the NIO connector's Poller processing incoming connections. I'm
a little surprised to see this appearing rather than application code
but given the relatively low % I'd not focus too much time on here just yet.

Mark


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Tomcat performance problem

[Tony Anecito <ad...@yahoo.com>]

Hi Alex,
 
Typically I have found the first issue is the response time of the database queries. Threads and sockets will start to reach a limit when this happens.
Then after that is looked at then the next issue will be understanding what is really possible with the system. Just going from EJB container to Tomcat on an I7 will take at least 5-6msec. Depending on EJB types caching at that layer and the Tomcat layer may be needed.
Sorry you are just starting tuning. It took me years to get to sub-millisecond response from Database to web service interface and to see it in my logs below a millisecond and then I started to see latency from network external to my company.
 
Good Hunting.
-Tony

--- On Sat, 11/24/12, Alex Moskvin <mo...@gmail.com> wrote:


From: Alex Moskvin <mo...@gmail.com>
Subject: Tomcat performance problem
To: users@tomcat.apache.org, users-help@tomcat.apache.org
Date: Saturday, November 24, 2012, 5:50 PM


Hi,

I am using Tomcat 7.0.30 and OpenEJB 4.5 to host high loaded app and at
some point found with VisualVM there is a bottleneck when there is a lot of
concurrent requests (about 400-600 req/sec). Requests and responses are
small (usually not larger than 2kb in JSON format) - almost all CPU time is
occupied by TaskQueue.poll() calls and general response latency becomes
very slow.
I tried APR/Nio connectors and different threading setting and that didn't
help anyhow. Tuning (increasing) amount of threads in thread pools didn't
help as well.
Tests were conducted on Intel Core i7 3450, 8Gb RAM, 100Mb network, RHEL
6.3 x64.

How to interpret this relatively low request rate? What I might be doing
wrong? What can be tuned in Tomcat to improve request handling?
Because we expect much larger request rate processing speed on the
mentioned hardware.

Here is a VisualVM top output:

Hot spots - method; self time [%]; self time; self time (cpu)

org.apache.tomcat.util.threads.TaskQueue.poll() 75.508965 57 668 249 ms
(75,5%) 0,000 ms
net.spy.memcached.internal.OperationFuture.get() 14.22508 10 864 079 ms
(14,2%) 6 798 ms
net.spy.memcached.internal.BulkGetFuture.internalGet() 5.6411114 4 308 270
ms (5,6%) 0,000 ms
org.apache.tomcat.util.net.NioEndpoint$Poller.run() 0.71728086 547 806 ms
(0,7%) 530 252 ms
org.quartz.core.QuartzSchedulerThread.run() 0.69727546 532 528 ms (0,7%)
0,000 ms

Regards,
Alex