You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Rhonny David <rh...@yahoo.com> on 2010/12/22 12:48:00 UTC
Tomcat Hangs with variable frequency
Hi ALL,
We are using tomcat 6.0 in production environment and facing a critical problem
that tomcat hangs with variable frequency in a week. We have tried to figure out
the problem but couldn't reach the root cause. To investigate the cause we used
the lambda probe tool. Lamba probe shows multiple threads are blocked when
tomcat hangs. Normally i have seen a similar type of stack trace for the thread
which lambda probe shows as blocked, I am pasting it as follows:
net.sourceforge.jtds.jdbc.SharedSocket.getNetPacket ( SharedSocket.java:693 )
net.sourceforge.jtds.jdbc.ResponseStream.getPacket ( ResponseStream.java:466 )
net.sourceforge.jtds.jdbc.ResponseStream.read ( ResponseStream.java:103 )
net.sourceforge.jtds.jdbc.TdsCore.nextToken ( TdsCore.java:2202 )
net.sourceforge.jtds.jdbc.TdsCore.getNextRow ( TdsCore.java:764 )
net.sourceforge.jtds.jdbc.JtdsResultSet.next ( JtdsResultSet.java:593 )
net.sourceforge.jtds.jdbc.JtdsResultSet.close ( JtdsResultSet.java:486 )
org.apache.tomcat.dbcp.dbcp.DelegatingResultSet.close (
DelegatingResultSet.java:187 )
org.apache.tomcat.dbcp.dbcp.DelegatingStatement.close (
DelegatingStatement.java:163 )
org.apache.tomcat.dbcp.dbcp.DelegatingConnection.passivate (
DelegatingConnection.java:426 )
org.apache.tomcat.dbcp.dbcp.DelegatingConnection.close (
DelegatingConnection.java:246 )
org.apache.tomcat.dbcp.dbcp.PoolableConnection.reallyClose (
PoolableConnection.java:122 )
org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.destroyObject (
PoolableConnectionFactory.java:628 )
org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.invalidateObject (
GenericObjectPool.java:1247 )
org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.invalidateObject (
AbandonedObjectPool.java:125 )
org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.removeAbandoned (
AbandonedObjectPool.java:158 )
org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.borrowObject (
AbandonedObjectPool.java:77 )
org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection (
PoolingDataSource.java:106 )
org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection (
BasicDataSource.java:1044 )
com.avanza.contactcenter.dbutil.DBConnection.getConnection (
DBConnection.java:154 )
com.avanza.contactcenter.db.TableFieldsMappingProcessor.getDynamicColumnsReportList
( TableFieldsMappingProcessor.java:696 )
com.avanza.contactcenter.processor.CustomerDetailActionProcessor.CustomerDetailInfoSearch
( CustomerDetailActionProcessor.java:187 )
com.avanza.contactcenter.processor.RdvInboundActionProcessor.responseCustomerDetailInfo
( RdvInboundActionProcessor.java:817 )
com.avanza.contactcenter.integration.RendezvousMessageSource.processIncomingMessage
( RendezvousMessageSource.java:431 )
com.avanza.contactcenter.integration.RendezvousMessageSource.run (
RendezvousMessageSource.java:136 )
As we have been investigating the issue from a month, so i would like to share
it with you though you can understand the scenario in depth. Initially we
thought that there are connection leak problems in our application because the
lambda probe showed us that db connections are continuously busy. We scanned
twice the DAO layer of our application to search any connection leaks, but we
didn't find any. At that time our application's Context.xml file didn't contain
the removeAbandoned property. After DAO layer scan, we decide to set
the removeAbandoned = true, so that connection pool refreshes at run time. But
this also didn't work for us. Lambda probe still showed us blocked threads when
tomcat hanged, but yes by setting the removeAbandoned = true connection pool
continued to refresh.
When tomcat hanged today we noticed following statistics on lambda probe, (along
with blocked threads for which stack trace is already provided above):
System OVerView
Memory utilization
-- Free: 469.95 MB Total: 595.5 MB Max: 1,016.12 MB
OS information
JVM: Java(TM) 2 Runtime Environment, Standard Edition 1.5.0_10-b03 Java
HotSpot(TM) Client VM)
OS: Windows 2003 (Service Pack 2) x86 5.2
Processors: 8
Current time: Wed Dec 22 13:00:48 GMT+05:00 2010
Working dir: C:\Program Files\Apache Software Foundation\Tomcat 6.0
Current memory usage
Used Committed Maximum Initial Group
Survivor Space: 1.26Mb 4.63Mb 7.88Mb 4.56Mb HEAP
Perm Gen: 34.28Mb 34.50Mb 64.00Mb 8.00Mb NON_HEAP
Tenured Gen: 115.20Mb 553.88Mb 945.25Mb 553.88Mb HEAP
Eden Space: 2.08Mb 37.00Mb 63.00Mb 37.00Mb HEAP
Code Cache 12.37Mb 12.47Mb 32.00Mb 192.00Kb NON_HEAP
Total 165.65Mb 642.47Mb 1.09Gb 603.63Mb TOTAL
OS information
OS Name: Windows 2003 OS Version: 5.2 Total RAM: 2.00Gb Free
RAM: 2.00Gb Committed JVM memory: 681.23Mb Total swap: 4.00Gb Free swap: 4.00Gb
(In above line,Free RAM is 2GB, it might be due to that operating system is 32
bit and hardware is also 32 bit)
STATUS:
ajp-8009
Current thread count: 0 Current threads busy: 0 Max threads: 200 Max spare
threads: 0 Min spare threads: 0
Max time (ms): 0 Processing time (ms): 0 Request count: 0 Error count: 0
Received: 0b Sent: 0b
No processors
http-8080
Current thread count: 60 Current threads busy: 5 Max threads: 200 Max spare
threads: 0 Min spare threads: 0
Max time (ms): 430500 Processing time (ms): 133505594 Request count: 390058
Error count: 8904 Received: 24Mb Sent: 4Gb
Context.xml
<Resource factory="org.apache.tomcat.dbcp.dbcp.BasicDataSourceFactory"
auth="Application" removeAbandoned="true" logAbandoned="true"
validationQuery="select 1" driverClassName="net.sourceforge.jtds.jdbc.Driver"
maxActive="60" maxIdle="20" maxWait="10000" name="DB_Pool"
type="javax.sql.DataSource" url="jdbc:jtds:sqlserver://10.22.4.29/XYZ" />
Note that we have noticed multiple times that there are no deadlocks at database
end when tomcat crashed and memory and CPU utilization of application server
machine and database machine is always normal. We are using SQL Server 2005
Enterprise in production environment. The number of concurrent users to use the
application simultaneously is 50.
Will appreciate if any one can give the idea by looking at above statistics that
what is area of problem. Will appreciate.
Regards,
David
Re: Tomcat Hangs with variable frequency
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Konstantin,
On 12/22/2010 11:56 AM, Konstantin Kolinko wrote:
> If closing the result set requires reading it in full (as JTDS does),
> it can take significant time.
Especially if the queries are something like
SELECT * FROM huge_table
> You should take thread dumps to see what the threads are doing and
> what they are waiting for. (Three thread dumps taken several seconds
> apart).
I would also inspect the queries that are in progress when these
connections appear to hang.
We had a problem with Oracle long ago where the table indexes had just
become horribly fragmented and performance slowed to a crawl. In an act
of desperation, someone ran "OPTIMIZE TABLE" on one of the tables
involved in particularly slow queries and basically everything got
better. From then on out, we called that "turning off the 'suck' bit" on
Oracle. :)
I would definitely like to see what queries are running on the server
and what their status is.
The problem is certainly /not/ Tomcat, here: threads are stuck in a
3rd-party library connecting to a 3rd-party database. Tomcat just
happens to be in the stack trace because that's the DBCP in use: the
driver appears to be hanging, not Tomcat's DBCP.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk0SNXUACgkQ9CaO5/Lv0PDAUQCfWgdgZY61qly2Z0xDP4vhZcbX
OyYAn14lv4chUAmiD4h4z3jqgDPxOGda
=Fqs8
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Hangs with variable frequency
Posted by Konstantin Kolinko <kn...@gmail.com>.
If closing the result set requires reading it in full (as JTDS does),
it can take significant time.
You should take thread dumps to see what the threads are doing and
what they are waiting for. (Three thread dumps taken several seconds
apart).
Post them inline, or somewhere else (like pastebin site). Attachments
are usually stripped off by the mailing list software.
2010/12/22 Rhonny David <rh...@yahoo.com>:
> I won't support your reasoning, because the same JTDS driver version we are
> using for our other site and there the number of concurrent users of the
> application is more than 800 and therefore the volume of concurrent requests is
> very high, but we never faced the issue of this nature. Secondly, we have
> recently started using JTD driver and tomcat jndi connection pooling for our
> current site and earlier we used microsoft driver and our own connection pooling
> mechaninsm, but the same issue of tomcat hanging was occurring at that time.
>
>
> Regards,
> David
>
>
>
> ________________________________
> From: Konstantin Kolinko <kn...@gmail.com>
> To: Tomcat Users List <us...@tomcat.apache.org>
> Sent: Wed, December 22, 2010 6:53:46 PM
> Subject: Re: Tomcat Hangs with variable frequency
>
>> net.sourceforge.jtds.jdbc.JtdsResultSet.close ( JtdsResultSet.java:486 )
>
> I think it is an issue with implementation of JtdsResultSet.close(), [1]
>
> 0485: if (!getConnection().isClosed()) {
> 0486: // Skip to end of result set
> 0487: // Could send cancel but this is safer as
> 0488: // cancel could kill other statements
> in a batch.
> 0489: while (next())
> 0490: ;
> 0491: }
>
> [1]
> http://www.java2s.com/Open-Source/Java-Document/Database-JDBC-Connection-Pool/jTDS/net/sourceforge/jtds/jdbc/JtdsResultSet.java.htm
>
>
> [2]
> http://jtds.svn.sourceforge.net/viewvc/jtds/branches/jTDS%201.2%20%28stable%29/src/main/net/sourceforge/jtds/jdbc/JtdsResultSet.java?revision=1107&view=markup#l467
>
>
> It tries to dispose the resultset by reading through it and hangs in
> the process (or maybe the resultset is too big and reading it takes a
> while).
>
> Best regards,
> Konstantin Kolinko
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Hangs with variable frequency
Posted by Rhonny David <rh...@yahoo.com>.
I won't support your reasoning, because the same JTDS driver version we are
using for our other site and there the number of concurrent users of the
application is more than 800 and therefore the volume of concurrent requests is
very high, but we never faced the issue of this nature. Secondly, we have
recently started using JTD driver and tomcat jndi connection pooling for our
current site and earlier we used microsoft driver and our own connection pooling
mechaninsm, but the same issue of tomcat hanging was occurring at that time.
Regards,
David
________________________________
From: Konstantin Kolinko <kn...@gmail.com>
To: Tomcat Users List <us...@tomcat.apache.org>
Sent: Wed, December 22, 2010 6:53:46 PM
Subject: Re: Tomcat Hangs with variable frequency
> net.sourceforge.jtds.jdbc.JtdsResultSet.close ( JtdsResultSet.java:486 )
I think it is an issue with implementation of JtdsResultSet.close(), [1]
0485: if (!getConnection().isClosed()) {
0486: // Skip to end of result set
0487: // Could send cancel but this is safer as
0488: // cancel could kill other statements
in a batch.
0489: while (next())
0490: ;
0491: }
[1]
http://www.java2s.com/Open-Source/Java-Document/Database-JDBC-Connection-Pool/jTDS/net/sourceforge/jtds/jdbc/JtdsResultSet.java.htm
[2]
http://jtds.svn.sourceforge.net/viewvc/jtds/branches/jTDS%201.2%20%28stable%29/src/main/net/sourceforge/jtds/jdbc/JtdsResultSet.java?revision=1107&view=markup#l467
It tries to dispose the resultset by reading through it and hangs in
the process (or maybe the resultset is too big and reading it takes a
while).
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Hangs with variable frequency
Posted by Konstantin Kolinko <kn...@gmail.com>.
> net.sourceforge.jtds.jdbc.JtdsResultSet.close ( JtdsResultSet.java:486 )
I think it is an issue with implementation of JtdsResultSet.close(), [1]
0485: if (!getConnection().isClosed()) {
0486: // Skip to end of result set
0487: // Could send cancel but this is safer as
0488: // cancel could kill other statements
in a batch.
0489: while (next())
0490: ;
0491: }
[1] http://www.java2s.com/Open-Source/Java-Document/Database-JDBC-Connection-Pool/jTDS/net/sourceforge/jtds/jdbc/JtdsResultSet.java.htm
[2] http://jtds.svn.sourceforge.net/viewvc/jtds/branches/jTDS%201.2%20%28stable%29/src/main/net/sourceforge/jtds/jdbc/JtdsResultSet.java?revision=1107&view=markup#l467
It tries to dispose the resultset by reading through it and hangs in
the process (or maybe the resultset is too big and reading it takes a
while).
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Hangs with variable frequency
Posted by Rhonny David <rh...@yahoo.com>.
I think i have already provided these except tomcat. Anyways we are using Tomcat
6.0.29, which is pointed to jre 1.5.0_10. And the OS is Microsoft Windows Server
2003, R2 , Standard Edition, Service Pack2.
Regards,
David
________________________________
From: Mark Thomas <ma...@apache.org>
To: Tomcat Users List <us...@tomcat.apache.org>
Sent: Wed, December 22, 2010 4:52:52 PM
Subject: Re: Tomcat Hangs with variable frequency
On 22/12/2010 11:48, Rhonny David wrote:
> Hi ALL,
>
>
>
> We are using tomcat 6.0 in production environment
Exactly which version? Also exact versions for JDK and OS please.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Hangs with variable frequency
Posted by Mark Thomas <ma...@apache.org>.
On 22/12/2010 11:48, Rhonny David wrote:
> Hi ALL,
>
>
>
> We are using tomcat 6.0 in production environment
Exactly which version? Also exact versions for JDK and OS please.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org