You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by tc <tc...@summerhost.net> on 2008/01/20 18:03:16 UTC
Tomcat App becomes Unresponsive
I have been asked to investigate a tomcat/java application.
The basic problem is that this application sometimes becomes unresponsive
if more than one user uses it at a time. Restarting Tomcat clears the
problem. I don't think it has ever occurred when only one person was
using the app.
It is a fairly straight forward data-entry app; the data is stored in a
mysql database.
The developers think there may some issue with the database. They have
also done multi-user testing without being able to reproduce the problem.
I'm looking for pointers to any resources on the web or elsewhere that can
help me track down the problem, or ideas about where to look, or what we
can do to improve post-freeze diagnosis of this problem.
I am also a developer, but I don't have a lot of experience with Java or
Tomcat. I'm trying to bring a fresh perspective to this problem.
Thanks.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat App becomes Unresponsive
Posted by Felix Schumacher <fe...@internetallee.de>.
Am Freitag, den 25.01.2008, 16:40 -0500 schrieb tc:
> The calls look like this:
>
> public void testXXX(DataSource ds, String login) {
> PreparedStatement ps=null;
> ResultSet rs=null;
>
> try {
> ...
> }catch (Exception e) {
> System.out.println("Exception: " + e);
> e.printStackTrace();
> }
> finally {
> try {
...
> try {
> if(conn != null || conn.isClosed()) conn.close();
You are closing the connection only, if it is not null and already
closed. So it will most likely be left open. You probably wanted
if ( !(conn == null || conn.isClosed()) ) { conn.close() }; // but why
should the connection be closed here?
If you think you have a datasource connection leak, you can search the
docs
http://tomcat.apache.org/tomcat-6.0-doc/jndi-datasource-examples-howto.html for
"Preventing dB connection pool leak" and setting the logAbandoned,
removeAbandoned and removeAbandonedTimeout parameters in your
datasource.
Bye
Felix
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat App becomes Unresponsive
Posted by tc <tc...@summerhost.net>.
> -----BEGIN PGP SIGNED MESSAGE-----
>
> Are you using a connection pool? If so, which one?
>
I'm trying to figure this out. I don't see any evidence of one configured
in the server.xml file. If not, I presume it would be a good idea for us
to start using one?
> You need to make sure that you call "close" on /everything/ when you're
> done with your Connection, Statement (and subclass), and ResultSet (and
> subclass) objects. Check to make sure that those close calls are done in
> "finally" blocks, and that a failure to close a ResultSet does not cause
> an exception to be thrown that avoids the Connection close.
>
> For example:
>
> Connection conn = getConnection();
>
> PreparedStatement ps = conn.prepareStatement("SELECT 1");
>
> ResultSet rs = ps.executeQuery();
>
> ...
>
> rs.close();
> ps.close();
> conn.close();
>
> return null;
>
> This is not good. What you really need is this:
>
> Connection conn = null;
> PreparedStatement ps = null;
> ResultSet rs = null;
>
> try
> {
> ~ conn = getConnection();
> ~ ps = ...;
> ~ rs = ...;
>
> ~ ...
> }
> finally
> {
> ~ if(null != rs)
> ~ try { rs.close(); } catch (SQLException sqle) { /* log */ }
> ~ if(null != ps)
> ~ try { ps.close(); } catch (SQLException sqle) { /* log */ }
> ~ if(null != conn)
> ~ try { conn.close(); } catch (SQLException sqle) { /* log */ }
> }
>
> Note how all the cleanup is done in the finally block (so the cleanup
> code will run even during an exception situation), and that each "close"
> is called in its own try/catch block: this prevents the failure to close
> the (e.g.) ResultSet from preventing the Statement or Connection from
> closing. Remember to log anything weird.
>
The calls look like this:
public void testXXX(DataSource ds, String login) {
PreparedStatement ps=null;
ResultSet rs=null;
try {
conn = ds.getconnection();
ps = conn.prepareStatement("SELECT * from test");
ps.setString(1,login);
rs = ps.executeQuery();
if(rs.next()){
//do something with the data
}
}catch (Exception e) {
System.out.println("Exception: " + e);
e.printStackTrace();
}
finally {
try {
try {
if(rs != null) rs.close();rs=null;
} catch (Exception e) {
e.printStackTrace();
}
try {
if (ps != null) ps.close();ps=null;
} catch (Exception e) {
e.printStackTrace();
}
try {
if(conn != null || conn.isClosed()) conn.close();
} catch (Exception e) {
e.printStackTrace();
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
I see that we are not using specifically catching "SQLException sqle" -
should we be?
Further info on our system (sorry for the earlier omission):
We are running Red Hat Enterprise Linux 4.0
Tomcat 5.5
Java 1.5.0_11
I also found two different mysql connector files. I'm not sure which one
(if either) is being used:
mysql-connector-java-3.1.14-bin.jar
mysql-connector-java-5.0.0-beta-bin.jar
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat App becomes Unresponsive
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Christopher Schultz [mailto:chris@christopherschultz.net]
> Subject: Re: Tomcat App becomes Unresponsive
>
> David Delbecq has already told you that sending a SIGQUIT to a running
> JVM will generate a thread dump on standard output.
I recently found another tool to take stack dumps of a running JVM:
http://java.sun.com/javase/6/docs/technotes/tools/share/jstack.html
Look here for some other interesting tips:
http://java.sun.com/javase/6/webnotes/trouble/index.html
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat App becomes Unresponsive
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Tc,
tc wrote:
| Thanks for the suggestions. I will have a look at the source and see how
| the app is interacting with the database.
|
| I don't think the whole tomcat server freezes; just the one application.
Okay. That could be an app-specific db pool, or something else.
| I'm not sure how to know what my threads are doing.
David Delbecq has already told you that sending a SIGQUIT to a running
JVM will generate a thread dump on standard output. Since you didn't
tell us what your platform you are on, we'll give you the easiest
response we can. <shrug>
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkeVSfUACgkQ9CaO5/Lv0PDK+ACeNyG8qShOnlVVQGQdWJlHNo8S
n9oAn01ubfjXbKLMe6YjgWguBa9juxMx
=zyAl
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat App becomes Unresponsive
Posted by Martin Gainty <mg...@hotmail.com>.
Is there a database listener installed? and
is it listening on the right port?
do you have the correct connection string?
do you have the correct driver (DB version) (JDBC version) installed?
M-
----- Original Message -----
From: "tc" <tc...@summerhost.net>
To: <us...@tomcat.apache.org>
Sent: Monday, January 21, 2008 8:22 PM
Subject: Re: Tomcat App becomes Unresponsive
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > Tc,
> >
> > tc wrote:
> > | The developers think there may some issue with the database. They
have
> > | also done multi-user testing without being able to reproduce the
> > problem.
> >
> > If they think it's the database, it's probably their code interacting
> > with the database. MySQL is pretty stable, although I have had some
> > legitimate problems with it in the past.
> >
> > The most likely problem is that you have code that is not properly
> > cleaning up after JDBC calls.
> >
> > Are you using a connection pool? If so, which one?
> >
> > You need to make sure that you call "close" on /everything/ when you're
> > done with your Connection, Statement (and subclass), and ResultSet (and
> > subclass) objects. Check to make sure that those close calls are done in
> > "finally" blocks, and that a failure to close a ResultSet does not cause
> > an exception to be thrown that avoids the Connection close.
> >
> > For example:
> >
> > Connection conn = getConnection();
> >
> > PreparedStatement ps = conn.prepareStatement("SELECT 1");
> >
> > ResultSet rs = ps.executeQuery();
> >
> > ...
> >
> > rs.close();
> > ps.close();
> > conn.close();
> >
> > return null;
> >
> > This is not good. What you really need is this:
> >
> > Connection conn = null;
> > PreparedStatement ps = null;
> > ResultSet rs = null;
> >
> > try
> > {
> > ~ conn = getConnection();
> > ~ ps = ...;
> > ~ rs = ...;
> >
> > ~ ...
> > }
> > finally
> > {
> > ~ if(null != rs)
> > ~ try { rs.close(); } catch (SQLException sqle) { /* log */ }
> > ~ if(null != ps)
> > ~ try { ps.close(); } catch (SQLException sqle) { /* log */ }
> > ~ if(null != conn)
> > ~ try { conn.close(); } catch (SQLException sqle) { /* log */ }
> > }
> >
> > Note how all the cleanup is done in the finally block (so the cleanup
> > code will run even during an exception situation), and that each "close"
> > is called in its own try/catch block: this prevents the failure to close
> > the (e.g.) ResultSet from preventing the Statement or Connection from
> > closing. Remember to log anything weird.
> >
> > | I'm looking for pointers to any resources on the web or elsewhere that
> > can
> > | help me track down the problem, or ideas about where to look, or what
we
> > | can do to improve post-freeze diagnosis of this problem.
> >
> > When the app freezes, what are yout threads doing? Does the server stop
> > serving requests entirely, or do some of them work while others do not?
> >
> > Good luck,
> > - -chris
>
>
> Thanks for the suggestions. I will have a look at the source and see how
> the app is interacting with the database.
>
> I don't think the whole tomcat server freezes; just the one application.
>
> I'm not sure how to know what my threads are doing.
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat App becomes Unresponsive
Posted by tc <tc...@summerhost.net>.
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Tc,
>
> tc wrote:
> | The developers think there may some issue with the database. They have
> | also done multi-user testing without being able to reproduce the
> problem.
>
> If they think it's the database, it's probably their code interacting
> with the database. MySQL is pretty stable, although I have had some
> legitimate problems with it in the past.
>
> The most likely problem is that you have code that is not properly
> cleaning up after JDBC calls.
>
> Are you using a connection pool? If so, which one?
>
> You need to make sure that you call "close" on /everything/ when you're
> done with your Connection, Statement (and subclass), and ResultSet (and
> subclass) objects. Check to make sure that those close calls are done in
> "finally" blocks, and that a failure to close a ResultSet does not cause
> an exception to be thrown that avoids the Connection close.
>
> For example:
>
> Connection conn = getConnection();
>
> PreparedStatement ps = conn.prepareStatement("SELECT 1");
>
> ResultSet rs = ps.executeQuery();
>
> ...
>
> rs.close();
> ps.close();
> conn.close();
>
> return null;
>
> This is not good. What you really need is this:
>
> Connection conn = null;
> PreparedStatement ps = null;
> ResultSet rs = null;
>
> try
> {
> ~ conn = getConnection();
> ~ ps = ...;
> ~ rs = ...;
>
> ~ ...
> }
> finally
> {
> ~ if(null != rs)
> ~ try { rs.close(); } catch (SQLException sqle) { /* log */ }
> ~ if(null != ps)
> ~ try { ps.close(); } catch (SQLException sqle) { /* log */ }
> ~ if(null != conn)
> ~ try { conn.close(); } catch (SQLException sqle) { /* log */ }
> }
>
> Note how all the cleanup is done in the finally block (so the cleanup
> code will run even during an exception situation), and that each "close"
> is called in its own try/catch block: this prevents the failure to close
> the (e.g.) ResultSet from preventing the Statement or Connection from
> closing. Remember to log anything weird.
>
> | I'm looking for pointers to any resources on the web or elsewhere that
> can
> | help me track down the problem, or ideas about where to look, or what we
> | can do to improve post-freeze diagnosis of this problem.
>
> When the app freezes, what are yout threads doing? Does the server stop
> serving requests entirely, or do some of them work while others do not?
>
> Good luck,
> - -chris
Thanks for the suggestions. I will have a look at the source and see how
the app is interacting with the database.
I don't think the whole tomcat server freezes; just the one application.
I'm not sure how to know what my threads are doing.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat App becomes Unresponsive
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Tc,
tc wrote:
| The developers think there may some issue with the database. They have
| also done multi-user testing without being able to reproduce the problem.
If they think it's the database, it's probably their code interacting
with the database. MySQL is pretty stable, although I have had some
legitimate problems with it in the past.
The most likely problem is that you have code that is not properly
cleaning up after JDBC calls.
Are you using a connection pool? If so, which one?
You need to make sure that you call "close" on /everything/ when you're
done with your Connection, Statement (and subclass), and ResultSet (and
subclass) objects. Check to make sure that those close calls are done in
"finally" blocks, and that a failure to close a ResultSet does not cause
an exception to be thrown that avoids the Connection close.
For example:
Connection conn = getConnection();
PreparedStatement ps = conn.prepareStatement("SELECT 1");
ResultSet rs = ps.executeQuery();
...
rs.close();
ps.close();
conn.close();
return null;
This is not good. What you really need is this:
Connection conn = null;
PreparedStatement ps = null;
ResultSet rs = null;
try
{
~ conn = getConnection();
~ ps = ...;
~ rs = ...;
~ ...
}
finally
{
~ if(null != rs)
~ try { rs.close(); } catch (SQLException sqle) { /* log */ }
~ if(null != ps)
~ try { ps.close(); } catch (SQLException sqle) { /* log */ }
~ if(null != conn)
~ try { conn.close(); } catch (SQLException sqle) { /* log */ }
}
Note how all the cleanup is done in the finally block (so the cleanup
code will run even during an exception situation), and that each "close"
is called in its own try/catch block: this prevents the failure to close
the (e.g.) ResultSet from preventing the Statement or Connection from
closing. Remember to log anything weird.
| I'm looking for pointers to any resources on the web or elsewhere that can
| help me track down the problem, or ideas about where to look, or what we
| can do to improve post-freeze diagnosis of this problem.
When the app freezes, what are yout threads doing? Does the server stop
serving requests entirely, or do some of them work while others do not?
Good luck,
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkeUt+4ACgkQ9CaO5/Lv0PD+yQCgpb1hfvSJVyQYml3xgIkSDRDw
5WAAn0n7GlGLX/g9WN2gypECeuy1ltAS
=LoXk
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat App becomes Unresponsive
Posted by tc <tc...@summerhost.net>.
Okay, sounds good. Thanks. Now if I could just figure out how to cause
it to happen...
> If you are on a UNIX jvm, issue a signal 3 (SIGQUIT) to the jvm process
> when tomcat stops responding. It will dump on standard-out (normaly
> redirected to catalina.out) the list of Threads with a stacktrace for
> each. By invstigating stacktrace you will get clues as where the problem
> is. I don't know if mysql is protected against deadlocks, but it may be
> user1 locks ressource X, user2 locks ressource Y, user1 wait for Y to
> free, user2 wait for X to free, all other users are waiting for X to
> free, filling in the ThreadQueue, tomcat then look like unresponsive.
>
> Of course the same kind of concurrency issue can occurs with
> synchronized{} blocks
>
> Note, you can also get such dump on windows, it requires you to press
> magic key combinaison in JVM terminal (don't remember which)
> tc a écrit :
>> I have been asked to investigate a tomcat/java application.
>>
>> The basic problem is that this application sometimes becomes
>> unresponsive
>> if more than one user uses it at a time. Restarting Tomcat clears the
>> problem. I don't think it has ever occurred when only one person was
>> using the app.
>>
>> It is a fairly straight forward data-entry app; the data is stored in a
>> mysql database.
>>
>> The developers think there may some issue with the database. They have
>> also done multi-user testing without being able to reproduce the
>> problem.
>>
>> I'm looking for pointers to any resources on the web or elsewhere that
>> can
>> help me track down the problem, or ideas about where to look, or what we
>> can do to improve post-freeze diagnosis of this problem.
>>
>> I am also a developer, but I don't have a lot of experience with Java or
>> Tomcat. I'm trying to bring a fresh perspective to this problem.
>>
>> Thanks.
>>
>>
>> ---------------------------------------------------------------------
>> To start a new topic, e-mail: users@tomcat.apache.org
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat App becomes Unresponsive
Posted by David Delbecq <de...@oma.be>.
If you are on a UNIX jvm, issue a signal 3 (SIGQUIT) to the jvm process
when tomcat stops responding. It will dump on standard-out (normaly
redirected to catalina.out) the list of Threads with a stacktrace for
each. By invstigating stacktrace you will get clues as where the problem
is. I don't know if mysql is protected against deadlocks, but it may be
user1 locks ressource X, user2 locks ressource Y, user1 wait for Y to
free, user2 wait for X to free, all other users are waiting for X to
free, filling in the ThreadQueue, tomcat then look like unresponsive.
Of course the same kind of concurrency issue can occurs with
synchronized{} blocks
Note, you can also get such dump on windows, it requires you to press
magic key combinaison in JVM terminal (don't remember which)
tc a écrit :
> I have been asked to investigate a tomcat/java application.
>
> The basic problem is that this application sometimes becomes unresponsive
> if more than one user uses it at a time. Restarting Tomcat clears the
> problem. I don't think it has ever occurred when only one person was
> using the app.
>
> It is a fairly straight forward data-entry app; the data is stored in a
> mysql database.
>
> The developers think there may some issue with the database. They have
> also done multi-user testing without being able to reproduce the problem.
>
> I'm looking for pointers to any resources on the web or elsewhere that can
> help me track down the problem, or ideas about where to look, or what we
> can do to improve post-freeze diagnosis of this problem.
>
> I am also a developer, but I don't have a lot of experience with Java or
> Tomcat. I'm trying to bring a fresh perspective to this problem.
>
> Thanks.
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org