You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Pid * <pi...@pidster.com> on 2012/01/27 10:03:43 UTC
Re: Tomcat 6.0.35-SocketException: Too many open files issue with
On 27 Jan 2012, at 05:32, gnath <ga...@yahoo.com> wrote:
> Hello Chris,
>
>
> After seeing the initial connection pool issue, i started searching online for help and i found this article :
> http://vigilbose.blogspot.com/2009/03/apache-commons-dbcp-and-tomcat-jdbc.html
> so, i thought may be tomcat's jar would bring some improvement. by the way, we had commons-dbcp-1.3.jar. Do you recommend upgrading to newer commons dbcp jar instead of using tomcat-jdbc.jar.
Tomcat ships with a DBCP implementing of its own.
How and where are you defining the database?
p
>
> Just because we are running tomcat-6.0.35, it did not come with tomcat-jdbc.jar, so we downloaded the 1.1.1 version or jar and dropped in WEB-INF/lib and started using it.
>
>
> I agree what you are saying about leaking the connection and will plan to set the logAbandoned flag as you suggested.
>
>
> However, i was about to file a new issue but would like to describe here as well. So we have 2 servers running tomcat (same code, same configuration). After we replaced tomcat-jdbc.jar and added 'removeAbandoned' flag to true, one of the servers is doing great (ofcourse i agree that pool is cleaning up the mess), but we saw one new issue on the second server. it hasn't been releasing the connections and was consistently growing slowly. So i collected thread dump and i saw a deadlock :
>
> Found one Java-level deadlock:
> =============================
> "catalina-exec-1":
> waiting to lock monitor 0x000000005d7944b8 (object 0x00000005bd522568, a com.mysql.jdbc.Connection),
> which is held by "[Pool-Cleaner]:Tomcat Connection Pool[1-1015483951]"
> "[Pool-Cleaner]:Tomcat Connection Pool[1-1015483951]":
> waiting to lock monitor 0x000000005dcdea28 (object 0x00000005bd659ce8, a com.mysql.jdbc.ResultSet),
> which is held by "catalina-exec-1"
>
> Java stack information for the threads listed above:
> ===================================================
> "catalina-exec-1":
> at com.mysql.jdbc.Connection.getCharsetConverter(Connection.java:3177)
> - waiting to lock <0x00000005bd522568> (a com.mysql.jdbc.Connection)
> at com.mysql.jdbc.Field.getStringFromBytes(Field.java:583)
> at com.mysql.jdbc.Field.getName(Field.java:487)
> at com.mysql.jdbc.ResultSet.buildIndexMapping(ResultSet.java:593)
> at com.mysql.jdbc.ResultSet.findColumn(ResultSet.java:926)
> - locked <0x00000005bd659ce8> (a com.mysql.jdbc.ResultSet)
> at com.mysql.jdbc.ResultSet.getInt(ResultSet.java:2401)
>
> "[Pool-Cleaner]:Tomcat Connection Pool[1-1015483951]":
> at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
> - waiting to lock <0x00000005bd659ce8> (a com.mysql.jdbc.ResultSet)
> at com.mysql.jdbc.Statement.realClose(Statement.java:1606)
> - locked <0x00000005bd522568> (a com.mysql.jdbc.Connection)
> - locked <0x00000005bd5e81c0> (a com.mysql.jdbc.ServerPreparedStatement)
> at com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:1703)
> at com.mysql.jdbc.ServerPreparedStatement.realClose(ServerPreparedStatement.java:901)
> - locked <0x00000005bd525ba0> (a java.lang.Object)
> - locked <0x00000005bd522568> (a com.mysql.jdbc.Connection)
> - locked <0x00000005bd5e81c0> (a com.mysql.jdbc.ServerPreparedStatement)
> at com.mysql.jdbc.Connection.closeAllOpenStatements(Connection.java:2126)
> at com.mysql.jdbc.Connection.realClose(Connection.java:4422)
> at com.mysql.jdbc.Connection.close(Connection.java:2098)
> - locked <0x00000005bd522568> (a com.mysql.jdbc.Connection)
> at org.apache.tomcat.jdbc.pool.PooledConnection.disconnect(PooledConnection.java:320)
>
>
>
> Please help us on this. Could it be a problem with tomcat-jdbc.jar?
>
> Thanks
> -G
>
>
> ________________________________
> From: Christopher Schultz <ch...@christopherschultz.net>
> To: Tomcat Users List <us...@tomcat.apache.org>
> Sent: Thursday, January 26, 2012 9:41 AM
> Subject: Re: Tomcat 6.0.35-SocketException: Too many open files issue with
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> G,
>
> On 1/25/12 11:53 PM, gnath wrote:
>> As you have suggested, i started collecting the thread dumps
>
> Thread dumps will set you free. Well, not really. Instead, they will
> tell you where your webapp is breaking, which usually means more work
> for you. But at least the hard part is done: finding out what's breaking.
>
>> when it happened again and we saw some kind of DBCP Connection pool
>> issues leading to 'Too Many open files' issue.
>
> That will definitely do it.
>
>> So we decided to replace the commons DBCP with tomcat-jdbc.jar
>> (with same configuration properties).
>
> Why?
>
>> After this change, it seemed for few hours but started seeing in
>> the logs where the Connection Pool jar could not give any
>> connections and seems to be all the connections are busy. So we
>> went ahead and added a configuration property
>> 'removeAbandoned=true' in our Datasource configuration.
>
> I would go back to DBCP unless you think you need to switch for some
> reason.
>
> I suspect you are leaking database connections and don't have a
> suitable timeout for removal of "lost" database connections (or maybe
> didn't have that set up in the first place).
>
> You really need to enable "logAbandoned" so you can find out where
> your connection leaks are, and fix them. In development, set
> maxActive="1" and leave it there, forever. Also, set
> logAbandoned="true" and always run like that in development. Running
> like that in production isn't a bad idea, either.
>
>> We are still watching the performance and the server behavior
>> after these changes. Will keep you posted on how things will turn
>> out or if i see any further issues.
>
> I suspect you are still leaking connections, but your pool is now
> silently cleaning-up after the mess your webapp is making. Instrument
> your pool. Fix your leaks.
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk8hkDkACgkQ9CaO5/Lv0PCxFgCgs+EiV/CNjmCNekeDwKHgnNtZ
> 5LYAoKZUkIAJOK0eItkoHBF3wScK9lQf
> =AyL4
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 6.0.35-SocketException: Too many open files issue with
Posted by Pid <pi...@pidster.com>.
This is top-posting, for anyone who's watching & doesn't know.
p
On 27/01/2012 14:26, gnath wrote:
> We defined our data sources in spring configuration file. We did not have any DB configuration defined on Server.
>
> ________________________________
> From: Pid * <pi...@pidster.com>
> To: Tomcat Users List <us...@tomcat.apache.org>
> Sent: Friday, January 27, 2012 1:03 AM
> Subject: Re: Tomcat 6.0.35-SocketException: Too many open files issue with
>
> On 27 Jan 2012, at 05:32, gnath <ga...@yahoo.com> wrote:
>
>> Hello Chris,
>>
>>
>> After seeing the initial connection pool issue, i started searching online for help and i found this article :
>> http://vigilbose.blogspot.com/2009/03/apache-commons-dbcp-and-tomcat-jdbc.html
>> so, i thought may be tomcat's jar would bring some improvement. by the way, we had commons-dbcp-1.3.jar. Do you recommend upgrading to newer commons dbcp jar instead of using tomcat-jdbc.jar.
>
> Tomcat ships with a DBCP implementing of its own.
>
> How and where are you defining the database?
>
>
> p
>
>
>
>>
>> Just because we are running tomcat-6.0.35, it did not come with tomcat-jdbc.jar, so we downloaded the 1.1.1 version or jar and dropped in WEB-INF/lib and started using it.
>>
>>
>> I agree what you are saying about leaking the connection and will plan to set the logAbandoned flag as you suggested.
>>
>>
>> However, i was about to file a new issue but would like to describe here as well. So we have 2 servers running tomcat (same code, same configuration). After we replaced tomcat-jdbc.jar and added 'removeAbandoned' flag to true, one of the servers is doing great (ofcourse i agree that pool is cleaning up the mess), but we saw one new issue on the second server. it hasn't been releasing the connections and was consistently growing slowly. So i collected thread dump and i saw a deadlock :
>>
>> Found one Java-level deadlock:
>> =============================
>> "catalina-exec-1":
>> waiting to lock monitor 0x000000005d7944b8 (object 0x00000005bd522568, a com.mysql.jdbc.Connection),
>> which is held by "[Pool-Cleaner]:Tomcat Connection Pool[1-1015483951]"
>> "[Pool-Cleaner]:Tomcat Connection Pool[1-1015483951]":
>> waiting to lock monitor 0x000000005dcdea28 (object 0x00000005bd659ce8, a com.mysql.jdbc.ResultSet),
>> which is held by "catalina-exec-1"
>>
>> Java stack information for the threads listed above:
>> ===================================================
>> "catalina-exec-1":
>> at com.mysql.jdbc.Connection.getCharsetConverter(Connection.java:3177)
>> - waiting to lock <0x00000005bd522568> (a com.mysql.jdbc.Connection)
>> at com.mysql.jdbc.Field.getStringFromBytes(Field.java:583)
>> at com.mysql.jdbc.Field.getName(Field.java:487)
>> at com.mysql.jdbc.ResultSet.buildIndexMapping(ResultSet.java:593)
>> at com.mysql.jdbc.ResultSet.findColumn(ResultSet.java:926)
>> - locked <0x00000005bd659ce8> (a com.mysql.jdbc.ResultSet)
>> at com.mysql.jdbc.ResultSet.getInt(ResultSet.java:2401)
>>
>> "[Pool-Cleaner]:Tomcat Connection Pool[1-1015483951]":
>> at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
>> - waiting to lock <0x00000005bd659ce8> (a com.mysql.jdbc.ResultSet)
>> at com.mysql.jdbc.Statement.realClose(Statement.java:1606)
>> - locked <0x00000005bd522568> (a com.mysql.jdbc.Connection)
>> - locked <0x00000005bd5e81c0> (a com.mysql.jdbc.ServerPreparedStatement)
>> at com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:1703)
>> at com.mysql.jdbc.ServerPreparedStatement.realClose(ServerPreparedStatement.java:901)
>> - locked <0x00000005bd525ba0> (a java.lang.Object)
>> - locked <0x00000005bd522568> (a com.mysql.jdbc.Connection)
>> - locked <0x00000005bd5e81c0> (a com.mysql.jdbc.ServerPreparedStatement)
>> at com.mysql.jdbc.Connection.closeAllOpenStatements(Connection.java:2126)
>> at com.mysql.jdbc.Connection.realClose(Connection.java:4422)
>> at com.mysql.jdbc.Connection.close(Connection.java:2098)
>> - locked <0x00000005bd522568> (a com.mysql.jdbc.Connection)
>> at org.apache.tomcat.jdbc.pool.PooledConnection.disconnect(PooledConnection.java:320)
>>
>>
>>
>> Please help us on this. Could it be a problem with tomcat-jdbc.jar?
>>
>> Thanks
>> -G
>>
>>
>> ________________________________
>> From: Christopher Schultz <ch...@christopherschultz.net>
>> To: Tomcat Users List <us...@tomcat.apache.org>
>> Sent: Thursday, January 26, 2012 9:41 AM
>> Subject: Re: Tomcat 6.0.35-SocketException: Too many open files issue with
>>
> G,
>
> On 1/25/12 11:53 PM, gnath wrote:
>>>> As you have suggested, i started collecting the thread dumps
>
> Thread dumps will set you free. Well, not really. Instead, they will
> tell you where your webapp is breaking, which usually means more work
> for you. But at least the hard part is done: finding out what's breaking.
>
>>>> when it happened again and we saw some kind of DBCP Connection pool
>>>> issues leading to 'Too Many open files' issue.
>
> That will definitely do it.
>
>>>> So we decided to replace the commons DBCP with tomcat-jdbc.jar
>>>> (with same configuration properties).
>
> Why?
>
>>>> After this change, it seemed for few hours but started seeing in
>>>> the logs where the Connection Pool jar could not give any
>>>> connections and seems to be all the connections are busy. So we
>>>> went ahead and added a configuration property
>>>> 'removeAbandoned=true' in our Datasource configuration.
>
> I would go back to DBCP unless you think you need to switch for some
> reason.
>
> I suspect you are leaking database connections and don't have a
> suitable timeout for removal of "lost" database connections (or maybe
> didn't have that set up in the first place).
>
> You really need to enable "logAbandoned" so you can find out where
> your connection leaks are, and fix them. In development, set
> maxActive="1" and leave it there, forever. Also, set
> logAbandoned="true" and always run like that in development. Running
> like that in production isn't a bad idea, either.
>
>>>> We are still watching the performance and the server behavior
>>>> after these changes. Will keep you posted on how things will turn
>>>> out or if i see any further issues.
>
> I suspect you are still leaking connections, but your pool is now
> silently cleaning-up after the mess your webapp is making. Instrument
> your pool. Fix your leaks.
>
> -chris
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
--
[key:62590808]
Re: Tomcat 6.0.35-SocketException: Too many open files issue with
Posted by gnath <ga...@yahoo.com>.
We defined our data sources in spring configuration file. We did not have any DB configuration defined on Server.
Thanks
-G
________________________________
From: Pid * <pi...@pidster.com>
To: Tomcat Users List <us...@tomcat.apache.org>
Sent: Friday, January 27, 2012 1:03 AM
Subject: Re: Tomcat 6.0.35-SocketException: Too many open files issue with
On 27 Jan 2012, at 05:32, gnath <ga...@yahoo.com> wrote:
> Hello Chris,
>
>
> After seeing the initial connection pool issue, i started searching online for help and i found this article :
> http://vigilbose.blogspot.com/2009/03/apache-commons-dbcp-and-tomcat-jdbc.html
> so, i thought may be tomcat's jar would bring some improvement. by the way, we had commons-dbcp-1.3.jar. Do you recommend upgrading to newer commons dbcp jar instead of using tomcat-jdbc.jar.
Tomcat ships with a DBCP implementing of its own.
How and where are you defining the database?
p
>
> Just because we are running tomcat-6.0.35, it did not come with tomcat-jdbc.jar, so we downloaded the 1.1.1 version or jar and dropped in WEB-INF/lib and started using it.
>
>
> I agree what you are saying about leaking the connection and will plan to set the logAbandoned flag as you suggested.
>
>
> However, i was about to file a new issue but would like to describe here as well. So we have 2 servers running tomcat (same code, same configuration). After we replaced tomcat-jdbc.jar and added 'removeAbandoned' flag to true, one of the servers is doing great (ofcourse i agree that pool is cleaning up the mess), but we saw one new issue on the second server. it hasn't been releasing the connections and was consistently growing slowly. So i collected thread dump and i saw a deadlock :
>
> Found one Java-level deadlock:
> =============================
> "catalina-exec-1":
> waiting to lock monitor 0x000000005d7944b8 (object 0x00000005bd522568, a com.mysql.jdbc.Connection),
> which is held by "[Pool-Cleaner]:Tomcat Connection Pool[1-1015483951]"
> "[Pool-Cleaner]:Tomcat Connection Pool[1-1015483951]":
> waiting to lock monitor 0x000000005dcdea28 (object 0x00000005bd659ce8, a com.mysql.jdbc.ResultSet),
> which is held by "catalina-exec-1"
>
> Java stack information for the threads listed above:
> ===================================================
> "catalina-exec-1":
> at com.mysql.jdbc.Connection.getCharsetConverter(Connection.java:3177)
> - waiting to lock <0x00000005bd522568> (a com.mysql.jdbc.Connection)
> at com.mysql.jdbc.Field.getStringFromBytes(Field.java:583)
> at com.mysql.jdbc.Field.getName(Field.java:487)
> at com.mysql.jdbc.ResultSet.buildIndexMapping(ResultSet.java:593)
> at com.mysql.jdbc.ResultSet.findColumn(ResultSet.java:926)
> - locked <0x00000005bd659ce8> (a com.mysql.jdbc.ResultSet)
> at com.mysql.jdbc.ResultSet.getInt(ResultSet.java:2401)
>
> "[Pool-Cleaner]:Tomcat Connection Pool[1-1015483951]":
> at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
> - waiting to lock <0x00000005bd659ce8> (a com.mysql.jdbc.ResultSet)
> at com.mysql.jdbc.Statement.realClose(Statement.java:1606)
> - locked <0x00000005bd522568> (a com.mysql.jdbc.Connection)
> - locked <0x00000005bd5e81c0> (a com.mysql.jdbc.ServerPreparedStatement)
> at com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:1703)
> at com.mysql.jdbc.ServerPreparedStatement.realClose(ServerPreparedStatement.java:901)
> - locked <0x00000005bd525ba0> (a java.lang.Object)
> - locked <0x00000005bd522568> (a com.mysql.jdbc.Connection)
> - locked <0x00000005bd5e81c0> (a com.mysql.jdbc.ServerPreparedStatement)
> at com.mysql.jdbc.Connection.closeAllOpenStatements(Connection.java:2126)
> at com.mysql.jdbc.Connection.realClose(Connection.java:4422)
> at com.mysql.jdbc.Connection.close(Connection.java:2098)
> - locked <0x00000005bd522568> (a com.mysql.jdbc.Connection)
> at org.apache.tomcat.jdbc.pool.PooledConnection.disconnect(PooledConnection.java:320)
>
>
>
> Please help us on this. Could it be a problem with tomcat-jdbc.jar?
>
> Thanks
> -G
>
>
> ________________________________
> From: Christopher Schultz <ch...@christopherschultz.net>
> To: Tomcat Users List <us...@tomcat.apache.org>
> Sent: Thursday, January 26, 2012 9:41 AM
> Subject: Re: Tomcat 6.0.35-SocketException: Too many open files issue with
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> G,
>
> On 1/25/12 11:53 PM, gnath wrote:
>> As you have suggested, i started collecting the thread dumps
>
> Thread dumps will set you free. Well, not really. Instead, they will
> tell you where your webapp is breaking, which usually means more work
> for you. But at least the hard part is done: finding out what's breaking.
>
>> when it happened again and we saw some kind of DBCP Connection pool
>> issues leading to 'Too Many open files' issue.
>
> That will definitely do it.
>
>> So we decided to replace the commons DBCP with tomcat-jdbc.jar
>> (with same configuration properties).
>
> Why?
>
>> After this change, it seemed for few hours but started seeing in
>> the logs where the Connection Pool jar could not give any
>> connections and seems to be all the connections are busy. So we
>> went ahead and added a configuration property
>> 'removeAbandoned=true' in our Datasource configuration.
>
> I would go back to DBCP unless you think you need to switch for some
> reason.
>
> I suspect you are leaking database connections and don't have a
> suitable timeout for removal of "lost" database connections (or maybe
> didn't have that set up in the first place).
>
> You really need to enable "logAbandoned" so you can find out where
> your connection leaks are, and fix them. In development, set
> maxActive="1" and leave it there, forever. Also, set
> logAbandoned="true" and always run like that in development. Running
> like that in production isn't a bad idea, either.
>
>> We are still watching the performance and the server behavior
>> after these changes. Will keep you posted on how things will turn
>> out or if i see any further issues.
>
> I suspect you are still leaking connections, but your pool is now
> silently cleaning-up after the mess your webapp is making. Instrument
> your pool. Fix your leaks.
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk8hkDkACgkQ9CaO5/Lv0PCxFgCgs+EiV/CNjmCNekeDwKHgnNtZ
> 5LYAoKZUkIAJOK0eItkoHBF3wScK9lQf
> =AyL4
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org