You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by "Scott,Tim" <Ti...@oclc.org> on 2023/08/25 08:08:40 UTC
DataSource Connection pool leak
Hi,
For various diagnostics, I tried Tomcat 9.0.79 recently on a development machine. It didn't solve the problem I was experiencing - that was later identified as a problem in IntelliJ with remote deployment and is not why I'm mailing.
I tweaked my Tomcat 9.0.79 configuration to start my application manually, as IntelliJ refused to deploy remotely.
This lead to it quickly exhausting its connection pool and then hanging* before the application could complete its startup activity.
* Each request for a connection from the pool timed out. The log shows that all 20 connections were allocated.
I pointed my system back to 9.0.68, and it started up fine.
I tried all versions I could find back from 9.0.78 through 9.0.71 - and ran into the same connection pool problem as 9.0.79 with every version.
v9.0.70 worked.
(well, at least didn't exhibit the same problem - the application completed its startup activity and was operational).
I would therefore conclude that something about the way I'm managing my database connection pool is preventing 9.0.71 onward from freeing up the connections.
My application uses javax.sql.DataSource with Corretto JDK 11 (11.0.16+8-LTS, to be specific).
It struck me as odd as that means that all releases in the last 8 months have this problem for my application. Has anyone else, here, run into a similar problem?
If a pool size of 20 is just too low, I think I'd need to set mine to a few hundred to complete the application startup and I'm not willing to try that without further insight.
Thanks,
Tim
--
Tim Scott (he/him/his)
OCLC * Lead Software Engineer / Technical Product Manager
cc: IT file
Re: Solved: DataSource Connection pool [non] leak
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Tim,
On 8/31/23 04:03, Scott,Tim wrote:
> Hi Chris,
>
>>> Hi all,
>>>
>>> Thanks for your responses. I think I've found the problem.
>>>
>>> My wrapping class which detects the invocation of the close() method to decrement its count is no longer decrementing its count because method.getDeclaringClass() has changed from java.sql.Connection to java.lang.AutoCloseable.
>>>
>>> Is it safe to check for either java.sql.Connection or java.lang.AutoCloseable?
>>>
>>> .. or should I just check for the "close()" method invocation, based on the fact that I'm not wrapping anything (here) that I shouldn't be counting?
>
>> That depends upon the point of the whole thing. What, um, is the point
>> of the whole thing?
>
> The wrapping class is to limit the number of active connections, waiting up to a configured time for one to become idle if necessary. I can't recall if this was because the timeout or the limit was not configurable when this was first written in 2007. The count is decremented if the called method is "close" and it's from the expected Class. Previously this was java.sql.Connection, now it also accepts "close" from java.lang.AutoCloseable to decrement the count.
What you describe above is the job of a connection pool, and Tomcat
provides two of them for your selection. There are a host of others
available for Java as well. I highly recommend that you use one of those
instead of trying to re-invent this particular wheel.
> As the count wasn't being decremented, the active concurrent limit was quickly (believed to be) reached.
>
> I discussed this with a colleague who had previously worked with this application. There was some consternation about the approach but it was agreed that this was the least risk answer - for an application we're dropping support for in December, it is not worth rewriting. This will at least enable deployments to address vulnerabilities fixed in 9.0.71+.
Okay :)
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Solved: DataSource Connection pool [non] leak
Posted by "Scott,Tim" <Ti...@oclc.org>.
Hi Chris,
> > Hi all,
> >
> > Thanks for your responses. I think I've found the problem.
> >
> > My wrapping class which detects the invocation of the close() method to decrement its count is no longer decrementing its count because method.getDeclaringClass() has changed from java.sql.Connection to java.lang.AutoCloseable.
> >
> > Is it safe to check for either java.sql.Connection or java.lang.AutoCloseable?
> >
> > .. or should I just check for the "close()" method invocation, based on the fact that I'm not wrapping anything (here) that I shouldn't be counting?
> That depends upon the point of the whole thing. What, um, is the point
> of the whole thing?
The wrapping class is to limit the number of active connections, waiting up to a configured time for one to become idle if necessary. I can't recall if this was because the timeout or the limit was not configurable when this was first written in 2007. The count is decremented if the called method is "close" and it's from the expected Class. Previously this was java.sql.Connection, now it also accepts "close" from java.lang.AutoCloseable to decrement the count.
As the count wasn't being decremented, the active concurrent limit was quickly (believed to be) reached.
I discussed this with a colleague who had previously worked with this application. There was some consternation about the approach but it was agreed that this was the least risk answer - for an application we're dropping support for in December, it is not worth rewriting. This will at least enable deployments to address vulnerabilities fixed in 9.0.71+.
Thanks,
Tim
Re: [EXTERNAL] RE: DataSource Connection pool leak
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Tim,
On 8/29/23 10:33, Scott,Tim wrote:
> Hi all,
>
> Thanks for your responses. I think I've found the problem.
>
> My wrapping class which detects the invocation of the close() method to decrement its count is no longer decrementing its count because method.getDeclaringClass() has changed from java.sql.Connection to java.lang.AutoCloseable.
>
> Is it safe to check for either java.sql.Connection or java.lang.AutoCloseable?
>
> .. or should I just check for the "close()" method invocation, based on the fact that I'm not wrapping anything (here) that I shouldn't be counting?
That depends upon the point of the whole thing. What, um, is the point
of the whole thing?
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: [EXTERNAL] RE: DataSource Connection pool leak
Posted by "Scott,Tim" <Ti...@oclc.org>.
Hi all,
Thanks for your responses. I think I've found the problem.
My wrapping class which detects the invocation of the close() method to decrement its count is no longer decrementing its count because method.getDeclaringClass() has changed from java.sql.Connection to java.lang.AutoCloseable.
Is it safe to check for either java.sql.Connection or java.lang.AutoCloseable?
.. or should I just check for the "close()" method invocation, based on the fact that I'm not wrapping anything (here) that I shouldn't be counting?
Thanks,
Tim
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [EXTERNAL] RE: DataSource Connection pool leak
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Tim,
On 8/25/23 10:48, Scott,Tim wrote:
> Hi John,
>
>> Why does your app need 20 connections just to start up? That's a
>> bit of a rhetorical question, but needing so many connections to
>> start up seems odd to me.
> It doesn't. It only needs 1-2 at a time, but it makes 100s of queries
> in loops, each time using a connection from the pool and passing it
> back.
Yeah, this is what makes it a suspected leak, eh?
>> Are you using the Tomcat pool or another one? If it's Tomcat, I
>> think you should use some of the abandoned connection settings
>> described here:
> There're a few wrappers around the org.apache.commons.dbcp classes. I
> have abandoned connection settings in place. The code - in fact, the
> build - remains completely unchanged throughout my testing of 9.0.68,
> 9.0.70 - 9.0.79 as it used the same .war file. Whilst that doesn't
> eliminate my code from the equation, it looks like a problem outside
> my code. I will happily revise that view if nobody else has run into
> a similar problem in the last 8 months - or if they have and found a
> cause outside Tomcat.
I don't use 9.x but I haven't noticed a problem in 8.5.x and they are
very similar if not identical when it comes to the pooling code.
>> If your app is reserving connections and not releasing them, they
>> will be considered abandoned. With logAbandoned=true, you'll get a
>> stack trace showing where the connection was obtained.
> With 9.0.68 and 9.0.70 (indeed with most Tomcat versions I've used
> since 8.0... ) we have worked hard to ensure that it does release
> connections back to the pool appropriately. These connections are not
> "abandoned" - a release was attempted.
Sometimes the "abandoned" configuration is not quite correct. Can you
post your <Resource> configuration for your application? Remove secrets
of course.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: [EXTERNAL] RE: DataSource Connection pool leak
Posted by "Scott,Tim" <Ti...@oclc.org>.
Hi John,
> Why does your app need 20 connections just to start up? That's a bit of a rhetorical question, but needing so many connections to start up seems odd to me.
It doesn't. It only needs 1-2 at a time, but it makes 100s of queries in loops, each time using a connection from the pool and passing it back.
> Are you using the Tomcat pool or another one? If it's Tomcat, I think you should use some of the abandoned connection settings described here:
There're a few wrappers around the org.apache.commons.dbcp classes. I have abandoned connection settings in place.
The code - in fact, the build - remains completely unchanged throughout my testing of 9.0.68, 9.0.70 - 9.0.79 as it used the same .war file.
Whilst that doesn't eliminate my code from the equation, it looks like a problem outside my code.
I will happily revise that view if nobody else has run into a similar problem in the last 8 months - or if they have and found a cause outside Tomcat.
> If your app is reserving connections and not releasing them, they will be considered abandoned. With logAbandoned=true, you'll get a stack trace showing where the connection was obtained.
With 9.0.68 and 9.0.70 (indeed with most Tomcat versions I've used since 8.0... ) we have worked hard to ensure that it does release connections back to the pool appropriately.
These connections are not "abandoned" - a release was attempted.
> Also, you can take thread dumps to see what your threads are doing. If they are actually using the connection, you might see a query in flight. Then you can ask yourself why they're taking so long.
I know what the SQL is. It's in the log file. The SQL isn't slow, it's called multiple times for different tables/data during startup but with the versions of Tomcat with which I am having problems, it only manages 20 calls before exhausting the pool.
The SQL is run serially, not in parallel.
Thanks,
Tim
----
> Hi,
>
> For various diagnostics, I tried Tomcat 9.0.79 recently on a development
> machine. It didn't solve the problem I was experiencing - that was later
> identified as a problem in IntelliJ with remote deployment and is not why I'm
> mailing.
>
> I tweaked my Tomcat 9.0.79 configuration to start my application manually, as
> IntelliJ refused to deploy remotely.
>
> This lead to it quickly exhausting its connection pool and then hanging*
> before the application could complete its startup activity.
> * Each request for a connection from the pool timed out. The log
> shows that all 20 connections were allocated.
>
> I pointed my system back to 9.0.68, and it started up fine.
>
> I tried all versions I could find back from 9.0.78 through 9.0.71 - and ran into
> the same connection pool problem as 9.0.79 with every version.
>
> v9.0.70 worked.
> (well, at least didn't exhibit the same problem - the application
> completed its startup activity and was operational).
>
> I would therefore conclude that something about the way I'm managing my
> database connection pool is preventing 9.0.71 onward from freeing up the
> connections.
>
> My application uses javax.sql.DataSource with Corretto JDK 11 (11.0.16+8-
> LTS, to be specific).
>
> It struck me as odd as that means that all releases in the last 8 months have
> this problem for my application. Has anyone else, here, run into a similar
> problem?
>
> If a pool size of 20 is just too low, I think I'd need to set mine to a few
> hundred to complete the application startup and I'm not willing to try that
> without further insight.
>
> Thanks,
> Tim
>
> --
> Tim Scott (he/him/his)
> OCLC * Lead Software Engineer / Technical Product Manager
>
>
> cc: IT file
Why does your app need 20 connections just to start up? That's a bit of a rhetorical question, but needing so many connections to start up seems odd to me.
Are you using the Tomcat pool or another one? If it's Tomcat, I think you should use some of the abandoned connection settings described here:
https://tomcat.apache.org/tomcat-9.0-doc/jdbc-pool.html
If your app is reserving connections and not releasing them, they will be considered abandoned. With logAbandoned=true, you'll get a stack trace showing where the connection was obtained.
Also, you can take thread dumps to see what your threads are doing. If they are actually using the connection, you might see a query in flight. Then you can ask yourself why they're taking so long.
John
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: DataSource Connection pool leak
Posted by Jo...@wellsfargo.com.INVALID.
Tim,
> -----Original Message-----
> From: Scott,Tim <Ti...@oclc.org>
> Sent: Friday, August 25, 2023 3:09 AM
> To: users@tomcat.apache.org
> Subject: DataSource Connection pool leak
>
> Hi,
>
> For various diagnostics, I tried Tomcat 9.0.79 recently on a development
> machine. It didn't solve the problem I was experiencing - that was later
> identified as a problem in IntelliJ with remote deployment and is not why I'm
> mailing.
>
> I tweaked my Tomcat 9.0.79 configuration to start my application manually, as
> IntelliJ refused to deploy remotely.
>
> This lead to it quickly exhausting its connection pool and then hanging*
> before the application could complete its startup activity.
> * Each request for a connection from the pool timed out. The log
> shows that all 20 connections were allocated.
>
> I pointed my system back to 9.0.68, and it started up fine.
>
> I tried all versions I could find back from 9.0.78 through 9.0.71 - and ran into
> the same connection pool problem as 9.0.79 with every version.
>
> v9.0.70 worked.
> (well, at least didn't exhibit the same problem - the application
> completed its startup activity and was operational).
>
> I would therefore conclude that something about the way I'm managing my
> database connection pool is preventing 9.0.71 onward from freeing up the
> connections.
>
> My application uses javax.sql.DataSource with Corretto JDK 11 (11.0.16+8-
> LTS, to be specific).
>
> It struck me as odd as that means that all releases in the last 8 months have
> this problem for my application. Has anyone else, here, run into a similar
> problem?
>
> If a pool size of 20 is just too low, I think I'd need to set mine to a few
> hundred to complete the application startup and I'm not willing to try that
> without further insight.
>
> Thanks,
> Tim
>
> --
> Tim Scott (he/him/his)
> OCLC * Lead Software Engineer / Technical Product Manager
>
>
> cc: IT file
Why does your app need 20 connections just to start up? That's a bit of a rhetorical question, but needing so many connections to start up seems odd to me.
Are you using the Tomcat pool or another one? If it's Tomcat, I think you should use some of the abandoned connection settings described here:
https://tomcat.apache.org/tomcat-9.0-doc/jdbc-pool.html
If your app is reserving connections and not releasing them, they will be considered abandoned. With logAbandoned=true, you'll get a stack trace showing where the connection was obtained.
Also, you can take thread dumps to see what your threads are doing. If they are actually using the connection, you might see a query in flight. Then you can ask yourself why they're taking so long.
John
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org