You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Akshay Mishra <ak...@nutanix.com> on 2023/04/17 23:07:04 UTC
Tips on identifying the DB connection leaks leading to the "Pool empty" error
Hello Tomcat Community!
Tomcat Version: 9.0.70
Our Tomcat web service started exhausting the max connections allowed to the backend datastore (Postgres).
From the logs, we don’t see any 1 thread requesting lot of connections. Nor do we see too many threads requesting a new connection.
We collected thread dumps (at a fixed interval of 5 mins) and they do not point to too many threads requesting connections.
We suspect someone could be leaking connections but we are unclear on how to narrow that down especially since the thread dump was inconclusive
We added parameters like “removeAbandoned”, “removeAbandonedTimeout” and “ResetAbandonedTimer” to our datasource config and the results were encouraging - we don't see the pool getting exhausted but are concerned this might mask a larger problem in case there are connection leaks in our service.
It would be great if the community has any thoughts on these new parameters. Or on way to narrow down the problem to isolate connection leaks.
Any help is appreciated. Thanks
Akshay
Re: Tips on identifying the DB connection leaks leading to the "Pool empty" error
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Torsten,
On 4/17/23 19:31, Torsten Krah wrote:
> Use logAbandoned as a boolean parameter
+1
remoteAbandoned is only a band-aid; you need to fix your application.
In development, I always run with maxTotal="1" and logAbandoned="true"
and maxWaitMillis="10000". This will help you find connection leaks very
quickly.
It will also help find connection deadlocks, where a single thread tries
to obtain two connections from the pool instead of using only one at a
time. This can cause a deadlock in the following situation:
0. Thread pool contains 2 connections
1. Thread 1 obtains a connection
2. Thread 2 obtains a connection
3. Thread 1 tries to obtain a connection and waits for it
4. Thread 2 tries to obtain a connection and waits for it
Both threads are now waiting on the other thread to release one of the
two available connections, and neither one ever will. Meanwhile, the
pool is empty and nobody else can get any work done, either.
I haven't posted this in a while, so:
https://blog.christopherschultz.net/2009/03/16/properly-handling-pooled-jdbc-connections/
-chris
> Akshay Mishra <ak...@nutanix.com> schrieb am Di., 18. Apr. 2023,
> 01:07:
>
>> Hello Tomcat Community!
>>
>>
>> Tomcat Version: 9.0.70
>>
>> Our Tomcat web service started exhausting the max connections allowed to
>> the backend datastore (Postgres).
>>
>> From the logs, we don’t see any 1 thread requesting lot of connections.
>> Nor do we see too many threads requesting a new connection.
>> We collected thread dumps (at a fixed interval of 5 mins) and they do not
>> point to too many threads requesting connections.
>>
>> We suspect someone could be leaking connections but we are unclear on how
>> to narrow that down especially since the thread dump was inconclusive
>>
>> We added parameters like “removeAbandoned”, “removeAbandonedTimeout” and
>> “ResetAbandonedTimer” to our datasource config and the results were
>> encouraging - we don't see the pool getting exhausted but are concerned
>> this might mask a larger problem in case there are connection leaks in our
>> service.
>>
>> It would be great if the community has any thoughts on these new
>> parameters. Or on way to narrow down the problem to isolate connection
>> leaks.
>>
>> Any help is appreciated. Thanks
>>
>> Akshay
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tips on identifying the DB connection leaks leading to the "Pool empty" error
Posted by Kevin Huntly <km...@gmail.com>.
We stumbled into it to be honest. There should be a fairly easy way though
if you're on a linux box... something like..
grep -r ".getConnection" * should get you all the classes that make a call
to something like sqlConn.getConnection(), and then once you have that you
can check for finally {} blocks and ensure you;re closing out
________________________________________________
Kevin Huntly
Email: kmhuntly@gmail.com
Cell: 716/424-3311
________________________________________________
-----BEGIN GEEK CODE BLOCK-----
Version: 1.0
GCS/IT d+ s a C++ UL+++$ P+(++) L+++ E---
W+++ N+ o K(+) w--- O- M-- V-- PS+ PE Y(+)
PGP++(+++) t+ 5-- X-- R+ tv+ b++ DI++ D++
G++ e(+) h--- r+++ y+++*
------END GEEK CODE BLOCK------
On Mon, Apr 17, 2023 at 7:32 PM Torsten Krah <kr...@gmail.com> wrote:
> Use logAbandoned as a boolean parameter, you will get stacktraces in the
> log. From those you can narrow down where you are leaking connections.
>
> Akshay Mishra <ak...@nutanix.com> schrieb am Di., 18. Apr. 2023,
> 01:07:
>
> > Hello Tomcat Community!
> >
> >
> > Tomcat Version: 9.0.70
> >
> > Our Tomcat web service started exhausting the max connections allowed to
> > the backend datastore (Postgres).
> >
> > From the logs, we don’t see any 1 thread requesting lot of connections.
> > Nor do we see too many threads requesting a new connection.
> > We collected thread dumps (at a fixed interval of 5 mins) and they do not
> > point to too many threads requesting connections.
> >
> > We suspect someone could be leaking connections but we are unclear on how
> > to narrow that down especially since the thread dump was inconclusive
> >
> > We added parameters like “removeAbandoned”, “removeAbandonedTimeout” and
> > “ResetAbandonedTimer” to our datasource config and the results were
> > encouraging - we don't see the pool getting exhausted but are concerned
> > this might mask a larger problem in case there are connection leaks in
> our
> > service.
> >
> > It would be great if the community has any thoughts on these new
> > parameters. Or on way to narrow down the problem to isolate connection
> > leaks.
> >
> > Any help is appreciated. Thanks
> >
> > Akshay
> >
>
Re: Tips on identifying the DB connection leaks leading to the "Pool empty" error
Posted by Torsten Krah <kr...@gmail.com>.
Use logAbandoned as a boolean parameter, you will get stacktraces in the
log. From those you can narrow down where you are leaking connections.
Akshay Mishra <ak...@nutanix.com> schrieb am Di., 18. Apr. 2023,
01:07:
> Hello Tomcat Community!
>
>
> Tomcat Version: 9.0.70
>
> Our Tomcat web service started exhausting the max connections allowed to
> the backend datastore (Postgres).
>
> From the logs, we don’t see any 1 thread requesting lot of connections.
> Nor do we see too many threads requesting a new connection.
> We collected thread dumps (at a fixed interval of 5 mins) and they do not
> point to too many threads requesting connections.
>
> We suspect someone could be leaking connections but we are unclear on how
> to narrow that down especially since the thread dump was inconclusive
>
> We added parameters like “removeAbandoned”, “removeAbandonedTimeout” and
> “ResetAbandonedTimer” to our datasource config and the results were
> encouraging - we don't see the pool getting exhausted but are concerned
> this might mask a larger problem in case there are connection leaks in our
> service.
>
> It would be great if the community has any thoughts on these new
> parameters. Or on way to narrow down the problem to isolate connection
> leaks.
>
> Any help is appreciated. Thanks
>
> Akshay
>
Re: Tips on identifying the DB connection leaks leading to the "Pool empty" error
Posted by Kevin Huntly <km...@gmail.com>.
my team saw this same thing in production - there was code that never
closed the SQL connection after it was created... 150+ connections spike
all at once, took out the pool. Not sure if that's your issue, but thought
i'd mention it
________________________________________________
Kevin Huntly
Email: kmhuntly@gmail.com
Cell: 716/424-3311
________________________________________________
-----BEGIN GEEK CODE BLOCK-----
Version: 1.0
GCS/IT d+ s a C++ UL+++$ P+(++) L+++ E---
W+++ N+ o K(+) w--- O- M-- V-- PS+ PE Y(+)
PGP++(+++) t+ 5-- X-- R+ tv+ b++ DI++ D++
G++ e(+) h--- r+++ y+++*
------END GEEK CODE BLOCK------
On Mon, Apr 17, 2023 at 7:07 PM Akshay Mishra <ak...@nutanix.com>
wrote:
> Hello Tomcat Community!
>
>
> Tomcat Version: 9.0.70
>
> Our Tomcat web service started exhausting the max connections allowed to
> the backend datastore (Postgres).
>
> From the logs, we don’t see any 1 thread requesting lot of connections.
> Nor do we see too many threads requesting a new connection.
> We collected thread dumps (at a fixed interval of 5 mins) and they do not
> point to too many threads requesting connections.
>
> We suspect someone could be leaking connections but we are unclear on how
> to narrow that down especially since the thread dump was inconclusive
>
> We added parameters like “removeAbandoned”, “removeAbandonedTimeout” and
> “ResetAbandonedTimer” to our datasource config and the results were
> encouraging - we don't see the pool getting exhausted but are concerned
> this might mask a larger problem in case there are connection leaks in our
> service.
>
> It would be great if the community has any thoughts on these new
> parameters. Or on way to narrow down the problem to isolate connection
> leaks.
>
> Any help is appreciated. Thanks
>
> Akshay
>