You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Jerry Malcolm <te...@malcolms.com> on 2021/12/05 04:06:17 UTC
RemoveAbandoned Problems
I had a db connection leak in my code where an error condition would
throw an exception and bypass the connection cleanup code. I found that
and fixed it. But before I found the problem, my program was
overrunning the max connections and locking out. It would take
sometimes 12 hours after a reboot to go from 0 connections to max.
Normal steady state connections should currently be under 50. The ramp
over several hours to max was very obvious in my numActive log. What
I'm confused about is why removeAbandoned didn't recover those
connections. Granted, if I write my code correctly, removeAbandoned
shouldn't be necessary. The coding problem is solved now. But
apparently my understanding/configuration of removeAbandoned is not
correct. I'd like to have that figured out in case there's a next time
(which sadly there probably will be....). Basically, with the
configuration below, I'm not getting any idle connections detected and
returned. This is TC 8.5.73. And the leak was happening on a basic
request/response (no threads involved). I requested the connection,
encountered an error, and returned without closing the connection.
Ideas? Thx.
<Resource name="jdbc/-----"
url="jdbc:mysql://db1.-----.---/-----"
maxTotal="320"
maxIdle="3"
username="-----------"
password="-----------"
auth="Container"
type="javax.sql.DataSource"
maxWaitMillis="30000"
removeAbandonedOnBorrow="true"
removeAbandonedOnMaintenance="true"
removeAbandonedTimeout="15"
logAbandoned="true"
driverClassName="com.mysql.cj.jdbc.Driver" />
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: RemoveAbandoned Problems
Posted by Phil Steitz <ph...@gmail.com>.
On 12/8/21 1:44 PM, Christopher Schultz wrote:
> Phil,
>
> On 12/8/21 15:23, Phil Steitz wrote:
>>
>>
>> On 12/8/21 6:36 AM, Christopher Schultz wrote:
>>> Jerry,
>>>
>>> On 12/7/21 20:59, Jerry Malcolm wrote:
>>>> Chris, The way I thought it worked was if I configured
>>>> 'RemoveAbandonedOnBorrow' and RemoveAbandonedTimeout="15" was that
>>>> each time I requested a new connection from the pool, any
>>>> connections that had been idle for >15 minutes and had not been
>>>> returned by my code to the pool would be recovered, returned to the
>>>> pool and logged (assuming logAbandoned was set).
>>>
>>> Nope. "removeAbandoned" causes any connection that isn't returned to
>>> the pool to be *removed from the pool*, and replaced with a new
>>> (presumably working) connection. The connection that was never
>>> returned ... stays out there, doing whatever it was doing.
>>
>> Not exactly. Abandoned connection cleanup does try to physically
>> close connections [1] that are deemed abandoned. It creates capacity
>> to create new ones but it does not create them immediately.
>>>
>>> "logAbandoned" just lets you know when the pool gives up. It doesn't
>>> "do" anything (other than the logging).
>>>
>>> The alternative would be for the pool to forcibly terminate the
>>> connection, which could cause all kinds of chaos, so it does the
>>> only thing it can reasonably do: forget the connection ever existed
>>> in the first place. If your code never closes it, and the Connection
>>> object never gets GC'd (and, presumably, closed in the process),
>>> then it just lived forever, wasting an open-connection to your db.
>>> Since you have limited your total connections (per user? per host?)
>>> you eventually run out due to the leak.
>>
>> This is why you need to be careful to set the abandoned timeout long
>> enough so that "chaos" does not ensue. The pool tries to physically
>> close abandoned connections when this is configured to happen. If
>> clients retain handles to them and later try to use them, they will
>> get exceptions. You can see this confirmed in DBCP's
>> TestAbandonedBasicDataSource unit tests.
>
> Oh, I had no idea DBCP was actually trying to kill those connections.
> I guess reasonable can disagree over whether or not those connections
> should be killed by DBCP.
Yeah it has been that way since inception. The problem with not
actually closing them is you end up creating problems on the DB side if
you leave them hanging. Also kind of defeats the purpose of the pool
and changes its contract vis a vis the factory (maxTotal means a pool
will not take up more than that many instances at a given time).
Phil
>
> I'm cool either way :)
>
> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: RemoveAbandoned Problems
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Phil,
On 12/8/21 15:23, Phil Steitz wrote:
>
>
> On 12/8/21 6:36 AM, Christopher Schultz wrote:
>> Jerry,
>>
>> On 12/7/21 20:59, Jerry Malcolm wrote:
>>> Chris, The way I thought it worked was if I configured
>>> 'RemoveAbandonedOnBorrow' and RemoveAbandonedTimeout="15" was that
>>> each time I requested a new connection from the pool, any connections
>>> that had been idle for >15 minutes and had not been returned by my
>>> code to the pool would be recovered, returned to the pool and logged
>>> (assuming logAbandoned was set).
>>
>> Nope. "removeAbandoned" causes any connection that isn't returned to
>> the pool to be *removed from the pool*, and replaced with a new
>> (presumably working) connection. The connection that was never
>> returned ... stays out there, doing whatever it was doing.
>
> Not exactly. Abandoned connection cleanup does try to physically close
> connections [1] that are deemed abandoned. It creates capacity to
> create new ones but it does not create them immediately.
>>
>> "logAbandoned" just lets you know when the pool gives up. It doesn't
>> "do" anything (other than the logging).
>>
>> The alternative would be for the pool to forcibly terminate the
>> connection, which could cause all kinds of chaos, so it does the only
>> thing it can reasonably do: forget the connection ever existed in the
>> first place. If your code never closes it, and the Connection object
>> never gets GC'd (and, presumably, closed in the process), then it just
>> lived forever, wasting an open-connection to your db. Since you have
>> limited your total connections (per user? per host?) you eventually
>> run out due to the leak.
>
> This is why you need to be careful to set the abandoned timeout long
> enough so that "chaos" does not ensue. The pool tries to physically
> close abandoned connections when this is configured to happen. If
> clients retain handles to them and later try to use them, they will get
> exceptions. You can see this confirmed in DBCP's
> TestAbandonedBasicDataSource unit tests.
Oh, I had no idea DBCP was actually trying to kill those connections. I
guess reasonable can disagree over whether or not those connections
should be killed by DBCP.
I'm cool either way :)
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: RemoveAbandoned Problems
Posted by Phil Steitz <ph...@gmail.com>.
On 12/8/21 6:36 AM, Christopher Schultz wrote:
> Jerry,
>
> On 12/7/21 20:59, Jerry Malcolm wrote:
>> Chris, The way I thought it worked was if I configured
>> 'RemoveAbandonedOnBorrow' and RemoveAbandonedTimeout="15" was that
>> each time I requested a new connection from the pool, any connections
>> that had been idle for >15 minutes and had not been returned by my
>> code to the pool would be recovered, returned to the pool and logged
>> (assuming logAbandoned was set).
>
> Nope. "removeAbandoned" causes any connection that isn't returned to
> the pool to be *removed from the pool*, and replaced with a new
> (presumably working) connection. The connection that was never
> returned ... stays out there, doing whatever it was doing.
Not exactly. Abandoned connection cleanup does try to physically close
connections [1] that are deemed abandoned. It creates capacity to
create new ones but it does not create them immediately.
>
> "logAbandoned" just lets you know when the pool gives up. It doesn't
> "do" anything (other than the logging).
>
> The alternative would be for the pool to forcibly terminate the
> connection, which could cause all kinds of chaos, so it does the only
> thing it can reasonably do: forget the connection ever existed in the
> first place. If your code never closes it, and the Connection object
> never gets GC'd (and, presumably, closed in the process), then it just
> lived forever, wasting an open-connection to your db. Since you have
> limited your total connections (per user? per host?) you eventually
> run out due to the leak.
This is why you need to be careful to set the abandoned timeout long
enough so that "chaos" does not ensue. The pool tries to physically
close abandoned connections when this is configured to happen. If
clients retain handles to them and later try to use them, they will get
exceptions. You can see this confirmed in DBCP's
TestAbandonedBasicDataSource unit tests.
>
>> Until a few days ago I had a code error that was bypassing the
>> closing of the connection in certain situations, and after 12-24
>> hours the pool had worked its way up to maxing out. My problem is
>> fixed now, and the numActive count is staying fairly flat during
>> normal activity. But the way I understood removeAbandonedOnBorrow
>> was that TC connection pooling code would not allow errant
>> connections to remain in use forever.
>>
>> I'm sure I'm just misunderstanding how it works. Again, not critical
>> at this moment. But I'd like to figure out where my understanding is
>> wrong for future situations.
I think that the reason that you did not see connections closed by the
pool may have been that you did not get close enough to the maxTotal
setting (see other response above) or you had idle connections in the
pool as it was leaking and you did not have
timeBetweenEvictionRunsMillis set. If you set
timeBetweenEvictionRunsMillis to a positive value, that will trigger
unconditional removal when it runs (i.e., it does not check how close
the pool is to maxTotal or how many idle connections there are). The
removeAbandonedOnBorrow setting is really more of a liveness than a
cleanup feature - basically trying to keep the pool ahead of demand by
cleaning up when it is exhausted or close to it.
Phil
[1] As of DBCP 2.9.0, abort is used in place of close. See
https://issues.apache.org/jira/browse/DBCP-567
>
> You thought the pool would "clean-up" the mess. IT doesn't. What it
> *does* do is allow the pool to continue to function and provide its
> service to the application, even when the application is leaking
> connections.
>
> So, rather than starving clients when connections leak, those
> connections are simply allowed to leak.
>
> I always recommend running with maxActive="1" in development, with
> removeAbandoned="false" and logAbandoned="true". You'll find any leaks
> VERY quickly. ;)
>
> -chris
>
>> On 12/7/2021 2:31 PM, Christopher Schultz wrote:
>>> Jerry,
>>>
>>> On 12/4/21 23:06, Jerry Malcolm wrote:
>>>> I had a db connection leak in my code where an error condition
>>>> would throw an exception and bypass the connection cleanup code. I
>>>> found that and fixed it. But before I found the problem, my
>>>> program was overrunning the max connections and locking out. It
>>>> would take sometimes 12 hours after a reboot to go from 0
>>>> connections to max. Normal steady state connections should
>>>> currently be under 50. The ramp over several hours to max was very
>>>> obvious in my numActive log. What I'm confused about is why
>>>> removeAbandoned didn't recover those connections.
>>>
>>> When you say "recover"... what exactly do you mean?
>>>
>>>> Granted, if I write my code correctly, removeAbandoned shouldn't be
>>>> necessary. The coding problem is solved now. But apparently my
>>>> understanding/configuration of removeAbandoned is not correct.
>>>
>>> Possibly, but you didn't state your expectations.
>>>
>>>> I'd like to have that figured out in case there's a next time
>>>> (which sadly there probably will be....). Basically, with the
>>>> configuration below, I'm not getting any idle connections detected
>>>> and returned. This is TC 8.5.73. And the leak was happening on a
>>>> basic request/response (no threads involved). I requested the
>>>> connection, encountered an error, and returned without closing the
>>>> connection. Ideas? Thx.
>>>
>>> -chris
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: RemoveAbandoned Problems
Posted by Jerry Malcolm <te...@malcolms.com>.
Chris,
That makes total sense. Thanks so much for clarifying this.
Fortunately, I think I've got all the current set of leaks under
control. But this will help greatly for the next time I start seeing leaks.
Jerry
On 12/8/2021 7:36 AM, Christopher Schultz wrote:
> Jerry,
>
> On 12/7/21 20:59, Jerry Malcolm wrote:
>> Chris, The way I thought it worked was if I configured
>> 'RemoveAbandonedOnBorrow' and RemoveAbandonedTimeout="15" was that
>> each time I requested a new connection from the pool, any connections
>> that had been idle for >15 minutes and had not been returned by my
>> code to the pool would be recovered, returned to the pool and logged
>> (assuming logAbandoned was set).
>
> Nope. "removeAbandoned" causes any connection that isn't returned to
> the pool to be *removed from the pool*, and replaced with a new
> (presumably working) connection. The connection that was never
> returned ... stays out there, doing whatever it was doing.
>
> "logAbandoned" just lets you know when the pool gives up. It doesn't
> "do" anything (other than the logging).
>
> The alternative would be for the pool to forcibly terminate the
> connection, which could cause all kinds of chaos, so it does the only
> thing it can reasonably do: forget the connection ever existed in the
> first place. If your code never closes it, and the Connection object
> never gets GC'd (and, presumably, closed in the process), then it just
> lived forever, wasting an open-connection to your db. Since you have
> limited your total connections (per user? per host?) you eventually
> run out due to the leak.
>
>> Until a few days ago I had a code error that was bypassing the
>> closing of the connection in certain situations, and after 12-24
>> hours the pool had worked its way up to maxing out. My problem is
>> fixed now, and the numActive count is staying fairly flat during
>> normal activity. But the way I understood removeAbandonedOnBorrow
>> was that TC connection pooling code would not allow errant
>> connections to remain in use forever.
>>
>> I'm sure I'm just misunderstanding how it works. Again, not critical
>> at this moment. But I'd like to figure out where my understanding is
>> wrong for future situations.
>
> You thought the pool would "clean-up" the mess. IT doesn't. What it
> *does* do is allow the pool to continue to function and provide its
> service to the application, even when the application is leaking
> connections.
>
> So, rather than starving clients when connections leak, those
> connections are simply allowed to leak.
>
> I always recommend running with maxActive="1" in development, with
> removeAbandoned="false" and logAbandoned="true". You'll find any leaks
> VERY quickly. ;)
>
> -chris
>
>> On 12/7/2021 2:31 PM, Christopher Schultz wrote:
>>> Jerry,
>>>
>>> On 12/4/21 23:06, Jerry Malcolm wrote:
>>>> I had a db connection leak in my code where an error condition
>>>> would throw an exception and bypass the connection cleanup code. I
>>>> found that and fixed it. But before I found the problem, my
>>>> program was overrunning the max connections and locking out. It
>>>> would take sometimes 12 hours after a reboot to go from 0
>>>> connections to max. Normal steady state connections should
>>>> currently be under 50. The ramp over several hours to max was very
>>>> obvious in my numActive log. What I'm confused about is why
>>>> removeAbandoned didn't recover those connections.
>>>
>>> When you say "recover"... what exactly do you mean?
>>>
>>>> Granted, if I write my code correctly, removeAbandoned shouldn't be
>>>> necessary. The coding problem is solved now. But apparently my
>>>> understanding/configuration of removeAbandoned is not correct.
>>>
>>> Possibly, but you didn't state your expectations.
>>>
>>>> I'd like to have that figured out in case there's a next time
>>>> (which sadly there probably will be....). Basically, with the
>>>> configuration below, I'm not getting any idle connections detected
>>>> and returned. This is TC 8.5.73. And the leak was happening on a
>>>> basic request/response (no threads involved). I requested the
>>>> connection, encountered an error, and returned without closing the
>>>> connection. Ideas? Thx.
>>>
>>> -chris
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: RemoveAbandoned Problems
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Jerry,
On 12/7/21 20:59, Jerry Malcolm wrote:
> Chris, The way I thought it worked was if I configured
> 'RemoveAbandonedOnBorrow' and RemoveAbandonedTimeout="15" was that each
> time I requested a new connection from the pool, any connections that
> had been idle for >15 minutes and had not been returned by my code to
> the pool would be recovered, returned to the pool and logged (assuming
> logAbandoned was set).
Nope. "removeAbandoned" causes any connection that isn't returned to the
pool to be *removed from the pool*, and replaced with a new (presumably
working) connection. The connection that was never returned ... stays
out there, doing whatever it was doing.
"logAbandoned" just lets you know when the pool gives up. It doesn't
"do" anything (other than the logging).
The alternative would be for the pool to forcibly terminate the
connection, which could cause all kinds of chaos, so it does the only
thing it can reasonably do: forget the connection ever existed in the
first place. If your code never closes it, and the Connection object
never gets GC'd (and, presumably, closed in the process), then it just
lived forever, wasting an open-connection to your db. Since you have
limited your total connections (per user? per host?) you eventually run
out due to the leak.
> Until a few days ago I had a code error that was bypassing the closing
> of the connection in certain situations, and after 12-24 hours the pool
> had worked its way up to maxing out. My problem is fixed now, and the
> numActive count is staying fairly flat during normal activity. But the
> way I understood removeAbandonedOnBorrow was that TC connection pooling
> code would not allow errant connections to remain in use forever.
>
> I'm sure I'm just misunderstanding how it works. Again, not critical at
> this moment. But I'd like to figure out where my understanding is wrong
> for future situations.
You thought the pool would "clean-up" the mess. IT doesn't. What it
*does* do is allow the pool to continue to function and provide its
service to the application, even when the application is leaking
connections.
So, rather than starving clients when connections leak, those
connections are simply allowed to leak.
I always recommend running with maxActive="1" in development, with
removeAbandoned="false" and logAbandoned="true". You'll find any leaks
VERY quickly. ;)
-chris
> On 12/7/2021 2:31 PM, Christopher Schultz wrote:
>> Jerry,
>>
>> On 12/4/21 23:06, Jerry Malcolm wrote:
>>> I had a db connection leak in my code where an error condition would
>>> throw an exception and bypass the connection cleanup code. I found
>>> that and fixed it. But before I found the problem, my program was
>>> overrunning the max connections and locking out. It would take
>>> sometimes 12 hours after a reboot to go from 0 connections to max.
>>> Normal steady state connections should currently be under 50. The
>>> ramp over several hours to max was very obvious in my numActive log.
>>> What I'm confused about is why removeAbandoned didn't recover those
>>> connections.
>>
>> When you say "recover"... what exactly do you mean?
>>
>>> Granted, if I write my code correctly, removeAbandoned shouldn't be
>>> necessary. The coding problem is solved now. But apparently my
>>> understanding/configuration of removeAbandoned is not correct.
>>
>> Possibly, but you didn't state your expectations.
>>
>>> I'd like to have that figured out in case there's a next time (which
>>> sadly there probably will be....). Basically, with the configuration
>>> below, I'm not getting any idle connections detected and returned.
>>> This is TC 8.5.73. And the leak was happening on a basic
>>> request/response (no threads involved). I requested the connection,
>>> encountered an error, and returned without closing the connection.
>>> Ideas? Thx.
>>
>> -chris
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: RemoveAbandoned Problems
Posted by Jerry Malcolm <te...@malcolms.com>.
Chris, The way I thought it worked was if I configured
'RemoveAbandonedOnBorrow' and RemoveAbandonedTimeout="15" was that each
time I requested a new connection from the pool, any connections that
had been idle for >15 minutes and had not been returned by my code to
the pool would be recovered, returned to the pool and logged (assuming
logAbandoned was set).
Until a few days ago I had a code error that was bypassing the closing
of the connection in certain situations, and after 12-24 hours the pool
had worked its way up to maxing out. My problem is fixed now, and the
numActive count is staying fairly flat during normal activity. But the
way I understood removeAbandonedOnBorrow was that TC connection pooling
code would not allow errant connections to remain in use forever.
I'm sure I'm just misunderstanding how it works. Again, not critical at
this moment. But I'd like to figure out where my understanding is wrong
for future situations.
Thanks
Jerry
On 12/7/2021 2:31 PM, Christopher Schultz wrote:
> Jerry,
>
> On 12/4/21 23:06, Jerry Malcolm wrote:
>> I had a db connection leak in my code where an error condition would
>> throw an exception and bypass the connection cleanup code. I found
>> that and fixed it. But before I found the problem, my program was
>> overrunning the max connections and locking out. It would take
>> sometimes 12 hours after a reboot to go from 0 connections to max.
>> Normal steady state connections should currently be under 50. The
>> ramp over several hours to max was very obvious in my numActive log.
>> What I'm confused about is why removeAbandoned didn't recover those
>> connections.
>
> When you say "recover"... what exactly do you mean?
>
>> Granted, if I write my code correctly, removeAbandoned shouldn't be
>> necessary. The coding problem is solved now. But apparently my
>> understanding/configuration of removeAbandoned is not correct.
>
> Possibly, but you didn't state your expectations.
>
>> I'd like to have that figured out in case there's a next time (which
>> sadly there probably will be....). Basically, with the configuration
>> below, I'm not getting any idle connections detected and returned.
>> This is TC 8.5.73. And the leak was happening on a basic
>> request/response (no threads involved). I requested the connection,
>> encountered an error, and returned without closing the connection.
>> Ideas? Thx.
>
> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: RemoveAbandoned Problems
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Jerry,
On 12/4/21 23:06, Jerry Malcolm wrote:
> I had a db connection leak in my code where an error condition would
> throw an exception and bypass the connection cleanup code. I found that
> and fixed it. But before I found the problem, my program was
> overrunning the max connections and locking out. It would take
> sometimes 12 hours after a reboot to go from 0 connections to max.
> Normal steady state connections should currently be under 50. The ramp
> over several hours to max was very obvious in my numActive log. What
> I'm confused about is why removeAbandoned didn't recover those
> connections.
When you say "recover"... what exactly do you mean?
> Granted, if I write my code correctly, removeAbandoned
> shouldn't be necessary. The coding problem is solved now. But
> apparently my understanding/configuration of removeAbandoned is not
> correct.
Possibly, but you didn't state your expectations.
> I'd like to have that figured out in case there's a next time
> (which sadly there probably will be....). Basically, with the
> configuration below, I'm not getting any idle connections detected and
> returned. This is TC 8.5.73. And the leak was happening on a basic
> request/response (no threads involved). I requested the connection,
> encountered an error, and returned without closing the connection.
> Ideas? Thx.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: RemoveAbandoned Problems
Posted by Phil Steitz <ph...@gmail.com>.
Sorry again. Docs are here (at the bottom in the abandoned config section):
https://commons.apache.org/proper/commons-dbcp/configuration.html
On 12/6/21 10:01 AM, Phil Steitz wrote:
>
>
> On 12/5/21 2:34 PM, Jerry Malcolm wrote:
>> Phil,
>>
>> Thanks for the response. I saw that note in the docs that said the
>> removeAbandonedOnMaintenance wouldn't do anything without an evictor
>> service. But removeAbandonedOnBorrow also requires an evictor
>> service to run in order remove on borrow? That's fine. Just a bit
>> confusing that on-borrow requires a timed eviction run. I'll do
>> whatever it takes. Again, just trying to figure it all out.
>
> I am sorry. My mistake. I saw the removeAbandonOnMaintenance and not
> the other one. You are correct that with removeAbandonedOnBorrow on
> you should not need to have the evictor turned on (though it will
> obviously only run on borrow). The docs could be clearer on this
> (seems some have been moved / deleted), but when
> removeAbandonedOnMaintenance is on, actual removal only happens when
> there are fewer than 2 idle objects available in the pool and
> getNumActive() > getMaxTotal() - 3.
>
> Phil
>>
>> Jerry
>>
>> On 12/5/2021 12:19 PM, Phil Steitz wrote:
>>> In order for abandoned connection cleanup to happen, you have to
>>> have configured a maintenance (aka "evictor") thread to run. You
>>> need to set the value of timeBetweenEvictionRunsMillis to a positive
>>> number.
>>>
>>> Phil
>>>
>>> On 12/4/21 9:06 PM, Jerry Malcolm wrote:
>>>> I had a db connection leak in my code where an error condition
>>>> would throw an exception and bypass the connection cleanup code. I
>>>> found that and fixed it. But before I found the problem, my
>>>> program was overrunning the max connections and locking out. It
>>>> would take sometimes 12 hours after a reboot to go from 0
>>>> connections to max. Normal steady state connections should
>>>> currently be under 50. The ramp over several hours to max was very
>>>> obvious in my numActive log. What I'm confused about is why
>>>> removeAbandoned didn't recover those connections. Granted, if I
>>>> write my code correctly, removeAbandoned shouldn't be necessary.
>>>> The coding problem is solved now. But apparently my
>>>> understanding/configuration of removeAbandoned is not correct. I'd
>>>> like to have that figured out in case there's a next time (which
>>>> sadly there probably will be....). Basically, with the
>>>> configuration below, I'm not getting any idle connections detected
>>>> and returned. This is TC 8.5.73. And the leak was happening on a
>>>> basic request/response (no threads involved). I requested the
>>>> connection, encountered an error, and returned without closing the
>>>> connection. Ideas? Thx.
>>>>
>>>> <Resource name="jdbc/-----"
>>>> url="jdbc:mysql://db1.-----.---/-----"
>>>> maxTotal="320"
>>>> maxIdle="3"
>>>> username="-----------"
>>>> password="-----------"
>>>> auth="Container"
>>>> type="javax.sql.DataSource"
>>>> maxWaitMillis="30000"
>>>> removeAbandonedOnBorrow="true"
>>>> removeAbandonedOnMaintenance="true"
>>>> removeAbandonedTimeout="15"
>>>> logAbandoned="true"
>>>> driverClassName="com.mysql.cj.jdbc.Driver" />
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: RemoveAbandoned Problems
Posted by Phil Steitz <ph...@gmail.com>.
On 12/5/21 2:34 PM, Jerry Malcolm wrote:
> Phil,
>
> Thanks for the response. I saw that note in the docs that said the
> removeAbandonedOnMaintenance wouldn't do anything without an evictor
> service. But removeAbandonedOnBorrow also requires an evictor service
> to run in order remove on borrow? That's fine. Just a bit confusing
> that on-borrow requires a timed eviction run. I'll do whatever it
> takes. Again, just trying to figure it all out.
I am sorry. My mistake. I saw the removeAbandonOnMaintenance and not
the other one. You are correct that with removeAbandonedOnBorrow on you
should not need to have the evictor turned on (though it will obviously
only run on borrow). The docs could be clearer on this (seems some have
been moved / deleted), but when removeAbandonedOnMaintenance is on,
actual removal only happens when there are fewer than 2 idle objects
available in the pool and getNumActive() > getMaxTotal() - 3.
Phil
>
> Jerry
>
> On 12/5/2021 12:19 PM, Phil Steitz wrote:
>> In order for abandoned connection cleanup to happen, you have to have
>> configured a maintenance (aka "evictor") thread to run. You need to
>> set the value of timeBetweenEvictionRunsMillis to a positive number.
>>
>> Phil
>>
>> On 12/4/21 9:06 PM, Jerry Malcolm wrote:
>>> I had a db connection leak in my code where an error condition would
>>> throw an exception and bypass the connection cleanup code. I found
>>> that and fixed it. But before I found the problem, my program was
>>> overrunning the max connections and locking out. It would take
>>> sometimes 12 hours after a reboot to go from 0 connections to max.
>>> Normal steady state connections should currently be under 50. The
>>> ramp over several hours to max was very obvious in my numActive log.
>>> What I'm confused about is why removeAbandoned didn't recover those
>>> connections. Granted, if I write my code correctly, removeAbandoned
>>> shouldn't be necessary. The coding problem is solved now. But
>>> apparently my understanding/configuration of removeAbandoned is not
>>> correct. I'd like to have that figured out in case there's a next
>>> time (which sadly there probably will be....). Basically, with the
>>> configuration below, I'm not getting any idle connections detected
>>> and returned. This is TC 8.5.73. And the leak was happening on a
>>> basic request/response (no threads involved). I requested the
>>> connection, encountered an error, and returned without closing the
>>> connection. Ideas? Thx.
>>>
>>> <Resource name="jdbc/-----"
>>> url="jdbc:mysql://db1.-----.---/-----"
>>> maxTotal="320"
>>> maxIdle="3"
>>> username="-----------"
>>> password="-----------"
>>> auth="Container"
>>> type="javax.sql.DataSource"
>>> maxWaitMillis="30000"
>>> removeAbandonedOnBorrow="true"
>>> removeAbandonedOnMaintenance="true"
>>> removeAbandonedTimeout="15"
>>> logAbandoned="true"
>>> driverClassName="com.mysql.cj.jdbc.Driver" />
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: RemoveAbandoned Problems
Posted by Jerry Malcolm <te...@malcolms.com>.
Phil,
Thanks for the response. I saw that note in the docs that said the
removeAbandonedOnMaintenance wouldn't do anything without an evictor
service. But removeAbandonedOnBorrow also requires an evictor service
to run in order remove on borrow? That's fine. Just a bit confusing
that on-borrow requires a timed eviction run. I'll do whatever it
takes. Again, just trying to figure it all out.
Jerry
On 12/5/2021 12:19 PM, Phil Steitz wrote:
> In order for abandoned connection cleanup to happen, you have to have
> configured a maintenance (aka "evictor") thread to run. You need to
> set the value of timeBetweenEvictionRunsMillis to a positive number.
>
> Phil
>
> On 12/4/21 9:06 PM, Jerry Malcolm wrote:
>> I had a db connection leak in my code where an error condition would
>> throw an exception and bypass the connection cleanup code. I found
>> that and fixed it. But before I found the problem, my program was
>> overrunning the max connections and locking out. It would take
>> sometimes 12 hours after a reboot to go from 0 connections to max.
>> Normal steady state connections should currently be under 50. The
>> ramp over several hours to max was very obvious in my numActive log.
>> What I'm confused about is why removeAbandoned didn't recover those
>> connections. Granted, if I write my code correctly, removeAbandoned
>> shouldn't be necessary. The coding problem is solved now. But
>> apparently my understanding/configuration of removeAbandoned is not
>> correct. I'd like to have that figured out in case there's a next
>> time (which sadly there probably will be....). Basically, with the
>> configuration below, I'm not getting any idle connections detected
>> and returned. This is TC 8.5.73. And the leak was happening on a
>> basic request/response (no threads involved). I requested the
>> connection, encountered an error, and returned without closing the
>> connection. Ideas? Thx.
>>
>> <Resource name="jdbc/-----"
>> url="jdbc:mysql://db1.-----.---/-----"
>> maxTotal="320"
>> maxIdle="3"
>> username="-----------"
>> password="-----------"
>> auth="Container"
>> type="javax.sql.DataSource"
>> maxWaitMillis="30000"
>> removeAbandonedOnBorrow="true"
>> removeAbandonedOnMaintenance="true"
>> removeAbandonedTimeout="15"
>> logAbandoned="true"
>> driverClassName="com.mysql.cj.jdbc.Driver" />
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: RemoveAbandoned Problems
Posted by Phil Steitz <ph...@gmail.com>.
In order for abandoned connection cleanup to happen, you have to have
configured a maintenance (aka "evictor") thread to run. You need to set
the value of timeBetweenEvictionRunsMillis to a positive number.
Phil
On 12/4/21 9:06 PM, Jerry Malcolm wrote:
> I had a db connection leak in my code where an error condition would
> throw an exception and bypass the connection cleanup code. I found
> that and fixed it. But before I found the problem, my program was
> overrunning the max connections and locking out. It would take
> sometimes 12 hours after a reboot to go from 0 connections to max.
> Normal steady state connections should currently be under 50. The
> ramp over several hours to max was very obvious in my numActive log.
> What I'm confused about is why removeAbandoned didn't recover those
> connections. Granted, if I write my code correctly, removeAbandoned
> shouldn't be necessary. The coding problem is solved now. But
> apparently my understanding/configuration of removeAbandoned is not
> correct. I'd like to have that figured out in case there's a next time
> (which sadly there probably will be....). Basically, with the
> configuration below, I'm not getting any idle connections detected and
> returned. This is TC 8.5.73. And the leak was happening on a basic
> request/response (no threads involved). I requested the connection,
> encountered an error, and returned without closing the connection.
> Ideas? Thx.
>
> <Resource name="jdbc/-----"
> url="jdbc:mysql://db1.-----.---/-----"
> maxTotal="320"
> maxIdle="3"
> username="-----------"
> password="-----------"
> auth="Container"
> type="javax.sql.DataSource"
> maxWaitMillis="30000"
> removeAbandonedOnBorrow="true"
> removeAbandonedOnMaintenance="true"
> removeAbandonedTimeout="15"
> logAbandoned="true"
> driverClassName="com.mysql.cj.jdbc.Driver" />
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org