You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Jerry Malcolm <te...@malcolms.com> on 2015/07/29 07:38:58 UTC
"Too Many Connections" exceptions after moving to Tomcat 8
I am trying to move my client to Tomcat 8 from 7. We've been on 7 for a
year or two and have not had any major issues. I read through the 7-->8
migration instructions. As far as I can tell, I have renamed all of the
resource config parameters that changed (maxTotal, maxWaitMillis,
removedAbandonedOnBorrow). But after running for a couple of hours, I
start getting "Too Many Connections" exceptions when connecting to the
db. Granted, there's a chance I have some connection leaks in the code,
and I'll look into those. But this hasn't been an issue on 7 for the
past year or two. And shouldn't removeAbandoned clean those up? I've
got logAbandoned set to 'true', and nothing is showing up in the logs.
Is this exception referring to running out of available connections in
the pool?
Is there something else I need to change in the configuration to move to
Tomcat 8 that I've overlooked?
Thanks.
Jerry
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: "Too Many Connections" exceptions after moving to Tomcat 8
Posted by Prabhu Mannu <pr...@fernridge.com>.
Hi jerry ,
It's probably issue with the application itself,
You might want to check your application for connections leaks.
I usually use this https://github.com/javamelody/javamelody/wiki
To monitor the application in both production and test environment.
It's easy to use and very efficient tool.
Hope it helps.
With regards
Prabhu
-----Original Message-----
From: Jerry Malcolm [mailto:techstuff@malcolms.com]
Sent: Wednesday, 29 July, 2015 1:39 PM
To: Tomcat Users List <us...@tomcat.apache.org>
Subject: "Too Many Connections" exceptions after moving to Tomcat 8
I am trying to move my client to Tomcat 8 from 7. We've been on 7 for a year or two and have not had any major issues. I read through the 7-->8 migration instructions. As far as I can tell, I have renamed all of the resource config parameters that changed (maxTotal, maxWaitMillis, removedAbandonedOnBorrow). But after running for a couple of hours, I start getting "Too Many Connections" exceptions when connecting to the db. Granted, there's a chance I have some connection leaks in the code, and I'll look into those. But this hasn't been an issue on 7 for the past year or two. And shouldn't removeAbandoned clean those up? I've got logAbandoned set to 'true', and nothing is showing up in the logs.
Is this exception referring to running out of available connections in the pool?
Is there something else I need to change in the configuration to move to Tomcat 8 that I've overlooked?
Thanks.
Jerry
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: "Too Many Connections" exceptions after moving to Tomcat 8
Posted by Jerry Malcolm <te...@malcolms.com>.
On 7/29/2015 3:39 PM, Mark Thomas wrote:
> On 29/07/2015 20:25, Jerry Malcolm wrote:
>> Well, it appears that we are slowly getting to the bottom of this. But
>> with every answer, I get a few more questions....
>>
>> First, I installed the latest TC8 on my laptop, copied my server.conf
>> and conf/catalina folder to it and started it up just to see what errors
>> I got. After changing out an obsolete listener, it came up. I found
>> all of the <resource> parm exceptions in stderr. So that question is
>> cleared up. Thanks for clarifying where to find that.
>>
>> The site is a wedding vendor advertiser site that spans two major
>> cities. There is no user login. Simply a very huge online catalog.
>> I'm certain it's deployed only once. Whether I need that many
>> connections is a valid question. As far as I know, I haven't hit the
>> limit in normal operation until now. Could possibly reduce the count if
>> I collect statistics.
>>
>> I've been monitoring the production server logs all day watching to be
>> sure connection pool doesn't dry up again. About an hour ago, there was
>> a single huge dump in stdout of approx 2000 'logAbandoned' exceptions.
>> They showed connections from 1am right after my last bounce of the
>> server thru 1:35pm. The good news is with the stack trace on one of
>> them I was able to see the bug causing the leak. But why did it decide
>> to wait over 12 hours accumulating abandoned connections before dumping
>> them back in the pool? I realize from now knowing the code bug that the
>> leak is a slow drip that is continually leaking on a regular basis. But
>> since that last 12-hour accumulated dump, the abandoning has returned to
>> silence. Since leaks are occurring regularly and would be timing out
>> regularly, shouldn't I see a similar 'slow drip' of logAbandoned entries
>> in stdout instead of a big dump every 12 hours?
>>
>> It's going to take a day or two to fix the leak, test, and deploy. If
>> indeed abandoned connections are now correctly being returned to the
>> pool, then I presume we are back to working the way it did on TC7.
>> Still not sure why it started working now. But I guess once I get the
>> leak fixed and if TC8 is now configured to handle abandoned connections,
>> I'm good. Still would like to know about the mega-dump vs. trickle of
>> abandoned connections being logged.
> That sounds like a bug that has just been fixed in Pool:
> https://issues.apache.org/jira/browse/POOL-300
>
> The fix for that should be in the next Tomcat 8 release.
>
> Mark
Thanks, Mark..... Yep that would definitely explain why with a smaller
count of abandoned connections, I wasn't seeing any output.
Jerry
>
>> Thanks again for all of the info.
>>
>> Jerry
>>
>> On 7/29/2015 1:15 PM, Christopher Schultz wrote:
>> Jerry,
>>
>> On 7/29/15 12:06 PM, Jerry Malcolm wrote:
>>>>> On 7/29/2015 10:11 AM, Christopher Schultz wrote:
>>>>>
>>>>> I guess I could move up to the latest 8. But it'll require some
>>>>> client down time, and they aren't thrilled with that at this
>>>>> point.
>> Understood, though trying various <Resource> configurations is
>> certainly resulting in downtime, no?
>>
>>>>>>> Off topic, but relevant.... it sure would be nice if the xml
>>>>>>> parser for <resource> tags and other config items would flag
>>>>>>> with errors any parameter misspellings such as using the old
>>>>>>> obsolete 'removeAbandoned' instead of the new
>>>>>>> 'removeAbandonedOnBorrow', etc. or even just typo misspells.
>>>>>> It does:
>>>>>>
>>>>>> Jul 02, 2015 10:17:25 AM
>>>>>> org.apache.tomcat.dbcp.dbcp2.BasicDataSourceFactory get
>>>>>> ObjectInstance WARNING: Name = diagnosis Property maxActive is
>>>>>> not used in DBCP2, use maxTotal instead. maxTotal default value
>>>>>> is 8. You have set value of "1" for "maxActive" property, which
>>>>>> is being ignored.
>>>>>>
>>>>>> ...
>>>>>>
>>>>>> WARNING: Name = diagnosis Property removeAbandoned is not used
>>>>>> in DBCP2, use one or both of removeAbandonedOnBorrow or
>>>>>> removeAbandonedOnMaintenance instead. Both have default value set
>>>>>> to false. You have set value of "true" for "removeAbandoned"
>>>>>> property, which is being ignored.
>>>>>>
>>>>>>
>>>>>> (I have a context.xml that I want to use between both Tomcat 7
>>>>>> and Tomcat 8, so I actually have both spellings of those
>>>>>> attributes in my XML.)
>>>>> This is really strange. Are the above log entries in stdout?
>> Yes, so they typically end up in catalina.out. Those logs are from my
>> dev system's catalina.out from a recent restart.
>>
>>>>> I had 'removeAbandoned' in my resource statements until yesterday.
>>>>> But I am not seeing anything in stdout like the statement above.
>>>>> And after changing removeAbandoned to the correct spelling, I'm
>>>>> also not seeing any abandoned log entries either.
>> Hmm.
>>
>>>>> Is there some sort of 'log level' that may be throttling back my
>>>>> log entries? If my code has leaks, and I'm running out of
>>>>> connections but not seeing any abandoned entries in the log, that
>>>>> should be a concern, right?
>> Possibly.
>>
>> Perhaps you are deploying your application more times than you
>> thought, and are making many more connections than you expected.
>>
>>>>>> What does your <Resource> element actually look like? (Remember
>>>>>> to remove sensitive information such as passwords.)
>>>>> <Resource name="jdbc/wgbe" testOnBorrow="true"
>>>>> validationQuery="SELECT 1"
>> validationQuery might not be necessary anymore. I think DBCP2 does a
>> lightweight connection test instead issuing an actual query, unless
>> you override it with a validationQuery.
>>
>> Also, for Connector/J, you can use "/* ping */SELECT 1" and the driver
>> itself will perform the lightweight connection test instead of
>> actually issuing a query. Much faster than going through the SQL
>> parser, etc.
>>
>>>>> auth="Container" type="javax.sql.DataSource" maxTotal="300"
>> That's a LOT of connections IMO. What is your total connection limit?
>> Is it per-user?
>>
>>>>> maxIdle="30" maxWaitMillis="10000" removeAbandonedOnBorrow="true"
>>>>> removeAbandonedOnMaintenance="true" removeAbandonedTimeout="30"
>>>>> logAbandoned="true" username="xxxxxxx" password="xxxxxxxxx"
>>>>> driverClassName="org.gjt.mm.mysql.Driver"
>>>>> url="jdbc:mysql://localhost:3306/wgbe?autoReconnect=true" />
>> Do you have the 'manager' web application installed? Can you check to
>> see what applications are actually deployed? If you have 2 copies
>> deployed, you might see 600 connections open.
>>
>> Also, do you have any stray context.xml files hanging around in
>> conf/*/* that you weren't expecting? If you are making changes to your
>> configuration and they don't appear to be having any effect, it's
>> possible that your new configuration isn't being applied because
>> another config is masking it.
>>
>> -chris
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>>
>>> -----
>>> No virus found in this message.
>>> Checked by AVG - www.avg.com
>>> Version: 2015.0.6081 / Virus Database: 4392/10332 - Release Date:
>>> 07/29/15
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
> -----
> No virus found in this message.
> Checked by AVG - www.avg.com
> Version: 2015.0.6081 / Virus Database: 4392/10332 - Release Date: 07/29/15
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: "Too Many Connections" exceptions after moving to Tomcat 8
Posted by Mark Thomas <ma...@apache.org>.
On 29/07/2015 20:25, Jerry Malcolm wrote:
> Well, it appears that we are slowly getting to the bottom of this. But
> with every answer, I get a few more questions....
>
> First, I installed the latest TC8 on my laptop, copied my server.conf
> and conf/catalina folder to it and started it up just to see what errors
> I got. After changing out an obsolete listener, it came up. I found
> all of the <resource> parm exceptions in stderr. So that question is
> cleared up. Thanks for clarifying where to find that.
>
> The site is a wedding vendor advertiser site that spans two major
> cities. There is no user login. Simply a very huge online catalog.
> I'm certain it's deployed only once. Whether I need that many
> connections is a valid question. As far as I know, I haven't hit the
> limit in normal operation until now. Could possibly reduce the count if
> I collect statistics.
>
> I've been monitoring the production server logs all day watching to be
> sure connection pool doesn't dry up again. About an hour ago, there was
> a single huge dump in stdout of approx 2000 'logAbandoned' exceptions.
> They showed connections from 1am right after my last bounce of the
> server thru 1:35pm. The good news is with the stack trace on one of
> them I was able to see the bug causing the leak. But why did it decide
> to wait over 12 hours accumulating abandoned connections before dumping
> them back in the pool? I realize from now knowing the code bug that the
> leak is a slow drip that is continually leaking on a regular basis. But
> since that last 12-hour accumulated dump, the abandoning has returned to
> silence. Since leaks are occurring regularly and would be timing out
> regularly, shouldn't I see a similar 'slow drip' of logAbandoned entries
> in stdout instead of a big dump every 12 hours?
>
> It's going to take a day or two to fix the leak, test, and deploy. If
> indeed abandoned connections are now correctly being returned to the
> pool, then I presume we are back to working the way it did on TC7.
> Still not sure why it started working now. But I guess once I get the
> leak fixed and if TC8 is now configured to handle abandoned connections,
> I'm good. Still would like to know about the mega-dump vs. trickle of
> abandoned connections being logged.
That sounds like a bug that has just been fixed in Pool:
https://issues.apache.org/jira/browse/POOL-300
The fix for that should be in the next Tomcat 8 release.
Mark
>
> Thanks again for all of the info.
>
> Jerry
>
> On 7/29/2015 1:15 PM, Christopher Schultz wrote:
> Jerry,
>
> On 7/29/15 12:06 PM, Jerry Malcolm wrote:
>>>> On 7/29/2015 10:11 AM, Christopher Schultz wrote:
>>>>
>>>> I guess I could move up to the latest 8. But it'll require some
>>>> client down time, and they aren't thrilled with that at this
>>>> point.
> Understood, though trying various <Resource> configurations is
> certainly resulting in downtime, no?
>
>>>>>> Off topic, but relevant.... it sure would be nice if the xml
>>>>>> parser for <resource> tags and other config items would flag
>>>>>> with errors any parameter misspellings such as using the old
>>>>>> obsolete 'removeAbandoned' instead of the new
>>>>>> 'removeAbandonedOnBorrow', etc. or even just typo misspells.
>>>>> It does:
>>>>>
>>>>> Jul 02, 2015 10:17:25 AM
>>>>> org.apache.tomcat.dbcp.dbcp2.BasicDataSourceFactory get
>>>>> ObjectInstance WARNING: Name = diagnosis Property maxActive is
>>>>> not used in DBCP2, use maxTotal instead. maxTotal default value
>>>>> is 8. You have set value of "1" for "maxActive" property, which
>>>>> is being ignored.
>>>>>
>>>>> ...
>>>>>
>>>>> WARNING: Name = diagnosis Property removeAbandoned is not used
>>>>> in DBCP2, use one or both of removeAbandonedOnBorrow or
>>>>> removeAbandonedOnMaintenance instead. Both have default value set
>>>>> to false. You have set value of "true" for "removeAbandoned"
>>>>> property, which is being ignored.
>>>>>
>>>>>
>>>>> (I have a context.xml that I want to use between both Tomcat 7
>>>>> and Tomcat 8, so I actually have both spellings of those
>>>>> attributes in my XML.)
>>>> This is really strange. Are the above log entries in stdout?
> Yes, so they typically end up in catalina.out. Those logs are from my
> dev system's catalina.out from a recent restart.
>
>>>> I had 'removeAbandoned' in my resource statements until yesterday.
>>>> But I am not seeing anything in stdout like the statement above.
>>>> And after changing removeAbandoned to the correct spelling, I'm
>>>> also not seeing any abandoned log entries either.
> Hmm.
>
>>>> Is there some sort of 'log level' that may be throttling back my
>>>> log entries? If my code has leaks, and I'm running out of
>>>> connections but not seeing any abandoned entries in the log, that
>>>> should be a concern, right?
> Possibly.
>
> Perhaps you are deploying your application more times than you
> thought, and are making many more connections than you expected.
>
>>>>> What does your <Resource> element actually look like? (Remember
>>>>> to remove sensitive information such as passwords.)
>>>> <Resource name="jdbc/wgbe" testOnBorrow="true"
>>>> validationQuery="SELECT 1"
> validationQuery might not be necessary anymore. I think DBCP2 does a
> lightweight connection test instead issuing an actual query, unless
> you override it with a validationQuery.
>
> Also, for Connector/J, you can use "/* ping */SELECT 1" and the driver
> itself will perform the lightweight connection test instead of
> actually issuing a query. Much faster than going through the SQL
> parser, etc.
>
>>>> auth="Container" type="javax.sql.DataSource" maxTotal="300"
> That's a LOT of connections IMO. What is your total connection limit?
> Is it per-user?
>
>>>> maxIdle="30" maxWaitMillis="10000" removeAbandonedOnBorrow="true"
>>>> removeAbandonedOnMaintenance="true" removeAbandonedTimeout="30"
>>>> logAbandoned="true" username="xxxxxxx" password="xxxxxxxxx"
>>>> driverClassName="org.gjt.mm.mysql.Driver"
>>>> url="jdbc:mysql://localhost:3306/wgbe?autoReconnect=true" />
> Do you have the 'manager' web application installed? Can you check to
> see what applications are actually deployed? If you have 2 copies
> deployed, you might see 600 connections open.
>
> Also, do you have any stray context.xml files hanging around in
> conf/*/* that you weren't expecting? If you are making changes to your
> configuration and they don't appear to be having any effect, it's
> possible that your new configuration isn't being applied because
> another config is masking it.
>
> -chris
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>>
>> -----
>> No virus found in this message.
>> Checked by AVG - www.avg.com
>> Version: 2015.0.6081 / Virus Database: 4392/10332 - Release Date:
>> 07/29/15
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [OT] "Too Many Connections" exceptions after moving to Tomcat
8
Posted by Phil Steitz <ph...@gmail.com>.
On 8/4/15 2:03 PM, Christopher Schultz wrote:
> Phil,
>
> On 8/1/15 9:34 PM, Phil Steitz wrote:
> > On 8/1/15 6:14 PM, Christopher Schultz wrote:
> >> Phil,
> >>
> >> On 8/1/15 11:12 AM, Phil Steitz wrote:
> >>> Sorry the docs are not that clear. The problem is that the
> >>> config properties work together to determine behavior and
> >>> documenting them individually makes it hard to put the whole
> >>> picture together. Improvement patches most welcome :) In any
> >>> case, here is how it works:
> >>
> >>> Connections are evaluated for abandonment when they are out in
> >>> circulation - checked out to clients. If you have set
> >>> timeBetweenEvictionRunsMillis to a positive value, pool
> >>> maintenance runs every timeBetweenEvictionRunsMillis
> >>> milliseconds. If you have removeAbandonedOnMaintenance set to
> >>> true, each time maintenance runs the pool removes abandoned
> >>> connections. If you have removeAbandonedOnBorrow set to true,
> >>> the pool removes abandoned connections if it is nearing
> >>> depletion when a borrow request arrives. In both cases, the
> >>> pool looks at the statistics that it maintains on the list of
> >>> all objects checked out by clients to determine which ones
> >>> appear to be abandoned. To appear abandoned means to be
> >>> checked out by a client but not used for longer than the
> >>> removeAbandonedTimeout. For DBCP, "used" means the connection
> >>> has been used.
> >>
> >> So when during all that does the abandoned connection get
> >> "logged"?
>
> > The log message is written when the abandoned connection is closed
> > by the pool.
>
> So if the client code never calls close(), does that mean the
> amandonment is never logged? IIRC, DBCP 1.x would log the stack trace
> as soon as removeAbandonedTimeout elapsed if the connection hadn't
> been returned.
The behavior is the same in 2.x as 1.x with the exception that in
2.x, pool maintenance can also trigger abandoned object removal. In
1.x, it is necessary to attempt a borrow to get abandoned
connections to be removed. Both 1.x and 2.x essentially remove
abandoned connections in batches, closing the connections and
logging the traces when the batches run. In 1.x, the only way to
trigger an abandoned cleanup batch is to request a connection when
the pool is near depletion. In 2.x, if removeAbandonedOnMaintenance
is true, an abandoned cleanup batch will run every
timeBetweenEvictionRunsMillis ms.
>
> > (The pool bug was basically delaying the effective write; but the
> > stack trace goes to the buffer when the close happens)
> >>
> >> If I have a pool with two connections in it, and the first one
> >> gets leaked (after, lets say, 10 seconds) but the second one is
> >> returned properly, and the application continues to borrow and
> >> return the second connection, what happens?
>
> > That depends on the settings. See below.
> >>
> >> Are you only checking connections for "abandonment" after they
> >> are returned and then when a client attempts to borrow that
> >> specific connection? Or, are all currently-borrowed connections
> >> checked for abandonment whenever any borrow operation is
> >> requested?
>
> > It is the latter, assuming removeAbandonedOnBorrow is true.
> > Removal on borrow also requires that the pool be near depletion
> > when the borrow happens (see the javadoc for precise description
> > of conditions). There are two things that can trigger abandoned
> > object removal - a borrow request and pool maintenance. In each
> > case, if the config setting to enable the check on that event is
> > on, all checked out objects are examined. The ones that appear
> > abandoned are destroyed (in DBCP that means physical connections
> > are closed).
>
> Why is there no option to dump a warning as soon as the timeout
> elapses?
Ideas / patches welcome on how to do that efficiently.
> I suppose if the pool is being used slightly more than
> /never/, the abandoned notice will happen pretty quickly, but for
> testing purposes, it means that a single-user will have to perform
> some operation, wait, and then perform another operation in order to
> get the feedback about the abandonment. That's surprising behavior if
> you ask me.
See above. If you set removeAbandonedOnMaintenance and
maxTimeBetweenEvictionRunsMillis, abandoned connection removal will
happen regularly.
Phil
>
> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [OT] "Too Many Connections" exceptions after moving to Tomcat 8
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Phil,
On 8/1/15 9:34 PM, Phil Steitz wrote:
> On 8/1/15 6:14 PM, Christopher Schultz wrote:
>> Phil,
>>
>> On 8/1/15 11:12 AM, Phil Steitz wrote:
>>> Sorry the docs are not that clear. The problem is that the
>>> config properties work together to determine behavior and
>>> documenting them individually makes it hard to put the whole
>>> picture together. Improvement patches most welcome :) In any
>>> case, here is how it works:
>>
>>> Connections are evaluated for abandonment when they are out in
>>> circulation - checked out to clients. If you have set
>>> timeBetweenEvictionRunsMillis to a positive value, pool
>>> maintenance runs every timeBetweenEvictionRunsMillis
>>> milliseconds. If you have removeAbandonedOnMaintenance set to
>>> true, each time maintenance runs the pool removes abandoned
>>> connections. If you have removeAbandonedOnBorrow set to true,
>>> the pool removes abandoned connections if it is nearing
>>> depletion when a borrow request arrives. In both cases, the
>>> pool looks at the statistics that it maintains on the list of
>>> all objects checked out by clients to determine which ones
>>> appear to be abandoned. To appear abandoned means to be
>>> checked out by a client but not used for longer than the
>>> removeAbandonedTimeout. For DBCP, "used" means the connection
>>> has been used.
>>
>> So when during all that does the abandoned connection get
>> "logged"?
>
> The log message is written when the abandoned connection is closed
> by the pool.
So if the client code never calls close(), does that mean the
amandonment is never logged? IIRC, DBCP 1.x would log the stack trace
as soon as removeAbandonedTimeout elapsed if the connection hadn't
been returned.
> (The pool bug was basically delaying the effective write; but the
> stack trace goes to the buffer when the close happens)
>>
>> If I have a pool with two connections in it, and the first one
>> gets leaked (after, lets say, 10 seconds) but the second one is
>> returned properly, and the application continues to borrow and
>> return the second connection, what happens?
>
> That depends on the settings. See below.
>>
>> Are you only checking connections for "abandonment" after they
>> are returned and then when a client attempts to borrow that
>> specific connection? Or, are all currently-borrowed connections
>> checked for abandonment whenever any borrow operation is
>> requested?
>
> It is the latter, assuming removeAbandonedOnBorrow is true.
> Removal on borrow also requires that the pool be near depletion
> when the borrow happens (see the javadoc for precise description
> of conditions). There are two things that can trigger abandoned
> object removal - a borrow request and pool maintenance. In each
> case, if the config setting to enable the check on that event is
> on, all checked out objects are examined. The ones that appear
> abandoned are destroyed (in DBCP that means physical connections
> are closed).
Why is there no option to dump a warning as soon as the timeout
elapses? I suppose if the pool is being used slightly more than
/never/, the abandoned notice will happen pretty quickly, but for
testing purposes, it means that a single-user will have to perform
some operation, wait, and then perform another operation in order to
get the feedback about the abandonment. That's surprising behavior if
you ask me.
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
iQIcBAEBCAAGBQJVwSifAAoJEBzwKT+lPKRYoaAP/jLgMP3BWm3j5OTvAFw55Sbm
GyDQel0X0LxlXC1Nn/YETjTvUGh3d1sv0FE0zyuNFfrmEGi35hX6vEIFA+aAx5wb
1LjUR4gUTbMR/4mYGbVyHb65Pt+7CY/oMuGbp22o+2DwXEy/+1sS7lHAXK/xKkbb
F+HcLN7bU2khLp/1JuSp7NfmSEpBKkpuitkTUyGR27ijorYRRqpLfM80/aJV8aTh
MSMhgFuJ6igkJv29oTMqTXY3k+xIfZeIpJuYQ0yFLFwKmZuxPgSP+9hSfOyQAbDs
fev2bcwE9yYmypWOJrPBFR+T7xt61BSLzi1LwUixuTDP6eZEOYrYDvXd0lrB+Ohv
jS5Md182lWiAwbF+8grMwALDTUCE4cCfq/BqO1gSWNaZSDJoFhmm0QmPPgB2iVU+
ZEX9rkwCfY0SKal2x7uJzMwCJYgtPlXq88Fr2aX+Wvg90nXKqwVucAMBf+zOyaP3
tEIuw4H20ug6OPgbfUmdKS1qYLyjS5V7XzP/LoZCNjzDK+pqrmdkRneNdSiIlTzv
2HIQKtIYFZXHEMOmnBGiMWrGfF8lUrAk/rpmFerXk2RI/e6H9ufAExSIz7xhKZEG
AToWrGhb7DOCBOAsABoGnU4R9qE27FuWBcceKdDmSRZzqThmT+9o2tLjXEGJfcc1
Bmt9TxMpWJYKboZkaj/w
=eTpv
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [OT] "Too Many Connections" exceptions after moving to Tomcat
8
Posted by Phil Steitz <ph...@gmail.com>.
On 8/1/15 6:14 PM, Christopher Schultz wrote:
> Phil,
>
> On 8/1/15 11:12 AM, Phil Steitz wrote:
> > Sorry the docs are not that clear. The problem is that the config
> > properties work together to determine behavior and documenting
> > them individually makes it hard to put the whole picture together.
> > Improvement patches most welcome :) In any case, here is how it
> > works:
>
> > Connections are evaluated for abandonment when they are out in
> > circulation - checked out to clients. If you have set
> > timeBetweenEvictionRunsMillis to a positive value, pool
> > maintenance runs every timeBetweenEvictionRunsMillis milliseconds.
> > If you have removeAbandonedOnMaintenance set to true, each time
> > maintenance runs the pool removes abandoned connections. If you
> > have removeAbandonedOnBorrow set to true, the pool removes
> > abandoned connections if it is nearing depletion when a borrow
> > request arrives. In both cases, the pool looks at the statistics
> > that it maintains on the list of all objects checked out by clients
> > to determine which ones appear to be abandoned. To appear
> > abandoned means to be checked out by a client but not used for
> > longer than the removeAbandonedTimeout. For DBCP, "used" means the
> > connection has been used.
>
> So when during all that does the abandoned connection get "logged"?
The log message is written when the abandoned connection is closed
by the pool. (The pool bug was basically delaying the effective
write; but the stack trace goes to the buffer when the close happens)
>
> If I have a pool with two connections in it, and the first one gets
> leaked (after, lets say, 10 seconds) but the second one is returned
> properly, and the application continues to borrow and return the
> second connection, what happens?
That depends on the settings. See below.
>
> Are you only checking connections for "abandonment" after they are
> returned and then when a client attempts to borrow that specific
> connection? Or, are all currently-borrowed connections checked for
> abandonment whenever any borrow operation is requested?
It is the latter, assuming removeAbandonedOnBorrow is true. Removal
on borrow also requires that the pool be near depletion when the
borrow happens (see the javadoc for precise description of
conditions). There are two things that can trigger abandoned object
removal - a borrow request and pool maintenance. In each case, if
the config setting to enable the check on that event is on, all
checked out objects are examined. The ones that appear abandoned
are destroyed (in DBCP that means physical connections are closed).
Phil
>
> Thanks,
> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [OT] "Too Many Connections" exceptions after moving to Tomcat 8
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Phil,
On 8/1/15 11:12 AM, Phil Steitz wrote:
> Sorry the docs are not that clear. The problem is that the config
> properties work together to determine behavior and documenting
> them individually makes it hard to put the whole picture together.
> Improvement patches most welcome :) In any case, here is how it
> works:
>
> Connections are evaluated for abandonment when they are out in
> circulation - checked out to clients. If you have set
> timeBetweenEvictionRunsMillis to a positive value, pool
> maintenance runs every timeBetweenEvictionRunsMillis milliseconds.
> If you have removeAbandonedOnMaintenance set to true, each time
> maintenance runs the pool removes abandoned connections. If you
> have removeAbandonedOnBorrow set to true, the pool removes
> abandoned connections if it is nearing depletion when a borrow
> request arrives. In both cases, the pool looks at the statistics
> that it maintains on the list of all objects checked out by clients
> to determine which ones appear to be abandoned. To appear
> abandoned means to be checked out by a client but not used for
> longer than the removeAbandonedTimeout. For DBCP, "used" means the
> connection has been used.
So when during all that does the abandoned connection get "logged"?
If I have a pool with two connections in it, and the first one gets
leaked (after, lets say, 10 seconds) but the second one is returned
properly, and the application continues to borrow and return the
second connection, what happens?
Are you only checking connections for "abandonment" after they are
returned and then when a client attempts to borrow that specific
connection? Or, are all currently-borrowed connections checked for
abandonment whenever any borrow operation is requested?
Thanks,
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
iQIcBAEBCAAGBQJVvW7pAAoJEBzwKT+lPKRYY0UQAIBXTrkBRNC9j7KBYw8DZFcJ
UY+U1WrHi7osxEseupCjSjP/xmmOHoC16LkEXqtil5k8rd5yQ9D5I1imPKYZSjiE
GBBkkjrbvlwDq/sRp3fxN8La6ueTed4yyh96J5DFfx0opcTtX06YMnpkJDTN2iTt
R3GUKbBo4dNZKIFNGhicR4j9JbSwerxbfJDVpEtb0igO7fQu1dRTDQh7gijyUsZ7
PPa0owq9WOBl5dr4ebsoipirArbLbB4WZqCtteOj5QpafEOYyGuKAjFs6PhodVAR
A2BlikA9+s6AUcKVnqP+CabZFNYO/1raZWl61q7vSWNR9YMrSSI6TapCsBgzNDQW
p8KloJYqjaZFNGtVW23eDtRjTehABnQqoXrB4OL2CEFqgZVjnfpCDl1K0b8wc01y
QNnA0eo40462XkwsRSyMOwtunLu02mGayOrzrn7blJ0f2Kj594+/U6HKqAttRlOX
0Sr2VnYJTaBRACHVjycKjjQh1A1lIIfRoIQ0enMBiw8kZ4rVpQsCvwk/5h54/oap
5osEzulFO7zwqhAxWaHigFG9hCeHJHGr7eT4XUXfUqw+/zg6vI//r5rLExIwSi4D
p/R214PYSkEMiz+i4uSu+sUOYpOJbaAnJH0R6kXSKkoDmaaylTyLuHujPvLKPF6b
ZV739LJI+k5cgcDEWmjv
=PKZ/
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: "Too Many Connections" exceptions after moving to Tomcat 8
Posted by Phil Steitz <ph...@gmail.com>.
On 7/31/15 8:16 AM, Christopher Schultz wrote:
> Phil,
>
> On 7/30/15 8:05 PM, Phil Steitz wrote:
> > On 7/30/15 9:03 AM, Christopher Schultz wrote:
> >> Jerry,
> >>
> >> On 7/29/15 3:25 PM, Jerry Malcolm wrote:
> >>> Well, it appears that we are slowly getting to the bottom of
> >>> this. But with every answer, I get a few more questions....
> >>
> >>> First, I installed the latest TC8 on my laptop, copied my
> >>> server.conf and conf/catalina folder to it and started it up
> >>> just to see what errors I got. After changing out an obsolete
> >>> listener, it came up. I found all of the <resource> parm
> >>> exceptions in stderr. So that question is cleared up. Thanks
> >>> for clarifying where to find that.
> >>
> >> If you have an obsolete Listener, you probably copied your
> >> server.xml from Tomcat 7 to Tomcat 8 which, while being less of a
> >> disaster than with previous version-pairs, is not good practice.
> >>
> >> Instead, start with the stock server.xml that comes with your
> >> Tomcat version and modify it to suit. These days, you should
> >> pretty much only have to configure the <Executor> and <Connector>
> >> elements, unless you have a particularly exotic <Host>
> >> configuration.
> >>
> >>> The site is a wedding vendor advertiser site that spans two
> >>> major cities. There is no user login. Simply a very huge
> >>> online catalog. I'm certain it's deployed only once. Whether I
> >>> need that many connections is a valid question. As far as I
> >>> know, I haven't hit the limit in normal operation until now.
> >>> Could possibly reduce the count if I collect statistics.
> >>
> >> Our user load is roughly 250 concurrently logged-in users per
> >> Tomcat node, and we have maxTotal="20". I never get alarms about
> >> hitting that maximum. Your requirements may be different.
> >>
> >>> I've been monitoring the production server logs all day
> >>> watching to be sure connection pool doesn't dry up again.
> >>> About an hour ago, there was a single huge dump in stdout of
> >>> approx 2000 'logAbandoned' exceptions. They showed connections
> >>> from 1am right after my last bounce of the server thru 1:35pm.
> >>
> >> It looks like your startup process (likely loading and caching
> >> stuff from the db on launch) is leaky. That can run-up your
> >> connection could quite quickly.
> >>
> >>> The good news is with the stack trace on one of them I was able
> >>> to see the bug causing the leak.
> >>
> >> Good.
> >>> But why did it decide to wait over 12 hours accumulating
> >>> abandoned connections before dumping them back in the pool?
> >>
> >> I was about to say the following, but markt says it might be a
> >> bug in DBCP.
>
> > The bug is in Commons Pool (POOL-300). It is not flushing its
> > abandoned object log. That means abandoned traces won't appear
> > (in the default System.out configuration) until some have been
> > accumulated.
>
> Thanks for the correction.
>
> >> I'll say it anyway:
> >>
> >> DBCP 2 looks like it only checks for abandoned connections "on
> >> borrow" so it might only log their abandonment when you see a
> >> flurry of connection-checkouts occurring, not when the
> >> connections are returned to the pool. DBCP 1 would complain
> >> pretty much immediately when the timeout was reached and the
> >> connection hadn't been returned.
>
> > When DBCP checks for abandoned connections depends in its
> > configuration properties. There are two relevant properties:
> > removeAbandonedOnBorrow and removeAbandonedOnMaintenance.
>
> Right. I think most people don't use the "maintenance" mode, so I was
> being sloppy. I haven't read the code, but the configuration options
> make it sound like the connection isn't checked until it's borrowed
> again from the pool, which could be a very long time after it would be
> expected to have been "abandoned".
Sorry the docs are not that clear. The problem is that the config
properties work together to determine behavior and documenting them
individually makes it hard to put the whole picture together.
Improvement patches most welcome :) In any case, here is how it works:
Connections are evaluated for abandonment when they are out in
circulation - checked out to clients. If you have set
timeBetweenEvictionRunsMillis to a positive value, pool maintenance
runs every timeBetweenEvictionRunsMillis milliseconds. If you have
removeAbandonedOnMaintenance set to true, each time maintenance runs
the pool removes abandoned connections. If you have
removeAbandonedOnBorrow set to true, the pool removes abandoned
connections if it is nearing depletion when a borrow request
arrives. In both cases, the pool looks at the statistics that it
maintains on the list of all objects checked out by clients to
determine which ones appear to be abandoned. To appear abandoned
means to be checked out by a client but not used for longer than the
removeAbandonedTimeout. For DBCP, "used" means the connection has
been used.
Phil
>
> >>> I realize from now knowing the code bug that the leak is a
> >>> slow drip that is continually leaking on a regular basis. But
> >>> since that last 12-hour accumulated dump, the abandoning has
> >>> returned to silence. Since leaks are occurring regularly and
> >>> would be timing out regularly, shouldn't I see a similar 'slow
> >>> drip' of logAbandoned entries in stdout instead of a big dump
> >>> every 12 hours?
> >>
> >>> It's going to take a day or two to fix the leak, test, and
> >>> deploy.
> >>
> >> For testing, set maxTotal="1". You'll find your leaks *very
> >> quickly* that way, because everything will come grinding to a
> >> halt when you try to fetch that second connection from the pool.
> >>
> >>> If indeed abandoned connections are now correctly being
> >>> returned to the pool, then I presume we are back to working the
> >>> way it did on TC7. Still not sure why it started working now.
> >>> But I guess once I get the leak fixed and if TC8 is now
> >>> configured to handle abandoned connections, I'm good. Still
> >>> would like to know about the mega-dump vs. trickle of abandoned
> >>> connections being logged.
> >>
> >> You should be able to run in testing with an upgraded DBCP 2.
> >> You might have to build it from trunk, though. I'm not sure if
> >> you are okay with that, but it might help you with your testing.
>
> > The thing to swap out is Commons Pool.
>
> Yep, thanks for the clarification.
>
> > There is a release VOTE in progress now for an RC including a fix
> > for POOL-300.
>
> Good news!
>
> > A workaround that should work is to get a reference to the
> > BasicDataSource instance, say, bds and do
>
> > bds.setAbandonedLogWriter(new PrintWriter(System.out, true));
>
> > before using the pool.
>
> > Not sure if this will work correctly to get the output properly
> > directed under tomcat; but it is worth a try.
>
> Definitely.
>
> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: "Too Many Connections" exceptions after moving to Tomcat 8
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Phil,
On 7/30/15 8:05 PM, Phil Steitz wrote:
> On 7/30/15 9:03 AM, Christopher Schultz wrote:
>> Jerry,
>>
>> On 7/29/15 3:25 PM, Jerry Malcolm wrote:
>>> Well, it appears that we are slowly getting to the bottom of
>>> this. But with every answer, I get a few more questions....
>>
>>> First, I installed the latest TC8 on my laptop, copied my
>>> server.conf and conf/catalina folder to it and started it up
>>> just to see what errors I got. After changing out an obsolete
>>> listener, it came up. I found all of the <resource> parm
>>> exceptions in stderr. So that question is cleared up. Thanks
>>> for clarifying where to find that.
>>
>> If you have an obsolete Listener, you probably copied your
>> server.xml from Tomcat 7 to Tomcat 8 which, while being less of a
>> disaster than with previous version-pairs, is not good practice.
>>
>> Instead, start with the stock server.xml that comes with your
>> Tomcat version and modify it to suit. These days, you should
>> pretty much only have to configure the <Executor> and <Connector>
>> elements, unless you have a particularly exotic <Host>
>> configuration.
>>
>>> The site is a wedding vendor advertiser site that spans two
>>> major cities. There is no user login. Simply a very huge
>>> online catalog. I'm certain it's deployed only once. Whether I
>>> need that many connections is a valid question. As far as I
>>> know, I haven't hit the limit in normal operation until now.
>>> Could possibly reduce the count if I collect statistics.
>>
>> Our user load is roughly 250 concurrently logged-in users per
>> Tomcat node, and we have maxTotal="20". I never get alarms about
>> hitting that maximum. Your requirements may be different.
>>
>>> I've been monitoring the production server logs all day
>>> watching to be sure connection pool doesn't dry up again.
>>> About an hour ago, there was a single huge dump in stdout of
>>> approx 2000 'logAbandoned' exceptions. They showed connections
>>> from 1am right after my last bounce of the server thru 1:35pm.
>>
>> It looks like your startup process (likely loading and caching
>> stuff from the db on launch) is leaky. That can run-up your
>> connection could quite quickly.
>>
>>> The good news is with the stack trace on one of them I was able
>>> to see the bug causing the leak.
>>
>> Good.
>>> But why did it decide to wait over 12 hours accumulating
>>> abandoned connections before dumping them back in the pool?
>>
>> I was about to say the following, but markt says it might be a
>> bug in DBCP.
>
> The bug is in Commons Pool (POOL-300). It is not flushing its
> abandoned object log. That means abandoned traces won't appear
> (in the default System.out configuration) until some have been
> accumulated.
Thanks for the correction.
>> I'll say it anyway:
>>
>> DBCP 2 looks like it only checks for abandoned connections "on
>> borrow" so it might only log their abandonment when you see a
>> flurry of connection-checkouts occurring, not when the
>> connections are returned to the pool. DBCP 1 would complain
>> pretty much immediately when the timeout was reached and the
>> connection hadn't been returned.
>
> When DBCP checks for abandoned connections depends in its
> configuration properties. There are two relevant properties:
> removeAbandonedOnBorrow and removeAbandonedOnMaintenance.
Right. I think most people don't use the "maintenance" mode, so I was
being sloppy. I haven't read the code, but the configuration options
make it sound like the connection isn't checked until it's borrowed
again from the pool, which could be a very long time after it would be
expected to have been "abandoned".
>>> I realize from now knowing the code bug that the leak is a
>>> slow drip that is continually leaking on a regular basis. But
>>> since that last 12-hour accumulated dump, the abandoning has
>>> returned to silence. Since leaks are occurring regularly and
>>> would be timing out regularly, shouldn't I see a similar 'slow
>>> drip' of logAbandoned entries in stdout instead of a big dump
>>> every 12 hours?
>>
>>> It's going to take a day or two to fix the leak, test, and
>>> deploy.
>>
>> For testing, set maxTotal="1". You'll find your leaks *very
>> quickly* that way, because everything will come grinding to a
>> halt when you try to fetch that second connection from the pool.
>>
>>> If indeed abandoned connections are now correctly being
>>> returned to the pool, then I presume we are back to working the
>>> way it did on TC7. Still not sure why it started working now.
>>> But I guess once I get the leak fixed and if TC8 is now
>>> configured to handle abandoned connections, I'm good. Still
>>> would like to know about the mega-dump vs. trickle of abandoned
>>> connections being logged.
>>
>> You should be able to run in testing with an upgraded DBCP 2.
>> You might have to build it from trunk, though. I'm not sure if
>> you are okay with that, but it might help you with your testing.
>
> The thing to swap out is Commons Pool.
Yep, thanks for the clarification.
> There is a release VOTE in progress now for an RC including a fix
> for POOL-300.
Good news!
> A workaround that should work is to get a reference to the
> BasicDataSource instance, say, bds and do
>
> bds.setAbandonedLogWriter(new PrintWriter(System.out, true));
>
> before using the pool.
>
> Not sure if this will work correctly to get the output properly
> directed under tomcat; but it is worth a try.
Definitely.
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
iQIcBAEBCAAGBQJVu5FhAAoJEBzwKT+lPKRY4iYQAJyk0anccACg2gINK7RomL7G
mG/q3ZCIfOTFaY+GJAJHk2iHnN8rzA+yoJ+od6xrJyF+TZDtu4X+AdUXsJLKzYYL
oKt7SGk1hBT1mUd2TxXhdyL/+i+dw4Knz2TImgB5EPknPiHQDc4w6yppvuz7LNdc
BnADCoqbW6HO6htzqhkuRDoMC8rErJgMRVBW+U9yXdejWF1x1vIWG4zLQu0GJF75
2IlwNjO4n8jHHfBqrXw/OkSTF448pJSomXUGq1MtwFk+D5/e1n7IIuz31xnx/TIF
rXcyPMW1ukYpSUH2lHHKOKqf+LCQOp1UlJwpCpFm3QvgI6dDwmfP1WUrzi80y+33
m+E/D/WfRZ7saIN2qFtGFo7S88YGeP0eJBJ9UNeuNLFpXGgaYdv+iuxBqd4isRPN
KfwL0J8Jg4xFO7IhV3CWRpPe37C6nsK3H2iTX4IFQ/fqWXICAm3UJgCz43OwpSWE
3vt7TLDicP0+wsFWmLCFELGcJvvpmS1MKD6I8zXV5ryt7uun8j5FdIwt6gJSkpCV
Vhm33vhC59L1ZqkQWo/s1TSKgQ0EMxJsgRE97+onOE9oL63bimlbXS1QZM6inSEC
xsC0V6FVf7kp0IonTsOLCqUUXW4yNlYrFYEZvqA5GR7xMRs8NPSoINyQ+57Ehvjn
6T4ANOg/gcdtd5Bni5cH
=npUe
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: "Too Many Connections" exceptions after moving to Tomcat 8
Posted by Phil Steitz <ph...@gmail.com>.
On 7/30/15 9:03 AM, Christopher Schultz wrote:
> Jerry,
>
> On 7/29/15 3:25 PM, Jerry Malcolm wrote:
> > Well, it appears that we are slowly getting to the bottom of this.
> > But with every answer, I get a few more questions....
>
> > First, I installed the latest TC8 on my laptop, copied my
> > server.conf and conf/catalina folder to it and started it up just
> > to see what errors I got. After changing out an obsolete listener,
> > it came up. I found all of the <resource> parm exceptions in
> > stderr. So that question is cleared up. Thanks for clarifying
> > where to find that.
>
> If you have an obsolete Listener, you probably copied your server.xml
> from Tomcat 7 to Tomcat 8 which, while being less of a disaster than
> with previous version-pairs, is not good practice.
>
> Instead, start with the stock server.xml that comes with your Tomcat
> version and modify it to suit. These days, you should pretty much only
> have to configure the <Executor> and <Connector> elements, unless you
> have a particularly exotic <Host> configuration.
>
> > The site is a wedding vendor advertiser site that spans two major
> > cities. There is no user login. Simply a very huge online
> > catalog. I'm certain it's deployed only once. Whether I need that
> > many connections is a valid question. As far as I know, I haven't
> > hit the limit in normal operation until now. Could possibly reduce
> > the count if I collect statistics.
>
> Our user load is roughly 250 concurrently logged-in users per Tomcat
> node, and we have maxTotal="20". I never get alarms about hitting that
> maximum. Your requirements may be different.
>
> > I've been monitoring the production server logs all day watching to
> > be sure connection pool doesn't dry up again. About an hour ago,
> > there was a single huge dump in stdout of approx 2000
> > 'logAbandoned' exceptions. They showed connections from 1am right
> > after my last bounce of the server thru 1:35pm.
>
> It looks like your startup process (likely loading and caching stuff
> from the db on launch) is leaky. That can run-up your connection could
> quite quickly.
>
> > The good news is with the stack trace on one of them I was able to
> > see the bug causing the leak.
>
> Good.
> > But why did it decide to wait over 12 hours accumulating abandoned
> > connections before dumping them back in the pool?
>
> I was about to say the following, but markt says it might be a bug in
> DBCP.
The bug is in Commons Pool (POOL-300). It is not flushing its
abandoned object log. That means abandoned traces won't appear (in
the default System.out configuration) until some have been
accumulated.
> I'll say it anyway:
>
> DBCP 2 looks like it only checks for abandoned connections "on borrow"
> so it might only log their abandonment when you see a flurry of
> connection-checkouts occurring, not when the connections are returned
> to the pool. DBCP 1 would complain pretty much immediately when the
> timeout was reached and the connection hadn't been returned.
When DBCP checks for abandoned connections depends in its
configuration properties. There are two relevant properties:
removeAbandonedOnBorrow and removeAbandonedOnMaintenance.
>
> > I realize from now knowing the code bug that the leak is a slow
> > drip that is continually leaking on a regular basis. But since that
> > last 12-hour accumulated dump, the abandoning has returned to
> > silence. Since leaks are occurring regularly and would be timing
> > out regularly, shouldn't I see a similar 'slow drip' of
> > logAbandoned entries in stdout instead of a big dump every 12
> > hours?
>
> > It's going to take a day or two to fix the leak, test, and deploy.
>
> For testing, set maxTotal="1". You'll find your leaks *very quickly*
> that way, because everything will come grinding to a halt when you try
> to fetch that second connection from the pool.
>
> > If indeed abandoned connections are now correctly being returned to
> > the pool, then I presume we are back to working the way it did on
> > TC7. Still not sure why it started working now. But I guess once I
> > get the leak fixed and if TC8 is now configured to handle abandoned
> > connections, I'm good. Still would like to know about the
> > mega-dump vs. trickle of abandoned connections being logged.
>
> You should be able to run in testing with an upgraded DBCP 2. You
> might have to build it from trunk, though. I'm not sure if you are
> okay with that, but it might help you with your testing.
The thing to swap out is Commons Pool. There is a release VOTE in
progress now for an RC including a fix for POOL-300.
A workaround that should work is to get a reference to the
BasicDataSource instance, say, bds and do
bds.setAbandonedLogWriter(new PrintWriter(System.out, true));
before using the pool.
Not sure if this will work correctly to get the output properly
directed under tomcat; but it is worth a try.
Phil
>
> Glad you are rooting-out some problems with your code.
>
> I highly recommend the use of findbugs:
> http://findbugs.sourceforge.net/
>
> It automatically detects stuff like this in your code, as well as a
> whole bunch of other things. It sometimes finds false positives, but
> you can knock-up a configuration file to exclude those things in very
> specific cases if need be.
>
> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: "Too Many Connections" exceptions after moving to Tomcat 8
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Jerry,
On 7/29/15 3:25 PM, Jerry Malcolm wrote:
> Well, it appears that we are slowly getting to the bottom of this.
> But with every answer, I get a few more questions....
>
> First, I installed the latest TC8 on my laptop, copied my
> server.conf and conf/catalina folder to it and started it up just
> to see what errors I got. After changing out an obsolete listener,
> it came up. I found all of the <resource> parm exceptions in
> stderr. So that question is cleared up. Thanks for clarifying
> where to find that.
If you have an obsolete Listener, you probably copied your server.xml
from Tomcat 7 to Tomcat 8 which, while being less of a disaster than
with previous version-pairs, is not good practice.
Instead, start with the stock server.xml that comes with your Tomcat
version and modify it to suit. These days, you should pretty much only
have to configure the <Executor> and <Connector> elements, unless you
have a particularly exotic <Host> configuration.
> The site is a wedding vendor advertiser site that spans two major
> cities. There is no user login. Simply a very huge online
> catalog. I'm certain it's deployed only once. Whether I need that
> many connections is a valid question. As far as I know, I haven't
> hit the limit in normal operation until now. Could possibly reduce
> the count if I collect statistics.
Our user load is roughly 250 concurrently logged-in users per Tomcat
node, and we have maxTotal="20". I never get alarms about hitting that
maximum. Your requirements may be different.
> I've been monitoring the production server logs all day watching to
> be sure connection pool doesn't dry up again. About an hour ago,
> there was a single huge dump in stdout of approx 2000
> 'logAbandoned' exceptions. They showed connections from 1am right
> after my last bounce of the server thru 1:35pm.
It looks like your startup process (likely loading and caching stuff
from the db on launch) is leaky. That can run-up your connection could
quite quickly.
> The good news is with the stack trace on one of them I was able to
> see the bug causing the leak.
Good.
> But why did it decide to wait over 12 hours accumulating abandoned
> connections before dumping them back in the pool?
I was about to say the following, but markt says it might be a bug in
DBCP. I'll say it anyway:
DBCP 2 looks like it only checks for abandoned connections "on borrow"
so it might only log their abandonment when you see a flurry of
connection-checkouts occurring, not when the connections are returned
to the pool. DBCP 1 would complain pretty much immediately when the
timeout was reached and the connection hadn't been returned.
> I realize from now knowing the code bug that the leak is a slow
> drip that is continually leaking on a regular basis. But since that
> last 12-hour accumulated dump, the abandoning has returned to
> silence. Since leaks are occurring regularly and would be timing
> out regularly, shouldn't I see a similar 'slow drip' of
> logAbandoned entries in stdout instead of a big dump every 12
> hours?
>
> It's going to take a day or two to fix the leak, test, and deploy.
For testing, set maxTotal="1". You'll find your leaks *very quickly*
that way, because everything will come grinding to a halt when you try
to fetch that second connection from the pool.
> If indeed abandoned connections are now correctly being returned to
> the pool, then I presume we are back to working the way it did on
> TC7. Still not sure why it started working now. But I guess once I
> get the leak fixed and if TC8 is now configured to handle abandoned
> connections, I'm good. Still would like to know about the
> mega-dump vs. trickle of abandoned connections being logged.
You should be able to run in testing with an upgraded DBCP 2. You
might have to build it from trunk, though. I'm not sure if you are
okay with that, but it might help you with your testing.
Glad you are rooting-out some problems with your code.
I highly recommend the use of findbugs:
http://findbugs.sourceforge.net/
It automatically detects stuff like this in your code, as well as a
whole bunch of other things. It sometimes finds false positives, but
you can knock-up a configuration file to exclude those things in very
specific cases if need be.
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
iQIcBAEBCAAGBQJVukrfAAoJEBzwKT+lPKRYY20QAJw+fAfLXhTiTFadMRTeZFcX
aU5oLi08BVPUq1/gkCAykZkRFTDGx+iGAo7QKPfjfwstX89wFmY50BWppo/MT6sA
S0/nGQxwzSHI+lT1rxMAPhY/vq6PJXg92izkKejaF+8sy3uu5vbvzOviOcqtUAdD
+3TDX2ybZpQI1rS4lCZRrCqVONSH0if8pooBLUeQBmnXuUDDuRfU5mQB/mT4XRcr
MTWqjEU7/WgCmpTgdHyIEIImGNkey/79PAzxa9t8QjecdHpkCO4cE+5lmcw+0eL1
i/f2/+ujt5nDSv/n/+4PE9SEj2w70O0q9HNwmEOKBi9bv3lPEwzI38ixnm44/bqa
rV+/BPoKd6mZUQuXsH4cJccGhaZ1Sn5MzC2BPgFfBSeK3yGnNm7k6U9iyQ7EoW2O
HzAJquikdaPHKLvtmd76heMzxdSckDrlu2Xi8Jfp1exf/aTh426niunTZHrSRRPI
lJFmUCRglpG9peOXxf+sH7pvO1DMcpt8ExMrtLvKXPbEQ6MRybHXS5Z0lWrlQRGg
XPngxL66BHUVCTmrd4Q0RpysYkbtuO70ZrVGSX6yYV/SBBFQtrwi/yzAKFQu6IFj
57WQkRquDSkSo1wCuECiKPgpmP1JGTI8pDxaTKcJ0wTPgaBU2Z0DQZeW24wk1Rd0
Bjg3aT7AwW1q3R5y7jlp
=tDW7
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: "Too Many Connections" exceptions after moving to Tomcat 8
Posted by Jerry Malcolm <te...@malcolms.com>.
Well, it appears that we are slowly getting to the bottom of this. But
with every answer, I get a few more questions....
First, I installed the latest TC8 on my laptop, copied my server.conf
and conf/catalina folder to it and started it up just to see what errors
I got. After changing out an obsolete listener, it came up. I found
all of the <resource> parm exceptions in stderr. So that question is
cleared up. Thanks for clarifying where to find that.
The site is a wedding vendor advertiser site that spans two major
cities. There is no user login. Simply a very huge online catalog.
I'm certain it's deployed only once. Whether I need that many
connections is a valid question. As far as I know, I haven't hit the
limit in normal operation until now. Could possibly reduce the count if
I collect statistics.
I've been monitoring the production server logs all day watching to be
sure connection pool doesn't dry up again. About an hour ago, there was
a single huge dump in stdout of approx 2000 'logAbandoned' exceptions.
They showed connections from 1am right after my last bounce of the
server thru 1:35pm. The good news is with the stack trace on one of
them I was able to see the bug causing the leak. But why did it decide
to wait over 12 hours accumulating abandoned connections before dumping
them back in the pool? I realize from now knowing the code bug that the
leak is a slow drip that is continually leaking on a regular basis. But
since that last 12-hour accumulated dump, the abandoning has returned to
silence. Since leaks are occurring regularly and would be timing out
regularly, shouldn't I see a similar 'slow drip' of logAbandoned entries
in stdout instead of a big dump every 12 hours?
It's going to take a day or two to fix the leak, test, and deploy. If
indeed abandoned connections are now correctly being returned to the
pool, then I presume we are back to working the way it did on TC7.
Still not sure why it started working now. But I guess once I get the
leak fixed and if TC8 is now configured to handle abandoned connections,
I'm good. Still would like to know about the mega-dump vs. trickle of
abandoned connections being logged.
Thanks again for all of the info.
Jerry
On 7/29/2015 1:15 PM, Christopher Schultz wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Jerry,
>
> On 7/29/15 12:06 PM, Jerry Malcolm wrote:
>> On 7/29/2015 10:11 AM, Christopher Schultz wrote:
>>
>> I guess I could move up to the latest 8. But it'll require some
>> client down time, and they aren't thrilled with that at this
>> point.
> Understood, though trying various <Resource> configurations is
> certainly resulting in downtime, no?
>
>>>> Off topic, but relevant.... it sure would be nice if the xml
>>>> parser for <resource> tags and other config items would flag
>>>> with errors any parameter misspellings such as using the old
>>>> obsolete 'removeAbandoned' instead of the new
>>>> 'removeAbandonedOnBorrow', etc. or even just typo misspells.
>>> It does:
>>>
>>> Jul 02, 2015 10:17:25 AM
>>> org.apache.tomcat.dbcp.dbcp2.BasicDataSourceFactory get
>>> ObjectInstance WARNING: Name = diagnosis Property maxActive is
>>> not used in DBCP2, use maxTotal instead. maxTotal default value
>>> is 8. You have set value of "1" for "maxActive" property, which
>>> is being ignored.
>>>
>>> ...
>>>
>>> WARNING: Name = diagnosis Property removeAbandoned is not used
>>> in DBCP2, use one or both of removeAbandonedOnBorrow or
>>> removeAbandonedOnMaintenance instead. Both have default value set
>>> to false. You have set value of "true" for "removeAbandoned"
>>> property, which is being ignored.
>>>
>>>
>>> (I have a context.xml that I want to use between both Tomcat 7
>>> and Tomcat 8, so I actually have both spellings of those
>>> attributes in my XML.)
>> This is really strange. Are the above log entries in stdout?
> Yes, so they typically end up in catalina.out. Those logs are from my
> dev system's catalina.out from a recent restart.
>
>> I had 'removeAbandoned' in my resource statements until yesterday.
>> But I am not seeing anything in stdout like the statement above.
>> And after changing removeAbandoned to the correct spelling, I'm
>> also not seeing any abandoned log entries either.
> Hmm.
>
>> Is there some sort of 'log level' that may be throttling back my
>> log entries? If my code has leaks, and I'm running out of
>> connections but not seeing any abandoned entries in the log, that
>> should be a concern, right?
> Possibly.
>
> Perhaps you are deploying your application more times than you
> thought, and are making many more connections than you expected.
>
>>> What does your <Resource> element actually look like? (Remember
>>> to remove sensitive information such as passwords.)
>> <Resource name="jdbc/wgbe" testOnBorrow="true"
>> validationQuery="SELECT 1"
> validationQuery might not be necessary anymore. I think DBCP2 does a
> lightweight connection test instead issuing an actual query, unless
> you override it with a validationQuery.
>
> Also, for Connector/J, you can use "/* ping */SELECT 1" and the driver
> itself will perform the lightweight connection test instead of
> actually issuing a query. Much faster than going through the SQL
> parser, etc.
>
>> auth="Container" type="javax.sql.DataSource" maxTotal="300"
> That's a LOT of connections IMO. What is your total connection limit?
> Is it per-user?
>
>> maxIdle="30" maxWaitMillis="10000" removeAbandonedOnBorrow="true"
>> removeAbandonedOnMaintenance="true" removeAbandonedTimeout="30"
>> logAbandoned="true" username="xxxxxxx" password="xxxxxxxxx"
>> driverClassName="org.gjt.mm.mysql.Driver"
>> url="jdbc:mysql://localhost:3306/wgbe?autoReconnect=true" />
> Do you have the 'manager' web application installed? Can you check to
> see what applications are actually deployed? If you have 2 copies
> deployed, you might see 600 connections open.
>
> Also, do you have any stray context.xml files hanging around in
> conf/*/* that you weren't expecting? If you are making changes to your
> configuration and they don't appear to be having any effect, it's
> possible that your new configuration isn't being applied because
> another config is masking it.
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Comment: GPGTools - http://gpgtools.org
>
> iQIcBAEBCAAGBQJVuRgxAAoJEBzwKT+lPKRY/2YQALiweWD9Le3Ip3LeyFL/c43d
> ngd50y3iqYcPGbA33IOAy5ThDmcrRDiSbs5cMG4hhAo3rFIU8QmrCJdSFrhLIXu4
> K0ZcZPtlmPrEL0CUUE2D6HIX4mg0FMiNB7c0N4XjB6iicd16VGEMBVs4Ck7Sr0DM
> A2zTFEYhH/YXlWoOIkTE8gduNfHNNM7y6aaspIgRy9gqAMRFvD4HZaXONCrXPyKD
> hh2o8dInaSZY/jv3HrGcZd0SG9LzggKUynN195SOhc9cy6tsvOeIRFiOZWKckYSz
> NpKg77+2q9SYFVM5GSKPdrFlzbcjavmsoAP3EvCtWV5HvFE7MHo/ajvI/rSb47Uy
> 3jYD2b7S9NZdvPu/d8EPlqL29gbUhK2a1ISIY0S3Dgyi/Junzps/sztmOfgib71X
> IRDa3KH6gww4IwOX9K8YmMfHi9On06M2NF9NOVYiLqEN4CkHPAQ8YhzpnQz7/LSg
> DsxLQ9D5lDdaBjE3kvL17kd9h969SgM2SJKHfHay2xOTYjOHQ9vHY1EjF/C+3smd
> BzChYmUmjy0CBWjSCfYxJTjShNkLvyRyenAE+5WAFZhpOGTGW2DxSCXf9zQ2rqlv
> vSk7xVf6AMrZXQEZ376MsbRXGlUipZ9ZlufXLj9Pdqmcad7WoLHGaKNNwIyJvdWy
> AVhayMcrSTj5Y/xBeSXQ
> =xsp7
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
> -----
> No virus found in this message.
> Checked by AVG - www.avg.com
> Version: 2015.0.6081 / Virus Database: 4392/10332 - Release Date: 07/29/15
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: "Too Many Connections" exceptions after moving to Tomcat 8
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Jerry,
On 7/29/15 12:06 PM, Jerry Malcolm wrote:
> On 7/29/2015 10:11 AM, Christopher Schultz wrote:
>
> I guess I could move up to the latest 8. But it'll require some
> client down time, and they aren't thrilled with that at this
> point.
Understood, though trying various <Resource> configurations is
certainly resulting in downtime, no?
>>> Off topic, but relevant.... it sure would be nice if the xml
>>> parser for <resource> tags and other config items would flag
>>> with errors any parameter misspellings such as using the old
>>> obsolete 'removeAbandoned' instead of the new
>>> 'removeAbandonedOnBorrow', etc. or even just typo misspells.
>>
>> It does:
>>
>> Jul 02, 2015 10:17:25 AM
>> org.apache.tomcat.dbcp.dbcp2.BasicDataSourceFactory get
>> ObjectInstance WARNING: Name = diagnosis Property maxActive is
>> not used in DBCP2, use maxTotal instead. maxTotal default value
>> is 8. You have set value of "1" for "maxActive" property, which
>> is being ignored.
>>
>> ...
>>
>> WARNING: Name = diagnosis Property removeAbandoned is not used
>> in DBCP2, use one or both of removeAbandonedOnBorrow or
>> removeAbandonedOnMaintenance instead. Both have default value set
>> to false. You have set value of "true" for "removeAbandoned"
>> property, which is being ignored.
>>
>>
>> (I have a context.xml that I want to use between both Tomcat 7
>> and Tomcat 8, so I actually have both spellings of those
>> attributes in my XML.)
>
> This is really strange. Are the above log entries in stdout?
Yes, so they typically end up in catalina.out. Those logs are from my
dev system's catalina.out from a recent restart.
> I had 'removeAbandoned' in my resource statements until yesterday.
> But I am not seeing anything in stdout like the statement above.
> And after changing removeAbandoned to the correct spelling, I'm
> also not seeing any abandoned log entries either.
Hmm.
> Is there some sort of 'log level' that may be throttling back my
> log entries? If my code has leaks, and I'm running out of
> connections but not seeing any abandoned entries in the log, that
> should be a concern, right?
Possibly.
Perhaps you are deploying your application more times than you
thought, and are making many more connections than you expected.
>> What does your <Resource> element actually look like? (Remember
>> to remove sensitive information such as passwords.)
>
> <Resource name="jdbc/wgbe" testOnBorrow="true"
> validationQuery="SELECT 1"
validationQuery might not be necessary anymore. I think DBCP2 does a
lightweight connection test instead issuing an actual query, unless
you override it with a validationQuery.
Also, for Connector/J, you can use "/* ping */SELECT 1" and the driver
itself will perform the lightweight connection test instead of
actually issuing a query. Much faster than going through the SQL
parser, etc.
> auth="Container" type="javax.sql.DataSource" maxTotal="300"
That's a LOT of connections IMO. What is your total connection limit?
Is it per-user?
> maxIdle="30" maxWaitMillis="10000" removeAbandonedOnBorrow="true"
> removeAbandonedOnMaintenance="true" removeAbandonedTimeout="30"
> logAbandoned="true" username="xxxxxxx" password="xxxxxxxxx"
> driverClassName="org.gjt.mm.mysql.Driver"
> url="jdbc:mysql://localhost:3306/wgbe?autoReconnect=true" />
Do you have the 'manager' web application installed? Can you check to
see what applications are actually deployed? If you have 2 copies
deployed, you might see 600 connections open.
Also, do you have any stray context.xml files hanging around in
conf/*/* that you weren't expecting? If you are making changes to your
configuration and they don't appear to be having any effect, it's
possible that your new configuration isn't being applied because
another config is masking it.
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
iQIcBAEBCAAGBQJVuRgxAAoJEBzwKT+lPKRY/2YQALiweWD9Le3Ip3LeyFL/c43d
ngd50y3iqYcPGbA33IOAy5ThDmcrRDiSbs5cMG4hhAo3rFIU8QmrCJdSFrhLIXu4
K0ZcZPtlmPrEL0CUUE2D6HIX4mg0FMiNB7c0N4XjB6iicd16VGEMBVs4Ck7Sr0DM
A2zTFEYhH/YXlWoOIkTE8gduNfHNNM7y6aaspIgRy9gqAMRFvD4HZaXONCrXPyKD
hh2o8dInaSZY/jv3HrGcZd0SG9LzggKUynN195SOhc9cy6tsvOeIRFiOZWKckYSz
NpKg77+2q9SYFVM5GSKPdrFlzbcjavmsoAP3EvCtWV5HvFE7MHo/ajvI/rSb47Uy
3jYD2b7S9NZdvPu/d8EPlqL29gbUhK2a1ISIY0S3Dgyi/Junzps/sztmOfgib71X
IRDa3KH6gww4IwOX9K8YmMfHi9On06M2NF9NOVYiLqEN4CkHPAQ8YhzpnQz7/LSg
DsxLQ9D5lDdaBjE3kvL17kd9h969SgM2SJKHfHay2xOTYjOHQ9vHY1EjF/C+3smd
BzChYmUmjy0CBWjSCfYxJTjShNkLvyRyenAE+5WAFZhpOGTGW2DxSCXf9zQ2rqlv
vSk7xVf6AMrZXQEZ376MsbRXGlUipZ9ZlufXLj9Pdqmcad7WoLHGaKNNwIyJvdWy
AVhayMcrSTj5Y/xBeSXQ
=xsp7
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: "Too Many Connections" exceptions after moving to Tomcat 8
Posted by Jerry Malcolm <te...@malcolms.com>.
On 7/29/2015 10:11 AM, Christopher Schultz wrote:
> Any reason not to upgrade to 8.0.24? 8.0.15 is nearly a year out of
> date and there have been a bunch of performance, security, and
> stability fixes made. It shouldn't have any effect on your current
> problem, but upgrading would be worth your time.
Quick history... we migrated to a new box in January. I installed the
latest TC8 at the time. I did not read the migration notes closely, and
didn't change any of the parameter names. As expected things went
downhill pretty quickly. I quickly installed TC 7 and moved back to it
to keep the boss happy until I figured out why 8 was crashing. I found
the info about parameter changes. But by then, client was fine with
TC7, so I just let it slide until last week when I uploaded some new
code and hit that EL issue with ${...} that I discussed in another
thread. So, a good time to move back to 8 which fixed the ${...}
problem. But now I'm back to connection problems.... So, I'm on 8.0.15
simply because it was already installed. I guess I could move up to the
latest 8. But it'll require some client down time, and they aren't
thrilled with that at this point.
>>>> We've been on 7 for a year or two and have not had any major
>>>> issues. I read through the 7-->8 migration instructions. As
>>>> far as I can tell, I have renamed all of the resource config
>>>> parameters that changed (maxTotal, maxWaitMillis,
>>>> removedAbandonedOnBorrow).
>>> There are two pool implementations in Tomcat 7 as well as in
>>> Tomcat 8. Which one are you using? (Your factory = ?)
>> I looked at my <resource> tags, and I'm not specifying a factory
>> attribute at all.
> Then you are using the default commons-DBCP-based connection pool. The
> configuration is different for each one, so it's important we know
> which one you have.
>
>> What has the default been in TC7 and TC8 without a factory being
>> specified?
> commons-DBCP for both, though TC7 used DBCP 1.2 and TC 8 uses DBCP 2.
> It's possible you have found a bug in DBCP2 but at this point, it's
> unlikely since nobody else is really having this kind of problem.
>
>> Could leaving this off actually be the cause of running out of
>> connections?
> No, it just chooses the connection pool factory.
>
>>>> But after running for a couple of hours, I start getting "Too
>>>> Many Connections" exceptions when connecting to the db.
>>>> Granted, there's a chance I have some connection leaks in the
>>>> code, and I'll look into those. But this hasn't been an issue
>>>> on 7 for the past year or two. And shouldn't removeAbandoned
>>>> clean those up? I've got logAbandoned set to 'true', and
>>>> nothing is showing up in the logs. Is this exception referring
>>>> to running out of available connections in the pool?
>>> Describe "this exception" in more detail.
>>>
>>> What component logs that message? (class, method name that
>>> accompany the log message) Stack trace?
>> Stack trace at the very bottom of this note
>>>> Is there something else I need to change in the configuration
>>>> to move to Tomcat 8 that I've overlooked?
>>> Read official documentation for Apache Commons DBCP pool
>>> http://commons.apache.org/proper/commons-dbcp/configuration.html
>>>
>>> Maybe you will notice some discrepancies with your
>>> configuration.
>> I'll keep looking. But I'm very curious that even though I might
>> have connection leaks, what is preventing Tomcat from abandoning
>> them and returning them to the pool as I believe I have it
>> configured to do?
> Do you have a test environment you can instrument? I highly recommend
> setting maxTotal="1" (or maxActive="1" for DBCP 1.2 for Tomcat 7) and
> using whatever spelling of logAbandoned="true" for your DBCP version.
> That will find leaks pretty quickly, especially the ones that can
> cause deadlocks in production if you aren't careful.
>
> http://blog.christopherschultz.net/index.php/2009/03/16/properly-handlin
> g-pooled-jdbc-connections/
I definitely want to find/address any leaks. But the fact that I had no
problems in TC7 and do have problems in TC8 leads me to believe I still
have some sort of configuration problem in TC8. I would like to
find/fix all config problems first before I start digging into the code
to locate/correct connection leak problems.
>
>> I've got logAbandoned set to true. But I'm not seeing a single
>> entry in the logs about connections being abandoned.
> What's the abandoned timeout?
>
>> Off topic, but relevant.... it sure would be nice if the xml parser
>> for <resource> tags and other config items would flag with errors
>> any parameter misspellings such as using the old obsolete
>> 'removeAbandoned' instead of the new 'removeAbandonedOnBorrow',
>> etc. or even just typo misspells.
> It does:
>
> Jul 02, 2015 10:17:25 AM
> org.apache.tomcat.dbcp.dbcp2.BasicDataSourceFactory get
> ObjectInstance
> WARNING: Name = diagnosis Property maxActive is not used in DBCP2, use
> maxTotal instead. maxTotal default value is 8. You have set value of
> "1" for "maxActive" property, which is being ignored.
>
> ...
>
> WARNING: Name = diagnosis Property removeAbandoned is not used in
> DBCP2, use one or both of removeAbandonedOnBorrow or
> removeAbandonedOnMaintenance instead. Both have default value set to
> false. You have set value of "true" for "removeAbandoned" property,
> which is being ignored.
>
>
> (I have a context.xml that I want to use between both Tomcat 7 and
> Tomcat 8, so I actually have both spellings of those attributes in my
> XML.)
This is really strange. Are the above log entries in stdout? I had
'removeAbandoned' in my resource statements until yesterday. But I am
not seeing anything in stdout like the statement above. And after
changing removeAbandoned to the correct spelling, I'm also not seeing
any abandoned log entries either. Is there some sort of 'log level'
that may be throttling back my log entries? If my code has leaks, and
I'm running out of connections but not seeing any abandoned entries in
the log, that should be a concern, right?
>
> What does your <Resource> element actually look like? (Remember to
> remove sensitive information such as passwords.)
<Resource name="jdbc/wgbe"
testOnBorrow="true"
validationQuery="SELECT 1"
auth="Container"
type="javax.sql.DataSource"
maxTotal="300"
maxIdle="30"
maxWaitMillis="10000"
removeAbandonedOnBorrow="true"
removeAbandonedOnMaintenance="true"
removeAbandonedTimeout="30"
logAbandoned="true"
username="xxxxxxx"
password="xxxxxxxxx"
driverClassName="org.gjt.mm.mysql.Driver"
url="jdbc:mysql://localhost:3306/wgbe?autoReconnect=true" />
>
> - -chris
> -
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: "Too Many Connections" exceptions after moving to Tomcat 8
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Jerry,
On 7/29/15 9:47 AM, Jerry Malcolm wrote:
> On 7/29/2015 3:49 AM, Konstantin Kolinko wrote:
>> 2015-07-29 8:38 GMT+03:00 Jerry Malcolm
>> <te...@malcolms.com>:
>>> I am trying to move my client to Tomcat 8 from 7.
>> Tomcat version = ?
> version 8.0.15
Any reason not to upgrade to 8.0.24? 8.0.15 is nearly a year out of
date and there have been a bunch of performance, security, and
stability fixes made. It shouldn't have any effect on your current
problem, but upgrading would be worth your time.
>>> We've been on 7 for a year or two and have not had any major
>>> issues. I read through the 7-->8 migration instructions. As
>>> far as I can tell, I have renamed all of the resource config
>>> parameters that changed (maxTotal, maxWaitMillis,
>>> removedAbandonedOnBorrow).
>> There are two pool implementations in Tomcat 7 as well as in
>> Tomcat 8. Which one are you using? (Your factory = ?)
> I looked at my <resource> tags, and I'm not specifying a factory
> attribute at all.
Then you are using the default commons-DBCP-based connection pool. The
configuration is different for each one, so it's important we know
which one you have.
> What has the default been in TC7 and TC8 without a factory being
> specified?
commons-DBCP for both, though TC7 used DBCP 1.2 and TC 8 uses DBCP 2.
It's possible you have found a bug in DBCP2 but at this point, it's
unlikely since nobody else is really having this kind of problem.
> Could leaving this off actually be the cause of running out of
> connections?
No, it just chooses the connection pool factory.
>>> But after running for a couple of hours, I start getting "Too
>>> Many Connections" exceptions when connecting to the db.
>>> Granted, there's a chance I have some connection leaks in the
>>> code, and I'll look into those. But this hasn't been an issue
>>> on 7 for the past year or two. And shouldn't removeAbandoned
>>> clean those up? I've got logAbandoned set to 'true', and
>>> nothing is showing up in the logs. Is this exception referring
>>> to running out of available connections in the pool?
>> Describe "this exception" in more detail.
>>
>> What component logs that message? (class, method name that
>> accompany the log message) Stack trace?
> Stack trace at the very bottom of this note
>>> Is there something else I need to change in the configuration
>>> to move to Tomcat 8 that I've overlooked?
>>
>> Read official documentation for Apache Commons DBCP pool
>> http://commons.apache.org/proper/commons-dbcp/configuration.html
>>
>> Maybe you will notice some discrepancies with your
>> configuration.
>
> I'll keep looking. But I'm very curious that even though I might
> have connection leaks, what is preventing Tomcat from abandoning
> them and returning them to the pool as I believe I have it
> configured to do?
Do you have a test environment you can instrument? I highly recommend
setting maxTotal="1" (or maxActive="1" for DBCP 1.2 for Tomcat 7) and
using whatever spelling of logAbandoned="true" for your DBCP version.
That will find leaks pretty quickly, especially the ones that can
cause deadlocks in production if you aren't careful.
http://blog.christopherschultz.net/index.php/2009/03/16/properly-handlin
g-pooled-jdbc-connections/
> I've got logAbandoned set to true. But I'm not seeing a single
> entry in the logs about connections being abandoned.
What's the abandoned timeout?
> Off topic, but relevant.... it sure would be nice if the xml parser
> for <resource> tags and other config items would flag with errors
> any parameter misspellings such as using the old obsolete
> 'removeAbandoned' instead of the new 'removeAbandonedOnBorrow',
> etc. or even just typo misspells.
It does:
Jul 02, 2015 10:17:25 AM
org.apache.tomcat.dbcp.dbcp2.BasicDataSourceFactory get
ObjectInstance
WARNING: Name = diagnosis Property maxActive is not used in DBCP2, use
maxTotal instead. maxTotal default value is 8. You have set value of
"1" for "maxActive" property, which is being ignored.
...
WARNING: Name = diagnosis Property removeAbandoned is not used in
DBCP2, use one or both of removeAbandonedOnBorrow or
removeAbandonedOnMaintenance instead. Both have default value set to
false. You have set value of "true" for "removeAbandoned" property,
which is being ignored.
(I have a context.xml that I want to use between both Tomcat 7 and
Tomcat 8, so I actually have both spellings of those attributes in my
XML.)
What does your <Resource> element actually look like? (Remember to
remove sensitive information such as passwords.)
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
iQIcBAEBCAAGBQJVuO08AAoJEBzwKT+lPKRYlNgQAI/MQUBV45kvHxv8Ma8Kj8EV
xQ3njp1FoCXmdqzfSptrJ+eQh7EkJZ4dJ5sqvTysfl71JgetgwdleD3muaJfW0ES
xmQysVfTzaj88bj18P0OE8vps6T/4+V/mAB/qqttsdvviY9pnFzMzKUq2PQNaoIn
H0L2SW6frseKVGSh8hJ63JvC42d1HGjPpLy4Mmx19/J3c+TSckj7ywyKMeTr90yd
tm1xlmX+EW2DFebmfJ+ywCgugt2um+Zr9G5c9pmmKyeVGAvhDE5+iygFjBWkwvhj
FXL25vmOxa3bGnuakEC5LAr7oet11IlYeN2/CARsRKeNXGEJTfC1AZ+GK96quFxS
DD5gakEn2m9WFQNW+kvZRPG/7Jr7b2zVoz/H0H4ahzDWv/jyOf2YSQxfb23+curh
V3sWt8LOnJNX0hetoEvnkLn+nWIxKpTFSVmT0dc9PbRRcKb0vZmOzcgPqOBeQa/c
dARLaaVUXNIciW+4W/V5N8THVTZ75Lk2E42r7A8eSvUssgsEVtDhv3tZKZCKV5B4
ry977SiEoDJICebADCQArtpQGaGh6oPk3Xl2qeAXahvoiQzYhIbN92vfJOVqOg4H
Z0vG28icUv7kLWvDV3gv1DeZ6wnjIjOoZ9icBl8+AdsNQNGqRDanTaBcQGwsxHbV
F/3q9SdAkwxjqIpGxD7O
=Tyk6
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: "Too Many Connections" exceptions after moving to Tomcat 8
Posted by Jerry Malcolm <te...@malcolms.com>.
On 7/29/2015 3:49 AM, Konstantin Kolinko wrote:
> 2015-07-29 8:38 GMT+03:00 Jerry Malcolm <te...@malcolms.com>:
>> I am trying to move my client to Tomcat 8 from 7.
> Tomcat version = ?
version 8.0.15
>
>> We've been on 7 for a
>> year or two and have not had any major issues. I read through the 7-->8
>> migration instructions. As far as I can tell, I have renamed all of the
>> resource config parameters that changed (maxTotal, maxWaitMillis,
>> removedAbandonedOnBorrow).
> There are two pool implementations in Tomcat 7 as well as in Tomcat 8.
> Which one are you using? (Your factory = ?)
I looked at my <resource> tags, and I'm not specifying a factory
attribute at all. The doc says: "...with the only difference being that
you have to specify the |factory| attribute and set the value to
|org.apache.tomcat.jdbc.pool.DataSourceFactory" Is this new in TC 8 or
have I just been misconfigured for years? I assume from the doc that I
need to go in and add that to all resource tags, correct?
What has the default been in TC7 and TC8 without a factory being
specified? Could leaving this off actually be the cause of running out
of connections?
|
>
>> But after running for a couple of hours, I start
>> getting "Too Many Connections" exceptions when connecting to the db.
>> Granted, there's a chance I have some connection leaks in the code, and I'll
>> look into those. But this hasn't been an issue on 7 for the past year or
>> two. And shouldn't removeAbandoned clean those up? I've got logAbandoned
>> set to 'true', and nothing is showing up in the logs. Is this exception
>> referring to running out of available connections in the pool?
> Describe "this exception" in more detail.
>
> What component logs that message? (class, method name that accompany
> the log message)
> Stack trace?
Stack trace at the very bottom of this note
>> Is there something else I need to change in the configuration to move to
>> Tomcat 8 that I've overlooked?
>
> Read official documentation for Apache Commons DBCP pool
> http://commons.apache.org/proper/commons-dbcp/configuration.html
>
> Maybe you will notice some discrepancies with your configuration.
I'll keep looking. But I'm very curious that even though I might have
connection leaks, what is preventing Tomcat from abandoning them and
returning them to the pool as I believe I have it configured to do?
I've got logAbandoned set to true. But I'm not seeing a single entry in
the logs about connections being abandoned.
Off topic, but relevant.... it sure would be nice if the xml parser for
<resource> tags and other config items would flag with errors any
parameter misspellings such as using the old obsolete 'removeAbandoned'
instead of the new 'removeAbandonedOnBorrow', etc. or even just typo
misspells.
>
>
> Best regards,
> Konstantin Kolinko
>
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
Could not create connection to database server. Attempted reconnect 3
times. Giving up.
at
sun.reflect.GeneratedConstructorAccessor65.newInstance(Unknown Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:377)
at com.mysql.jdbc.Util.getInstance(Util.java:360)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:935)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:924)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:870)
at
com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2144)
at
com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2069)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:790)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:44)
at
sun.reflect.GeneratedConstructorAccessor53.newInstance(Unknown Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:377)
at
com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:395)
at
com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:325)
at
org.apache.tomcat.dbcp.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
at
org.apache.tomcat.dbcp.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:204)
at
org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:860)
at
org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:432)
at
org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:360)
at
org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:118)
at
org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1412)
at jwm.servletdb.ServletDBData.getInstance(ServletDBData.java:117)
at jwm.servletdb.ServletDBData.getInstance(ServletDBData.java:101)
at jwm.wgbe.PageRouterFilter.doFilter(PageRouterFilter.java:151)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at
org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:610)
at
org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:291)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:537)
at
org.apache.coyote.ajp.AbstractAjpProcessor.process(AbstractAjpProcessor.java:831)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:658)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1556)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1513)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Unknown Source)
Caused by:
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
Data source rejected establishment of connection, message from server:
"Too many connections"
at
sun.reflect.GeneratedConstructorAccessor65.newInstance(Unknown Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:377)
at com.mysql.jdbc.Util.getInstance(Util.java:360)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:935)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:924)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1037)
at
com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2234)
at
com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2085)
... 38 more
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: "Too Many Connections" exceptions after moving to Tomcat 8
Posted by Konstantin Kolinko <kn...@gmail.com>.
2015-07-29 8:38 GMT+03:00 Jerry Malcolm <te...@malcolms.com>:
> I am trying to move my client to Tomcat 8 from 7.
Tomcat version = ?
> We've been on 7 for a
> year or two and have not had any major issues. I read through the 7-->8
> migration instructions. As far as I can tell, I have renamed all of the
> resource config parameters that changed (maxTotal, maxWaitMillis,
> removedAbandonedOnBorrow).
There are two pool implementations in Tomcat 7 as well as in Tomcat 8.
Which one are you using? (Your factory = ?)
> But after running for a couple of hours, I start
> getting "Too Many Connections" exceptions when connecting to the db.
> Granted, there's a chance I have some connection leaks in the code, and I'll
> look into those. But this hasn't been an issue on 7 for the past year or
> two. And shouldn't removeAbandoned clean those up? I've got logAbandoned
> set to 'true', and nothing is showing up in the logs. Is this exception
> referring to running out of available connections in the pool?
Describe "this exception" in more detail.
What component logs that message? (class, method name that accompany
the log message)
Stack trace?
> Is there something else I need to change in the configuration to move to
> Tomcat 8 that I've overlooked?
Read official documentation for Apache Commons DBCP pool
http://commons.apache.org/proper/commons-dbcp/configuration.html
Maybe you will notice some discrepancies with your configuration.
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org