You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Yuval Schwartz <yu...@gmail.com> on 2016/09/12 10:54:11 UTC
SQLExceptions after stopping Tomcat
Hello,
Tomcat: 8.0.22
Java: jdk1.8.0_05
Server: Amazon Linux
My web application looses access to the database after about 24hrs of
running (probably dependent on amount of users, not time).
No messages/exceptions are printed to the console or log files.
When I stop tomcat, then I get a few "INFO" message
("org.apache.catalina.core.StandardWrapper.unload Waiting for [number]
instance(s) to be deallocated for Servlet")
followed by a long list of all the SQLExceptions ("cannot get a connection,
general error") and Warnings
("org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads
The web application is still processing a request that has yet to
finish...").
I think the Servlet specified in the INFO messages, the stack trace of the
SQLExceptions and the stack trace of the Warnings is just specific to
whichever Servlet was called and not to which Servlet/method is at the root
of what is causing this issue.
I assume this is something in my application. I have looked over changes
that I made to the code recently and haven't found anything (all
connections, Statements, ResultSets are closed after being used).
The beginning of the stack trace that appears after the warning:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
org.apache.tomcat.dbcp.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:582)
org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:439)
org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:360)
org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:118)
org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1412)
The beginning of the stack trace that appears after the SQLException:
at
org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:130)
at
org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1412)
at data.ConnectionPool.getConnection(ConnectionPool.java:64)
...
and then:
Caused by: java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
at
org.apache.tomcat.dbcp.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:582)
at
org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:439)
at
org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:360)
at
org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:118)
Any ideas as to what is going on?
I took a thread dump, but I'm not too good at analyzing those. If anyone
has any recommendations as to what to look out for in the thread dump I'd
be interested in hearing it.
Thank you.
Re: SQLExceptions after stopping Tomcat
Posted by Mark Thomas <ma...@apache.org>.
On 15/09/2016 09:30, Yuval Schwartz wrote:
<snip/>
> Questions:
> 1) What order does a thread dump print in? That is, I'd like to know what
> the first thread was in my thread dump (temporally) so that I might know
> which thread caused the actual issue. Is this the beginning of the thread
> dump or the end?
Neither. There is no way (from the thread dump) to determine execution
order amongst the threads.
> 2) The thread dump was so long that I can't see the beginning of it. Does
> anyone know how to get the thread dump in bits and pieces so the beginning
> doesn't get cut off when it's long?
It should be written to standard out which in normally redirected to a file.
> Possible causes:
> Is it possible that someone is acting maliciously and intentionally hitting
> certain pages many times to cause the pool to lock up?
The access logs will tell you if that is the case.
Mark
> I think this because if I just take a thread dump when my application is
> functioning, then it doesn't show anything suspicious.
> This leads me to believe that the problem is not cumulative but rather
> happens all at once.
> Then when I take a look at the problematic thread dump and see so many
> stack traces that access my database, it leads me to think that they are
> accessed all at once at the time that the pool locks up.
>
> Any thoughts?
>
> Thanks a lot.
> _
>
>
> On Mon, Sep 12, 2016 at 9:37 PM, Mark Thomas <ma...@apache.org> wrote:
>
>> On 12/09/2016 19:02, Yuval Schwartz wrote:
>>> Hey Mark, thanks a lot.
>>>
>>> On Mon, Sep 12, 2016 at 4:42 PM, Mark Thomas <ma...@apache.org> wrote:
>>>
>>>> On 12/09/2016 11:54, Yuval Schwartz wrote:
>>
>> <snip/>
>>
>>>> It might also be a bug in the connection pool that has been fixed.
>>>> Upgrading to the latest 8.0.x (or better still the latest 8.5.x) should
>>>> address that.
>>>>
>>>
>>> I'll look for a bug report on this (although I haven't found anything as
>> of
>>> yet).
>>
>> There was one around XA connections that could result in a leak. The
>> others, you'd need to dig into the DBCP change log and Jira for.
>>
>>> I wouldn't mind upgrading but do you think this could be a bug? I've been
>>> running my application with this setup for about 8 months; the problem
>> only
>>> started in the last week.
>>
>> That begs the question what changed a week ago?
>>
>> Mark
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: SQLExceptions after stopping Tomcat
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Yuval,
On 9/19/16 2:55 AM, Yuval Schwartz wrote:
> Hello Chris,
>
> On Sat, Sep 17, 2016 at 7:40 PM, Christopher Schultz <
> chris@christopherschultz.net> wrote:
>
> Yuval,
>
> On 9/16/16 2:33 PM, Yuval Schwartz wrote:
>>>> Hey Chris,
>>>>
>>>> You have a line in your link that reads:
>>>>
>>>> // if you want to issue multiple queries and/or // work with
>>>> multiple result sets, either declare more Statement // and
>>>> ResultSet locals above and duplicate all cleanup logic for
>>>> them, // or make sure you close each object along the way
>>>> before you // try to re-use the reference
>>>>
>>>> I have the following code in one of my data accessing
>>>> methods:
>>>>
>>>> [same ps and rs instantiating as you]
>>>>
>>>> try { ps = connection.prepareStatement(query1); rs =
>>>> ps.executeQuery();
>>>>
>>>> ...[process rs. (I dont close the ps or the rs at this
>>>> point)]
>>>>
>>>> ps = connection.prepareStatement(query2); rs =
>>>> ps.executeQuery(); .... }
>>>>
>>>> [catch and finally blocks as you specify]
>>>>
>>>> As you can see I simply reuse the ps and rs variables,
>>>> without closing them beforehand. Is this a problem?
>
> That depends upon your database driver. I seem to recall Oracle
> specifically being very unforgiving when it comes to failing to
> close resources. I would add explicit "close" calls to every object
> you are expecting to be finished with.
>
>>>> The reason this never raised a flag before is because I have
>>>> another database accessing method that executes two queries
>>>> in the same fashion and it never caused any issues.
>
> Are there any other factors involved? Maybe that other accessing
> method doesn't get called very often? Is it in a separate process
> where cleanup happens when the process completes? Do you have some
> other kind of pooling mechanism that will auto-close resources?
>
> I think it's always a good idea to follow the API's
> recommendations about closing resources when you are done with
> them. In the worst case, the close() method is a no-op and it will
> eventually be JIT'd away. In the best case, you get clean resource
> management.
>
>
>> Thanks, I found information about this in forums and made the
>> change. I'll also look for it in formal documentation when I have
>> some more time. Thanks for your help, this was most likely the
>> issue.
The best advice I can give you is to run your connection pool with
exactly 1 connection in the pool in development. You'll find all kinds
of problems with your app very quickly if it hasn't been written with
careful attention to resource-management.
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJX4EfvAAoJEBzwKT+lPKRYygIQAIt6kVOIUuJ0f9/e5mUyRFCG
G0ZczqsZ5BC68ebL6pB/LzxcDdxKVii6mnBjG0MuAXsiy8tnJXl8nlP6WFfmf8+L
0wIws3sSjLzW/VXlLnHsW0d4SUC88qgZh/MkeZvO63uL+hyw6TG3sbClaS1kUh0O
HI1KIIoCOQbr/pcS6Uu4xQ+bUuiWMAOpq+nVSFSUU04nRmcyAtemDm539RRq1GBY
pyfgE9P3IZix5ZvjAN/r0WVV2Mwmuk45YDKSoSjeOyB3SJTWHxIj3z0aod0NJQCW
XU+OVdozpAXjyH8s7ydlDYSHFtsXsOIy66xei0DEV6zKqvsBQUBha00ew6cYjYqx
2dMUoKY6wNwR02AbwzmVP5Yvzlpv8q/VKMBwAoTHYYpRRdJVIlPPDVll0ldW0wKO
n/PbpcWtZLDeOFDHh3I1mR1eRg0h8iQKfDNeeIIRxV8tFU1V3xG+B2CiWp1kGRy7
Gnkup9kCu8RnnPBpvhDW/Z7EPt78qgUrhL2gnFVqUvRAWXkPmnwPUrHs4kOdPtg7
ZkyWOZlu7cGXA6kV8hqY9OkmDUYjVrNAbLcWc917twVUe7NM4KnY3LOmclPbV0Uq
dxxaSgPjgVlB23dmCp43OyJkbXwlZR5QmtI2JvSeNBZNINZTs7OgbjSYZ8UqCXTG
TnRKvI5crM9XNp8B5yTh
=Ueel
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: SQLExceptions after stopping Tomcat
Posted by Yuval Schwartz <yu...@gmail.com>.
Hello Chris,
On Sat, Sep 17, 2016 at 7:40 PM, Christopher Schultz <
chris@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Yuval,
>
> On 9/16/16 2:33 PM, Yuval Schwartz wrote:
> > Hey Chris,
> >
> > You have a line in your link that reads:
> >
> > // if you want to issue multiple queries and/or // work with
> > multiple result sets, either declare more Statement // and
> > ResultSet locals above and duplicate all cleanup logic for them, //
> > or make sure you close each object along the way before you // try
> > to re-use the reference
> >
> > I have the following code in one of my data accessing methods:
> >
> > [same ps and rs instantiating as you]
> >
> > try { ps = connection.prepareStatement(query1); rs =
> > ps.executeQuery();
> >
> > ...[process rs. (I dont close the ps or the rs at this point)]
> >
> > ps = connection.prepareStatement(query2); rs = ps.executeQuery();
> > .... }
> >
> > [catch and finally blocks as you specify]
> >
> > As you can see I simply reuse the ps and rs variables, without
> > closing them beforehand. Is this a problem?
>
> That depends upon your database driver. I seem to recall Oracle
> specifically being very unforgiving when it comes to failing to close
> resources. I would add explicit "close" calls to every object you are
> expecting to be finished with.
>
> > The reason this never raised a flag before is because I have
> > another database accessing method that executes two queries in the
> > same fashion and it never caused any issues.
>
> Are there any other factors involved? Maybe that other accessing
> method doesn't get called very often? Is it in a separate process
> where cleanup happens when the process completes? Do you have some
> other kind of pooling mechanism that will auto-close resources?
>
> I think it's always a good idea to follow the API's recommendations
> about closing resources when you are done with them. In the worst
> case, the close() method is a no-op and it will eventually be JIT'd
> away. In the best case, you get clean resource management.
>
>
Thanks, I found information about this in forums and made the change.
I'll also look for it in formal documentation when I have some more time.
Thanks for your help, this was most likely the issue.
> - -chris
>
> -----BEGIN PGP SIGNATURE-----
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIcBAEBCAAGBQJX3XIPAAoJEBzwKT+lPKRYhKsQAJo/+GhhRAQ2y6Mqkr54Ufb5
> Wz22WHvJ15er8z/LdZ8FLzGchD7WsmgBZj8v3yeEptzoxK7rhjvoxrO7w6ZTtz7x
> 6Qogj+qa+/HeUqELnr2SkQdyrMp+aal/DEXQvMvFp+VAKrEq+x0OKeMGBvzaimsf
> xt9AWHeD5YsfRyGqvW/RFDIPFcSq83VDha2owQEcvvzNaw/DRTPTi/OubHoDuFtF
> +MCkLjXRkbYThg/ljElWivEDd2NZJP5yDtwTBAsnQeydvKVpN7vzA47tQOznbxHN
> eVhM1aPxQim1Jt3++jXT+ByGHNpiHLuhKkxwVPm8ZD6WyWgFGHp7R6w9eddnMLth
> mUhzS5HPCLYOS0x3XcuheuhybEQdWNiBh4d6dQS59+rkdU3CASPWrbwFsdEPwMq1
> 0OdDJAZ436t+EP+XJjObnT8KV9wBWFzEN3ywxhjHWdus6bmJR+O6dxlbFXYHoYdX
> fOemuIsJvjb4vl0mw9nql4kKm1MsU46gAqjP89fiCaIVPXGJjGTyqLeD48HUiNNy
> YwhSCEFcj1ucQp6vG+MU+AnvQbxwnnrymjcuDWsgMehG772zWmRsg5mT3NaK4EZr
> rbPnfyZSMkNMgGMjwCZQl8GlKlrsyIgnS/5zozPiSmldqJ0dEyKhPgxIfWpAIJiG
> K0/k2szfVtoSBaje5eIG
> =+Il1
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: SQLExceptions after stopping Tomcat
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Yuval,
On 9/16/16 2:33 PM, Yuval Schwartz wrote:
> Hey Chris,
>
> You have a line in your link that reads:
>
> // if you want to issue multiple queries and/or // work with
> multiple result sets, either declare more Statement // and
> ResultSet locals above and duplicate all cleanup logic for them, //
> or make sure you close each object along the way before you // try
> to re-use the reference
>
> I have the following code in one of my data accessing methods:
>
> [same ps and rs instantiating as you]
>
> try { ps = connection.prepareStatement(query1); rs =
> ps.executeQuery();
>
> ...[process rs. (I dont close the ps or the rs at this point)]
>
> ps = connection.prepareStatement(query2); rs = ps.executeQuery();
> .... }
>
> [catch and finally blocks as you specify]
>
> As you can see I simply reuse the ps and rs variables, without
> closing them beforehand. Is this a problem?
That depends upon your database driver. I seem to recall Oracle
specifically being very unforgiving when it comes to failing to close
resources. I would add explicit "close" calls to every object you are
expecting to be finished with.
> The reason this never raised a flag before is because I have
> another database accessing method that executes two queries in the
> same fashion and it never caused any issues.
Are there any other factors involved? Maybe that other accessing
method doesn't get called very often? Is it in a separate process
where cleanup happens when the process completes? Do you have some
other kind of pooling mechanism that will auto-close resources?
I think it's always a good idea to follow the API's recommendations
about closing resources when you are done with them. In the worst
case, the close() method is a no-op and it will eventually be JIT'd
away. In the best case, you get clean resource management.
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJX3XIPAAoJEBzwKT+lPKRYhKsQAJo/+GhhRAQ2y6Mqkr54Ufb5
Wz22WHvJ15er8z/LdZ8FLzGchD7WsmgBZj8v3yeEptzoxK7rhjvoxrO7w6ZTtz7x
6Qogj+qa+/HeUqELnr2SkQdyrMp+aal/DEXQvMvFp+VAKrEq+x0OKeMGBvzaimsf
xt9AWHeD5YsfRyGqvW/RFDIPFcSq83VDha2owQEcvvzNaw/DRTPTi/OubHoDuFtF
+MCkLjXRkbYThg/ljElWivEDd2NZJP5yDtwTBAsnQeydvKVpN7vzA47tQOznbxHN
eVhM1aPxQim1Jt3++jXT+ByGHNpiHLuhKkxwVPm8ZD6WyWgFGHp7R6w9eddnMLth
mUhzS5HPCLYOS0x3XcuheuhybEQdWNiBh4d6dQS59+rkdU3CASPWrbwFsdEPwMq1
0OdDJAZ436t+EP+XJjObnT8KV9wBWFzEN3ywxhjHWdus6bmJR+O6dxlbFXYHoYdX
fOemuIsJvjb4vl0mw9nql4kKm1MsU46gAqjP89fiCaIVPXGJjGTyqLeD48HUiNNy
YwhSCEFcj1ucQp6vG+MU+AnvQbxwnnrymjcuDWsgMehG772zWmRsg5mT3NaK4EZr
rbPnfyZSMkNMgGMjwCZQl8GlKlrsyIgnS/5zozPiSmldqJ0dEyKhPgxIfWpAIJiG
K0/k2szfVtoSBaje5eIG
=+Il1
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: SQLExceptions after stopping Tomcat
Posted by Yuval Schwartz <yu...@gmail.com>.
Hey Chris,
You have a line in your link that reads:
// if you want to issue multiple queries and/or
// work with multiple result sets, either declare more Statement
// and ResultSet locals above and duplicate all cleanup logic for them,
// or make sure you close each object along the way before you
// try to re-use the reference
I have the following code in one of my data accessing methods:
[same ps and rs instantiating as you]
try {
ps = connection.prepareStatement(query1);
rs = ps.executeQuery();
...[process rs. (I dont close the ps or the rs at this point)]
ps = connection.prepareStatement(query2);
rs = ps.executeQuery();
....
}
[catch and finally blocks as you specify]
As you can see I simply reuse the ps and rs variables, without closing them
beforehand. Is this a problem?
The reason this never raised a flag before is because I have another
database accessing method that executes two queries in the same fashion and
it never caused any issues.
Thanks a lot.
_
On Fri, Sep 16, 2016 at 7:18 PM, Yuval Schwartz <yu...@gmail.com>
wrote:
> Thanks Chris, I'll have a look.
>
> On Fri, Sep 16, 2016 at 6:41 PM, Christopher Schultz <
> chris@christopherschultz.net> wrote:
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA256
>>
>> Yuval,
>>
>> On 9/16/16 3:32 AM, Yuval Schwartz wrote:
>> > A few more words/questions on my issue:
>> >
>> > My Resource element used the attribute "removeAbandoned" rather
>> > than "removeAbandonedOnBorrow". I have now corrected this. Does
>> > this affect what gets logged (ie: I've had "logAbandoned" on this
>> > whole time but have not seen anything in the logs about abandoned
>> > connections, is it possible that I will now see information about
>> > abandoned threads in my logs)?
>> >
>> > "removeAbandonedOnBorrow" doesn't really solve my issue since it
>> > doesn't get to the core of the problem, how will I know to
>> > recognize when an abandoned connection has been removed so that I
>> > can correct the bad code in my application.
>>
>> My bet is on a code path that simply fails to return a connection, or
>> a code path that requests two connections and therefore deadlocks when
>> things get busy.
>>
>> Have a look at this:
>> http://blog.christopherschultz.net/index.php/2009/03/16/properly-handlin
>> g-pooled-jdbc-connections/
>> <http://blog.christopherschultz.net/index.php/2009/03/16/properly-handling-pooled-jdbc-connections/>
>>
>> It might help you track down what's going on.
>>
>> Hope that helps,
>> - -chris
>> -----BEGIN PGP SIGNATURE-----
>> Comment: GPGTools - http://gpgtools.org
>> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>>
>> iQIcBAEBCAAGBQJX3BKlAAoJEBzwKT+lPKRY3RoP/AxYmD6buD7kkQF0DTWLjDa2
>> dmJSVqAj4mtxDoIeRfbMmNAUz7Y4kVq7AmHj3coPKMukk8CU4676IYgKjqXCtB7K
>> a91cR/yC5FKlGuZwIxtG1oqIUQDvg5e5j2k1hX918nFdNMjJLLoF+7xNkNauKi3/
>> rafsVD12GI34jXZLxna2gfFO6kkLAjXkHSyTU9p2+6gIzHcTd6hqAsVlqkCPGDGT
>> ODcC2nFWxqLx+snTdJRuGiU/LVxNPDBObqqKWZ3CKQQaWIQ0a7aHXe7bGq3aWsBA
>> GJHMtWpLpwyB9dpRzbTXE/Phj+WkvifUDTVtBSin1YkrQcLhaAqwViszRKl2wXnd
>> wtwYXiYo86QsHwfXLeq1GDfH0FmnGphdpVYe0J6nnSxgNb7kSTaX+FPkOaqKQD0g
>> phs41jcMipgFyOrdDVPbt2ntHOoP3VmFUqNmHnhdqOJx0xI+gQKglB4NAY/BQGvt
>> /yRdmHx3AGVLIp7nNGNKSrUK7vy/j6tynGYYMizRlnpOSZISbmVpAeDqjn5/Q27S
>> VYSPwA8UdrmB9tKAgSpXQMirTzt+PRVVD+zna4Dbe/thDyNbP/sNQn14nskCgf9y
>> aRmhC6R1j3TkjS7rzd7LLhlw69agjWm8s0roj+AVxCOX63xpm6RgDmaOUkoG+IZS
>> BJa1vpRmY21wlq6IaW5g
>> =Qvhj
>> -----END PGP SIGNATURE-----
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
Re: SQLExceptions after stopping Tomcat
Posted by Yuval Schwartz <yu...@gmail.com>.
Thanks Chris, I'll have a look.
On Fri, Sep 16, 2016 at 6:41 PM, Christopher Schultz <
chris@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Yuval,
>
> On 9/16/16 3:32 AM, Yuval Schwartz wrote:
> > A few more words/questions on my issue:
> >
> > My Resource element used the attribute "removeAbandoned" rather
> > than "removeAbandonedOnBorrow". I have now corrected this. Does
> > this affect what gets logged (ie: I've had "logAbandoned" on this
> > whole time but have not seen anything in the logs about abandoned
> > connections, is it possible that I will now see information about
> > abandoned threads in my logs)?
> >
> > "removeAbandonedOnBorrow" doesn't really solve my issue since it
> > doesn't get to the core of the problem, how will I know to
> > recognize when an abandoned connection has been removed so that I
> > can correct the bad code in my application.
>
> My bet is on a code path that simply fails to return a connection, or
> a code path that requests two connections and therefore deadlocks when
> things get busy.
>
> Have a look at this:
> http://blog.christopherschultz.net/index.php/2009/03/16/properly-handlin
> g-pooled-jdbc-connections/
>
> It might help you track down what's going on.
>
> Hope that helps,
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIcBAEBCAAGBQJX3BKlAAoJEBzwKT+lPKRY3RoP/AxYmD6buD7kkQF0DTWLjDa2
> dmJSVqAj4mtxDoIeRfbMmNAUz7Y4kVq7AmHj3coPKMukk8CU4676IYgKjqXCtB7K
> a91cR/yC5FKlGuZwIxtG1oqIUQDvg5e5j2k1hX918nFdNMjJLLoF+7xNkNauKi3/
> rafsVD12GI34jXZLxna2gfFO6kkLAjXkHSyTU9p2+6gIzHcTd6hqAsVlqkCPGDGT
> ODcC2nFWxqLx+snTdJRuGiU/LVxNPDBObqqKWZ3CKQQaWIQ0a7aHXe7bGq3aWsBA
> GJHMtWpLpwyB9dpRzbTXE/Phj+WkvifUDTVtBSin1YkrQcLhaAqwViszRKl2wXnd
> wtwYXiYo86QsHwfXLeq1GDfH0FmnGphdpVYe0J6nnSxgNb7kSTaX+FPkOaqKQD0g
> phs41jcMipgFyOrdDVPbt2ntHOoP3VmFUqNmHnhdqOJx0xI+gQKglB4NAY/BQGvt
> /yRdmHx3AGVLIp7nNGNKSrUK7vy/j6tynGYYMizRlnpOSZISbmVpAeDqjn5/Q27S
> VYSPwA8UdrmB9tKAgSpXQMirTzt+PRVVD+zna4Dbe/thDyNbP/sNQn14nskCgf9y
> aRmhC6R1j3TkjS7rzd7LLhlw69agjWm8s0roj+AVxCOX63xpm6RgDmaOUkoG+IZS
> BJa1vpRmY21wlq6IaW5g
> =Qvhj
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: SQLExceptions after stopping Tomcat
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Yuval,
On 9/16/16 3:32 AM, Yuval Schwartz wrote:
> A few more words/questions on my issue:
>
> My Resource element used the attribute "removeAbandoned" rather
> than "removeAbandonedOnBorrow". I have now corrected this. Does
> this affect what gets logged (ie: I've had "logAbandoned" on this
> whole time but have not seen anything in the logs about abandoned
> connections, is it possible that I will now see information about
> abandoned threads in my logs)?
>
> "removeAbandonedOnBorrow" doesn't really solve my issue since it
> doesn't get to the core of the problem, how will I know to
> recognize when an abandoned connection has been removed so that I
> can correct the bad code in my application.
My bet is on a code path that simply fails to return a connection, or
a code path that requests two connections and therefore deadlocks when
things get busy.
Have a look at this:
http://blog.christopherschultz.net/index.php/2009/03/16/properly-handlin
g-pooled-jdbc-connections/
It might help you track down what's going on.
Hope that helps,
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJX3BKlAAoJEBzwKT+lPKRY3RoP/AxYmD6buD7kkQF0DTWLjDa2
dmJSVqAj4mtxDoIeRfbMmNAUz7Y4kVq7AmHj3coPKMukk8CU4676IYgKjqXCtB7K
a91cR/yC5FKlGuZwIxtG1oqIUQDvg5e5j2k1hX918nFdNMjJLLoF+7xNkNauKi3/
rafsVD12GI34jXZLxna2gfFO6kkLAjXkHSyTU9p2+6gIzHcTd6hqAsVlqkCPGDGT
ODcC2nFWxqLx+snTdJRuGiU/LVxNPDBObqqKWZ3CKQQaWIQ0a7aHXe7bGq3aWsBA
GJHMtWpLpwyB9dpRzbTXE/Phj+WkvifUDTVtBSin1YkrQcLhaAqwViszRKl2wXnd
wtwYXiYo86QsHwfXLeq1GDfH0FmnGphdpVYe0J6nnSxgNb7kSTaX+FPkOaqKQD0g
phs41jcMipgFyOrdDVPbt2ntHOoP3VmFUqNmHnhdqOJx0xI+gQKglB4NAY/BQGvt
/yRdmHx3AGVLIp7nNGNKSrUK7vy/j6tynGYYMizRlnpOSZISbmVpAeDqjn5/Q27S
VYSPwA8UdrmB9tKAgSpXQMirTzt+PRVVD+zna4Dbe/thDyNbP/sNQn14nskCgf9y
aRmhC6R1j3TkjS7rzd7LLhlw69agjWm8s0roj+AVxCOX63xpm6RgDmaOUkoG+IZS
BJa1vpRmY21wlq6IaW5g
=Qvhj
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: SQLExceptions after stopping Tomcat
Posted by Mark Thomas <ma...@apache.org>.
On 16/09/2016 08:32, Yuval Schwartz wrote:
> A few more words/questions on my issue:
>
> My Resource element used the attribute "removeAbandoned" rather than
> "removeAbandonedOnBorrow". I have now corrected this. Does this affect what
> gets logged (ie: I've had "logAbandoned" on this whole time but have not
> seen anything in the logs about abandoned connections, is it possible that
> I will now see information about abandoned threads in my logs)?
>
> "removeAbandonedOnBorrow" doesn't really solve my issue since it doesn't
> get to the core of the problem, how will I know to recognize when an
> abandoned connection has been removed so that I can correct the bad code in
> my application.
You should see information in the logs.
Mark
>
> Thanks.
>
> On Thu, Sep 15, 2016 at 11:30 AM, Yuval Schwartz <yu...@gmail.com>
> wrote:
>
>> Hello,
>>
>> I'd rather narrow down the problem a bit more before upgrading (to make
>> sure that that's the issue).
>>
>> It happened again, this time I took a thread dump of the instance that
>> went down (actually, it's only the DB connection pool that becomes
>> unresponsive but I remove the instance from my load balancer when this
>> happens).
>>
>> The thread dump shows pretty much the same things that the logs showed
>> when I shutdown tomcat: a very long list of a lot of stack traces which all
>> get stuck when trying to access the connection pool. An example from the
>> thread dump:
>>
>> "http-apr-8080-exec-805" #840 daemon prio=5 os_prio=0
>> tid=0x00007f50d00c1800 nid=0x5fbc waiting on condition [0x00007f50c62e7000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for <0x00000000ec44cf38> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.
>> java:175)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> at org.apache.tomcat.dbcp.pool2.impl.LinkedBlockingDeque.
>> takeFirst(LinkedBlockingDeque.java:582)
>> at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.
>> borrowObject(GenericObjectPool.java:439)
>> at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.
>> borrowObject(GenericObjectPool.java:360)
>> at org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(
>> PoolingDataSource.java:118)
>> at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(
>> BasicDataSource.java:1412)
>>
>> Questions:
>> 1) What order does a thread dump print in? That is, I'd like to know what
>> the first thread was in my thread dump (temporally) so that I might know
>> which thread caused the actual issue. Is this the beginning of the thread
>> dump or the end?
>>
>> 2) The thread dump was so long that I can't see the beginning of it. Does
>> anyone know how to get the thread dump in bits and pieces so the beginning
>> doesn't get cut off when it's long?
>>
>> Possible causes:
>> Is it possible that someone is acting maliciously and intentionally
>> hitting certain pages many times to cause the pool to lock up?
>> I think this because if I just take a thread dump when my application is
>> functioning, then it doesn't show anything suspicious.
>> This leads me to believe that the problem is not cumulative but rather
>> happens all at once.
>> Then when I take a look at the problematic thread dump and see so many
>> stack traces that access my database, it leads me to think that they are
>> accessed all at once at the time that the pool locks up.
>>
>> Any thoughts?
>>
>> Thanks a lot.
>> _
>>
>>
>> On Mon, Sep 12, 2016 at 9:37 PM, Mark Thomas <ma...@apache.org> wrote:
>>
>>> On 12/09/2016 19:02, Yuval Schwartz wrote:
>>>> Hey Mark, thanks a lot.
>>>>
>>>> On Mon, Sep 12, 2016 at 4:42 PM, Mark Thomas <ma...@apache.org> wrote:
>>>>
>>>>> On 12/09/2016 11:54, Yuval Schwartz wrote:
>>>
>>> <snip/>
>>>
>>>>> It might also be a bug in the connection pool that has been fixed.
>>>>> Upgrading to the latest 8.0.x (or better still the latest 8.5.x) should
>>>>> address that.
>>>>>
>>>>
>>>> I'll look for a bug report on this (although I haven't found anything
>>> as of
>>>> yet).
>>>
>>> There was one around XA connections that could result in a leak. The
>>> others, you'd need to dig into the DBCP change log and Jira for.
>>>
>>>> I wouldn't mind upgrading but do you think this could be a bug? I've
>>> been
>>>> running my application with this setup for about 8 months; the problem
>>> only
>>>> started in the last week.
>>>
>>> That begs the question what changed a week ago?
>>>
>>> Mark
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: SQLExceptions after stopping Tomcat
Posted by Yuval Schwartz <yu...@gmail.com>.
A few more words/questions on my issue:
My Resource element used the attribute "removeAbandoned" rather than
"removeAbandonedOnBorrow". I have now corrected this. Does this affect what
gets logged (ie: I've had "logAbandoned" on this whole time but have not
seen anything in the logs about abandoned connections, is it possible that
I will now see information about abandoned threads in my logs)?
"removeAbandonedOnBorrow" doesn't really solve my issue since it doesn't
get to the core of the problem, how will I know to recognize when an
abandoned connection has been removed so that I can correct the bad code in
my application.
Thanks.
On Thu, Sep 15, 2016 at 11:30 AM, Yuval Schwartz <yu...@gmail.com>
wrote:
> Hello,
>
> I'd rather narrow down the problem a bit more before upgrading (to make
> sure that that's the issue).
>
> It happened again, this time I took a thread dump of the instance that
> went down (actually, it's only the DB connection pool that becomes
> unresponsive but I remove the instance from my load balancer when this
> happens).
>
> The thread dump shows pretty much the same things that the logs showed
> when I shutdown tomcat: a very long list of a lot of stack traces which all
> get stuck when trying to access the connection pool. An example from the
> thread dump:
>
> "http-apr-8080-exec-805" #840 daemon prio=5 os_prio=0
> tid=0x00007f50d00c1800 nid=0x5fbc waiting on condition [0x00007f50c62e7000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ec44cf38> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at org.apache.tomcat.dbcp.pool2.impl.LinkedBlockingDeque.
> takeFirst(LinkedBlockingDeque.java:582)
> at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.
> borrowObject(GenericObjectPool.java:439)
> at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.
> borrowObject(GenericObjectPool.java:360)
> at org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(
> PoolingDataSource.java:118)
> at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(
> BasicDataSource.java:1412)
>
> Questions:
> 1) What order does a thread dump print in? That is, I'd like to know what
> the first thread was in my thread dump (temporally) so that I might know
> which thread caused the actual issue. Is this the beginning of the thread
> dump or the end?
>
> 2) The thread dump was so long that I can't see the beginning of it. Does
> anyone know how to get the thread dump in bits and pieces so the beginning
> doesn't get cut off when it's long?
>
> Possible causes:
> Is it possible that someone is acting maliciously and intentionally
> hitting certain pages many times to cause the pool to lock up?
> I think this because if I just take a thread dump when my application is
> functioning, then it doesn't show anything suspicious.
> This leads me to believe that the problem is not cumulative but rather
> happens all at once.
> Then when I take a look at the problematic thread dump and see so many
> stack traces that access my database, it leads me to think that they are
> accessed all at once at the time that the pool locks up.
>
> Any thoughts?
>
> Thanks a lot.
> _
>
>
> On Mon, Sep 12, 2016 at 9:37 PM, Mark Thomas <ma...@apache.org> wrote:
>
>> On 12/09/2016 19:02, Yuval Schwartz wrote:
>> > Hey Mark, thanks a lot.
>> >
>> > On Mon, Sep 12, 2016 at 4:42 PM, Mark Thomas <ma...@apache.org> wrote:
>> >
>> >> On 12/09/2016 11:54, Yuval Schwartz wrote:
>>
>> <snip/>
>>
>> >> It might also be a bug in the connection pool that has been fixed.
>> >> Upgrading to the latest 8.0.x (or better still the latest 8.5.x) should
>> >> address that.
>> >>
>> >
>> > I'll look for a bug report on this (although I haven't found anything
>> as of
>> > yet).
>>
>> There was one around XA connections that could result in a leak. The
>> others, you'd need to dig into the DBCP change log and Jira for.
>>
>> > I wouldn't mind upgrading but do you think this could be a bug? I've
>> been
>> > running my application with this setup for about 8 months; the problem
>> only
>> > started in the last week.
>>
>> That begs the question what changed a week ago?
>>
>> Mark
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
Re: SQLExceptions after stopping Tomcat
Posted by Yuval Schwartz <yu...@gmail.com>.
Hello,
I'd rather narrow down the problem a bit more before upgrading (to make
sure that that's the issue).
It happened again, this time I took a thread dump of the instance that went
down (actually, it's only the DB connection pool that becomes unresponsive
but I remove the instance from my load balancer when this happens).
The thread dump shows pretty much the same things that the logs showed when
I shutdown tomcat: a very long list of a lot of stack traces which all get
stuck when trying to access the connection pool. An example from the thread
dump:
"http-apr-8080-exec-805" #840 daemon prio=5 os_prio=0
tid=0x00007f50d00c1800 nid=0x5fbc waiting on condition [0x00007f50c62e7000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ec44cf38> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
org.apache.tomcat.dbcp.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:582)
at
org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:439)
at
org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:360)
at
org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:118)
at
org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1412)
Questions:
1) What order does a thread dump print in? That is, I'd like to know what
the first thread was in my thread dump (temporally) so that I might know
which thread caused the actual issue. Is this the beginning of the thread
dump or the end?
2) The thread dump was so long that I can't see the beginning of it. Does
anyone know how to get the thread dump in bits and pieces so the beginning
doesn't get cut off when it's long?
Possible causes:
Is it possible that someone is acting maliciously and intentionally hitting
certain pages many times to cause the pool to lock up?
I think this because if I just take a thread dump when my application is
functioning, then it doesn't show anything suspicious.
This leads me to believe that the problem is not cumulative but rather
happens all at once.
Then when I take a look at the problematic thread dump and see so many
stack traces that access my database, it leads me to think that they are
accessed all at once at the time that the pool locks up.
Any thoughts?
Thanks a lot.
_
On Mon, Sep 12, 2016 at 9:37 PM, Mark Thomas <ma...@apache.org> wrote:
> On 12/09/2016 19:02, Yuval Schwartz wrote:
> > Hey Mark, thanks a lot.
> >
> > On Mon, Sep 12, 2016 at 4:42 PM, Mark Thomas <ma...@apache.org> wrote:
> >
> >> On 12/09/2016 11:54, Yuval Schwartz wrote:
>
> <snip/>
>
> >> It might also be a bug in the connection pool that has been fixed.
> >> Upgrading to the latest 8.0.x (or better still the latest 8.5.x) should
> >> address that.
> >>
> >
> > I'll look for a bug report on this (although I haven't found anything as
> of
> > yet).
>
> There was one around XA connections that could result in a leak. The
> others, you'd need to dig into the DBCP change log and Jira for.
>
> > I wouldn't mind upgrading but do you think this could be a bug? I've been
> > running my application with this setup for about 8 months; the problem
> only
> > started in the last week.
>
> That begs the question what changed a week ago?
>
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: SQLExceptions after stopping Tomcat
Posted by Mark Thomas <ma...@apache.org>.
On 12/09/2016 19:02, Yuval Schwartz wrote:
> Hey Mark, thanks a lot.
>
> On Mon, Sep 12, 2016 at 4:42 PM, Mark Thomas <ma...@apache.org> wrote:
>
>> On 12/09/2016 11:54, Yuval Schwartz wrote:
<snip/>
>> It might also be a bug in the connection pool that has been fixed.
>> Upgrading to the latest 8.0.x (or better still the latest 8.5.x) should
>> address that.
>>
>
> I'll look for a bug report on this (although I haven't found anything as of
> yet).
There was one around XA connections that could result in a leak. The
others, you'd need to dig into the DBCP change log and Jira for.
> I wouldn't mind upgrading but do you think this could be a bug? I've been
> running my application with this setup for about 8 months; the problem only
> started in the last week.
That begs the question what changed a week ago?
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: SQLExceptions after stopping Tomcat
Posted by Yuval Schwartz <yu...@gmail.com>.
Hey Mark, thanks a lot.
On Mon, Sep 12, 2016 at 4:42 PM, Mark Thomas <ma...@apache.org> wrote:
> On 12/09/2016 11:54, Yuval Schwartz wrote:
> > Hello,
> >
> > Tomcat: 8.0.22
> > Java: jdk1.8.0_05
> > Server: Amazon Linux
> >
> > My web application looses access to the database after about 24hrs of
> > running (probably dependent on amount of users, not time).
> >
> > No messages/exceptions are printed to the console or log files.
>
> <snip/>
>
> > Any ideas as to what is going on?
>
> Some form of connection leak?
>
> For configuration options to fix the leak and track down the source see:
>
> http://tomcat.apache.org/tomcat-8.0-doc/jndi-datasource-examples-howto.
> html#Preventing_database_connection_pool_leaks
>
> You probably want to use logAbandoned="true" to find the source of any
>
I actually already have this enabled (and had it enabled when reporting the
issue to this group. But good to know that it slows things down, I will
disable in the future.
> leaks (it will slow things down so disable once you have fixed any leaks)
>
> It might also be a bug in the connection pool that has been fixed.
> Upgrading to the latest 8.0.x (or better still the latest 8.5.x) should
> address that.
>
I'll look for a bug report on this (although I haven't found anything as of
yet).
I wouldn't mind upgrading but do you think this could be a bug? I've been
running my application with this setup for about 8 months; the problem only
started in the last week.
>
> Mark
>
> > I took a thread dump, but I'm not too good at analyzing those. If anyone
> > has any recommendations as to what to look out for in the thread dump I'd
> > be interested in hearing it.
> >
> > Thank you.
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: SQLExceptions after stopping Tomcat
Posted by Mark Thomas <ma...@apache.org>.
On 12/09/2016 11:54, Yuval Schwartz wrote:
> Hello,
>
> Tomcat: 8.0.22
> Java: jdk1.8.0_05
> Server: Amazon Linux
>
> My web application looses access to the database after about 24hrs of
> running (probably dependent on amount of users, not time).
>
> No messages/exceptions are printed to the console or log files.
<snip/>
> Any ideas as to what is going on?
Some form of connection leak?
For configuration options to fix the leak and track down the source see:
http://tomcat.apache.org/tomcat-8.0-doc/jndi-datasource-examples-howto.html#Preventing_database_connection_pool_leaks
You probably want to use logAbandoned="true" to find the source of any
leaks (it will slow things down so disable once you have fixed any leaks)
It might also be a bug in the connection pool that has been fixed.
Upgrading to the latest 8.0.x (or better still the latest 8.5.x) should
address that.
Mark
> I took a thread dump, but I'm not too good at analyzing those. If anyone
> has any recommendations as to what to look out for in the thread dump I'd
> be interested in hearing it.
>
> Thank you.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org