You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user-java@ibatis.apache.org by Michael Schall <mi...@gmail.com> on 2008/07/17 17:55:15 UTC
iBATIS lock timeouts
Production Environment:
We are using a JNDI datasource in WS 6.1 (fixpack 15) to connect to a DB2
9.1 (fixpack 2) database on a separate cluster.
We recently upgraded our Java version of iBATIS after falling way behind
from 2.0.9 to the latest release 2.3.1.
The dev and test environment showed no issues with the upgrade.
When rolling out to production we started getting lock timeouts that would
bring the system down under heavy load.
We did not recreate the JNDI datasource or replace any database drivers on
the WebSphere machines or make any configuration changes within DB2 (other
than new tables/columns) during the latest release.
We first thought the default IsolationLevel had changed from "Cursor
Stability" to "Read Stability", but that is actually the default
IsolationLevel when connecting from WS to DB2 using JNDI. However, as a
troubleshooting step, we set the IsolationLevel within the JNDI datasource
to 2 (Cursor Stability), which is allowing our system to avoid the lock
timeouts.
I have looked throught the change log and nothing seems to point to the
issue we are having. Any ideas on where our problem might be?
Thanks for your time.
Mike
Re: iBATIS lock timeouts
Posted by Clinton Begin <cl...@gmail.com>.
Haha... see, even I screw it up! I reversed up sessions and transactions.
My example here, while it would work, would be kind of silly.
The most important thing is the Note section in the book. I forgot we put
that warning of future deprecation in there. :-)
If we find that iBATIS does need throttling in the future, we'd likely put
it at the top level and only have one setting.
Clinton
On Mon, Jul 21, 2008 at 8:31 AM, Hugh Ross <hh...@gmail.com> wrote:
> Thanks. Very helpful. Agree that these things are more properly the
> responsibility of "containers."
>
> Just reread the section of the SQL Maps Developer Guide that covers these
> settings. Your maxTransactions example below seems to differ from the
> documented usage.
>
> With earlier versions of iBATIS, can we set them to 0 (or something) to
> suppress their action?
>
>
> On 7/20/08, Clinton Begin <cl...@gmail.com> wrote:
>>
>> Unfortunately it was a pretty deep implementation detail. Essentially
>> there used to be 3 settings:
>>
>> maxSessions
>> maxTransactions
>> maxRequests
>>
>> A session was a user thread accessing nearly any part of the framework. A
>> transaction was a database connection and transaction. A request was an
>> executing SQL statement. So essentially you could configure something
>> like: 5 sessions, 15 transactions (avg. two per session), and 60 requests
>> (avg. four per transaction). Note that these are limits on how many of
>> these could be active at any one time, not how many can be created etc.
>>
>> A Throttle was the implementation of the limitation. In a nutshell, it
>> was the Guarded Suspension pattern that blocked with a wait until a session,
>> transaction or request became available.
>>
>> This is similar to the throttling that EJBs and Spring (?) would normally
>> do. Typical usages were to avoid hitting the database too hard, avoid
>> running out of open cursors in oracle, or to avoid opening too many
>> connections at once.
>>
>> The problems were:
>>
>> * The settings were confusing, most people didn't understand them (not
>> their fault) and therefore they were often set incorrectly.
>> * The behavior was duplicated, as the app server of container framework
>> (Spring), and even the datasource were likely throttling or limiting such
>> activity already.
>> * The throttle features hid the underlying problems which were usually
>> caused by not properly ending transactions or closing connections.
>> (try/finally)
>> * Some complained about the blocking and synchronization required to
>> implement this feature.
>>
>> So the behavior was superfluous and the errors were vague. All for
>> something that wasn't really the responsibility of iBATIS to handle. So it
>> was removed.
>>
>> The caveat is that you will now know if your application has a problem.
>>
>> * If you fail to end transactions (i.e. close connections) your
>> datasource or database will complain.
>> * If you hit your database too hard, you'll see performance degrade as
>> threads contend for resources.
>> * If you try to run too many statements, your database may complain
>> about too many open cursors.
>>
>> The advantage is that:
>>
>> * The throttling is no longer duplicated in various layers of your
>> application.
>> * Performance should be generally better once you tune some other
>> throttle or fix mismanaged connections.
>> * You'll get the proper error that will tell you what the real problem
>> is.
>>
>> If you want to throttle iBATIS yourself, you can do so fairly simply with
>> a dynamic proxy between the SqlMapClient interface and the SqlMapClientImpl
>> -- perhaps I should release something of the sort as a plug-in?
>>
>> But I highly recommend you hunt down the actual problem with your
>> application.. If you're getting deadlocks, that could either be unclosed
>> connections, poorly written stored procedures or resources that are
>> allocated out of order. The problems were always there, iBATIS was just
>> hiding them "for you".... hope that helps.
>>
>> Cheers,
>> Clinton
>>
>>
>>
>> On Sun, Jul 20, 2008 at 8:01 AM, Hugh Ross <hh...@gmail.com> wrote:
>>
>>> Clinton,
>>>
>>> Can you send us a link to more detail on this thread throttling, perhaps
>>> in wiki or release notes?
>>>
>>> Thanks...
>>>
>>> On 7/17/08, Clinton Begin <cl...@gmail.com> wrote:
>>>>
>>>> I've removed all thread throttling from iBATIS. It was causing a great
>>>> deal of confusing and people complained about performance. The thread
>>>> throttling was quite advanced and had a lot of scalability testing behind
>>>> it, it was able to hide a lot of problems like this.
>>>>
>>>> The downside is that you now have to make sure you're not throwing too
>>>> much at your database. The way you've solved it is probably the way it
>>>> should have been originally. iBATIS was hiding the problem for you, which
>>>> was sometimes a feature, and other times a bug. At least now you know
>>>> what's really going on... :-)
>>>>
>>>> That said, it's also possible that you're not closing your transactions
>>>> properly. The thread throttling in past releases also did a good job of
>>>> hiding that from you as well.. but now you'll start to see the database
>>>> complain about such errors. Again, sometimes a feature, other times a bug.
>>>>
>>>> Clinton
>>>>
>>>> On Thu, Jul 17, 2008 at 9:55 AM, Michael Schall <mi...@gmail.com>
>>>> wrote:
>>>>
>>>>> Production Environment:
>>>>> We are using a JNDI datasource in WS 6.1 (fixpack 15) to connect to a
>>>>> DB2 9.1 (fixpack 2) database on a separate cluster.
>>>>>
>>>>> We recently upgraded our Java version of iBATIS after falling way
>>>>> behind from 2.0.9 to the latest release 2.3.1.
>>>>>
>>>>> The dev and test environment showed no issues with the upgrade.
>>>>>
>>>>> When rolling out to production we started getting lock timeouts that
>>>>> would bring the system down under heavy load.
>>>>>
>>>>> We did not recreate the JNDI datasource or replace any database drivers
>>>>> on the WebSphere machines or make any configuration changes within DB2
>>>>> (other than new tables/columns) during the latest release.
>>>>>
>>>>> We first thought the default IsolationLevel had changed from "Cursor
>>>>> Stability" to "Read Stability", but that is actually the default
>>>>> IsolationLevel when connecting from WS to DB2 using JNDI. However, as a
>>>>> troubleshooting step, we set the IsolationLevel within the JNDI datasource
>>>>> to 2 (Cursor Stability), which is allowing our system to avoid the lock
>>>>> timeouts.
>>>>>
>>>>> I have looked throught the change log and nothing seems to point to the
>>>>> issue we are having. Any ideas on where our problem might be?
>>>>>
>>>>> Thanks for your time.
>>>>>
>>>>> Mike
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
Re: iBATIS lock timeouts
Posted by Hugh Ross <hh...@gmail.com>.
Thanks. Very helpful. Agree that these things are more properly the
responsibility of "containers."
Just reread the section of the SQL Maps Developer Guide that covers these
settings. Your maxTransactions example below seems to differ from the
documented usage.
With earlier versions of iBATIS, can we set them to 0 (or something) to
suppress their action?
On 7/20/08, Clinton Begin <cl...@gmail.com> wrote:
>
> Unfortunately it was a pretty deep implementation detail. Essentially
> there used to be 3 settings:
>
> maxSessions
> maxTransactions
> maxRequests
>
> A session was a user thread accessing nearly any part of the framework. A
> transaction was a database connection and transaction. A request was an
> executing SQL statement. So essentially you could configure something
> like: 5 sessions, 15 transactions (avg. two per session), and 60 requests
> (avg. four per transaction). Note that these are limits on how many of
> these could be active at any one time, not how many can be created etc.
>
> A Throttle was the implementation of the limitation. In a nutshell, it was
> the Guarded Suspension pattern that blocked with a wait until a session,
> transaction or request became available.
>
> This is similar to the throttling that EJBs and Spring (?) would normally
> do. Typical usages were to avoid hitting the database too hard, avoid
> running out of open cursors in oracle, or to avoid opening too many
> connections at once.
>
> The problems were:
>
> * The settings were confusing, most people didn't understand them (not
> their fault) and therefore they were often set incorrectly.
> * The behavior was duplicated, as the app server of container framework
> (Spring), and even the datasource were likely throttling or limiting such
> activity already.
> * The throttle features hid the underlying problems which were usually
> caused by not properly ending transactions or closing connections.
> (try/finally)
> * Some complained about the blocking and synchronization required to
> implement this feature.
>
> So the behavior was superfluous and the errors were vague. All for
> something that wasn't really the responsibility of iBATIS to handle. So it
> was removed.
>
> The caveat is that you will now know if your application has a problem.
>
> * If you fail to end transactions (i.e. close connections) your
> datasource or database will complain.
> * If you hit your database too hard, you'll see performance degrade as
> threads contend for resources.
> * If you try to run too many statements, your database may complain about
> too many open cursors.
>
> The advantage is that:
>
> * The throttling is no longer duplicated in various layers of your
> application.
> * Performance should be generally better once you tune some other
> throttle or fix mismanaged connections.
> * You'll get the proper error that will tell you what the real problem
> is.
>
> If you want to throttle iBATIS yourself, you can do so fairly simply with a
> dynamic proxy between the SqlMapClient interface and the SqlMapClientImpl --
> perhaps I should release something of the sort as a plug-in?
>
> But I highly recommend you hunt down the actual problem with your
> application.. If you're getting deadlocks, that could either be unclosed
> connections, poorly written stored procedures or resources that are
> allocated out of order. The problems were always there, iBATIS was just
> hiding them "for you".... hope that helps.
>
> Cheers,
> Clinton
>
>
>
> On Sun, Jul 20, 2008 at 8:01 AM, Hugh Ross <hh...@gmail.com> wrote:
>
>> Clinton,
>>
>> Can you send us a link to more detail on this thread throttling, perhaps
>> in wiki or release notes?
>>
>> Thanks...
>>
>> On 7/17/08, Clinton Begin <cl...@gmail.com> wrote:
>>>
>>> I've removed all thread throttling from iBATIS. It was causing a great
>>> deal of confusing and people complained about performance. The thread
>>> throttling was quite advanced and had a lot of scalability testing behind
>>> it, it was able to hide a lot of problems like this.
>>>
>>> The downside is that you now have to make sure you're not throwing too
>>> much at your database. The way you've solved it is probably the way it
>>> should have been originally. iBATIS was hiding the problem for you, which
>>> was sometimes a feature, and other times a bug. At least now you know
>>> what's really going on... :-)
>>>
>>> That said, it's also possible that you're not closing your transactions
>>> properly. The thread throttling in past releases also did a good job of
>>> hiding that from you as well.. but now you'll start to see the database
>>> complain about such errors. Again, sometimes a feature, other times a bug.
>>>
>>> Clinton
>>>
>>> On Thu, Jul 17, 2008 at 9:55 AM, Michael Schall <mi...@gmail.com>
>>> wrote:
>>>
>>>> Production Environment:
>>>> We are using a JNDI datasource in WS 6.1 (fixpack 15) to connect to a
>>>> DB2 9.1 (fixpack 2) database on a separate cluster.
>>>>
>>>> We recently upgraded our Java version of iBATIS after falling way behind
>>>> from 2.0.9 to the latest release 2.3.1.
>>>>
>>>> The dev and test environment showed no issues with the upgrade.
>>>>
>>>> When rolling out to production we started getting lock timeouts that
>>>> would bring the system down under heavy load.
>>>>
>>>> We did not recreate the JNDI datasource or replace any database drivers
>>>> on the WebSphere machines or make any configuration changes within DB2
>>>> (other than new tables/columns) during the latest release.
>>>>
>>>> We first thought the default IsolationLevel had changed from "Cursor
>>>> Stability" to "Read Stability", but that is actually the default
>>>> IsolationLevel when connecting from WS to DB2 using JNDI. However, as a
>>>> troubleshooting step, we set the IsolationLevel within the JNDI datasource
>>>> to 2 (Cursor Stability), which is allowing our system to avoid the lock
>>>> timeouts.
>>>>
>>>> I have looked throught the change log and nothing seems to point to the
>>>> issue we are having. Any ideas on where our problem might be?
>>>>
>>>> Thanks for your time.
>>>>
>>>> Mike
>>>>
>>>>
>>>
>>>
>>
>>
>
>
Re: iBATIS lock timeouts
Posted by Clinton Begin <cl...@gmail.com>.
Unfortunately it was a pretty deep implementation detail. Essentially there
used to be 3 settings:
maxSessions
maxTransactions
maxRequests
A session was a user thread accessing nearly any part of the framework. A
transaction was a database connection and transaction. A request was an
executing SQL statement. So essentially you could configure something
like: 5 sessions, 15 transactions (avg. two per session), and 60 requests
(avg. four per transaction). Note that these are limits on how many of
these could be active at any one time, not how many can be created etc.
A Throttle was the implementation of the limitation. In a nutshell, it was
the Guarded Suspension pattern that blocked with a wait until a session,
transaction or request became available.
This is similar to the throttling that EJBs and Spring (?) would normally
do. Typical usages were to avoid hitting the database too hard, avoid
running out of open cursors in oracle, or to avoid opening too many
connections at once.
The problems were:
* The settings were confusing, most people didn't understand them (not
their fault) and therefore they were often set incorrectly.
* The behavior was duplicated, as the app server of container framework
(Spring), and even the datasource were likely throttling or limiting such
activity already.
* The throttle features hid the underlying problems which were usually
caused by not properly ending transactions or closing connections.
(try/finally)
* Some complained about the blocking and synchronization required to
implement this feature.
So the behavior was superfluous and the errors were vague. All for
something that wasn't really the responsibility of iBATIS to handle. So it
was removed.
The caveat is that you will now know if your application has a problem.
* If you fail to end transactions (i.e. close connections) your datasource
or database will complain.
* If you hit your database too hard, you'll see performance degrade as
threads contend for resources.
* If you try to run too many statements, your database may complain about
too many open cursors.
The advantage is that:
* The throttling is no longer duplicated in various layers of your
application.
* Performance should be generally better once you tune some other throttle
or fix mismanaged connections.
* You'll get the proper error that will tell you what the real problem is.
If you want to throttle iBATIS yourself, you can do so fairly simply with a
dynamic proxy between the SqlMapClient interface and the SqlMapClientImpl --
perhaps I should release something of the sort as a plug-in?
But I highly recommend you hunt down the actual problem with your
application.. If you're getting deadlocks, that could either be unclosed
connections, poorly written stored procedures or resources that are
allocated out of order. The problems were always there, iBATIS was just
hiding them "for you".... hope that helps.
Cheers,
Clinton
On Sun, Jul 20, 2008 at 8:01 AM, Hugh Ross <hh...@gmail.com> wrote:
> Clinton,
>
> Can you send us a link to more detail on this thread throttling, perhaps in
> wiki or release notes?
>
> Thanks...
>
> On 7/17/08, Clinton Begin <cl...@gmail.com> wrote:
>>
>> I've removed all thread throttling from iBATIS. It was causing a great
>> deal of confusing and people complained about performance. The thread
>> throttling was quite advanced and had a lot of scalability testing behind
>> it, it was able to hide a lot of problems like this.
>>
>> The downside is that you now have to make sure you're not throwing too
>> much at your database. The way you've solved it is probably the way it
>> should have been originally. iBATIS was hiding the problem for you, which
>> was sometimes a feature, and other times a bug. At least now you know
>> what's really going on... :-)
>>
>> That said, it's also possible that you're not closing your transactions
>> properly. The thread throttling in past releases also did a good job of
>> hiding that from you as well.. but now you'll start to see the database
>> complain about such errors. Again, sometimes a feature, other times a bug.
>>
>> Clinton
>>
>> On Thu, Jul 17, 2008 at 9:55 AM, Michael Schall <mi...@gmail.com>
>> wrote:
>>
>>> Production Environment:
>>> We are using a JNDI datasource in WS 6.1 (fixpack 15) to connect to a DB2
>>> 9.1 (fixpack 2) database on a separate cluster.
>>>
>>> We recently upgraded our Java version of iBATIS after falling way behind
>>> from 2.0.9 to the latest release 2.3.1.
>>>
>>> The dev and test environment showed no issues with the upgrade.
>>>
>>> When rolling out to production we started getting lock timeouts that
>>> would bring the system down under heavy load.
>>>
>>> We did not recreate the JNDI datasource or replace any database drivers
>>> on the WebSphere machines or make any configuration changes within DB2
>>> (other than new tables/columns) during the latest release.
>>>
>>> We first thought the default IsolationLevel had changed from "Cursor
>>> Stability" to "Read Stability", but that is actually the default
>>> IsolationLevel when connecting from WS to DB2 using JNDI. However, as a
>>> troubleshooting step, we set the IsolationLevel within the JNDI datasource
>>> to 2 (Cursor Stability), which is allowing our system to avoid the lock
>>> timeouts.
>>>
>>> I have looked throught the change log and nothing seems to point to the
>>> issue we are having. Any ideas on where our problem might be?
>>>
>>> Thanks for your time.
>>>
>>> Mike
>>>
>>>
>>
>>
>
Re: iBATIS lock timeouts
Posted by Hugh Ross <hh...@gmail.com>.
Clinton,
Can you send us a link to more detail on this thread throttling, perhaps in
wiki or release notes?
Thanks...
On 7/17/08, Clinton Begin <cl...@gmail.com> wrote:
>
> I've removed all thread throttling from iBATIS. It was causing a great
> deal of confusing and people complained about performance. The thread
> throttling was quite advanced and had a lot of scalability testing behind
> it, it was able to hide a lot of problems like this.
>
> The downside is that you now have to make sure you're not throwing too much
> at your database. The way you've solved it is probably the way it should
> have been originally. iBATIS was hiding the problem for you, which was
> sometimes a feature, and other times a bug. At least now you know what's
> really going on... :-)
>
> That said, it's also possible that you're not closing your transactions
> properly. The thread throttling in past releases also did a good job of
> hiding that from you as well.. but now you'll start to see the database
> complain about such errors. Again, sometimes a feature, other times a bug.
>
> Clinton
>
> On Thu, Jul 17, 2008 at 9:55 AM, Michael Schall <mi...@gmail.com>
> wrote:
>
>> Production Environment:
>> We are using a JNDI datasource in WS 6.1 (fixpack 15) to connect to a DB2
>> 9.1 (fixpack 2) database on a separate cluster.
>>
>> We recently upgraded our Java version of iBATIS after falling way behind
>> from 2.0.9 to the latest release 2.3.1.
>>
>> The dev and test environment showed no issues with the upgrade.
>>
>> When rolling out to production we started getting lock timeouts that would
>> bring the system down under heavy load.
>>
>> We did not recreate the JNDI datasource or replace any database drivers on
>> the WebSphere machines or make any configuration changes within DB2 (other
>> than new tables/columns) during the latest release.
>>
>> We first thought the default IsolationLevel had changed from "Cursor
>> Stability" to "Read Stability", but that is actually the default
>> IsolationLevel when connecting from WS to DB2 using JNDI. However, as a
>> troubleshooting step, we set the IsolationLevel within the JNDI datasource
>> to 2 (Cursor Stability), which is allowing our system to avoid the lock
>> timeouts.
>>
>> I have looked throught the change log and nothing seems to point to the
>> issue we are having. Any ideas on where our problem might be?
>>
>> Thanks for your time.
>>
>> Mike
>>
>>
>
>
Re: iBATIS lock timeouts
Posted by Clinton Begin <cl...@gmail.com>.
I've removed all thread throttling from iBATIS. It was causing a great deal
of confusing and people complained about performance. The thread
throttling was quite advanced and had a lot of scalability testing behind
it, it was able to hide a lot of problems like this.
The downside is that you now have to make sure you're not throwing too much
at your database. The way you've solved it is probably the way it should
have been originally. iBATIS was hiding the problem for you, which was
sometimes a feature, and other times a bug. At least now you know what's
really going on... :-)
That said, it's also possible that you're not closing your transactions
properly. The thread throttling in past releases also did a good job of
hiding that from you as well.. but now you'll start to see the database
complain about such errors. Again, sometimes a feature, other times a bug.
Clinton
On Thu, Jul 17, 2008 at 9:55 AM, Michael Schall <mi...@gmail.com>
wrote:
> Production Environment:
> We are using a JNDI datasource in WS 6.1 (fixpack 15) to connect to a DB2
> 9.1 (fixpack 2) database on a separate cluster.
>
> We recently upgraded our Java version of iBATIS after falling way behind
> from 2.0.9 to the latest release 2.3.1.
>
> The dev and test environment showed no issues with the upgrade.
>
> When rolling out to production we started getting lock timeouts that would
> bring the system down under heavy load.
>
> We did not recreate the JNDI datasource or replace any database drivers on
> the WebSphere machines or make any configuration changes within DB2 (other
> than new tables/columns) during the latest release.
>
> We first thought the default IsolationLevel had changed from "Cursor
> Stability" to "Read Stability", but that is actually the default
> IsolationLevel when connecting from WS to DB2 using JNDI. However, as a
> troubleshooting step, we set the IsolationLevel within the JNDI datasource
> to 2 (Cursor Stability), which is allowing our system to avoid the lock
> timeouts.
>
> I have looked throught the change log and nothing seems to point to the
> issue we are having. Any ideas on where our problem might be?
>
> Thanks for your time.
>
> Mike
>