You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cayenne.apache.org by Joe Baldwin <jf...@earthlink.net> on 2010/05/12 16:39:43 UTC

Cayenne Connection Pool Monitoring - to solve "out of connections" error

This is a follow-up on the "out of connections" error I am periodically receiving from the production MySQL DBMS.

If you will recall, where we left it was that the evidence appears to support the interim conclusion that it is not Cayenne and that it may be the JDBC Driver or even MySQL.

Actions Taken (based on previous input):
I upgraded the connector to "mysql-connector-java-5.1.12" and the MySQL server was reluctantly upgraded by the webhost tech-person to "MySQL/5.0.88"  (it appears they do not like MySQL 5.1 for some reason).  I also perused my code for any "open" connections (which I am not sure that I have any control over since all DBMS connections are managed by Cayenne).

Results:
It appears that the problem continues.  It even occurs with *very* light usage. There does not seem to be any obvious trigger. This causes sever symptoms of a request for a webpage taking 30-60 sec to load.  (It appears that the 10 used-connections somehow magically allow the user to access after this 30-60 sec delay.  ????????????)

Question:
I am pretty good at tracking down bugs, but this one is really difficult to devise a methodical plan to solve.  My only idea right now is to monitor the Cayenne managed connection pool (with the hopes that I can monitor open connections and associate these connections with the cause).  I have not been able to figure out how to do this via the Cayenne API.  I am convinced that monitoring this connection pool will provide insight into a problem.

Therefore: can you provide some example code of how to monitor open (and closed) connections in the Cayenne managed connection pool (so that I can display the result in a JSP)?

Thanks,
Joe



Re: Cayenne Connection Pool Monitoring - to solve "out of connections" error

Posted by Joe Baldwin <jf...@earthlink.net>.
Andrus,

As you suggested, the webhost tech-person sent me two stack traces (attached below) that were executed during delay-events (that I mentioned previously).  Those are included below. (These are very difficult to get since the problem is intermittent and lasts for only a couple of minutes at a time, then disappears.

 I am not familiar enough with these traces to discern any problems. (The webhost tech believes there might be some important data at the end of the second file.)  Do you see anything that looks out of the ordinary?

Thanks
Joe



On May 12, 2010, at 11:19 AM, Andrus Adamchik wrote:

> 
> On May 12, 2010, at 6:17 PM, Joe Baldwin wrote:
> 
>> 1. Will this thread dump report anything that the MySQL exception does not?
> 
> Yes, it will report who is holding to all those connections.
> 
> Andrus
> 

Re: Cayenne Connection Pool Monitoring - to solve "out of connections" error

Posted by Joe Baldwin <jf...@earthlink.net>.
Ari,

My original question was how can I query the Cayenne Connection Pool Manger to find out how many connections are currently open and how many are currently closed?

Is there a way to do this via the Cayenne API?  (If there is I have not been able to determine how to access the PoolManager from the DataContext, or whatever means is available.)

I would like to start with the standard API first if possible.

Joe




On May 12, 2010, at 12:16 PM, Aristedes Maniatis wrote:

> On 13/05/10 1:46 AM, Joe Baldwin wrote:
>> I don't believe I can execute the steps outlined in your reference (1. the problem is not reproducible on my development machine and 2. the webhost prevents command line access like this on the Linux server.)
>> 
>> Any other ideas?:)
> 
> Attach a profiler like YourKit to the running application: that also lets you get a thread dump at any time. And Andrus mentioned subclassing DriverDataSourceFactory earlier.
> 
> Ari
> 
> -- 
> -------------------------->
> Aristedes Maniatis
> GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A


Re: Cayenne Connection Pool Monitoring - to solve "out of connections" error

Posted by Aristedes Maniatis <ar...@maniatis.org>.
On 13/05/10 1:46 AM, Joe Baldwin wrote:
> I don't believe I can execute the steps outlined in your reference (1. the problem is not reproducible on my development machine and 2. the webhost prevents command line access like this on the Linux server.)
>
> Any other ideas?:)

Attach a profiler like YourKit to the running application: that also lets you get a thread dump at any time. And Andrus mentioned subclassing DriverDataSourceFactory earlier.

Ari

-- 
-------------------------->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A

Re: Cayenne Connection Pool Monitoring - to solve "out of connections" error

Posted by Andrus Adamchik <an...@objectstyle.org>.
Currently Cayenne has no built in monitoring API. We'll likely be  
building one at some point, based on JMX. But you can use JMX Java  
tools right away. E.g. to take a peek at thread dump remotely:

http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html

If that also doesn't work for you, I suggest overriding  
DriverDataSourceFactory and adding logging.

Andrus

On May 12, 2010, at 6:46 PM, Joe Baldwin wrote:

> Andrus,
>
> I thought you were referring to
> 	Thread.dumpStack()
>
> I don't believe I can execute the steps outlined in your reference  
> (1. the problem is not reproducible on my development machine and 2.  
> the webhost prevents command line access like this on the Linux  
> server.)
>
> Any other ideas? :)
>
> Joe
>
>
>
> On May 12, 2010, at 11:33 AM, Andrus Adamchik wrote:
>
>> I think we are talking about different things. I am referring to  
>> this (sorry the URL with lots of ads, just something i googled  
>> quickly) :
>>
>> http://www.crazysquirrel.com/computing/java/basics/java-thread-dump.jspx
>>
>> Andrus
>>
>> On May 12, 2010, at 6:28 PM, Joe Baldwin wrote:
>>
>>> OK, I will implement this.  Do I need to make the query from the  
>>> same thread/session or will I get the same results from any session?
>>>
>>>
>>> On May 12, 2010, at 11:19 AM, Andrus Adamchik wrote:
>>>
>>>>
>>>> On May 12, 2010, at 6:17 PM, Joe Baldwin wrote:
>>>>
>>>>> 1. Will this thread dump report anything that the MySQL  
>>>>> exception does not?
>>>>
>>>> Yes, it will report who is holding to all those connections.
>>>>
>>>> Andrus
>>>>
>>>
>>>
>>
>
>


Re: Cayenne Connection Pool Monitoring - to solve "out of connections" error

Posted by Joe Baldwin <jf...@earthlink.net>.
Andrus,

I thought you were referring to
	Thread.dumpStack()

I don't believe I can execute the steps outlined in your reference (1. the problem is not reproducible on my development machine and 2. the webhost prevents command line access like this on the Linux server.)

Any other ideas? :)

Joe



On May 12, 2010, at 11:33 AM, Andrus Adamchik wrote:

> I think we are talking about different things. I am referring to this (sorry the URL with lots of ads, just something i googled quickly) :
> 
> http://www.crazysquirrel.com/computing/java/basics/java-thread-dump.jspx
> 
> Andrus
> 
> On May 12, 2010, at 6:28 PM, Joe Baldwin wrote:
> 
>> OK, I will implement this.  Do I need to make the query from the same thread/session or will I get the same results from any session?
>> 
>> 
>> On May 12, 2010, at 11:19 AM, Andrus Adamchik wrote:
>> 
>>> 
>>> On May 12, 2010, at 6:17 PM, Joe Baldwin wrote:
>>> 
>>>> 1. Will this thread dump report anything that the MySQL exception does not?
>>> 
>>> Yes, it will report who is holding to all those connections.
>>> 
>>> Andrus
>>> 
>> 
>> 
> 


Re: Cayenne Connection Pool Monitoring - to solve "out of connections" error

Posted by Andrus Adamchik <an...@objectstyle.org>.
I think we are talking about different things. I am referring to this  
(sorry the URL with lots of ads, just something i googled quickly) :

http://www.crazysquirrel.com/computing/java/basics/java-thread-dump.jspx

Andrus

On May 12, 2010, at 6:28 PM, Joe Baldwin wrote:

> OK, I will implement this.  Do I need to make the query from the  
> same thread/session or will I get the same results from any session?
>
>
> On May 12, 2010, at 11:19 AM, Andrus Adamchik wrote:
>
>>
>> On May 12, 2010, at 6:17 PM, Joe Baldwin wrote:
>>
>>> 1. Will this thread dump report anything that the MySQL exception  
>>> does not?
>>
>> Yes, it will report who is holding to all those connections.
>>
>> Andrus
>>
>
>


Re: Cayenne Connection Pool Monitoring - to solve "out of connections" error

Posted by Joe Baldwin <jf...@earthlink.net>.
OK, I will implement this.  Do I need to make the query from the same thread/session or will I get the same results from any session?


On May 12, 2010, at 11:19 AM, Andrus Adamchik wrote:

> 
> On May 12, 2010, at 6:17 PM, Joe Baldwin wrote:
> 
>> 1. Will this thread dump report anything that the MySQL exception does not?
> 
> Yes, it will report who is holding to all those connections.
> 
> Andrus
> 


Re: Cayenne Connection Pool Monitoring - to solve "out of connections" error

Posted by Andrus Adamchik <an...@objectstyle.org>.
On May 12, 2010, at 6:17 PM, Joe Baldwin wrote:

> 1. Will this thread dump report anything that the MySQL exception  
> does not?

Yes, it will report who is holding to all those connections.

Andrus


Re: Cayenne Connection Pool Monitoring - to solve "out of connections" error

Posted by Joe Baldwin <jf...@earthlink.net>.
Andrus,

I am not sure I follow you.

While I am not an expert on stack traces, the "out of connections" exception was discovered by perusing the Catalina logs for errors near the time that the loading-delays occurred.  This generated a pretty detailed stack trace of its own, (which revealed nothing more than MySQL was reporting that there were no more open connections).

1. Will this thread dump report anything that the MySQL exception does not?
2. DriverDataSourceFactory - I was actually looking for something that was more "data-pull" than "data-push", i.e. I would like to query the connection pool class to return information at the moment it is requested.

Thanks,
Joe



On May 12, 2010, at 10:45 AM, Andrus Adamchik wrote:

> Don't remember if you went through that already, but the easiest thing you can do to understand what's going on is to take a thread dump of the running (stuck) application.
> 
> There are other debugging options available (such as subclassing DriverDataSourceFactory to log when connections are opened and closed), but try the thread dump first.
> 
> Andrus
> 
> 
> On May 12, 2010, at 5:39 PM, Joe Baldwin wrote:
> 
>> This is a follow-up on the "out of connections" error I am periodically receiving from the production MySQL DBMS.
>> 
>> If you will recall, where we left it was that the evidence appears to support the interim conclusion that it is not Cayenne and that it may be the JDBC Driver or even MySQL.
>> 
>> Actions Taken (based on previous input):
>> I upgraded the connector to "mysql-connector-java-5.1.12" and the MySQL server was reluctantly upgraded by the webhost tech-person to "MySQL/5.0.88"  (it appears they do not like MySQL 5.1 for some reason).  I also perused my code for any "open" connections (which I am not sure that I have any control over since all DBMS connections are managed by Cayenne).
>> 
>> Results:
>> It appears that the problem continues.  It even occurs with *very* light usage. There does not seem to be any obvious trigger. This causes sever symptoms of a request for a webpage taking 30-60 sec to load.  (It appears that the 10 used-connections somehow magically allow the user to access after this 30-60 sec delay.  ????????????)
>> 
>> Question:
>> I am pretty good at tracking down bugs, but this one is really difficult to devise a methodical plan to solve.  My only idea right now is to monitor the Cayenne managed connection pool (with the hopes that I can monitor open connections and associate these connections with the cause).  I have not been able to figure out how to do this via the Cayenne API.  I am convinced that monitoring this connection pool will provide insight into a problem.
>> 
>> Therefore: can you provide some example code of how to monitor open (and closed) connections in the Cayenne managed connection pool (so that I can display the result in a JSP)?
>> 
>> Thanks,
>> Joe
>> 
>> 
> 


Re: Cayenne Connection Pool Monitoring - to solve "out of connections" error

Posted by Andrus Adamchik <an...@objectstyle.org>.
Don't remember if you went through that already, but the easiest thing  
you can do to understand what's going on is to take a thread dump of  
the running (stuck) application.

There are other debugging options available (such as subclassing  
DriverDataSourceFactory to log when connections are opened and  
closed), but try the thread dump first.

Andrus


On May 12, 2010, at 5:39 PM, Joe Baldwin wrote:

> This is a follow-up on the "out of connections" error I am  
> periodically receiving from the production MySQL DBMS.
>
> If you will recall, where we left it was that the evidence appears  
> to support the interim conclusion that it is not Cayenne and that it  
> may be the JDBC Driver or even MySQL.
>
> Actions Taken (based on previous input):
> I upgraded the connector to "mysql-connector-java-5.1.12" and the  
> MySQL server was reluctantly upgraded by the webhost tech-person to  
> "MySQL/5.0.88"  (it appears they do not like MySQL 5.1 for some  
> reason).  I also perused my code for any "open" connections (which I  
> am not sure that I have any control over since all DBMS connections  
> are managed by Cayenne).
>
> Results:
> It appears that the problem continues.  It even occurs with *very*  
> light usage. There does not seem to be any obvious trigger. This  
> causes sever symptoms of a request for a webpage taking 30-60 sec to  
> load.  (It appears that the 10 used-connections somehow magically  
> allow the user to access after this 30-60 sec delay.  ????????????)
>
> Question:
> I am pretty good at tracking down bugs, but this one is really  
> difficult to devise a methodical plan to solve.  My only idea right  
> now is to monitor the Cayenne managed connection pool (with the  
> hopes that I can monitor open connections and associate these  
> connections with the cause).  I have not been able to figure out how  
> to do this via the Cayenne API.  I am convinced that monitoring this  
> connection pool will provide insight into a problem.
>
> Therefore: can you provide some example code of how to monitor open  
> (and closed) connections in the Cayenne managed connection pool (so  
> that I can display the result in a JSP)?
>
> Thanks,
> Joe
>
>