You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-user@db.apache.org by Suraj Batuwana <sb...@virtusa.com> on 2006/11/15 08:23:18 UTC
Error when executing query:com.ibm.websphere.ce.cm.StaleConnectionException: Meta-data for Container org.apache.derby.impl.store.raw.data.RAFContainer@6fd148fc could not be accessed
I'm getting the following fun stack trace (in attached derby.log) when
using embedded network server derby 10.2.1.6 and was would like to know
if anyone knows what might cause this to error.
I am using IBM JDK 1.4.2 with the Websphere 6.0.2.5 as the application
server on Windows XP with service pack 2. Error comes when I run my
applications unit test cases.
I am getting following client side errors from my junit test cases as
Error when executing
query:com.ibm.websphere.ce.cm.StaleConnectionException: Meta-data for
Container org.apache.derby.impl.store.raw.data.RAFContainer@6fd148fc
could not be accessed
junit.framework.AssertionFailedError: Error when executing
query:com.ibm.websphere.ce.cm.StaleConnectionException: Meta-data for
Container org.apache.derby.impl.store.raw.data.RAFContainer@6fd148fc
could not be accessed
Other than that I am seeing some of the errors from websphere server.log
as
[11/15/06 11:04:47:738 IST] 0000002c SystemErr R
java.sql.SQLException: Failed to start database
'E:\Cloud_Branch\TestDB', see the next exception for details.DSRA0010E:
SQL State = XJ040, Error Code = 40,000DSRA0010E: SQL State = XJ040,
Error Code = 40,000
at sun.reflect.GeneratedConstructorAccessor243.newInstance(Unknown
Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCons
tructorAccessorImpl.java(Compiled Code))
at
java.lang.reflect.Constructor.newInstance(Constructor.java(Compiled
Code))
at
com.ibm.websphere.rsadapter.GenericDataStoreHelper.mapExceptionHelper(Ge
nericDataStoreHelper.java:501)
at
com.ibm.websphere.rsadapter.GenericDataStoreHelper.mapException(GenericD
ataStoreHelper.java:544)
at
com.ibm.ws.rsadapter.spi.WSRdbDataSource.getPooledConnection(WSRdbDataSo
urce.java:1037)
at
com.ibm.ws.rsadapter.spi.WSManagedConnectionFactoryImpl.createManagedCon
nection(WSManagedConnectionFactoryImpl.java:957)
at
com.ibm.ejs.j2c.poolmanager.FreePool.createManagedConnectionWithMCWrappe
r(FreePool.java:1551)
at
com.ibm.ejs.j2c.poolmanager.FreePool.createOrWaitForConnection(FreePool.
java:1343)
at
com.ibm.ejs.j2c.poolmanager.PoolManager.reserve(PoolManager.java(Compile
d Code))
at
com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.ja
va(Compiled Code))
at
com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.j
ava(Compiled Code))
at
com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSourc
e.java(Compiled Code))
at
com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSourc
e.java(Compiled Code))
Also when creating the data source in websphere I have used following
classes as well
Implementing class name
"org.apache.derby.jdbc.EmbeddedConnectionPoolDataSource"
Implementing class name for XA
"org.apache.derby.jdbc.EmbeddedXADataSource"
Datasource Helper Class Name
"com.ibm.websphere.rsadapter.DerbyDataStoreHelper"
Driver Class Path
"c:\jars\derby-10.2.1.6.jar;c:\jars\derbynet-10.2.1.6.jar;c:\jars\derbyt
ools-10.2.1.6.jar"
Unfortunately I could not reproduce this error from any sample
application other than my application
Websphere defines StaleConnectionException as
When an application receives a StaleConnectionException on a database
operation, it indicates that the connection currently held is no longer
valid. While it is possible to get a StaleConnectionException on any
database operation, the most common time to see a
StaleConnectionException thrown is the first time that a connection is
used, just after it is retrieved. Because connections are pooled, a
database failure is not detected until the operation immediately
following its retrieval from the pool, which is the first time
communication to the database is attempted. It is only when a failure is
detected that the connection is marked stale. StaleConnectionException
occurs less often if each method that accesses the database gets a new
connection from the pool.
(http://publib.boulder.ibm.com/infocenter/wasinfo/v5r1//index.jsp?topic=
/com.ibm.websphere.base.doc/info/aes/ae/rdat_stalconexp.html)
But I could not find out why this kind of situation occurred.
Also shutdown all the java processors in the machine I can connect to
the derby database using ij
I would like to know if anyone knows what might cause this to error
Regards,
Suraj
Re: Error when executing
query:com.ibm.websphere.ce.cm.StaleConnectionException: Meta-data for
Container org.apache.derby.impl.store.raw.data.RAFContainer@6fd148fc could
not be accessed
Posted by Suraj Batuwana <sb...@virtusa.com>.
Hi Suresh,
Many thanks for the reply. Yes, as you said jdk15 does not have the above
issue. But now the big problem is my application does not support jdk15. Is
there a way we can have a fix for the above issue with using the jdk14 ?
-Suraj
Suresh Thalamati wrote:
>
> Suraj Batuwana wrote:
>> createDB.sql in the attached zip file has all the sqls for database
>> tables,
>> indexes and views
>> Error comes when run SELECT * FROM
>> vwDerbyBasePackage_derbygen_DerbyRepositoryObject67ceb178
>>
>> Can use the CreateDatabase.bat to create the derby database and
>> TestIssue.bat can use to test the issue.
>> derby.log has the error I am getting.
>>
>> Above was tested with derby 10.2.1.6 but get the same results
>>
>> Please let me know if any one of you able to reproduce the issue
>>
>>
>
> Suraj ,
>
> Thanks a lot for providing scripts to reproduce the error. I am able
> to reproduce the problem you are seeing on jdk14 and jdk13. Query work
> s fine on jdk15. As I mentioned earlier , I think this error is
> happening because too many files are kept open(> 2000). On jdk14 ,you
> are hitting the jvm bug.
> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4189011, that cause
> the above error, when a container file opens.
>
> Now the obvious question is why derby is keeping so many files open to
> compile the query ?
>
> view "vwDerbyBasePackage_derbygen_DerbyRepositoryObject67ceb178" is
> pretty huge and uses other views. Total of no of user
> tables are 1197, and there are 2301 index containers. This query
> opened 2078 container files when I ran on jdk15. I could not find any
> cases in rawstore where files are not getting closed correctly.
>
> By doing quick scan of the code, I think during the query
> optimization Derby opens all the tables and all the indexes on each
> table to estimate the costs and those containers are not closed until
> the nested transaction used to compile the query is closed. Because
> the containers used to estimate the cost are not closed,
> corresponding container files can not be closed, so the container
> cache (file handle cache ) keep growing. May be it is
> not really required to keep all containers open until the end of
> compiling the query. Optimizer gurus , please correct me if I am
> saying something that is incorrect here.
>
>
> Thanks
> -suresh
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
--
View this message in context: http://www.nabble.com/Error-when-executing-query%3Acom.ibm.websphere.ce.cm.StaleConnectionException%3A-Meta-data-for-Container-org.apache.derby.impl.store.raw.data.RAFContainer%406fd148fc-could-not-be-accessed-tf2634449.html#a7634026
Sent from the Apache Derby Users mailing list archive at Nabble.com.
Re: Error when executing
query:com.ibm.websphere.ce.cm.StaleConnectionException: Meta-data for
Container org.apache.derby.impl.store.raw.data.RAFContainer@6fd148fc could
not be accessed
Posted by Suraj Batuwana <sb...@virtusa.com>.
I have file a JIRA issue at
https://issues.apache.org/jira/browse/DERBY-2144
Army-4 wrote:
>
> Suresh Thalamati wrote:
>> By doing quick scan of the code, I think during the query optimization
>> Derby opens all the tables and all the indexes on each table to estimate
>> the costs and those containers are not closed until the nested
>> transaction used to compile the query is closed. Because the containers
>> used to estimate the cost are not closed, corresponding container files
>> can not be closed, so the container cache (file handle cache ) keep
>> growing.
>
> I'm not very familiar with the costing mechanism of the optimizer (most of
> the
> work I've done just assumes that the costs are correct), but I spent some
> time
> looking at the cost estimation code and I agree with Suresh: as far as I
> can
> tell, it looks like the optimizer opens the containers for base tables and
> indexes in the query but does not close them until compilation completes.
>
>> May be it is not really required to keep all containers open
>> until the end of compiling the query.
>
> It may be possible to add logic that closes and re-opens the containers
> during
> optimization as they are needed. Note, though, that the containers are
> deliberately cached in the current code and the assumption appears to be
> that
> they will remain open throughout optimization. So I think any changes in
> this
> area could be non-trivial and would have to take into consideration
> potential
> performance effects. But that said, I willingly admit that I know very
> little
> about the relevant parts of the Derby code, so I could be wrong.
>
> In any event, maybe the best thing to do here is to file a Jira
> enhancement for
> tracking purposes? If you (Suraj) choose to do so, the zip file
> containing the
> reproduction could be attached to the Jira issue for ease of reference.
>
> > But now the big problem is my application does not support jdk15. Is
> > there a way we can have a fix for the above issue with using the jdk14
> ?
>
> The first step toward resolving the problem is to file a Jira issue:
>
> http://db.apache.org/derby/DerbyBugGuidelines.html
>
> Once that's done, the easiest way to get a fix into the codeline is to
> contribute the changes yourself :) If you have the time and inclination
> to work
> on this issue, you should feel free to join the community:
>
> http://wiki.apache.org/db-derby/DerbyDev
>
> Otherwise you can just file the Jira issue and then anyone who is
> interested can
> pick it up and work to resolve it...
>
> Army
>
>
>
--
View this message in context: http://www.nabble.com/Error-when-executing-query%3Acom.ibm.websphere.ce.cm.StaleConnectionException%3A-Meta-data-for-Container-org.apache.derby.impl.store.raw.data.RAFContainer%406fd148fc-could-not-be-accessed-tf2634449.html#a7694568
Sent from the Apache Derby Users mailing list archive at Nabble.com.
Re: Error when executing query:com.ibm.websphere.ce.cm.StaleConnectionException:
Meta-data for Container org.apache.derby.impl.store.raw.data.RAFContainer@6fd148fc
could not be accessed
Posted by Army <qo...@gmail.com>.
Suresh Thalamati wrote:
> By doing quick scan of the code, I think during the query optimization
> Derby opens all the tables and all the indexes on each table to estimate
> the costs and those containers are not closed until the nested
> transaction used to compile the query is closed. Because the containers
> used to estimate the cost are not closed, corresponding container files
> can not be closed, so the container cache (file handle cache ) keep
> growing.
I'm not very familiar with the costing mechanism of the optimizer (most of the
work I've done just assumes that the costs are correct), but I spent some time
looking at the cost estimation code and I agree with Suresh: as far as I can
tell, it looks like the optimizer opens the containers for base tables and
indexes in the query but does not close them until compilation completes.
> May be it is not really required to keep all containers open
> until the end of compiling the query.
It may be possible to add logic that closes and re-opens the containers during
optimization as they are needed. Note, though, that the containers are
deliberately cached in the current code and the assumption appears to be that
they will remain open throughout optimization. So I think any changes in this
area could be non-trivial and would have to take into consideration potential
performance effects. But that said, I willingly admit that I know very little
about the relevant parts of the Derby code, so I could be wrong.
In any event, maybe the best thing to do here is to file a Jira enhancement for
tracking purposes? If you (Suraj) choose to do so, the zip file containing the
reproduction could be attached to the Jira issue for ease of reference.
> But now the big problem is my application does not support jdk15. Is
> there a way we can have a fix for the above issue with using the jdk14 ?
The first step toward resolving the problem is to file a Jira issue:
http://db.apache.org/derby/DerbyBugGuidelines.html
Once that's done, the easiest way to get a fix into the codeline is to
contribute the changes yourself :) If you have the time and inclination to work
on this issue, you should feel free to join the community:
http://wiki.apache.org/db-derby/DerbyDev
Otherwise you can just file the Jira issue and then anyone who is interested can
pick it up and work to resolve it...
Army
Re: Error when executing query:com.ibm.websphere.ce.cm.StaleConnectionException:
Meta-data for Container org.apache.derby.impl.store.raw.data.RAFContainer@6fd148fc
could not be accessed
Posted by Suresh Thalamati <su...@gmail.com>.
Suraj Batuwana wrote:
> createDB.sql in the attached zip file has all the sqls for database tables,
> indexes and views
> Error comes when run SELECT * FROM
> vwDerbyBasePackage_derbygen_DerbyRepositoryObject67ceb178
>
> Can use the CreateDatabase.bat to create the derby database and
> TestIssue.bat can use to test the issue.
> derby.log has the error I am getting.
>
> Above was tested with derby 10.2.1.6 but get the same results
>
> Please let me know if any one of you able to reproduce the issue
>
>
Suraj ,
Thanks a lot for providing scripts to reproduce the error. I am able
to reproduce the problem you are seeing on jdk14 and jdk13. Query work
s fine on jdk15. As I mentioned earlier , I think this error is
happening because too many files are kept open(> 2000). On jdk14 ,you
are hitting the jvm bug.
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4189011, that cause
the above error, when a container file opens.
Now the obvious question is why derby is keeping so many files open to
compile the query ?
view "vwDerbyBasePackage_derbygen_DerbyRepositoryObject67ceb178" is
pretty huge and uses other views. Total of no of user
tables are 1197, and there are 2301 index containers. This query
opened 2078 container files when I ran on jdk15. I could not find any
cases in rawstore where files are not getting closed correctly.
By doing quick scan of the code, I think during the query
optimization Derby opens all the tables and all the indexes on each
table to estimate the costs and those containers are not closed until
the nested transaction used to compile the query is closed. Because
the containers used to estimate the cost are not closed,
corresponding container files can not be closed, so the container
cache (file handle cache ) keep growing. May be it is
not really required to keep all containers open until the end of
compiling the query. Optimizer gurus , please correct me if I am
saying something that is incorrect here.
Thanks
-suresh
Re: Error when executing query:com.ibm.websphere.ce.cm.StaleConnectionException:
Meta-data for Container org.apache.derby.impl.store.raw.data.RAFContainer@6fd148fc
could not be accessed
Posted by Suresh Thalamati <su...@gmail.com>.
Suraj Batuwana wrote:
> createDB.sql in the attached zip file has all the sqls for database tables,
> indexes and views
> Error comes when run SELECT * FROM
> vwDerbyBasePackage_derbygen_DerbyRepositoryObject67ceb178
>
> Can use the CreateDatabase.bat to create the derby database and
> TestIssue.bat can use to test the issue.
> derby.log has the error I am getting.
>
> Above was tested with derby 10.2.1.6 but get the same results
>
> Please let me know if any one of you able to reproduce the issue
>
>
>
> http://www.nabble.com/file/4227/DerbyIssue.zip DerbyIssue.zip
Suraj,
Great, I will try the repro. For some reason, I am not able to access
the attached zip file. Could you please file a Jira for this issue
and attach the repro to it.
Following link has information about filing Jira entry:
http://db.apache.org/derby/DerbyBugGuidelines.html#File+your+bug
Thanks
-suresh
Re: Error when executing
query:com.ibm.websphere.ce.cm.StaleConnectionException: Meta-data for
Container org.apache.derby.impl.store.raw.data.RAFContainer@6fd148fc could
not be accessed
Posted by Suraj Batuwana <sb...@virtusa.com>.
createDB.sql in the attached zip file has all the sqls for database tables,
indexes and views
Error comes when run SELECT * FROM
vwDerbyBasePackage_derbygen_DerbyRepositoryObject67ceb178
Can use the CreateDatabase.bat to create the derby database and
TestIssue.bat can use to test the issue.
derby.log has the error I am getting.
Above was tested with derby 10.2.1.6 but get the same results
Please let me know if any one of you able to reproduce the issue
Bryan Pendleton wrote:
>
>> I have execute the simple select on this particular view as
>> SELECT * FROM vwDerbyBaseView using a small java client for all the
>> derby modes (Embedded and Network)
>>
>> Then the same error Meta-data for Container
>> org.apache.derby.impl.store.raw.data.RAFContainer@6fd148fc could not be
>> accessed was given in the derby.log
>
> Great! It sounds like you've managed to isolate a reproducible test case.
>
> Can you package up the CREATE TABLE, CREATE INDEX, and CREATE VIEW
> statements
> for this particular view, together with the SELECT statement, into a
> single
> SQL script file, so that others can confirm the behavior in other
> configurations?
>
>
> thanks,
>
> bryan
>
> P.S. It definitely feels like a resource shortage problem. As a
> workaround, have
> you tried giving the JVM more resources?
>
> I have change the java Xmx and Xms but did get the same results.
>
>
>
http://www.nabble.com/file/4227/DerbyIssue.zip DerbyIssue.zip
--
View this message in context: http://www.nabble.com/Error-when-executing-query%3Acom.ibm.websphere.ce.cm.StaleConnectionException%3A-Meta-data-for-Container-org.apache.derby.impl.store.raw.data.RAFContainer%406fd148fc-could-not-be-accessed-tf2634449.html#a7440385
Sent from the Apache Derby Users mailing list archive at Nabble.com.