You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-user@db.apache.org by Eric Floehr <Er...@3x.com> on 2009/11/17 19:42:10 UTC
OOM with millions of weakly-referenced Derby objects
All,
I have a Derby table with up to a million rows. Some large subset of those rows may be returned by a SELECT query.
I am using IBATIS quertyForList with the embedded Derby driver. I am using the version which has a maxRows parameter. I call that method and receive back the proper "maxRows" java.sql.ResultSet objects. For example, the select would match 100,000 rows, but I only get the first 200. That all works fine.
However, I am getting OOM's when the select would match nearly a million rows and after profiling with JProfiler the culprit seems to be millions of weakly referenced org.apache.derby.iapi.types.SQL objects (SQLLongInt, for example) that appear to be created far faster than they can be GC'ed. This particular performance test run would return a million records, of which I am only wanting to retrieve the first 200.
My questions are:
1. Is this expected behavior?
2. Is IBATIS properly using a cursor, or what is the right way to limit the number of Derby objects created (the ResultSet count is correct)
3. Are there Derby or IBATIS parameters that can be employed to correct this behavior?
Derby version is 10.5.3, IBATIS version is 2.3.4.726, and Java version is 1.6. I have cross-posted to both the Derby and IBATIS lists since I don't know where the root problem resides.
Thanks so much for your help!
Best Regards,
Eric
Re: OOM with millions of weakly-referenced Derby objects
Posted by Bryan Pendleton <bp...@amberpoint.com>.
> I will attempt running the command via ODBC directly, with the FETCH syntax, and produce a heap histogram (is there anything in particular you are looking for or a specific tool output -- I generally use jprofiler).
This sounds like a great next step, please let us know what you find.
As another useful step, you could run that command using the Derby "ij"
tool, and set derby.language.logQueryPlan=true, and then post the
query plan information from your derby.log.
thanks,
bryan
Re: AW: AW: AW: OOM with millions of weakly-referenced Derby objects
Posted by Peter Ondruška <pe...@gmail.com>.
I develop on Windows and deploy to AIX 5.3 using Java 6 (_17 on Windows) and
SR4
http://www.ibm.com/developerworks/java/jdk/aix/j664/Java6_64.fixinfo.html and
never encountered anything unusual.
On Fri, Nov 27, 2009 at 4:30 PM, <Ma...@de.equens.com> wrote:
> Hi Dag,
> Thanks for your effort. The problem is as far I understand a special
> behavoiour on AIX Java Mashine. Does anybody of the developing or testing
> people got access on something like that?
>
>
> -----Ursprüngliche Nachricht-----
> Von: Dag.Wanvik@Sun.COM [mailto:Dag.Wanvik@Sun.COM]
> Gesendet: Freitag, 27. November 2009 16:15
> An: Derby Discussion
> Betreff: Re: AW: AW: AW: OOM with millions of weakly-referenced Derby
> objects
>
>
> Hi Malte,
>
> Malte.Kempff@de.equens.com writes:
>
>
> > Hi Dag, The problem in my case is, that I don't have access to any
> > AIX-Computer, and that is really a pity, because I cannot
> > reconstruct the scenario on my own and watch it with my own eyes.
> > That all happened on a production system and the provider denies
> > trying it out again. But attached you can find the SQL-script
> > without being UTF8. So that Problem should occur in the same way
> > used on AIX (5.3) BTW: is there a way to be kind of independent of
> > the input format?
>
> I converted the script you enclosed to UTF-8 and ran it through ij
> without any problems.
>
> I also tried to run it with the enclosed program, and it gave me no
> errors when the inout file was encoded with UTF-8 (Comments had
> non-ASCII German character), and it printed no errors on my
> OpenSolaris system (I don't have access to AIX).
>
> In any case, should you see this problem again, please feel free to
> file a bug report(*), preferably with a repro and/or logs.
>
> As for independence of the input format, since runScript requires that
> you specify the encoding, you could do some encoding auto-detect logic
> on the script, I guess..
>
> (*) http://db.apache.org/derby/DerbyBugGuidelines.html
>
> Hope this helps,
> Dag
>
>
> import org.apache.derby.tools.ij;
> import java.sql.*;
> import java.io.*;
>
> public class Foo {
>
> public Foo() {};
>
> static public void main(String[] args)
> throws SQLException,
> UnsupportedEncodingException,
> FileNotFoundException,
> ClassNotFoundException {
> Class.forName("org.apache.derby.jdbc.EmbeddedDriver");
> Connection c =
> DriverManager.getConnection("jdbc:derby:wombat;create=true");
>
> FileInputStream fs = new FileInputStream(args[0]);
>
> int errs = ij.runScript(c,
> fs,
> "UTF-8",
> System.out,
> "UTF-8");
>
> System.out.println("\n\nerrs=" + errs);
> c.close();
> }
> }
>
>
>
>
> >
> > Malte
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Dag.Wanvik@Sun.COM [mailto:Dag.Wanvik@Sun.COM]
> > Gesendet: Freitag, 27. November 2009 01:19
> > An: Derby Discussion
> > Betreff: Re: AW: AW: OOM with millions of weakly-referenced Derby objects
> >
> > Malte.Kempff@de.equens.com writes:
> >
> >> Yes this is the API I am reffering to. Well actually it says that,
> >> but in my case it did not do it. It needed a long time, several
> >> minutes. Reason as it turned later out, after getting the output in
> >> a file a lot of IOExcceptions. I never got those on windows nor on
> >> Solaris. Thr original logic of the routine calling the runScript
> >> Method was asking for the return code but either it never hit this
> >> if statement and crashed somewhere within ij without giving the
> >> Exception to the caller or ij gave something like 0 so the output
> >> never was given to the logger. So there is something phony with
> >> ij.runScript using it with AIX 5.3 and a SQL-Script not coded UTF8
> >> but declaring in the parameters as UTF8. Of course that is a fault
> >> by my own, but it should tel me about it.
> >
> > I looked at the implementation for runScript in
> > org.apache.derby.impl.tools.ij.utilMain#runScriptGuts and it does seem
> > to catch exceptions, count them and return them. Are you able to
> > provide a repro script showing that 0 or -1 is returned from runScript
> > for the failing scenario?
> >
> > Thanks,
> > Dag
> >
> >>
> >>
> >>
> >> -----Ursprüngliche Nachricht-----
> >> Von: Dag.Wanvik@Sun.COM [mailto:Dag.Wanvik@Sun.COM]
> >> Gesendet: Freitag, 20. November 2009 19:54
> >> An: Derby Discussion
> >> Betreff: Re: AW: OOM with millions of weakly-referenced Derby objects
> >>
> >> Malte.Kempff@de.equens.com writes:
> >>
> >>> In other words I would count that situation as not nice behaviour of
> ij, since the return code should have been given as I would suppose it
> looking at the method signature.
> >>
> >> Is this the API signature you are referring to?
> >>
> >>
> http://db.apache.org/derby/docs/10.5/publishedapi/jdbc4/org/apache/derby/tools/ij.html#runScript(java.sql.Connection,%20java.io.InputStream,%20java.lang.String,%20java.io.OutputStream,%20java.lang.String)
> >>
> >> It says:
> >>
> >> Returns:h
> >> Number of SQLExceptions thrown during the execution, -1 if not
> >> known.
> >>
> >> If so, are you seeing 0 or -1 returned here?
> >>
> >> Dag
>
AW: AW: AW: AW: OOM with millions of weakly-referenced Derby objects
Posted by Ma...@de.equens.com.
Hi Dag,
Thanks for your effort. The problem is as far I understand a special behavoiour on AIX Java Mashine. Does anybody of the developing or testing people got access on something like that?
-----Ursprüngliche Nachricht-----
Von: Dag.Wanvik@Sun.COM [mailto:Dag.Wanvik@Sun.COM]
Gesendet: Freitag, 27. November 2009 16:15
An: Derby Discussion
Betreff: Re: AW: AW: AW: OOM with millions of weakly-referenced Derby objects
Hi Malte,
Malte.Kempff@de.equens.com writes:
> Hi Dag, The problem in my case is, that I don't have access to any
> AIX-Computer, and that is really a pity, because I cannot
> reconstruct the scenario on my own and watch it with my own eyes.
> That all happened on a production system and the provider denies
> trying it out again. But attached you can find the SQL-script
> without being UTF8. So that Problem should occur in the same way
> used on AIX (5.3) BTW: is there a way to be kind of independent of
> the input format?
I converted the script you enclosed to UTF-8 and ran it through ij
without any problems.
I also tried to run it with the enclosed program, and it gave me no
errors when the inout file was encoded with UTF-8 (Comments had
non-ASCII German character), and it printed no errors on my
OpenSolaris system (I don't have access to AIX).
In any case, should you see this problem again, please feel free to
file a bug report(*), preferably with a repro and/or logs.
As for independence of the input format, since runScript requires that
you specify the encoding, you could do some encoding auto-detect logic
on the script, I guess..
(*) http://db.apache.org/derby/DerbyBugGuidelines.html
Hope this helps,
Dag
import org.apache.derby.tools.ij;
import java.sql.*;
import java.io.*;
public class Foo {
public Foo() {};
static public void main(String[] args)
throws SQLException,
UnsupportedEncodingException,
FileNotFoundException,
ClassNotFoundException {
Class.forName("org.apache.derby.jdbc.EmbeddedDriver");
Connection c = DriverManager.getConnection("jdbc:derby:wombat;create=true");
FileInputStream fs = new FileInputStream(args[0]);
int errs = ij.runScript(c,
fs,
"UTF-8",
System.out,
"UTF-8");
System.out.println("\n\nerrs=" + errs);
c.close();
}
}
>
> Malte
>
> -----Ursprüngliche Nachricht-----
> Von: Dag.Wanvik@Sun.COM [mailto:Dag.Wanvik@Sun.COM]
> Gesendet: Freitag, 27. November 2009 01:19
> An: Derby Discussion
> Betreff: Re: AW: AW: OOM with millions of weakly-referenced Derby objects
>
> Malte.Kempff@de.equens.com writes:
>
>> Yes this is the API I am reffering to. Well actually it says that,
>> but in my case it did not do it. It needed a long time, several
>> minutes. Reason as it turned later out, after getting the output in
>> a file a lot of IOExcceptions. I never got those on windows nor on
>> Solaris. Thr original logic of the routine calling the runScript
>> Method was asking for the return code but either it never hit this
>> if statement and crashed somewhere within ij without giving the
>> Exception to the caller or ij gave something like 0 so the output
>> never was given to the logger. So there is something phony with
>> ij.runScript using it with AIX 5.3 and a SQL-Script not coded UTF8
>> but declaring in the parameters as UTF8. Of course that is a fault
>> by my own, but it should tel me about it.
>
> I looked at the implementation for runScript in
> org.apache.derby.impl.tools.ij.utilMain#runScriptGuts and it does seem
> to catch exceptions, count them and return them. Are you able to
> provide a repro script showing that 0 or -1 is returned from runScript
> for the failing scenario?
>
> Thanks,
> Dag
>
>>
>>
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Dag.Wanvik@Sun.COM [mailto:Dag.Wanvik@Sun.COM]
>> Gesendet: Freitag, 20. November 2009 19:54
>> An: Derby Discussion
>> Betreff: Re: AW: OOM with millions of weakly-referenced Derby objects
>>
>> Malte.Kempff@de.equens.com writes:
>>
>>> In other words I would count that situation as not nice behaviour of ij, since the return code should have been given as I would suppose it looking at the method signature.
>>
>> Is this the API signature you are referring to?
>>
>> http://db.apache.org/derby/docs/10.5/publishedapi/jdbc4/org/apache/derby/tools/ij.html#runScript(java.sql.Connection,%20java.io.InputStream,%20java.lang.String,%20java.io.OutputStream,%20java.lang.String)
>>
>> It says:
>>
>> Returns:h
>> Number of SQLExceptions thrown during the execution, -1 if not
>> known.
>>
>> If so, are you seeing 0 or -1 returned here?
>>
>> Dag
Re: AW: AW: AW: OOM with millions of weakly-referenced Derby objects
Posted by "Dag H. Wanvik" <Da...@Sun.COM>.
Hi Malte,
Malte.Kempff@de.equens.com writes:
> Hi Dag, The problem in my case is, that I don't have access to any
> AIX-Computer, and that is really a pity, because I cannot
> reconstruct the scenario on my own and watch it with my own eyes.
> That all happened on a production system and the provider denies
> trying it out again. But attached you can find the SQL-script
> without being UTF8. So that Problem should occur in the same way
> used on AIX (5.3) BTW: is there a way to be kind of independent of
> the input format?
I converted the script you enclosed to UTF-8 and ran it through ij
without any problems.
I also tried to run it with the enclosed program, and it gave me no
errors when the inout file was encoded with UTF-8 (Comments had
non-ASCII German character), and it printed no errors on my
OpenSolaris system (I don't have access to AIX).
In any case, should you see this problem again, please feel free to
file a bug report(*), preferably with a repro and/or logs.
As for independence of the input format, since runScript requires that
you specify the encoding, you could do some encoding auto-detect logic
on the script, I guess..
(*) http://db.apache.org/derby/DerbyBugGuidelines.html
Hope this helps,
Dag
import org.apache.derby.tools.ij;
import java.sql.*;
import java.io.*;
public class Foo {
public Foo() {};
static public void main(String[] args)
throws SQLException,
UnsupportedEncodingException,
FileNotFoundException,
ClassNotFoundException {
Class.forName("org.apache.derby.jdbc.EmbeddedDriver");
Connection c = DriverManager.getConnection("jdbc:derby:wombat;create=true");
FileInputStream fs = new FileInputStream(args[0]);
int errs = ij.runScript(c,
fs,
"UTF-8",
System.out,
"UTF-8");
System.out.println("\n\nerrs=" + errs);
c.close();
}
}
>
> Malte
>
> -----Ursprüngliche Nachricht-----
> Von: Dag.Wanvik@Sun.COM [mailto:Dag.Wanvik@Sun.COM]
> Gesendet: Freitag, 27. November 2009 01:19
> An: Derby Discussion
> Betreff: Re: AW: AW: OOM with millions of weakly-referenced Derby objects
>
> Malte.Kempff@de.equens.com writes:
>
>> Yes this is the API I am reffering to. Well actually it says that,
>> but in my case it did not do it. It needed a long time, several
>> minutes. Reason as it turned later out, after getting the output in
>> a file a lot of IOExcceptions. I never got those on windows nor on
>> Solaris. Thr original logic of the routine calling the runScript
>> Method was asking for the return code but either it never hit this
>> if statement and crashed somewhere within ij without giving the
>> Exception to the caller or ij gave something like 0 so the output
>> never was given to the logger. So there is something phony with
>> ij.runScript using it with AIX 5.3 and a SQL-Script not coded UTF8
>> but declaring in the parameters as UTF8. Of course that is a fault
>> by my own, but it should tel me about it.
>
> I looked at the implementation for runScript in
> org.apache.derby.impl.tools.ij.utilMain#runScriptGuts and it does seem
> to catch exceptions, count them and return them. Are you able to
> provide a repro script showing that 0 or -1 is returned from runScript
> for the failing scenario?
>
> Thanks,
> Dag
>
>>
>>
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Dag.Wanvik@Sun.COM [mailto:Dag.Wanvik@Sun.COM]
>> Gesendet: Freitag, 20. November 2009 19:54
>> An: Derby Discussion
>> Betreff: Re: AW: OOM with millions of weakly-referenced Derby objects
>>
>> Malte.Kempff@de.equens.com writes:
>>
>>> In other words I would count that situation as not nice behaviour of ij, since the return code should have been given as I would suppose it looking at the method signature.
>>
>> Is this the API signature you are referring to?
>>
>> http://db.apache.org/derby/docs/10.5/publishedapi/jdbc4/org/apache/derby/tools/ij.html#runScript(java.sql.Connection,%20java.io.InputStream,%20java.lang.String,%20java.io.OutputStream,%20java.lang.String)
>>
>> It says:
>>
>> Returns:h
>> Number of SQLExceptions thrown during the execution, -1 if not
>> known.
>>
>> If so, are you seeing 0 or -1 returned here?
>>
>> Dag
AW: AW: AW: OOM with millions of weakly-referenced Derby objects
Posted by Ma...@de.equens.com.
Hi Dag,
The problem in my case is, that I don't have access to any AIX-Computer, and that is really a pity, because I cannot reconstruct the scenario on my own and watch it with my own eyes.
That all happened on a production system and the provider denies trying it out again.
But attached you can find the SQL-script without being UTF8.
So that Problem should occur in the same way used on AIX (5.3)
BTW: is there a way to be kind of independent of the input format?
Malte
-----Ursprüngliche Nachricht-----
Von: Dag.Wanvik@Sun.COM [mailto:Dag.Wanvik@Sun.COM]
Gesendet: Freitag, 27. November 2009 01:19
An: Derby Discussion
Betreff: Re: AW: AW: OOM with millions of weakly-referenced Derby objects
Malte.Kempff@de.equens.com writes:
> Yes this is the API I am reffering to. Well actually it says that,
> but in my case it did not do it. It needed a long time, several
> minutes. Reason as it turned later out, after getting the output in
> a file a lot of IOExcceptions. I never got those on windows nor on
> Solaris. Thr original logic of the routine calling the runScript
> Method was asking for the return code but either it never hit this
> if statement and crashed somewhere within ij without giving the
> Exception to the caller or ij gave something like 0 so the output
> never was given to the logger. So there is something phony with
> ij.runScript using it with AIX 5.3 and a SQL-Script not coded UTF8
> but declaring in the parameters as UTF8. Of course that is a fault
> by my own, but it should tel me about it.
I looked at the implementation for runScript in
org.apache.derby.impl.tools.ij.utilMain#runScriptGuts and it does seem
to catch exceptions, count them and return them. Are you able to
provide a repro script showing that 0 or -1 is returned from runScript
for the failing scenario?
Thanks,
Dag
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: Dag.Wanvik@Sun.COM [mailto:Dag.Wanvik@Sun.COM]
> Gesendet: Freitag, 20. November 2009 19:54
> An: Derby Discussion
> Betreff: Re: AW: OOM with millions of weakly-referenced Derby objects
>
> Malte.Kempff@de.equens.com writes:
>
>> In other words I would count that situation as not nice behaviour of ij, since the return code should have been given as I would suppose it looking at the method signature.
>
> Is this the API signature you are referring to?
>
> http://db.apache.org/derby/docs/10.5/publishedapi/jdbc4/org/apache/derby/tools/ij.html#runScript(java.sql.Connection,%20java.io.InputStream,%20java.lang.String,%20java.io.OutputStream,%20java.lang.String)
>
> It says:
>
> Returns:
> Number of SQLExceptions thrown during the execution, -1 if not
> known.
>
> If so, are you seeing 0 or -1 returned here?
>
> Dag
>
--
Dag H. Wanvik, staff engineer
Sun Microsystems, Java Core and Desktop - Java DB/Derby
Haakon VII gt. 7b, N-7485 Trondheim, Norway
Tel: x43496/+47 73842196, Fax: +47 73842101
Sun IM: dw136674, Yahoo IM: dag_h_wanvik
Re: AW: AW: OOM with millions of weakly-referenced Derby objects
Posted by "Dag H. Wanvik" <Da...@Sun.COM>.
Malte.Kempff@de.equens.com writes:
> Yes this is the API I am reffering to. Well actually it says that,
> but in my case it did not do it. It needed a long time, several
> minutes. Reason as it turned later out, after getting the output in
> a file a lot of IOExcceptions. I never got those on windows nor on
> Solaris. Thr original logic of the routine calling the runScript
> Method was asking for the return code but either it never hit this
> if statement and crashed somewhere within ij without giving the
> Exception to the caller or ij gave something like 0 so the output
> never was given to the logger. So there is something phony with
> ij.runScript using it with AIX 5.3 and a SQL-Script not coded UTF8
> but declaring in the parameters as UTF8. Of course that is a fault
> by my own, but it should tel me about it.
I looked at the implementation for runScript in
org.apache.derby.impl.tools.ij.utilMain#runScriptGuts and it does seem
to catch exceptions, count them and return them. Are you able to
provide a repro script showing that 0 or -1 is returned from runScript
for the failing scenario?
Thanks,
Dag
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: Dag.Wanvik@Sun.COM [mailto:Dag.Wanvik@Sun.COM]
> Gesendet: Freitag, 20. November 2009 19:54
> An: Derby Discussion
> Betreff: Re: AW: OOM with millions of weakly-referenced Derby objects
>
> Malte.Kempff@de.equens.com writes:
>
>> In other words I would count that situation as not nice behaviour of ij, since the return code should have been given as I would suppose it looking at the method signature.
>
> Is this the API signature you are referring to?
>
> http://db.apache.org/derby/docs/10.5/publishedapi/jdbc4/org/apache/derby/tools/ij.html#runScript(java.sql.Connection,%20java.io.InputStream,%20java.lang.String,%20java.io.OutputStream,%20java.lang.String)
>
> It says:
>
> Returns:
> Number of SQLExceptions thrown during the execution, -1 if not
> known.
>
> If so, are you seeing 0 or -1 returned here?
>
> Dag
>
--
Dag H. Wanvik, staff engineer
Sun Microsystems, Java Core and Desktop - Java DB/Derby
Haakon VII gt. 7b, N-7485 Trondheim, Norway
Tel: x43496/+47 73842196, Fax: +47 73842101
Sun IM: dw136674, Yahoo IM: dag_h_wanvik
AW: AW: OOM with millions of weakly-referenced Derby objects
Posted by Ma...@de.equens.com.
Yes this is the API I am reffering to.
Well actually it says that, but in my case it did not do it.
It needed a long time, several minutes. Reason as it turned later out, after getting the output in a file a lot of IOExcceptions. I never got those on windows nor on Solaris. Thr original logic of the routine calling the runScript Method was asking for the return code but either it never hit this if statement and crashed somewhere within ij without giving the Exception to the caller or ij gave something like 0 so the output never was given to the logger.
So there is something phony with ij.runScript using it with AIX 5.3 and a SQL-Script not coded UTF8 but declaring in the parameters as UTF8. Of course that is a fault by my own, but it should tel me about it.
-----Ursprüngliche Nachricht-----
Von: Dag.Wanvik@Sun.COM [mailto:Dag.Wanvik@Sun.COM]
Gesendet: Freitag, 20. November 2009 19:54
An: Derby Discussion
Betreff: Re: AW: OOM with millions of weakly-referenced Derby objects
Malte.Kempff@de.equens.com writes:
> In other words I would count that situation as not nice behaviour of ij, since the return code should have been given as I would suppose it looking at the method signature.
Is this the API signature you are referring to?
http://db.apache.org/derby/docs/10.5/publishedapi/jdbc4/org/apache/derby/tools/ij.html#runScript(java.sql.Connection,%20java.io.InputStream,%20java.lang.String,%20java.io.OutputStream,%20java.lang.String)
It says:
Returns:
Number of SQLExceptions thrown during the execution, -1 if not
known.
If so, are you seeing 0 or -1 returned here?
Dag
Re: AW: OOM with millions of weakly-referenced Derby objects
Posted by "Dag H. Wanvik" <Da...@Sun.COM>.
Malte.Kempff@de.equens.com writes:
> In other words I would count that situation as not nice behaviour of ij, since the return code should have been given as I would suppose it looking at the method signature.
Is this the API signature you are referring to?
http://db.apache.org/derby/docs/10.5/publishedapi/jdbc4/org/apache/derby/tools/ij.html#runScript(java.sql.Connection,%20java.io.InputStream,%20java.lang.String,%20java.io.OutputStream,%20java.lang.String)
It says:
Returns:
Number of SQLExceptions thrown during the execution, -1 if not
known.
If so, are you seeing 0 or -1 returned here?
Dag
AW: OOM with millions of weakly-referenced Derby objects
Posted by Ma...@de.equens.com.
Hi to all again,
I could analyse and solve the problem.
Originally I thought that if ij findes something phony it would give a return code as its signature suggests it.
Actually after finding out that the trouble stays somehow within ij after minutes of processing and that I have to set another IO-Stream and put it out I found exceptions-traces like this
2009-11-20 09:43:20,232 DEBUG [main] de.equens.fed.converting.data.FieldMemory: IJ ERROR: IOException: null
2009-11-20 09:43:20,241 DEBUG [main] de.equens.fed.converting.data.FieldMemory: org.apache.derby.impl.tools.ij.ijException: IOException: null
2009-11-20 09:43:20,251 DEBUG [main] de.equens.fed.converting.data.FieldMemory: at org.apache.derby.impl.tools.ij.ijException.iOException(Unknown Source)
2009-11-20 09:43:20,257 DEBUG [main] de.equens.fed.converting.data.FieldMemory: at org.apache.derby.impl.tools.ij.StatementFinder.peekChar(Unknown Source)
2009-11-20 09:43:20,264 DEBUG [main] de.equens.fed.converting.data.FieldMemory: at org.apache.derby.impl.tools.ij.StatementFinder.nextStatement(Unknown Source)
2009-11-20 09:43:20,270 DEBUG [main] de.equens.fed.converting.data.FieldMemory: at org.apache.derby.impl.tools.ij.utilMain.runScriptGuts(Unknown Source)
2009-11-20 09:43:20,285 DEBUG [main] de.equens.fed.converting.data.FieldMemory: at org.apache.derby.impl.tools.ij.utilMain.goScript(Unknown Source)
2009-11-20 09:43:20,293 DEBUG [main] de.equens.fed.converting.data.FieldMemory: at org.apache.derby.tools.ij.runScript(Unknown Source)
2009-11-20 09:43:20,302 DEBUG [main] de.equens.fed.converting.data.FieldMemory: at de.equens.fed.converting.data.FieldMemory.runScript(FieldMemory.java:506)
2009-11-20 09:43:20,311 DEBUG [main] de.equens.fed.converting.data.FieldMemory: at de.equens.fed.converting.data.FieldMemory.reorganize(FieldMemory.java:397)
2009-11-20 09:43:20,320 DEBUG [main] de.equens.fed.converting.data.FieldMemory: at de.equens.fed.converting.pacs2swift.Pacs08ToMT103Converter.<init>(Pacs08ToMT103Converter.java:53)
2009-11-20 09:43:20,330 DEBUG [main] de.equens.fed.converting.data.FieldMemory: at java.lang.Class.newInstanceImpl(Native Method)
2009-11-20 09:43:20,338 DEBUG [main] de.equens.fed.converting.data.FieldMemory: at java.lang.Class.newInstance(Class.java:1243)
2009-11-20 09:43:20,344 DEBUG [main] de.equens.fed.converting.data.FieldMemory: at de.equens.fed.converting.pacs2swift.PacsToMT103ConvFactory.newInstance(PacsToMT103ConvFactory.java:22)
2009-11-20 09:43:20,348 DEBUG [main] de.equens.fed.converting.data.FieldMemory: at de.equens.fed.converting.pacs2swift.PacsToMT103Handler.startElement(PacsToMT103Handler.java:102)
2009-11-20 09:43:20,352 DEBUG [main] de.equens.fed.converting.data.FieldMemory: at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown Source)
2009-11-20 09:43:20,357 DEBUG [main] de.equens.fed.converting.data.FieldMemory: at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source)
2009-11-20 09:43:20,361 DEBUG [main] de.equens.fed.converting.data.FieldMemory: at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
2009-11-20 09:43:20,366 DEBUG [main] de.equens.fed.converting.data.FieldMemory: at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
2009-11-20 09:43:20,370 DEBUG [main] de.equens.fed.converting.data.FieldMemory: at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
2009-11-20 09:43:20,374 DEBUG [main] de.equens.fed.converting.data.FieldMemory: at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
2009-11-20 09:43:20,377 DEBUG [main] de.equens.fed.converting.data.FieldMemory: at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
2009-11-20 09:43:20,380 DEBUG [main] de.equens.fed.converting.data.FieldMemory: at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
2009-11-20 09:43:20,383 DEBUG [main] de.equens.fed.converting.data.FieldMemory: at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
2009-11-20 09:43:20,385 DEBUG [main] de.equens.fed.converting.data.FieldMemory: at javax.xml.parsers.SAXParser.parse(Unknown Source)
2009-11-20 09:43:20,387 DEBUG [main] de.equens.fed.converting.data.FieldMemory: at de.equens.fed.converting.Pacs_MT103.run(Pacs_MT103.java:76)
2009-11-20 09:43:20,388 DEBUG [main] de.equens.fed.converting.data.FieldMemory: at de.equens.fed.converting.Pacs_MT103.main(Pacs_MT103.java:174)
That gave me the Idea that the sql-script I am trying to play on the database is probably not UTF-8.
And .. voilà ... after converting this script into UTF-8 all works fine, finally.
There are 2 things to learn about.
1. IJ seems not work with problems as I would aspect it as normal user, intuitively there should be either an exception or a returncode that tells you that something is going wrong instead of producing 5 mb ouput for a 3 kb script in my case.
2. here is obviously an os-specific java behaviour (or may be its just the behaviour of the different JVM) on how to read text files since the very same sql-script worked on Solaris and Windows with a Sun-JVM (JDK 5) that seems to be a bit more tolerant.
In other words I would count that situation as not nice behaviour of ij, since the return code should have been given as I would suppose it looking at the method signature.
________________________________
Von: Malte.Kempff@de.equens.com [mailto:Malte.Kempff@de.equens.com]
Gesendet: Donnerstag, 19. November 2009 13:26
An: derby-user@db.apache.org
Betreff: AW: OOM with millions of weakly-referenced Derby objects
Hi to all,
I have somehow a problem using derby on AIX 5.3
Derby is used as embedded DB by a little program. The database is constructed when it is not there using the specific driver property. That seems to work. Then I am using ij to create the tables etc if the first statement crashes because of having not table. This strategy works perfectly on Windows and Sun-Solaris as well, but now I am facing trouble on an AIX-machine.
Following program method uses ij
private void runScript(String fileName) throws FileNotFoundException, UnsupportedEncodingException, IOException, SQLException
{
BufferedInputStream iStream = null;
PrintStream pStream = null;
try
{
FileInputStream fileStream = new FileInputStream(fileName);
iStream = new BufferedInputStream(fileStream);
ByteArrayOutputStream bOut = new ByteArrayOutputStream(512);
pStream = new PrintStream(bOut);
logger.info("running Script '"+fileName+"'");
int result = ij.runScript(m_conn, iStream, "UTF-8", pStream, "UTF-8");
logger.debug("Result code is: " + result);
pStream.flush();
if (result!=0)
{
logger.error(bOut.toString());
throw new RuntimeException("problems running "+fileName+" rc:"+result+"\n see logs for more information");
}
}
finally
{
if (pStream!=null)
{
pStream.close();
}
if (iStream!=null)
{
iStream.close();
}
}
}
Obviously concerning the logs ij is called as in this line
int result = ij.runScript(m_conn, iStream, "UTF-8", pStream, "UTF-8");
after several minutes the program seems to crash without any exception I could see the derby.log got following entries (I usually expect
but does not show any of ij as it does when I use it on Window or Solaris)
following JVM is used on our AIX 5.3:
$ /usr/java5_64/bin/java -version
java version "1.5.0"
Java(TM) 2 Runtime Environment, Standard Edition (build pap64dev-20061003a (SR3))
IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 AIX ppc64-64 j9vmap6423-20061003 (JIT enabled)
J9VM - 20060915_08260_BHdSMr
JIT - 20060908_1811_r8
GC - 20060906_AA)
JCL - 20061003
Questions:
1. have derby and ij ever run successfully on AIX (e.g. 5.3) with Java5 (64 bit)?
2. is there a possibility to activate logs from ij so we can better see what is happening?
3. Did anybody face a similar problem and maybe got a workaround?
Thanks in advance
Malte
AW: OOM with millions of weakly-referenced Derby objects
Posted by Ma...@de.equens.com.
Hi to all,
I have somehow a problem using derby on AIX 5.3
Derby is used as embedded DB by a little program. The database is
constructed when it is not there using the specific driver property.
That seems to work. Then I am using ij to create the tables etc if the
first statement crashes because of having not table. This strategy works
perfectly on Windows and Sun-Solaris as well, but now I am facing
trouble on an AIX-machine.
Following program method uses ij
private void runScript(String fileName) throws FileNotFoundException,
UnsupportedEncodingException, IOException, SQLException
{
BufferedInputStream iStream = null;
PrintStream pStream = null;
try
{
FileInputStream fileStream = new
FileInputStream(fileName);
iStream = new
BufferedInputStream(fileStream);
ByteArrayOutputStream bOut = new
ByteArrayOutputStream(512);
pStream = new PrintStream(bOut);
logger.info("running Script '"+fileName+"'");
int result = ij.runScript(m_conn, iStream, "UTF-8", pStream,
"UTF-8");
logger.debug("Result code is: " + result);
pStream.flush();
if (result!=0)
{
logger.error(bOut.toString());
throw new RuntimeException("problems running
"+fileName+" rc:"+result+"\n see logs for more information");
}
}
finally
{
if (pStream!=null)
{
pStream.close();
}
if (iStream!=null)
{
iStream.close();
}
}
}
Obviously concerning the logs ij is called as in this line
int result = ij.runScript(m_conn, iStream, "UTF-8", pStream, "UTF-8");
after several minutes the program seems to crash without any exception I
could see the derby.log got following entries (I usually expect
but does not show any of ij as it does when I use it on Window or
Solaris)
following JVM is used on our AIX 5.3:
$ /usr/java5_64/bin/java -version
java version "1.5.0"
Java(TM) 2 Runtime Environment, Standard Edition (build
pap64dev-20061003a (SR3))
IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 AIX ppc64-64
j9vmap6423-20061003 (JIT enabled)
J9VM - 20060915_08260_BHdSMr
JIT - 20060908_1811_r8
GC - 20060906_AA)
JCL - 20061003
Questions:
1. have derby and ij ever run successfully on AIX (e.g. 5.3) with
Java5 (64 bit)?
2. is there a possibility to activate logs from ij so we can better
see what is happening?
3. Did anybody face a similar problem and maybe got a workaround?
Thanks in advance
Malte
RE: OOM with millions of weakly-referenced Derby objects
Posted by Eric Floehr <Er...@3x.com>.
Kristian,
The query is approximately this:
SELECT * FROM t WHERE (some criteria) ORDER BY y;
'y' is defined as a foreign key so has an index generated. Could it be that Derby is bringing the rows internal to do the sorting?
As for maximum page cache size, it’s the default. Is there a good reference for Derby performance tuning, because I sure could use some help with that. It is currently being run as an embedded server with vm parameters -Xmx256m.
I will attempt running the command via ODBC directly, with the FETCH syntax, and produce a heap histogram (is there anything in particular you are looking for or a specific tool output -- I generally use jprofiler).
Thanks so much for your help!
Eric
-----Original Message-----
From: Kristian.Waagan@Sun.COM [mailto:Kristian.Waagan@Sun.COM]
Sent: Wednesday, November 18, 2009 2:50 AM
To: Derby Discussion
Cc: user-java@ibatis.apache.org
Subject: Re: OOM with millions of weakly-referenced Derby objects
Larry Meadors wrote:
> The expected behavior is really dependent upon the underlying JDBC driver.
>
> Using the maxRows parameter is a last resort option, IMO.
>
> Limit the results with the SQL if at all possible, here's an example:
>
> http://old.nabble.com/top-N-reporting-with-derby-td17221545.html
>
Hi,
Derby now supports the OFFSET/FETCH clause. See [1] for some examples,
or consult the reference manual. I think some work is also being done on
improving the ROW_NUMBER implementation.
Regarding your OOM, that is generally not to be expected. However, it is
hard to say if it's a Derby bug without knowing what is going on in your
application, and how Derby is configured.
For instance, what is your maximum page cache size set to, and what size
is the heap of the JVM?
Posting the heap histogram might give us some clues.
Have you tried reproducing this without iBATIS?
Regards,
--
Kristian
[1] http://blogs.sun.com/kah/entry/derby_10_5_preview_fetch
> Larry
>
>
> On Tue, Nov 17, 2009 at 11:42 AM, Eric Floehr <Er...@3x.com> wrote:
>
>> All,
>>
>>
>>
>> I have a Derby table with up to a million rows. Some large subset of those
>> rows may be returned by a SELECT query.
>>
>>
>>
>> I am using IBATIS quertyForList with the embedded Derby driver. I am using
>> the version which has a maxRows parameter. I call that method and receive
>> back the proper “maxRows” java.sql.ResultSet objects. For example, the
>> select would match 100,000 rows, but I only get the first 200. That all
>> works fine.
>>
>>
>>
>> However, I am getting OOM’s when the select would match nearly a million
>> rows and after profiling with JProfiler the culprit seems to be millions of
>> weakly referenced org.apache.derby.iapi.types.SQL objects (SQLLongInt, for
>> example) that appear to be created far faster than they can be GC’ed. This
>> particular performance test run would return a million records, of which I
>> am only wanting to retrieve the first 200.
>>
>>
>>
>> My questions are:
>>
>> 1. Is this expected behavior?
>>
>> 2. Is IBATIS properly using a cursor, or what is the right way to
>> limit the number of Derby objects created (the ResultSet count is correct)
>>
>> 3. Are there Derby or IBATIS parameters that can be employed to
>> correct this behavior?
>>
>>
>>
>> Derby version is 10.5.3, IBATIS version is 2.3.4.726, and Java version is
>> 1.6. I have cross-posted to both the Derby and IBATIS lists since I don’t
>> know where the root problem resides.
>>
>>
>>
>> Thanks so much for your help!
>>
>>
>>
>> Best Regards,
>>
>> Eric
>>
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
For additional commands, e-mail: user-java-help@ibatis.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
For additional commands, e-mail: user-java-help@ibatis.apache.org
RE: OOM with millions of weakly-referenced Derby objects
Posted by Eric Floehr <Er...@3x.com>.
Kristian,
The query is approximately this:
SELECT * FROM t WHERE (some criteria) ORDER BY y;
'y' is defined as a foreign key so has an index generated. Could it be that Derby is bringing the rows internal to do the sorting?
As for maximum page cache size, it’s the default. Is there a good reference for Derby performance tuning, because I sure could use some help with that. It is currently being run as an embedded server with vm parameters -Xmx256m.
I will attempt running the command via ODBC directly, with the FETCH syntax, and produce a heap histogram (is there anything in particular you are looking for or a specific tool output -- I generally use jprofiler).
Thanks so much for your help!
Eric
-----Original Message-----
From: Kristian.Waagan@Sun.COM [mailto:Kristian.Waagan@Sun.COM]
Sent: Wednesday, November 18, 2009 2:50 AM
To: Derby Discussion
Cc: user-java@ibatis.apache.org
Subject: Re: OOM with millions of weakly-referenced Derby objects
Larry Meadors wrote:
> The expected behavior is really dependent upon the underlying JDBC driver.
>
> Using the maxRows parameter is a last resort option, IMO.
>
> Limit the results with the SQL if at all possible, here's an example:
>
> http://old.nabble.com/top-N-reporting-with-derby-td17221545.html
>
Hi,
Derby now supports the OFFSET/FETCH clause. See [1] for some examples,
or consult the reference manual. I think some work is also being done on
improving the ROW_NUMBER implementation.
Regarding your OOM, that is generally not to be expected. However, it is
hard to say if it's a Derby bug without knowing what is going on in your
application, and how Derby is configured.
For instance, what is your maximum page cache size set to, and what size
is the heap of the JVM?
Posting the heap histogram might give us some clues.
Have you tried reproducing this without iBATIS?
Regards,
--
Kristian
[1] http://blogs.sun.com/kah/entry/derby_10_5_preview_fetch
> Larry
>
>
> On Tue, Nov 17, 2009 at 11:42 AM, Eric Floehr <Er...@3x.com> wrote:
>
>> All,
>>
>>
>>
>> I have a Derby table with up to a million rows. Some large subset of those
>> rows may be returned by a SELECT query.
>>
>>
>>
>> I am using IBATIS quertyForList with the embedded Derby driver. I am using
>> the version which has a maxRows parameter. I call that method and receive
>> back the proper “maxRows” java.sql.ResultSet objects. For example, the
>> select would match 100,000 rows, but I only get the first 200. That all
>> works fine.
>>
>>
>>
>> However, I am getting OOM’s when the select would match nearly a million
>> rows and after profiling with JProfiler the culprit seems to be millions of
>> weakly referenced org.apache.derby.iapi.types.SQL objects (SQLLongInt, for
>> example) that appear to be created far faster than they can be GC’ed. This
>> particular performance test run would return a million records, of which I
>> am only wanting to retrieve the first 200.
>>
>>
>>
>> My questions are:
>>
>> 1. Is this expected behavior?
>>
>> 2. Is IBATIS properly using a cursor, or what is the right way to
>> limit the number of Derby objects created (the ResultSet count is correct)
>>
>> 3. Are there Derby or IBATIS parameters that can be employed to
>> correct this behavior?
>>
>>
>>
>> Derby version is 10.5.3, IBATIS version is 2.3.4.726, and Java version is
>> 1.6. I have cross-posted to both the Derby and IBATIS lists since I don’t
>> know where the root problem resides.
>>
>>
>>
>> Thanks so much for your help!
>>
>>
>>
>> Best Regards,
>>
>> Eric
>>
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
For additional commands, e-mail: user-java-help@ibatis.apache.org
Re: OOM with millions of weakly-referenced Derby objects
Posted by Kristian Waagan <Kr...@Sun.COM>.
Larry Meadors wrote:
> The expected behavior is really dependent upon the underlying JDBC driver.
>
> Using the maxRows parameter is a last resort option, IMO.
>
> Limit the results with the SQL if at all possible, here's an example:
>
> http://old.nabble.com/top-N-reporting-with-derby-td17221545.html
>
Hi,
Derby now supports the OFFSET/FETCH clause. See [1] for some examples,
or consult the reference manual. I think some work is also being done on
improving the ROW_NUMBER implementation.
Regarding your OOM, that is generally not to be expected. However, it is
hard to say if it's a Derby bug without knowing what is going on in your
application, and how Derby is configured.
For instance, what is your maximum page cache size set to, and what size
is the heap of the JVM?
Posting the heap histogram might give us some clues.
Have you tried reproducing this without iBATIS?
Regards,
--
Kristian
[1] http://blogs.sun.com/kah/entry/derby_10_5_preview_fetch
> Larry
>
>
> On Tue, Nov 17, 2009 at 11:42 AM, Eric Floehr <Er...@3x.com> wrote:
>
>> All,
>>
>>
>>
>> I have a Derby table with up to a million rows. Some large subset of those
>> rows may be returned by a SELECT query.
>>
>>
>>
>> I am using IBATIS quertyForList with the embedded Derby driver. I am using
>> the version which has a maxRows parameter. I call that method and receive
>> back the proper “maxRows” java.sql.ResultSet objects. For example, the
>> select would match 100,000 rows, but I only get the first 200. That all
>> works fine.
>>
>>
>>
>> However, I am getting OOM’s when the select would match nearly a million
>> rows and after profiling with JProfiler the culprit seems to be millions of
>> weakly referenced org.apache.derby.iapi.types.SQL objects (SQLLongInt, for
>> example) that appear to be created far faster than they can be GC’ed. This
>> particular performance test run would return a million records, of which I
>> am only wanting to retrieve the first 200.
>>
>>
>>
>> My questions are:
>>
>> 1. Is this expected behavior?
>>
>> 2. Is IBATIS properly using a cursor, or what is the right way to
>> limit the number of Derby objects created (the ResultSet count is correct)
>>
>> 3. Are there Derby or IBATIS parameters that can be employed to
>> correct this behavior?
>>
>>
>>
>> Derby version is 10.5.3, IBATIS version is 2.3.4.726, and Java version is
>> 1.6. I have cross-posted to both the Derby and IBATIS lists since I don’t
>> know where the root problem resides.
>>
>>
>>
>> Thanks so much for your help!
>>
>>
>>
>> Best Regards,
>>
>> Eric
>>
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
For additional commands, e-mail: user-java-help@ibatis.apache.org
Re: OOM with millions of weakly-referenced Derby objects
Posted by Kristian Waagan <Kr...@Sun.COM>.
Larry Meadors wrote:
> The expected behavior is really dependent upon the underlying JDBC driver.
>
> Using the maxRows parameter is a last resort option, IMO.
>
> Limit the results with the SQL if at all possible, here's an example:
>
> http://old.nabble.com/top-N-reporting-with-derby-td17221545.html
>
Hi,
Derby now supports the OFFSET/FETCH clause. See [1] for some examples,
or consult the reference manual. I think some work is also being done on
improving the ROW_NUMBER implementation.
Regarding your OOM, that is generally not to be expected. However, it is
hard to say if it's a Derby bug without knowing what is going on in your
application, and how Derby is configured.
For instance, what is your maximum page cache size set to, and what size
is the heap of the JVM?
Posting the heap histogram might give us some clues.
Have you tried reproducing this without iBATIS?
Regards,
--
Kristian
[1] http://blogs.sun.com/kah/entry/derby_10_5_preview_fetch
> Larry
>
>
> On Tue, Nov 17, 2009 at 11:42 AM, Eric Floehr <Er...@3x.com> wrote:
>
>> All,
>>
>>
>>
>> I have a Derby table with up to a million rows. Some large subset of those
>> rows may be returned by a SELECT query.
>>
>>
>>
>> I am using IBATIS quertyForList with the embedded Derby driver. I am using
>> the version which has a maxRows parameter. I call that method and receive
>> back the proper “maxRows” java.sql.ResultSet objects. For example, the
>> select would match 100,000 rows, but I only get the first 200. That all
>> works fine.
>>
>>
>>
>> However, I am getting OOM’s when the select would match nearly a million
>> rows and after profiling with JProfiler the culprit seems to be millions of
>> weakly referenced org.apache.derby.iapi.types.SQL objects (SQLLongInt, for
>> example) that appear to be created far faster than they can be GC’ed. This
>> particular performance test run would return a million records, of which I
>> am only wanting to retrieve the first 200.
>>
>>
>>
>> My questions are:
>>
>> 1. Is this expected behavior?
>>
>> 2. Is IBATIS properly using a cursor, or what is the right way to
>> limit the number of Derby objects created (the ResultSet count is correct)
>>
>> 3. Are there Derby or IBATIS parameters that can be employed to
>> correct this behavior?
>>
>>
>>
>> Derby version is 10.5.3, IBATIS version is 2.3.4.726, and Java version is
>> 1.6. I have cross-posted to both the Derby and IBATIS lists since I don’t
>> know where the root problem resides.
>>
>>
>>
>> Thanks so much for your help!
>>
>>
>>
>> Best Regards,
>>
>> Eric
>>
>>
>>
Re: OOM with millions of weakly-referenced Derby objects
Posted by Larry Meadors <la...@gmail.com>.
The expected behavior is really dependent upon the underlying JDBC driver.
Using the maxRows parameter is a last resort option, IMO.
Limit the results with the SQL if at all possible, here's an example:
http://old.nabble.com/top-N-reporting-with-derby-td17221545.html
Larry
On Tue, Nov 17, 2009 at 11:42 AM, Eric Floehr <Er...@3x.com> wrote:
> All,
>
>
>
> I have a Derby table with up to a million rows. Some large subset of those
> rows may be returned by a SELECT query.
>
>
>
> I am using IBATIS quertyForList with the embedded Derby driver. I am using
> the version which has a maxRows parameter. I call that method and receive
> back the proper “maxRows” java.sql.ResultSet objects. For example, the
> select would match 100,000 rows, but I only get the first 200. That all
> works fine.
>
>
>
> However, I am getting OOM’s when the select would match nearly a million
> rows and after profiling with JProfiler the culprit seems to be millions of
> weakly referenced org.apache.derby.iapi.types.SQL objects (SQLLongInt, for
> example) that appear to be created far faster than they can be GC’ed. This
> particular performance test run would return a million records, of which I
> am only wanting to retrieve the first 200.
>
>
>
> My questions are:
>
> 1. Is this expected behavior?
>
> 2. Is IBATIS properly using a cursor, or what is the right way to
> limit the number of Derby objects created (the ResultSet count is correct)
>
> 3. Are there Derby or IBATIS parameters that can be employed to
> correct this behavior?
>
>
>
> Derby version is 10.5.3, IBATIS version is 2.3.4.726, and Java version is
> 1.6. I have cross-posted to both the Derby and IBATIS lists since I don’t
> know where the root problem resides.
>
>
>
> Thanks so much for your help!
>
>
>
> Best Regards,
>
> Eric
>
>
Re: OOM with millions of weakly-referenced Derby objects
Posted by Larry Meadors <la...@gmail.com>.
The expected behavior is really dependent upon the underlying JDBC driver.
Using the maxRows parameter is a last resort option, IMO.
Limit the results with the SQL if at all possible, here's an example:
http://old.nabble.com/top-N-reporting-with-derby-td17221545.html
Larry
On Tue, Nov 17, 2009 at 11:42 AM, Eric Floehr <Er...@3x.com> wrote:
> All,
>
>
>
> I have a Derby table with up to a million rows. Some large subset of those
> rows may be returned by a SELECT query.
>
>
>
> I am using IBATIS quertyForList with the embedded Derby driver. I am using
> the version which has a maxRows parameter. I call that method and receive
> back the proper “maxRows” java.sql.ResultSet objects. For example, the
> select would match 100,000 rows, but I only get the first 200. That all
> works fine.
>
>
>
> However, I am getting OOM’s when the select would match nearly a million
> rows and after profiling with JProfiler the culprit seems to be millions of
> weakly referenced org.apache.derby.iapi.types.SQL objects (SQLLongInt, for
> example) that appear to be created far faster than they can be GC’ed. This
> particular performance test run would return a million records, of which I
> am only wanting to retrieve the first 200.
>
>
>
> My questions are:
>
> 1. Is this expected behavior?
>
> 2. Is IBATIS properly using a cursor, or what is the right way to
> limit the number of Derby objects created (the ResultSet count is correct)
>
> 3. Are there Derby or IBATIS parameters that can be employed to
> correct this behavior?
>
>
>
> Derby version is 10.5.3, IBATIS version is 2.3.4.726, and Java version is
> 1.6. I have cross-posted to both the Derby and IBATIS lists since I don’t
> know where the root problem resides.
>
>
>
> Thanks so much for your help!
>
>
>
> Best Regards,
>
> Eric
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
For additional commands, e-mail: user-java-help@ibatis.apache.org