You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user-java@ibatis.apache.org by "K. Arnold" <ak...@comcast.net> on 2009/09/14 21:47:43 UTC
ResultHandler - OutOfMemory Exception
I am trying to iterate over a result set of 2million records, for a large
bulk load and transformation into a new ODS. It appears that I am getting
an OutOfMemoryException because the DefaultResultSetHandler is caching the
object in the nestedResultObjects property. Is there some property I should
set or statement/ method call I should be using that will allow me to
process one line at a time and not have the nestedResultObjects store each
object?
My goal:
* Grab a row
* Send it to be processed
* Once processed, move on to the next row.
Note: Once a row is processed I no longer need a tie back to the object.
I have included the custom ResultHandler, the unit test and the
configuration file. Please let me know if you need other information.
package com.primetherapeutics.benplanmgr.entity.rxclaim;
import org.apache.ibatis.executor.result.ResultContext;
import org.apache.ibatis.executor.result.ResultHandler;
import org.apache.log4j.Logger;
/**
* @author kjarnold
*
*/
public class GroupEligibilityResultHandler implements ResultHandler {
Logger logger = Logger.getLogger(GroupEligibilityResultHandler.class);
int count = 0;
public void handleResult(ResultContext context) {
if(context.getResultObject() != null) {
count++;
logger.debug(count);
}
//context.stop();
}
public int getCount() {
return count;
}
}
@Test
public void getGroupElibibilitiesByResultHandler() {
Map<String, String> parameterMap = new HashMap<String, String>();
parameterMap.put("gelThruDate", "1090101");
parameterMap.put("addDate", "1090911");
parameterMap.put("chgDate", "1090911");
parameterMap.put("planDate", "1090101");
try {
GroupEligibilityResultHandler handler = new
GroupEligibilityResultHandler();
session.select("com.primetherapeutics.benplanmgr.entity.rxclaim.data.GroupEligibilityMapper.getGroupEligibilities",
parameterMap, handler);
logger.debug(handler.getCount());
} finally {
session.close();
}
}
Here are my mapping files:
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE configuration PUBLIC "-//ibatis.apache.org//DTD Config 3.0//EN"
"http://ibatis.apache.org/dtd/ibatis-3-config.dtd">
<configuration>
<settings>
<setting name="multipleResultSetsEnabled" value="false"/>
<setting name="defaultExecutorType" value="BATCH"/>
</settings>
<mappers>
<mapper
resource="com/primetherapeutics/benplanmgr/entity/rxclaim/data/BenefitMaxSchedule.xml"/>
<mapper
resource="com/primetherapeutics/benplanmgr/entity/rxclaim/data/GroupEligibility.xml"/>
</mappers>
</configuration>
--
View this message in context: http://www.nabble.com/ResultHandler---OutOfMemory-Exception-tp25442025p25442025.html
Sent from the iBATIS - User - Java mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
For additional commands, e-mail: user-java-help@ibatis.apache.org
Re: ResultHandler - OutOfMemory Exception
Posted by Clinton Begin <cl...@gmail.com>.
Okay, even though you didn't have a Jira ticket open for this, as an added
bonus of rewriting the entire result handler, there's now an optimized
implementation that should be able to handle any number of rows, as long as
you're not using nested result mapping (which makes it impossible to
avoid).
Clinton
On Tue, Sep 15, 2009 at 4:34 PM, Clinton Begin <cl...@gmail.com>wrote:
> Hmmm... I don't think so. I suppose it would make sense to add that ! :-)
> Jira ticket!
>
>
> On Tue, Sep 15, 2009 at 11:57 AM, K. Arnold <ak...@comcast.net> wrote:
>
>>
>> Out of curiosity, is it possible to create my own DefaultSetResultHandler?
>> I
>> wrote an plugin that would intercept the call, and I was going to walk
>> through the result set myself. Unfortunately I'm not sure how to get the
>> parameters that are used in the constructor. Looking at the code it seems
>> to jump through some hoops passing the objects through. Is there a
>> context
>> that contains the references to those that I could use?
>>
>>
>>
>> Clinton Begin wrote:
>> >
>> > Ah, then yes, iBATIS 3 might be more aggressive about using the nested
>> > objects cache...(even without join mapping), I can't remember if iBATIS
>> 2
>> > would behave the same way. Might be worth a try... if it works in 2,
>> then
>> > I'll revisit the design to see if I can get 3 to only cache if it
>> detects
>> > a
>> > nested result map.
>> > Clinton
>> >
>> > On Tue, Sep 15, 2009 at 10:44 AM, K. Arnold <ak...@comcast.net>
>> wrote:
>> >
>> >>
>> >> I have not used iBATIS 2 to do a comparison. It is very well that the
>> >> assumption given to me about how RowHandler works is false. I am in
>> the
>> >> process of trying to figure out a good limit size to pick utilizing an
>> >> AS400
>> >> jdbc driver working on a DB2 database.
>> >>
>> >> We have decided that utilizing the select statement with the limit is
>> the
>> >> better decision choice for us. Now we need to figure out how to
>> optimize
>> >> it.
>> >>
>> >> Any suggestions would be appreciated :ninja:
>> >>
>> >> Just to make sure I understand your reply for any future work. I don't
>> >> believe I am using a join mapper. I have a POJO that maps directly to
>> a
>> >> custom (yet extremely complicated) sql statement. The POJO doesn't
>> have
>> >> any
>> >> collections, and is basically stand alone. It is my understanding from
>> >> your
>> >> code, that my type of POJO isn't going to utilize the join mapper
>> method,
>> >> since the next POJO will be unique.
>> >>
>> >> Clinton Begin wrote:
>> >> >
>> >> > The difference between RowHandler and ResultHandler is only in name.
>> >> They
>> >> > did the same thing.
>> >> > I'd be interested to know if iBATIS 2.x works for you in this regard.
>> >> As
>> >> > far
>> >> > as I can recall, 2.x had the same design if you invoked the join
>> >> mapper...
>> >> >
>> >> > See, as soon as you try to use a JOIN to load a complex object graph,
>> >> we
>> >> > need to cache the results so that iBATIS knows to append to the
>> parent,
>> >> > rather than instantiate a new object.
>> >> >
>> >> > Clinton
>> >> >
>> >> > On Tue, Sep 15, 2009 at 9:04 AM, K. Arnold <ak...@comcast.net>
>> >> wrote:
>> >> >
>> >> >>
>> >> >> Thank you for the reply. Is it possible to make the join mapping
>> >> cache
>> >> a
>> >> >> configurable attribute on the mapping file? This is my first time
>> >> using
>> >> >> Ibatis and I was confidently informed that we could walk row by row
>> >> over
>> >> >> a
>> >> >> result set. I was told to look up the RowHandler interface. It is
>> my
>> >> >> understanding that in Ibatis 3 that the ResultHandler has replaced
>> the
>> >> >> RowHandler. Was there a specific design decision made not to allow
>> a
>> >> row
>> >> >> by
>> >> >> row walk through?
>> >> >>
>> >> >> As a work around I was looking into the Plugin feature. It looks
>> like
>> >> I
>> >> >> can
>> >> >> Intercept the resultSetsHandler method, and write my own code to
>> >> process
>> >> >> it.
>> >> >> In theory I could utilize the DefaultResultSetHandler code and
>> remove
>> >> the
>> >> >> cache feature in my implementation. However I am not sure if I have
>> >> >> access
>> >> >> to the same constructor parameters at the point of the intercept.
>> >> >>
>> >> >> There is some concern around the performance of also introducing
>> >> multiple
>> >> >> call backs to the Database for each "set" of data to process. I
>> need
>> >> to
>> >> >> balance the design choice of writing my own resultSetsHandler with
>> the
>> >> >> number of sql calls being made to the database via an offset Select
>> >> call.
>> >> >>
>> >> >> I appreciate your replies to this subject.
>> >> >>
>> >> >> Clinton Begin wrote:
>> >> >> >
>> >> >> > The nestedResultObjects is necessary for join mapping. One way to
>> >> deal
>> >> >> > with
>> >> >> > this though, is to use batches of reads as well as writes. Use
>> the
>> >> >> > pagination facilities and possibly even the proprietary
>> offset/limit
>> >> >> > features of your database to grab subsets of the results.
>> >> >> > Incidentally I'm rewriting the DefaultResultSetHandler to be
>> easier
>> >> to
>> >> >> > understand. But I don't see the need for that cache going away
>> >> anytime
>> >> >> > soon...
>> >> >> >
>> >> >> > Clinton
>> >> >> >
>> >> >> > On Mon, Sep 14, 2009 at 1:47 PM, K. Arnold <ak...@comcast.net>
>> >> >> wrote:
>> >> >> >
>> >> >> >>
>> >> >> >> I am trying to iterate over a result set of 2million records, for
>> a
>> >> >> large
>> >> >> >> bulk load and transformation into a new ODS. It appears that I
>> am
>> >> >> >> getting
>> >> >> >> an OutOfMemoryException because the DefaultResultSetHandler is
>> >> caching
>> >> >> >> the
>> >> >> >> object in the nestedResultObjects property. Is there some
>> property
>> >> I
>> >> >> >> should
>> >> >> >> set or statement/ method call I should be using that will allow
>> me
>> >> to
>> >> >> >> process one line at a time and not have the nestedResultObjects
>> >> store
>> >> >> >> each
>> >> >> >> object?
>> >> >> >>
>> >> >> >> My goal:
>> >> >> >> * Grab a row
>> >> >> >> * Send it to be processed
>> >> >> >> * Once processed, move on to the next row.
>> >> >> >> Note: Once a row is processed I no longer need a tie back to the
>> >> >> object.
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> I have included the custom ResultHandler, the unit test and the
>> >> >> >> configuration file. Please let me know if you need other
>> >> information.
>> >> >> >>
>> >> >> >> package com.primetherapeutics.benplanmgr.entity.rxclaim;
>> >> >> >>
>> >> >> >> import org.apache.ibatis.executor.result.ResultContext;
>> >> >> >> import org.apache.ibatis.executor.result.ResultHandler;
>> >> >> >> import org.apache.log4j.Logger;
>> >> >> >>
>> >> >> >> /**
>> >> >> >> * @author kjarnold
>> >> >> >> *
>> >> >> >> */
>> >> >> >> public class GroupEligibilityResultHandler implements
>> ResultHandler
>> >> {
>> >> >> >> Logger logger =
>> >> >> >> Logger.getLogger(GroupEligibilityResultHandler.class);
>> >> >> >>
>> >> >> >> int count = 0;
>> >> >> >>
>> >> >> >> public void handleResult(ResultContext context) {
>> >> >> >> if(context.getResultObject() != null) {
>> >> >> >> count++;
>> >> >> >> logger.debug(count);
>> >> >> >> }
>> >> >> >> //context.stop();
>> >> >> >> }
>> >> >> >>
>> >> >> >> public int getCount() {
>> >> >> >> return count;
>> >> >> >> }
>> >> >> >>
>> >> >> >> }
>> >> >> >>
>> >> >> >> @Test
>> >> >> >> public void getGroupElibibilitiesByResultHandler() {
>> >> >> >> Map<String, String> parameterMap = new
>> >> HashMap<String,
>> >> >> >> String>();
>> >> >> >> parameterMap.put("gelThruDate", "1090101");
>> >> >> >> parameterMap.put("addDate", "1090911");
>> >> >> >> parameterMap.put("chgDate", "1090911");
>> >> >> >> parameterMap.put("planDate", "1090101");
>> >> >> >> try {
>> >> >> >> GroupEligibilityResultHandler handler =
>> new
>> >> >> >> GroupEligibilityResultHandler();
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >>
>> >>
>> session.select("com.primetherapeutics.benplanmgr.entity.rxclaim.data.GroupEligibilityMapper.getGroupEligibilities",
>> >> >> >> parameterMap, handler);
>> >> >> >> logger.debug(handler.getCount());
>> >> >> >>
>> >> >> >> } finally {
>> >> >> >> session.close();
>> >> >> >> }
>> >> >> >>
>> >> >> >> }
>> >> >> >>
>> >> >> >>
>> >> >> >> Here are my mapping files:
>> >> >> >>
>> >> >> >> <?xml version="1.0" encoding="UTF-8" ?>
>> >> >> >> <!DOCTYPE configuration PUBLIC "-//ibatis.apache.org//DTD Config
>> >> >> 3.0//EN"
>> >> >> >> "http://ibatis.apache.org/dtd/ibatis-3-config.dtd">
>> >> >> >> <configuration>
>> >> >> >> <settings>
>> >> >> >> <setting name="multipleResultSetsEnabled" value="false"/>
>> >> >> >> <setting name="defaultExecutorType" value="BATCH"/>
>> >> >> >> </settings>
>> >> >> >> <mappers>
>> >> >> >> <mapper
>> >> >> >>
>> >> >> >>
>> >> >>
>> >>
>> resource="com/primetherapeutics/benplanmgr/entity/rxclaim/data/BenefitMaxSchedule.xml"/>
>> >> >> >> <mapper
>> >> >> >>
>> >> >> >>
>> >> >>
>> >>
>> resource="com/primetherapeutics/benplanmgr/entity/rxclaim/data/GroupEligibility.xml"/>
>> >> >> >> </mappers>
>> >> >> >> </configuration>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> --
>> >> >> >> View this message in context:
>> >> >> >>
>> >> >>
>> >>
>> http://www.nabble.com/ResultHandler---OutOfMemory-Exception-tp25442025p25442025.html
>> >> >> >> Sent from the iBATIS - User - Java mailing list archive at
>> >> Nabble.com.
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> ---------------------------------------------------------------------
>> >> >> >> To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
>> >> >> >> For additional commands, e-mail:
>> user-java-help@ibatis.apache.org
>> >> >> >>
>> >> >> >>
>> >> >> >
>> >> >> >
>> >> >>
>> >> >> --
>> >> >> View this message in context:
>> >> >>
>> >>
>> http://www.nabble.com/ResultHandler---OutOfMemory-Exception-tp25442025p25455710.html
>> >> >> Sent from the iBATIS - User - Java mailing list archive at
>> Nabble.com.
>> >> >>
>> >> >>
>> >> >>
>> ---------------------------------------------------------------------
>> >> >> To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
>> >> >> For additional commands, e-mail: user-java-help@ibatis.apache.org
>> >> >>
>> >> >>
>> >> >
>> >> >
>> >>
>> >> --
>> >> View this message in context:
>> >>
>> http://www.nabble.com/ResultHandler---OutOfMemory-Exception-tp25442025p25457565.html
>> >> Sent from the iBATIS - User - Java mailing list archive at Nabble.com.
>> >>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
>> >> For additional commands, e-mail: user-java-help@ibatis.apache.org
>> >>
>> >>
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/ResultHandler---OutOfMemory-Exception-tp25442025p25458817.html
>> Sent from the iBATIS - User - Java mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
>> For additional commands, e-mail: user-java-help@ibatis.apache.org
>>
>>
>
Re: ResultHandler - OutOfMemory Exception
Posted by Clinton Begin <cl...@gmail.com>.
Hmmm... I don't think so. I suppose it would make sense to add that ! :-)
Jira ticket!
On Tue, Sep 15, 2009 at 11:57 AM, K. Arnold <ak...@comcast.net> wrote:
>
> Out of curiosity, is it possible to create my own DefaultSetResultHandler?
> I
> wrote an plugin that would intercept the call, and I was going to walk
> through the result set myself. Unfortunately I'm not sure how to get the
> parameters that are used in the constructor. Looking at the code it seems
> to jump through some hoops passing the objects through. Is there a context
> that contains the references to those that I could use?
>
>
>
> Clinton Begin wrote:
> >
> > Ah, then yes, iBATIS 3 might be more aggressive about using the nested
> > objects cache...(even without join mapping), I can't remember if iBATIS 2
> > would behave the same way. Might be worth a try... if it works in 2,
> then
> > I'll revisit the design to see if I can get 3 to only cache if it detects
> > a
> > nested result map.
> > Clinton
> >
> > On Tue, Sep 15, 2009 at 10:44 AM, K. Arnold <ak...@comcast.net>
> wrote:
> >
> >>
> >> I have not used iBATIS 2 to do a comparison. It is very well that the
> >> assumption given to me about how RowHandler works is false. I am in the
> >> process of trying to figure out a good limit size to pick utilizing an
> >> AS400
> >> jdbc driver working on a DB2 database.
> >>
> >> We have decided that utilizing the select statement with the limit is
> the
> >> better decision choice for us. Now we need to figure out how to
> optimize
> >> it.
> >>
> >> Any suggestions would be appreciated :ninja:
> >>
> >> Just to make sure I understand your reply for any future work. I don't
> >> believe I am using a join mapper. I have a POJO that maps directly to a
> >> custom (yet extremely complicated) sql statement. The POJO doesn't have
> >> any
> >> collections, and is basically stand alone. It is my understanding from
> >> your
> >> code, that my type of POJO isn't going to utilize the join mapper
> method,
> >> since the next POJO will be unique.
> >>
> >> Clinton Begin wrote:
> >> >
> >> > The difference between RowHandler and ResultHandler is only in name.
> >> They
> >> > did the same thing.
> >> > I'd be interested to know if iBATIS 2.x works for you in this regard.
> >> As
> >> > far
> >> > as I can recall, 2.x had the same design if you invoked the join
> >> mapper...
> >> >
> >> > See, as soon as you try to use a JOIN to load a complex object graph,
> >> we
> >> > need to cache the results so that iBATIS knows to append to the
> parent,
> >> > rather than instantiate a new object.
> >> >
> >> > Clinton
> >> >
> >> > On Tue, Sep 15, 2009 at 9:04 AM, K. Arnold <ak...@comcast.net>
> >> wrote:
> >> >
> >> >>
> >> >> Thank you for the reply. Is it possible to make the join mapping
> >> cache
> >> a
> >> >> configurable attribute on the mapping file? This is my first time
> >> using
> >> >> Ibatis and I was confidently informed that we could walk row by row
> >> over
> >> >> a
> >> >> result set. I was told to look up the RowHandler interface. It is
> my
> >> >> understanding that in Ibatis 3 that the ResultHandler has replaced
> the
> >> >> RowHandler. Was there a specific design decision made not to allow a
> >> row
> >> >> by
> >> >> row walk through?
> >> >>
> >> >> As a work around I was looking into the Plugin feature. It looks
> like
> >> I
> >> >> can
> >> >> Intercept the resultSetsHandler method, and write my own code to
> >> process
> >> >> it.
> >> >> In theory I could utilize the DefaultResultSetHandler code and remove
> >> the
> >> >> cache feature in my implementation. However I am not sure if I have
> >> >> access
> >> >> to the same constructor parameters at the point of the intercept.
> >> >>
> >> >> There is some concern around the performance of also introducing
> >> multiple
> >> >> call backs to the Database for each "set" of data to process. I need
> >> to
> >> >> balance the design choice of writing my own resultSetsHandler with
> the
> >> >> number of sql calls being made to the database via an offset Select
> >> call.
> >> >>
> >> >> I appreciate your replies to this subject.
> >> >>
> >> >> Clinton Begin wrote:
> >> >> >
> >> >> > The nestedResultObjects is necessary for join mapping. One way to
> >> deal
> >> >> > with
> >> >> > this though, is to use batches of reads as well as writes. Use the
> >> >> > pagination facilities and possibly even the proprietary
> offset/limit
> >> >> > features of your database to grab subsets of the results.
> >> >> > Incidentally I'm rewriting the DefaultResultSetHandler to be easier
> >> to
> >> >> > understand. But I don't see the need for that cache going away
> >> anytime
> >> >> > soon...
> >> >> >
> >> >> > Clinton
> >> >> >
> >> >> > On Mon, Sep 14, 2009 at 1:47 PM, K. Arnold <ak...@comcast.net>
> >> >> wrote:
> >> >> >
> >> >> >>
> >> >> >> I am trying to iterate over a result set of 2million records, for
> a
> >> >> large
> >> >> >> bulk load and transformation into a new ODS. It appears that I am
> >> >> >> getting
> >> >> >> an OutOfMemoryException because the DefaultResultSetHandler is
> >> caching
> >> >> >> the
> >> >> >> object in the nestedResultObjects property. Is there some
> property
> >> I
> >> >> >> should
> >> >> >> set or statement/ method call I should be using that will allow me
> >> to
> >> >> >> process one line at a time and not have the nestedResultObjects
> >> store
> >> >> >> each
> >> >> >> object?
> >> >> >>
> >> >> >> My goal:
> >> >> >> * Grab a row
> >> >> >> * Send it to be processed
> >> >> >> * Once processed, move on to the next row.
> >> >> >> Note: Once a row is processed I no longer need a tie back to the
> >> >> object.
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> I have included the custom ResultHandler, the unit test and the
> >> >> >> configuration file. Please let me know if you need other
> >> information.
> >> >> >>
> >> >> >> package com.primetherapeutics.benplanmgr.entity.rxclaim;
> >> >> >>
> >> >> >> import org.apache.ibatis.executor.result.ResultContext;
> >> >> >> import org.apache.ibatis.executor.result.ResultHandler;
> >> >> >> import org.apache.log4j.Logger;
> >> >> >>
> >> >> >> /**
> >> >> >> * @author kjarnold
> >> >> >> *
> >> >> >> */
> >> >> >> public class GroupEligibilityResultHandler implements
> ResultHandler
> >> {
> >> >> >> Logger logger =
> >> >> >> Logger.getLogger(GroupEligibilityResultHandler.class);
> >> >> >>
> >> >> >> int count = 0;
> >> >> >>
> >> >> >> public void handleResult(ResultContext context) {
> >> >> >> if(context.getResultObject() != null) {
> >> >> >> count++;
> >> >> >> logger.debug(count);
> >> >> >> }
> >> >> >> //context.stop();
> >> >> >> }
> >> >> >>
> >> >> >> public int getCount() {
> >> >> >> return count;
> >> >> >> }
> >> >> >>
> >> >> >> }
> >> >> >>
> >> >> >> @Test
> >> >> >> public void getGroupElibibilitiesByResultHandler() {
> >> >> >> Map<String, String> parameterMap = new
> >> HashMap<String,
> >> >> >> String>();
> >> >> >> parameterMap.put("gelThruDate", "1090101");
> >> >> >> parameterMap.put("addDate", "1090911");
> >> >> >> parameterMap.put("chgDate", "1090911");
> >> >> >> parameterMap.put("planDate", "1090101");
> >> >> >> try {
> >> >> >> GroupEligibilityResultHandler handler = new
> >> >> >> GroupEligibilityResultHandler();
> >> >> >>
> >> >> >>
> >> >> >>
> >> >>
> >>
> session.select("com.primetherapeutics.benplanmgr.entity.rxclaim.data.GroupEligibilityMapper.getGroupEligibilities",
> >> >> >> parameterMap, handler);
> >> >> >> logger.debug(handler.getCount());
> >> >> >>
> >> >> >> } finally {
> >> >> >> session.close();
> >> >> >> }
> >> >> >>
> >> >> >> }
> >> >> >>
> >> >> >>
> >> >> >> Here are my mapping files:
> >> >> >>
> >> >> >> <?xml version="1.0" encoding="UTF-8" ?>
> >> >> >> <!DOCTYPE configuration PUBLIC "-//ibatis.apache.org//DTD Config
> >> >> 3.0//EN"
> >> >> >> "http://ibatis.apache.org/dtd/ibatis-3-config.dtd">
> >> >> >> <configuration>
> >> >> >> <settings>
> >> >> >> <setting name="multipleResultSetsEnabled" value="false"/>
> >> >> >> <setting name="defaultExecutorType" value="BATCH"/>
> >> >> >> </settings>
> >> >> >> <mappers>
> >> >> >> <mapper
> >> >> >>
> >> >> >>
> >> >>
> >>
> resource="com/primetherapeutics/benplanmgr/entity/rxclaim/data/BenefitMaxSchedule.xml"/>
> >> >> >> <mapper
> >> >> >>
> >> >> >>
> >> >>
> >>
> resource="com/primetherapeutics/benplanmgr/entity/rxclaim/data/GroupEligibility.xml"/>
> >> >> >> </mappers>
> >> >> >> </configuration>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >> View this message in context:
> >> >> >>
> >> >>
> >>
> http://www.nabble.com/ResultHandler---OutOfMemory-Exception-tp25442025p25442025.html
> >> >> >> Sent from the iBATIS - User - Java mailing list archive at
> >> Nabble.com.
> >> >> >>
> >> >> >>
> >> >> >>
> >> ---------------------------------------------------------------------
> >> >> >> To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
> >> >> >> For additional commands, e-mail: user-java-help@ibatis.apache.org
> >> >> >>
> >> >> >>
> >> >> >
> >> >> >
> >> >>
> >> >> --
> >> >> View this message in context:
> >> >>
> >>
> http://www.nabble.com/ResultHandler---OutOfMemory-Exception-tp25442025p25455710.html
> >> >> Sent from the iBATIS - User - Java mailing list archive at
> Nabble.com.
> >> >>
> >> >>
> >> >> ---------------------------------------------------------------------
> >> >> To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
> >> >> For additional commands, e-mail: user-java-help@ibatis.apache.org
> >> >>
> >> >>
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/ResultHandler---OutOfMemory-Exception-tp25442025p25457565.html
> >> Sent from the iBATIS - User - Java mailing list archive at Nabble.com.
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
> >> For additional commands, e-mail: user-java-help@ibatis.apache.org
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/ResultHandler---OutOfMemory-Exception-tp25442025p25458817.html
> Sent from the iBATIS - User - Java mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
> For additional commands, e-mail: user-java-help@ibatis.apache.org
>
>
Re: ResultHandler - OutOfMemory Exception
Posted by "K. Arnold" <ak...@comcast.net>.
Out of curiosity, is it possible to create my own DefaultSetResultHandler? I
wrote an plugin that would intercept the call, and I was going to walk
through the result set myself. Unfortunately I'm not sure how to get the
parameters that are used in the constructor. Looking at the code it seems
to jump through some hoops passing the objects through. Is there a context
that contains the references to those that I could use?
Clinton Begin wrote:
>
> Ah, then yes, iBATIS 3 might be more aggressive about using the nested
> objects cache...(even without join mapping), I can't remember if iBATIS 2
> would behave the same way. Might be worth a try... if it works in 2, then
> I'll revisit the design to see if I can get 3 to only cache if it detects
> a
> nested result map.
> Clinton
>
> On Tue, Sep 15, 2009 at 10:44 AM, K. Arnold <ak...@comcast.net> wrote:
>
>>
>> I have not used iBATIS 2 to do a comparison. It is very well that the
>> assumption given to me about how RowHandler works is false. I am in the
>> process of trying to figure out a good limit size to pick utilizing an
>> AS400
>> jdbc driver working on a DB2 database.
>>
>> We have decided that utilizing the select statement with the limit is the
>> better decision choice for us. Now we need to figure out how to optimize
>> it.
>>
>> Any suggestions would be appreciated :ninja:
>>
>> Just to make sure I understand your reply for any future work. I don't
>> believe I am using a join mapper. I have a POJO that maps directly to a
>> custom (yet extremely complicated) sql statement. The POJO doesn't have
>> any
>> collections, and is basically stand alone. It is my understanding from
>> your
>> code, that my type of POJO isn't going to utilize the join mapper method,
>> since the next POJO will be unique.
>>
>> Clinton Begin wrote:
>> >
>> > The difference between RowHandler and ResultHandler is only in name.
>> They
>> > did the same thing.
>> > I'd be interested to know if iBATIS 2.x works for you in this regard.
>> As
>> > far
>> > as I can recall, 2.x had the same design if you invoked the join
>> mapper...
>> >
>> > See, as soon as you try to use a JOIN to load a complex object graph,
>> we
>> > need to cache the results so that iBATIS knows to append to the parent,
>> > rather than instantiate a new object.
>> >
>> > Clinton
>> >
>> > On Tue, Sep 15, 2009 at 9:04 AM, K. Arnold <ak...@comcast.net>
>> wrote:
>> >
>> >>
>> >> Thank you for the reply. Is it possible to make the join mapping
>> cache
>> a
>> >> configurable attribute on the mapping file? This is my first time
>> using
>> >> Ibatis and I was confidently informed that we could walk row by row
>> over
>> >> a
>> >> result set. I was told to look up the RowHandler interface. It is my
>> >> understanding that in Ibatis 3 that the ResultHandler has replaced the
>> >> RowHandler. Was there a specific design decision made not to allow a
>> row
>> >> by
>> >> row walk through?
>> >>
>> >> As a work around I was looking into the Plugin feature. It looks like
>> I
>> >> can
>> >> Intercept the resultSetsHandler method, and write my own code to
>> process
>> >> it.
>> >> In theory I could utilize the DefaultResultSetHandler code and remove
>> the
>> >> cache feature in my implementation. However I am not sure if I have
>> >> access
>> >> to the same constructor parameters at the point of the intercept.
>> >>
>> >> There is some concern around the performance of also introducing
>> multiple
>> >> call backs to the Database for each "set" of data to process. I need
>> to
>> >> balance the design choice of writing my own resultSetsHandler with the
>> >> number of sql calls being made to the database via an offset Select
>> call.
>> >>
>> >> I appreciate your replies to this subject.
>> >>
>> >> Clinton Begin wrote:
>> >> >
>> >> > The nestedResultObjects is necessary for join mapping. One way to
>> deal
>> >> > with
>> >> > this though, is to use batches of reads as well as writes. Use the
>> >> > pagination facilities and possibly even the proprietary offset/limit
>> >> > features of your database to grab subsets of the results.
>> >> > Incidentally I'm rewriting the DefaultResultSetHandler to be easier
>> to
>> >> > understand. But I don't see the need for that cache going away
>> anytime
>> >> > soon...
>> >> >
>> >> > Clinton
>> >> >
>> >> > On Mon, Sep 14, 2009 at 1:47 PM, K. Arnold <ak...@comcast.net>
>> >> wrote:
>> >> >
>> >> >>
>> >> >> I am trying to iterate over a result set of 2million records, for a
>> >> large
>> >> >> bulk load and transformation into a new ODS. It appears that I am
>> >> >> getting
>> >> >> an OutOfMemoryException because the DefaultResultSetHandler is
>> caching
>> >> >> the
>> >> >> object in the nestedResultObjects property. Is there some property
>> I
>> >> >> should
>> >> >> set or statement/ method call I should be using that will allow me
>> to
>> >> >> process one line at a time and not have the nestedResultObjects
>> store
>> >> >> each
>> >> >> object?
>> >> >>
>> >> >> My goal:
>> >> >> * Grab a row
>> >> >> * Send it to be processed
>> >> >> * Once processed, move on to the next row.
>> >> >> Note: Once a row is processed I no longer need a tie back to the
>> >> object.
>> >> >>
>> >> >>
>> >> >>
>> >> >> I have included the custom ResultHandler, the unit test and the
>> >> >> configuration file. Please let me know if you need other
>> information.
>> >> >>
>> >> >> package com.primetherapeutics.benplanmgr.entity.rxclaim;
>> >> >>
>> >> >> import org.apache.ibatis.executor.result.ResultContext;
>> >> >> import org.apache.ibatis.executor.result.ResultHandler;
>> >> >> import org.apache.log4j.Logger;
>> >> >>
>> >> >> /**
>> >> >> * @author kjarnold
>> >> >> *
>> >> >> */
>> >> >> public class GroupEligibilityResultHandler implements ResultHandler
>> {
>> >> >> Logger logger =
>> >> >> Logger.getLogger(GroupEligibilityResultHandler.class);
>> >> >>
>> >> >> int count = 0;
>> >> >>
>> >> >> public void handleResult(ResultContext context) {
>> >> >> if(context.getResultObject() != null) {
>> >> >> count++;
>> >> >> logger.debug(count);
>> >> >> }
>> >> >> //context.stop();
>> >> >> }
>> >> >>
>> >> >> public int getCount() {
>> >> >> return count;
>> >> >> }
>> >> >>
>> >> >> }
>> >> >>
>> >> >> @Test
>> >> >> public void getGroupElibibilitiesByResultHandler() {
>> >> >> Map<String, String> parameterMap = new
>> HashMap<String,
>> >> >> String>();
>> >> >> parameterMap.put("gelThruDate", "1090101");
>> >> >> parameterMap.put("addDate", "1090911");
>> >> >> parameterMap.put("chgDate", "1090911");
>> >> >> parameterMap.put("planDate", "1090101");
>> >> >> try {
>> >> >> GroupEligibilityResultHandler handler = new
>> >> >> GroupEligibilityResultHandler();
>> >> >>
>> >> >>
>> >> >>
>> >>
>> session.select("com.primetherapeutics.benplanmgr.entity.rxclaim.data.GroupEligibilityMapper.getGroupEligibilities",
>> >> >> parameterMap, handler);
>> >> >> logger.debug(handler.getCount());
>> >> >>
>> >> >> } finally {
>> >> >> session.close();
>> >> >> }
>> >> >>
>> >> >> }
>> >> >>
>> >> >>
>> >> >> Here are my mapping files:
>> >> >>
>> >> >> <?xml version="1.0" encoding="UTF-8" ?>
>> >> >> <!DOCTYPE configuration PUBLIC "-//ibatis.apache.org//DTD Config
>> >> 3.0//EN"
>> >> >> "http://ibatis.apache.org/dtd/ibatis-3-config.dtd">
>> >> >> <configuration>
>> >> >> <settings>
>> >> >> <setting name="multipleResultSetsEnabled" value="false"/>
>> >> >> <setting name="defaultExecutorType" value="BATCH"/>
>> >> >> </settings>
>> >> >> <mappers>
>> >> >> <mapper
>> >> >>
>> >> >>
>> >>
>> resource="com/primetherapeutics/benplanmgr/entity/rxclaim/data/BenefitMaxSchedule.xml"/>
>> >> >> <mapper
>> >> >>
>> >> >>
>> >>
>> resource="com/primetherapeutics/benplanmgr/entity/rxclaim/data/GroupEligibility.xml"/>
>> >> >> </mappers>
>> >> >> </configuration>
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> View this message in context:
>> >> >>
>> >>
>> http://www.nabble.com/ResultHandler---OutOfMemory-Exception-tp25442025p25442025.html
>> >> >> Sent from the iBATIS - User - Java mailing list archive at
>> Nabble.com.
>> >> >>
>> >> >>
>> >> >>
>> ---------------------------------------------------------------------
>> >> >> To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
>> >> >> For additional commands, e-mail: user-java-help@ibatis.apache.org
>> >> >>
>> >> >>
>> >> >
>> >> >
>> >>
>> >> --
>> >> View this message in context:
>> >>
>> http://www.nabble.com/ResultHandler---OutOfMemory-Exception-tp25442025p25455710.html
>> >> Sent from the iBATIS - User - Java mailing list archive at Nabble.com.
>> >>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
>> >> For additional commands, e-mail: user-java-help@ibatis.apache.org
>> >>
>> >>
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/ResultHandler---OutOfMemory-Exception-tp25442025p25457565.html
>> Sent from the iBATIS - User - Java mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
>> For additional commands, e-mail: user-java-help@ibatis.apache.org
>>
>>
>
>
--
View this message in context: http://www.nabble.com/ResultHandler---OutOfMemory-Exception-tp25442025p25458817.html
Sent from the iBATIS - User - Java mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
For additional commands, e-mail: user-java-help@ibatis.apache.org
Re: ResultHandler - OutOfMemory Exception
Posted by Clinton Begin <cl...@gmail.com>.
Ah, then yes, iBATIS 3 might be more aggressive about using the nested
objects cache...(even without join mapping), I can't remember if iBATIS 2
would behave the same way. Might be worth a try... if it works in 2, then
I'll revisit the design to see if I can get 3 to only cache if it detects a
nested result map.
Clinton
On Tue, Sep 15, 2009 at 10:44 AM, K. Arnold <ak...@comcast.net> wrote:
>
> I have not used iBATIS 2 to do a comparison. It is very well that the
> assumption given to me about how RowHandler works is false. I am in the
> process of trying to figure out a good limit size to pick utilizing an
> AS400
> jdbc driver working on a DB2 database.
>
> We have decided that utilizing the select statement with the limit is the
> better decision choice for us. Now we need to figure out how to optimize
> it.
>
> Any suggestions would be appreciated :ninja:
>
> Just to make sure I understand your reply for any future work. I don't
> believe I am using a join mapper. I have a POJO that maps directly to a
> custom (yet extremely complicated) sql statement. The POJO doesn't have
> any
> collections, and is basically stand alone. It is my understanding from
> your
> code, that my type of POJO isn't going to utilize the join mapper method,
> since the next POJO will be unique.
>
> Clinton Begin wrote:
> >
> > The difference between RowHandler and ResultHandler is only in name.
> They
> > did the same thing.
> > I'd be interested to know if iBATIS 2.x works for you in this regard. As
> > far
> > as I can recall, 2.x had the same design if you invoked the join
> mapper...
> >
> > See, as soon as you try to use a JOIN to load a complex object graph, we
> > need to cache the results so that iBATIS knows to append to the parent,
> > rather than instantiate a new object.
> >
> > Clinton
> >
> > On Tue, Sep 15, 2009 at 9:04 AM, K. Arnold <ak...@comcast.net> wrote:
> >
> >>
> >> Thank you for the reply. Is it possible to make the join mapping cache
> a
> >> configurable attribute on the mapping file? This is my first time
> using
> >> Ibatis and I was confidently informed that we could walk row by row over
> >> a
> >> result set. I was told to look up the RowHandler interface. It is my
> >> understanding that in Ibatis 3 that the ResultHandler has replaced the
> >> RowHandler. Was there a specific design decision made not to allow a
> row
> >> by
> >> row walk through?
> >>
> >> As a work around I was looking into the Plugin feature. It looks like I
> >> can
> >> Intercept the resultSetsHandler method, and write my own code to process
> >> it.
> >> In theory I could utilize the DefaultResultSetHandler code and remove
> the
> >> cache feature in my implementation. However I am not sure if I have
> >> access
> >> to the same constructor parameters at the point of the intercept.
> >>
> >> There is some concern around the performance of also introducing
> multiple
> >> call backs to the Database for each "set" of data to process. I need to
> >> balance the design choice of writing my own resultSetsHandler with the
> >> number of sql calls being made to the database via an offset Select
> call.
> >>
> >> I appreciate your replies to this subject.
> >>
> >> Clinton Begin wrote:
> >> >
> >> > The nestedResultObjects is necessary for join mapping. One way to
> deal
> >> > with
> >> > this though, is to use batches of reads as well as writes. Use the
> >> > pagination facilities and possibly even the proprietary offset/limit
> >> > features of your database to grab subsets of the results.
> >> > Incidentally I'm rewriting the DefaultResultSetHandler to be easier to
> >> > understand. But I don't see the need for that cache going away
> anytime
> >> > soon...
> >> >
> >> > Clinton
> >> >
> >> > On Mon, Sep 14, 2009 at 1:47 PM, K. Arnold <ak...@comcast.net>
> >> wrote:
> >> >
> >> >>
> >> >> I am trying to iterate over a result set of 2million records, for a
> >> large
> >> >> bulk load and transformation into a new ODS. It appears that I am
> >> >> getting
> >> >> an OutOfMemoryException because the DefaultResultSetHandler is
> caching
> >> >> the
> >> >> object in the nestedResultObjects property. Is there some property I
> >> >> should
> >> >> set or statement/ method call I should be using that will allow me to
> >> >> process one line at a time and not have the nestedResultObjects store
> >> >> each
> >> >> object?
> >> >>
> >> >> My goal:
> >> >> * Grab a row
> >> >> * Send it to be processed
> >> >> * Once processed, move on to the next row.
> >> >> Note: Once a row is processed I no longer need a tie back to the
> >> object.
> >> >>
> >> >>
> >> >>
> >> >> I have included the custom ResultHandler, the unit test and the
> >> >> configuration file. Please let me know if you need other
> information.
> >> >>
> >> >> package com.primetherapeutics.benplanmgr.entity.rxclaim;
> >> >>
> >> >> import org.apache.ibatis.executor.result.ResultContext;
> >> >> import org.apache.ibatis.executor.result.ResultHandler;
> >> >> import org.apache.log4j.Logger;
> >> >>
> >> >> /**
> >> >> * @author kjarnold
> >> >> *
> >> >> */
> >> >> public class GroupEligibilityResultHandler implements ResultHandler {
> >> >> Logger logger =
> >> >> Logger.getLogger(GroupEligibilityResultHandler.class);
> >> >>
> >> >> int count = 0;
> >> >>
> >> >> public void handleResult(ResultContext context) {
> >> >> if(context.getResultObject() != null) {
> >> >> count++;
> >> >> logger.debug(count);
> >> >> }
> >> >> //context.stop();
> >> >> }
> >> >>
> >> >> public int getCount() {
> >> >> return count;
> >> >> }
> >> >>
> >> >> }
> >> >>
> >> >> @Test
> >> >> public void getGroupElibibilitiesByResultHandler() {
> >> >> Map<String, String> parameterMap = new HashMap<String,
> >> >> String>();
> >> >> parameterMap.put("gelThruDate", "1090101");
> >> >> parameterMap.put("addDate", "1090911");
> >> >> parameterMap.put("chgDate", "1090911");
> >> >> parameterMap.put("planDate", "1090101");
> >> >> try {
> >> >> GroupEligibilityResultHandler handler = new
> >> >> GroupEligibilityResultHandler();
> >> >>
> >> >>
> >> >>
> >>
> session.select("com.primetherapeutics.benplanmgr.entity.rxclaim.data.GroupEligibilityMapper.getGroupEligibilities",
> >> >> parameterMap, handler);
> >> >> logger.debug(handler.getCount());
> >> >>
> >> >> } finally {
> >> >> session.close();
> >> >> }
> >> >>
> >> >> }
> >> >>
> >> >>
> >> >> Here are my mapping files:
> >> >>
> >> >> <?xml version="1.0" encoding="UTF-8" ?>
> >> >> <!DOCTYPE configuration PUBLIC "-//ibatis.apache.org//DTD Config
> >> 3.0//EN"
> >> >> "http://ibatis.apache.org/dtd/ibatis-3-config.dtd">
> >> >> <configuration>
> >> >> <settings>
> >> >> <setting name="multipleResultSetsEnabled" value="false"/>
> >> >> <setting name="defaultExecutorType" value="BATCH"/>
> >> >> </settings>
> >> >> <mappers>
> >> >> <mapper
> >> >>
> >> >>
> >>
> resource="com/primetherapeutics/benplanmgr/entity/rxclaim/data/BenefitMaxSchedule.xml"/>
> >> >> <mapper
> >> >>
> >> >>
> >>
> resource="com/primetherapeutics/benplanmgr/entity/rxclaim/data/GroupEligibility.xml"/>
> >> >> </mappers>
> >> >> </configuration>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> View this message in context:
> >> >>
> >>
> http://www.nabble.com/ResultHandler---OutOfMemory-Exception-tp25442025p25442025.html
> >> >> Sent from the iBATIS - User - Java mailing list archive at
> Nabble.com.
> >> >>
> >> >>
> >> >> ---------------------------------------------------------------------
> >> >> To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
> >> >> For additional commands, e-mail: user-java-help@ibatis.apache.org
> >> >>
> >> >>
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/ResultHandler---OutOfMemory-Exception-tp25442025p25455710.html
> >> Sent from the iBATIS - User - Java mailing list archive at Nabble.com.
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
> >> For additional commands, e-mail: user-java-help@ibatis.apache.org
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/ResultHandler---OutOfMemory-Exception-tp25442025p25457565.html
> Sent from the iBATIS - User - Java mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
> For additional commands, e-mail: user-java-help@ibatis.apache.org
>
>
Re: ResultHandler - OutOfMemory Exception
Posted by "K. Arnold" <ak...@comcast.net>.
I have not used iBATIS 2 to do a comparison. It is very well that the
assumption given to me about how RowHandler works is false. I am in the
process of trying to figure out a good limit size to pick utilizing an AS400
jdbc driver working on a DB2 database.
We have decided that utilizing the select statement with the limit is the
better decision choice for us. Now we need to figure out how to optimize
it.
Any suggestions would be appreciated :ninja:
Just to make sure I understand your reply for any future work. I don't
believe I am using a join mapper. I have a POJO that maps directly to a
custom (yet extremely complicated) sql statement. The POJO doesn't have any
collections, and is basically stand alone. It is my understanding from your
code, that my type of POJO isn't going to utilize the join mapper method,
since the next POJO will be unique.
Clinton Begin wrote:
>
> The difference between RowHandler and ResultHandler is only in name. They
> did the same thing.
> I'd be interested to know if iBATIS 2.x works for you in this regard. As
> far
> as I can recall, 2.x had the same design if you invoked the join mapper...
>
> See, as soon as you try to use a JOIN to load a complex object graph, we
> need to cache the results so that iBATIS knows to append to the parent,
> rather than instantiate a new object.
>
> Clinton
>
> On Tue, Sep 15, 2009 at 9:04 AM, K. Arnold <ak...@comcast.net> wrote:
>
>>
>> Thank you for the reply. Is it possible to make the join mapping cache a
>> configurable attribute on the mapping file? This is my first time using
>> Ibatis and I was confidently informed that we could walk row by row over
>> a
>> result set. I was told to look up the RowHandler interface. It is my
>> understanding that in Ibatis 3 that the ResultHandler has replaced the
>> RowHandler. Was there a specific design decision made not to allow a row
>> by
>> row walk through?
>>
>> As a work around I was looking into the Plugin feature. It looks like I
>> can
>> Intercept the resultSetsHandler method, and write my own code to process
>> it.
>> In theory I could utilize the DefaultResultSetHandler code and remove the
>> cache feature in my implementation. However I am not sure if I have
>> access
>> to the same constructor parameters at the point of the intercept.
>>
>> There is some concern around the performance of also introducing multiple
>> call backs to the Database for each "set" of data to process. I need to
>> balance the design choice of writing my own resultSetsHandler with the
>> number of sql calls being made to the database via an offset Select call.
>>
>> I appreciate your replies to this subject.
>>
>> Clinton Begin wrote:
>> >
>> > The nestedResultObjects is necessary for join mapping. One way to deal
>> > with
>> > this though, is to use batches of reads as well as writes. Use the
>> > pagination facilities and possibly even the proprietary offset/limit
>> > features of your database to grab subsets of the results.
>> > Incidentally I'm rewriting the DefaultResultSetHandler to be easier to
>> > understand. But I don't see the need for that cache going away anytime
>> > soon...
>> >
>> > Clinton
>> >
>> > On Mon, Sep 14, 2009 at 1:47 PM, K. Arnold <ak...@comcast.net>
>> wrote:
>> >
>> >>
>> >> I am trying to iterate over a result set of 2million records, for a
>> large
>> >> bulk load and transformation into a new ODS. It appears that I am
>> >> getting
>> >> an OutOfMemoryException because the DefaultResultSetHandler is caching
>> >> the
>> >> object in the nestedResultObjects property. Is there some property I
>> >> should
>> >> set or statement/ method call I should be using that will allow me to
>> >> process one line at a time and not have the nestedResultObjects store
>> >> each
>> >> object?
>> >>
>> >> My goal:
>> >> * Grab a row
>> >> * Send it to be processed
>> >> * Once processed, move on to the next row.
>> >> Note: Once a row is processed I no longer need a tie back to the
>> object.
>> >>
>> >>
>> >>
>> >> I have included the custom ResultHandler, the unit test and the
>> >> configuration file. Please let me know if you need other information.
>> >>
>> >> package com.primetherapeutics.benplanmgr.entity.rxclaim;
>> >>
>> >> import org.apache.ibatis.executor.result.ResultContext;
>> >> import org.apache.ibatis.executor.result.ResultHandler;
>> >> import org.apache.log4j.Logger;
>> >>
>> >> /**
>> >> * @author kjarnold
>> >> *
>> >> */
>> >> public class GroupEligibilityResultHandler implements ResultHandler {
>> >> Logger logger =
>> >> Logger.getLogger(GroupEligibilityResultHandler.class);
>> >>
>> >> int count = 0;
>> >>
>> >> public void handleResult(ResultContext context) {
>> >> if(context.getResultObject() != null) {
>> >> count++;
>> >> logger.debug(count);
>> >> }
>> >> //context.stop();
>> >> }
>> >>
>> >> public int getCount() {
>> >> return count;
>> >> }
>> >>
>> >> }
>> >>
>> >> @Test
>> >> public void getGroupElibibilitiesByResultHandler() {
>> >> Map<String, String> parameterMap = new HashMap<String,
>> >> String>();
>> >> parameterMap.put("gelThruDate", "1090101");
>> >> parameterMap.put("addDate", "1090911");
>> >> parameterMap.put("chgDate", "1090911");
>> >> parameterMap.put("planDate", "1090101");
>> >> try {
>> >> GroupEligibilityResultHandler handler = new
>> >> GroupEligibilityResultHandler();
>> >>
>> >>
>> >>
>> session.select("com.primetherapeutics.benplanmgr.entity.rxclaim.data.GroupEligibilityMapper.getGroupEligibilities",
>> >> parameterMap, handler);
>> >> logger.debug(handler.getCount());
>> >>
>> >> } finally {
>> >> session.close();
>> >> }
>> >>
>> >> }
>> >>
>> >>
>> >> Here are my mapping files:
>> >>
>> >> <?xml version="1.0" encoding="UTF-8" ?>
>> >> <!DOCTYPE configuration PUBLIC "-//ibatis.apache.org//DTD Config
>> 3.0//EN"
>> >> "http://ibatis.apache.org/dtd/ibatis-3-config.dtd">
>> >> <configuration>
>> >> <settings>
>> >> <setting name="multipleResultSetsEnabled" value="false"/>
>> >> <setting name="defaultExecutorType" value="BATCH"/>
>> >> </settings>
>> >> <mappers>
>> >> <mapper
>> >>
>> >>
>> resource="com/primetherapeutics/benplanmgr/entity/rxclaim/data/BenefitMaxSchedule.xml"/>
>> >> <mapper
>> >>
>> >>
>> resource="com/primetherapeutics/benplanmgr/entity/rxclaim/data/GroupEligibility.xml"/>
>> >> </mappers>
>> >> </configuration>
>> >>
>> >>
>> >>
>> >> --
>> >> View this message in context:
>> >>
>> http://www.nabble.com/ResultHandler---OutOfMemory-Exception-tp25442025p25442025.html
>> >> Sent from the iBATIS - User - Java mailing list archive at Nabble.com.
>> >>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
>> >> For additional commands, e-mail: user-java-help@ibatis.apache.org
>> >>
>> >>
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/ResultHandler---OutOfMemory-Exception-tp25442025p25455710.html
>> Sent from the iBATIS - User - Java mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
>> For additional commands, e-mail: user-java-help@ibatis.apache.org
>>
>>
>
>
--
View this message in context: http://www.nabble.com/ResultHandler---OutOfMemory-Exception-tp25442025p25457565.html
Sent from the iBATIS - User - Java mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
For additional commands, e-mail: user-java-help@ibatis.apache.org
Re: ResultHandler - OutOfMemory Exception
Posted by Clinton Begin <cl...@gmail.com>.
The difference between RowHandler and ResultHandler is only in name. They
did the same thing.
I'd be interested to know if iBATIS 2.x works for you in this regard. As far
as I can recall, 2.x had the same design if you invoked the join mapper...
See, as soon as you try to use a JOIN to load a complex object graph, we
need to cache the results so that iBATIS knows to append to the parent,
rather than instantiate a new object.
Clinton
On Tue, Sep 15, 2009 at 9:04 AM, K. Arnold <ak...@comcast.net> wrote:
>
> Thank you for the reply. Is it possible to make the join mapping cache a
> configurable attribute on the mapping file? This is my first time using
> Ibatis and I was confidently informed that we could walk row by row over a
> result set. I was told to look up the RowHandler interface. It is my
> understanding that in Ibatis 3 that the ResultHandler has replaced the
> RowHandler. Was there a specific design decision made not to allow a row
> by
> row walk through?
>
> As a work around I was looking into the Plugin feature. It looks like I
> can
> Intercept the resultSetsHandler method, and write my own code to process
> it.
> In theory I could utilize the DefaultResultSetHandler code and remove the
> cache feature in my implementation. However I am not sure if I have access
> to the same constructor parameters at the point of the intercept.
>
> There is some concern around the performance of also introducing multiple
> call backs to the Database for each "set" of data to process. I need to
> balance the design choice of writing my own resultSetsHandler with the
> number of sql calls being made to the database via an offset Select call.
>
> I appreciate your replies to this subject.
>
> Clinton Begin wrote:
> >
> > The nestedResultObjects is necessary for join mapping. One way to deal
> > with
> > this though, is to use batches of reads as well as writes. Use the
> > pagination facilities and possibly even the proprietary offset/limit
> > features of your database to grab subsets of the results.
> > Incidentally I'm rewriting the DefaultResultSetHandler to be easier to
> > understand. But I don't see the need for that cache going away anytime
> > soon...
> >
> > Clinton
> >
> > On Mon, Sep 14, 2009 at 1:47 PM, K. Arnold <ak...@comcast.net> wrote:
> >
> >>
> >> I am trying to iterate over a result set of 2million records, for a
> large
> >> bulk load and transformation into a new ODS. It appears that I am
> >> getting
> >> an OutOfMemoryException because the DefaultResultSetHandler is caching
> >> the
> >> object in the nestedResultObjects property. Is there some property I
> >> should
> >> set or statement/ method call I should be using that will allow me to
> >> process one line at a time and not have the nestedResultObjects store
> >> each
> >> object?
> >>
> >> My goal:
> >> * Grab a row
> >> * Send it to be processed
> >> * Once processed, move on to the next row.
> >> Note: Once a row is processed I no longer need a tie back to the object.
> >>
> >>
> >>
> >> I have included the custom ResultHandler, the unit test and the
> >> configuration file. Please let me know if you need other information.
> >>
> >> package com.primetherapeutics.benplanmgr.entity.rxclaim;
> >>
> >> import org.apache.ibatis.executor.result.ResultContext;
> >> import org.apache.ibatis.executor.result.ResultHandler;
> >> import org.apache.log4j.Logger;
> >>
> >> /**
> >> * @author kjarnold
> >> *
> >> */
> >> public class GroupEligibilityResultHandler implements ResultHandler {
> >> Logger logger =
> >> Logger.getLogger(GroupEligibilityResultHandler.class);
> >>
> >> int count = 0;
> >>
> >> public void handleResult(ResultContext context) {
> >> if(context.getResultObject() != null) {
> >> count++;
> >> logger.debug(count);
> >> }
> >> //context.stop();
> >> }
> >>
> >> public int getCount() {
> >> return count;
> >> }
> >>
> >> }
> >>
> >> @Test
> >> public void getGroupElibibilitiesByResultHandler() {
> >> Map<String, String> parameterMap = new HashMap<String,
> >> String>();
> >> parameterMap.put("gelThruDate", "1090101");
> >> parameterMap.put("addDate", "1090911");
> >> parameterMap.put("chgDate", "1090911");
> >> parameterMap.put("planDate", "1090101");
> >> try {
> >> GroupEligibilityResultHandler handler = new
> >> GroupEligibilityResultHandler();
> >>
> >>
> >>
> session.select("com.primetherapeutics.benplanmgr.entity.rxclaim.data.GroupEligibilityMapper.getGroupEligibilities",
> >> parameterMap, handler);
> >> logger.debug(handler.getCount());
> >>
> >> } finally {
> >> session.close();
> >> }
> >>
> >> }
> >>
> >>
> >> Here are my mapping files:
> >>
> >> <?xml version="1.0" encoding="UTF-8" ?>
> >> <!DOCTYPE configuration PUBLIC "-//ibatis.apache.org//DTD Config
> 3.0//EN"
> >> "http://ibatis.apache.org/dtd/ibatis-3-config.dtd">
> >> <configuration>
> >> <settings>
> >> <setting name="multipleResultSetsEnabled" value="false"/>
> >> <setting name="defaultExecutorType" value="BATCH"/>
> >> </settings>
> >> <mappers>
> >> <mapper
> >>
> >>
> resource="com/primetherapeutics/benplanmgr/entity/rxclaim/data/BenefitMaxSchedule.xml"/>
> >> <mapper
> >>
> >>
> resource="com/primetherapeutics/benplanmgr/entity/rxclaim/data/GroupEligibility.xml"/>
> >> </mappers>
> >> </configuration>
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/ResultHandler---OutOfMemory-Exception-tp25442025p25442025.html
> >> Sent from the iBATIS - User - Java mailing list archive at Nabble.com.
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
> >> For additional commands, e-mail: user-java-help@ibatis.apache.org
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/ResultHandler---OutOfMemory-Exception-tp25442025p25455710.html
> Sent from the iBATIS - User - Java mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
> For additional commands, e-mail: user-java-help@ibatis.apache.org
>
>
Re: ResultHandler - OutOfMemory Exception
Posted by "K. Arnold" <ak...@comcast.net>.
Thank you for the reply. Is it possible to make the join mapping cache a
configurable attribute on the mapping file? This is my first time using
Ibatis and I was confidently informed that we could walk row by row over a
result set. I was told to look up the RowHandler interface. It is my
understanding that in Ibatis 3 that the ResultHandler has replaced the
RowHandler. Was there a specific design decision made not to allow a row by
row walk through?
As a work around I was looking into the Plugin feature. It looks like I can
Intercept the resultSetsHandler method, and write my own code to process it.
In theory I could utilize the DefaultResultSetHandler code and remove the
cache feature in my implementation. However I am not sure if I have access
to the same constructor parameters at the point of the intercept.
There is some concern around the performance of also introducing multiple
call backs to the Database for each "set" of data to process. I need to
balance the design choice of writing my own resultSetsHandler with the
number of sql calls being made to the database via an offset Select call.
I appreciate your replies to this subject.
Clinton Begin wrote:
>
> The nestedResultObjects is necessary for join mapping. One way to deal
> with
> this though, is to use batches of reads as well as writes. Use the
> pagination facilities and possibly even the proprietary offset/limit
> features of your database to grab subsets of the results.
> Incidentally I'm rewriting the DefaultResultSetHandler to be easier to
> understand. But I don't see the need for that cache going away anytime
> soon...
>
> Clinton
>
> On Mon, Sep 14, 2009 at 1:47 PM, K. Arnold <ak...@comcast.net> wrote:
>
>>
>> I am trying to iterate over a result set of 2million records, for a large
>> bulk load and transformation into a new ODS. It appears that I am
>> getting
>> an OutOfMemoryException because the DefaultResultSetHandler is caching
>> the
>> object in the nestedResultObjects property. Is there some property I
>> should
>> set or statement/ method call I should be using that will allow me to
>> process one line at a time and not have the nestedResultObjects store
>> each
>> object?
>>
>> My goal:
>> * Grab a row
>> * Send it to be processed
>> * Once processed, move on to the next row.
>> Note: Once a row is processed I no longer need a tie back to the object.
>>
>>
>>
>> I have included the custom ResultHandler, the unit test and the
>> configuration file. Please let me know if you need other information.
>>
>> package com.primetherapeutics.benplanmgr.entity.rxclaim;
>>
>> import org.apache.ibatis.executor.result.ResultContext;
>> import org.apache.ibatis.executor.result.ResultHandler;
>> import org.apache.log4j.Logger;
>>
>> /**
>> * @author kjarnold
>> *
>> */
>> public class GroupEligibilityResultHandler implements ResultHandler {
>> Logger logger =
>> Logger.getLogger(GroupEligibilityResultHandler.class);
>>
>> int count = 0;
>>
>> public void handleResult(ResultContext context) {
>> if(context.getResultObject() != null) {
>> count++;
>> logger.debug(count);
>> }
>> //context.stop();
>> }
>>
>> public int getCount() {
>> return count;
>> }
>>
>> }
>>
>> @Test
>> public void getGroupElibibilitiesByResultHandler() {
>> Map<String, String> parameterMap = new HashMap<String,
>> String>();
>> parameterMap.put("gelThruDate", "1090101");
>> parameterMap.put("addDate", "1090911");
>> parameterMap.put("chgDate", "1090911");
>> parameterMap.put("planDate", "1090101");
>> try {
>> GroupEligibilityResultHandler handler = new
>> GroupEligibilityResultHandler();
>>
>>
>> session.select("com.primetherapeutics.benplanmgr.entity.rxclaim.data.GroupEligibilityMapper.getGroupEligibilities",
>> parameterMap, handler);
>> logger.debug(handler.getCount());
>>
>> } finally {
>> session.close();
>> }
>>
>> }
>>
>>
>> Here are my mapping files:
>>
>> <?xml version="1.0" encoding="UTF-8" ?>
>> <!DOCTYPE configuration PUBLIC "-//ibatis.apache.org//DTD Config 3.0//EN"
>> "http://ibatis.apache.org/dtd/ibatis-3-config.dtd">
>> <configuration>
>> <settings>
>> <setting name="multipleResultSetsEnabled" value="false"/>
>> <setting name="defaultExecutorType" value="BATCH"/>
>> </settings>
>> <mappers>
>> <mapper
>>
>> resource="com/primetherapeutics/benplanmgr/entity/rxclaim/data/BenefitMaxSchedule.xml"/>
>> <mapper
>>
>> resource="com/primetherapeutics/benplanmgr/entity/rxclaim/data/GroupEligibility.xml"/>
>> </mappers>
>> </configuration>
>>
>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/ResultHandler---OutOfMemory-Exception-tp25442025p25442025.html
>> Sent from the iBATIS - User - Java mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
>> For additional commands, e-mail: user-java-help@ibatis.apache.org
>>
>>
>
>
--
View this message in context: http://www.nabble.com/ResultHandler---OutOfMemory-Exception-tp25442025p25455710.html
Sent from the iBATIS - User - Java mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
For additional commands, e-mail: user-java-help@ibatis.apache.org
Re: ResultHandler - OutOfMemory Exception
Posted by Clinton Begin <cl...@gmail.com>.
The nestedResultObjects is necessary for join mapping. One way to deal with
this though, is to use batches of reads as well as writes. Use the
pagination facilities and possibly even the proprietary offset/limit
features of your database to grab subsets of the results.
Incidentally I'm rewriting the DefaultResultSetHandler to be easier to
understand. But I don't see the need for that cache going away anytime
soon...
Clinton
On Mon, Sep 14, 2009 at 1:47 PM, K. Arnold <ak...@comcast.net> wrote:
>
> I am trying to iterate over a result set of 2million records, for a large
> bulk load and transformation into a new ODS. It appears that I am getting
> an OutOfMemoryException because the DefaultResultSetHandler is caching the
> object in the nestedResultObjects property. Is there some property I
> should
> set or statement/ method call I should be using that will allow me to
> process one line at a time and not have the nestedResultObjects store each
> object?
>
> My goal:
> * Grab a row
> * Send it to be processed
> * Once processed, move on to the next row.
> Note: Once a row is processed I no longer need a tie back to the object.
>
>
>
> I have included the custom ResultHandler, the unit test and the
> configuration file. Please let me know if you need other information.
>
> package com.primetherapeutics.benplanmgr.entity.rxclaim;
>
> import org.apache.ibatis.executor.result.ResultContext;
> import org.apache.ibatis.executor.result.ResultHandler;
> import org.apache.log4j.Logger;
>
> /**
> * @author kjarnold
> *
> */
> public class GroupEligibilityResultHandler implements ResultHandler {
> Logger logger =
> Logger.getLogger(GroupEligibilityResultHandler.class);
>
> int count = 0;
>
> public void handleResult(ResultContext context) {
> if(context.getResultObject() != null) {
> count++;
> logger.debug(count);
> }
> //context.stop();
> }
>
> public int getCount() {
> return count;
> }
>
> }
>
> @Test
> public void getGroupElibibilitiesByResultHandler() {
> Map<String, String> parameterMap = new HashMap<String,
> String>();
> parameterMap.put("gelThruDate", "1090101");
> parameterMap.put("addDate", "1090911");
> parameterMap.put("chgDate", "1090911");
> parameterMap.put("planDate", "1090101");
> try {
> GroupEligibilityResultHandler handler = new
> GroupEligibilityResultHandler();
>
>
> session.select("com.primetherapeutics.benplanmgr.entity.rxclaim.data.GroupEligibilityMapper.getGroupEligibilities",
> parameterMap, handler);
> logger.debug(handler.getCount());
>
> } finally {
> session.close();
> }
>
> }
>
>
> Here are my mapping files:
>
> <?xml version="1.0" encoding="UTF-8" ?>
> <!DOCTYPE configuration PUBLIC "-//ibatis.apache.org//DTD Config 3.0//EN"
> "http://ibatis.apache.org/dtd/ibatis-3-config.dtd">
> <configuration>
> <settings>
> <setting name="multipleResultSetsEnabled" value="false"/>
> <setting name="defaultExecutorType" value="BATCH"/>
> </settings>
> <mappers>
> <mapper
>
> resource="com/primetherapeutics/benplanmgr/entity/rxclaim/data/BenefitMaxSchedule.xml"/>
> <mapper
>
> resource="com/primetherapeutics/benplanmgr/entity/rxclaim/data/GroupEligibility.xml"/>
> </mappers>
> </configuration>
>
>
>
> --
> View this message in context:
> http://www.nabble.com/ResultHandler---OutOfMemory-Exception-tp25442025p25442025.html
> Sent from the iBATIS - User - Java mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
> For additional commands, e-mail: user-java-help@ibatis.apache.org
>
>