You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user-java@ibatis.apache.org by Dave Rafkind <da...@gmail.com> on 2010/01/14 18:02:44 UTC
ibatis 3 memory leak
Hi ibatis list, I'm new to ibatis so perhaps this is a noob question. I'm
using Ibatis 3 (ibatis-3-core-3.0.0.216.jar) with a somewhat complicated
schema (plenty of circular links etc).
I'm doing something like this:
List<MyIdObject> ids = session.selectList("getAll");
for (id : ids) {
MyObject o = session.select("getOne", id.getActualId());
}
The first query returns a list about 2k big, and the second query in the for
loop returns objects that are somewhat large (have several collections in
them, a discriminator, etc).
The problem I have is that as the for loop marches on it uses an
ever-increasing amount of memory. I would assuming that when the objects in
the body of the for loop go out of scope they can get garbage collected, but
apparently that never happens; is there some weird interaction with the
"first-level cache"? Should I be going about this a different way?
Thanks!
Dave
Re: ibatis 3 memory leak
Posted by Clinton Begin <cl...@gmail.com>.
That's exactly what should happen. Here's the catch: you should only
depend on lazy loading OR caching... not both. I would suggest caching.
Try disabling lazy loading and use one sqlSession instance.
Clinton
On Mon, Jan 18, 2010 at 9:37 AM, Dave Rafkind <da...@gmail.com>wrote:
> Unfortunately the data model is very thin (it's a database representation
> of a very large series of XSD schema documents), with "types" that have
> "properties", where each "type" has a "parent", and each "property" has a
> "type" as well (among other things).
>
> Big objects and N+1 selects don't bother me too much, but since a given
> "type" or "property" may be involved in several relationships I was hoping
> that information about them could be pulled from the cache instead of
> re-selected, during lazy-load joins.
>
>
> On Mon, Jan 18, 2010 at 11:10 AM, Clinton Begin <cl...@gmail.com>wrote:
>
>> Hmm... it looks to me like your situation is a perfect storm.
>>
>> * You're loading lots of objects.
>> * You're loading big objects.
>> * You are guaranteeing an N+1 selects problem with that for loop.
>>
>> No one of these situations should create a major problem, but all three
>> together are a guaranteed disappointment.
>>
>> That said, I'm surprised that 2000 rows would be that much of a problem...
>> how big are these objects really?
>>
>> Also, memory wise, there should be almost no difference between using one
>> session or multiple. And no, I would not open multiple as in your second
>> example.
>>
>> As for the serialization error, that might be a bug if LoadPair wasn't
>> serializable.
>>
>> Clinton
>>
>>
>> On Mon, Jan 18, 2010 at 8:24 AM, Dave Rafkind <da...@gmail.com>wrote:
>>
>>> Ok, that's some good information, I understand that you should marshall
>>> your objects with care. Unfortunately for me at some point I will have to
>>> marshall every single one of my database rows into an object graph. However
>>> what I took from your advice was not to be so stingy in re-using the session
>>> objects. So if I do something like this:
>>>
>>> for (id : ids) {
>>> SqlSession session2 = sessionMapper.openSession();
>>> MyObject o = session2.select("getOne", id.getActualId());
>>> session2.close();
>>> }
>>>
>>> Now memory usage is pretty good! However now performance is terrible. In
>>> attempting to rectify this I tried to put in a custom cache implementation,
>>> logging cache access to see how it was performing. I noticed it was never
>>> actually using the cache (never doing cache.putObject).
>>> I noticed that according to this discussion:
>>>
>>>
>>> http://markmail.org/search/?q=select+commit+cache+list%3Aorg.apache.ibatis.user-java#query:select%20commit%20cache%20list%3Aorg.apache.ibatis.user-java+page:1+mid:4mmwki3dnp57gweu+state:results
>>>
>>> ...commits don't occur (and thus cache fill doesn't occur) unless a
>>> transaction is committed. So for a pure read-only use case of the db the
>>> cache won't be very useful, right?
>>>
>>> Additionally, putting in a "session2.commit()" before the call to close()
>>> causes the following error (which I assume, but don't know is caused by
>>> committing when nothing actually needs to be comitted:)
>>>
>>> ### Error committing transaction. Cause:
>>> org.apache.ibatis.cache.CacheException: Error serializing object. Cause:
>>> java.io.NotSerializableException:
>>> org.apache.ibatis.executor.loader.ResultLoaderRegistry$LoadPair
>>>
>>>
>>>
>>> On Thu, Jan 14, 2010 at 2:44 PM, Clinton Begin <cl...@gmail.com>wrote:
>>>
>>>> By nested results, yes, I mean collections and associations. And by
>>>> "flattening" I mean avoiding the use of those.
>>>>
>>>> iBATIS exhibits this behavior, the same way any ORM would, because the
>>>> object instances need to be cached to preserve object identity. So as
>>>> you're result set is being read through, each unique object is stored.
>>>> iBATIS isn't quite as effcient as something like Hibernate can be with
>>>> circular references though, in that depending on how you map it out, you may
>>>> end up with multiple instances of the same data (parent/child relationships
>>>> mapped with resultMap).
>>>>
>>>> So with iBATIS, the most memory efficient approach is to use nested
>>>> select associations/collections. But for query performance, nested
>>>> resultMaps are ideal. I often find I need to use a combination of both to
>>>> get the best optimization. But if I often don't load lists of complete
>>>> objects either. If I'm loading a large list, I'll use a lighter weight
>>>> representation, and then only load the complete object graph for individual
>>>> instances. I've never found a case where this wasn't a good idea anyway.
>>>> Even when working with something like Rails, a rich domain ORM, I would
>>>> often write optimized lightweight queries for large lists of flatter
>>>> objects.
>>>>
>>>> The memory should not be significantly more than will ultimately be
>>>> required to store your final result set. And any additional memory used
>>>> should be released upon the closing of the SqlSession.
>>>>
>>>> If the memory isn't being released at the end of the session, that's a
>>>> different story... but otherwise, this is normal behavior.
>>>>
>>>> Clinton
>>>>
>>>>
>>>> On Thu, Jan 14, 2010 at 11:20 AM, Dave Rafkind <da...@gmail.com>wrote:
>>>>
>>>>> Thanks for the reply. What do you mean by nested result maps or
>>>>> selects? Do you mean collections or associations with their own selects and
>>>>> result maps? Why would ibatis exhibit this behavior in that case?
>>>>>
>>>>> And by flattening, you mean the same kind of stuff used to avoid the
>>>>> n+1 select problem?
>>>>>
>>>>> On Thu, Jan 14, 2010 at 1:02 PM, Clinton Begin <
>>>>> clinton.begin@gmail.com> wrote:
>>>>>
>>>>>> If it uses nested result maps or nested selects, I'm afraid you're out
>>>>>> of luck. You'll need to reduce the query results, or flatten out the
>>>>>> results.
>>>>>>
>>>>>> Clinton
>>>>>>
>>>>>> On 2010-01-14, Dave Rafkind <da...@gmail.com> wrote:
>>>>>> > Hi ibatis list, I'm new to ibatis so perhaps this is a noob
>>>>>> question. I'm
>>>>>> > using Ibatis 3 (ibatis-3-core-3.0.0.216.jar) with a somewhat
>>>>>> complicated
>>>>>> > schema (plenty of circular links etc).
>>>>>> >
>>>>>> > I'm doing something like this:
>>>>>> >
>>>>>> > List<MyIdObject> ids = session.selectList("getAll");
>>>>>> >
>>>>>> > for (id : ids) {
>>>>>> > MyObject o = session.select("getOne", id.getActualId());
>>>>>> > }
>>>>>> >
>>>>>> > The first query returns a list about 2k big, and the second query in
>>>>>> the for
>>>>>> > loop returns objects that are somewhat large (have several
>>>>>> collections in
>>>>>> > them, a discriminator, etc).
>>>>>> >
>>>>>> > The problem I have is that as the for loop marches on it uses an
>>>>>> > ever-increasing amount of memory. I would assuming that when the
>>>>>> objects in
>>>>>> > the body of the for loop go out of scope they can get garbage
>>>>>> collected, but
>>>>>> > apparently that never happens; is there some weird interaction with
>>>>>> the
>>>>>> > "first-level cache"? Should I be going about this a different way?
>>>>>> >
>>>>>> > Thanks!
>>>>>> > Dave
>>>>>> >
>>>>>>
>>>>>> --
>>>>>> Sent from my mobile device
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
>>>>>> For additional commands, e-mail: user-java-help@ibatis.apache.org
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: ibatis 3 memory leak
Posted by Dave Rafkind <da...@gmail.com>.
Unfortunately the data model is very thin (it's a database representation of
a very large series of XSD schema documents), with "types" that have
"properties", where each "type" has a "parent", and each "property" has a
"type" as well (among other things).
Big objects and N+1 selects don't bother me too much, but since a given
"type" or "property" may be involved in several relationships I was hoping
that information about them could be pulled from the cache instead of
re-selected, during lazy-load joins.
On Mon, Jan 18, 2010 at 11:10 AM, Clinton Begin <cl...@gmail.com>wrote:
> Hmm... it looks to me like your situation is a perfect storm.
>
> * You're loading lots of objects.
> * You're loading big objects.
> * You are guaranteeing an N+1 selects problem with that for loop.
>
> No one of these situations should create a major problem, but all three
> together are a guaranteed disappointment.
>
> That said, I'm surprised that 2000 rows would be that much of a problem...
> how big are these objects really?
>
> Also, memory wise, there should be almost no difference between using one
> session or multiple. And no, I would not open multiple as in your second
> example.
>
> As for the serialization error, that might be a bug if LoadPair wasn't
> serializable.
>
> Clinton
>
>
> On Mon, Jan 18, 2010 at 8:24 AM, Dave Rafkind <da...@gmail.com>wrote:
>
>> Ok, that's some good information, I understand that you should marshall
>> your objects with care. Unfortunately for me at some point I will have to
>> marshall every single one of my database rows into an object graph. However
>> what I took from your advice was not to be so stingy in re-using the session
>> objects. So if I do something like this:
>>
>> for (id : ids) {
>> SqlSession session2 = sessionMapper.openSession();
>> MyObject o = session2.select("getOne", id.getActualId());
>> session2.close();
>> }
>>
>> Now memory usage is pretty good! However now performance is terrible. In
>> attempting to rectify this I tried to put in a custom cache implementation,
>> logging cache access to see how it was performing. I noticed it was never
>> actually using the cache (never doing cache.putObject).
>> I noticed that according to this discussion:
>>
>>
>> http://markmail.org/search/?q=select+commit+cache+list%3Aorg.apache.ibatis.user-java#query:select%20commit%20cache%20list%3Aorg.apache.ibatis.user-java+page:1+mid:4mmwki3dnp57gweu+state:results
>>
>> ...commits don't occur (and thus cache fill doesn't occur) unless a
>> transaction is committed. So for a pure read-only use case of the db the
>> cache won't be very useful, right?
>>
>> Additionally, putting in a "session2.commit()" before the call to close()
>> causes the following error (which I assume, but don't know is caused by
>> committing when nothing actually needs to be comitted:)
>>
>> ### Error committing transaction. Cause:
>> org.apache.ibatis.cache.CacheException: Error serializing object. Cause:
>> java.io.NotSerializableException:
>> org.apache.ibatis.executor.loader.ResultLoaderRegistry$LoadPair
>>
>>
>>
>> On Thu, Jan 14, 2010 at 2:44 PM, Clinton Begin <cl...@gmail.com>wrote:
>>
>>> By nested results, yes, I mean collections and associations. And by
>>> "flattening" I mean avoiding the use of those.
>>>
>>> iBATIS exhibits this behavior, the same way any ORM would, because the
>>> object instances need to be cached to preserve object identity. So as
>>> you're result set is being read through, each unique object is stored.
>>> iBATIS isn't quite as effcient as something like Hibernate can be with
>>> circular references though, in that depending on how you map it out, you may
>>> end up with multiple instances of the same data (parent/child relationships
>>> mapped with resultMap).
>>>
>>> So with iBATIS, the most memory efficient approach is to use nested
>>> select associations/collections. But for query performance, nested
>>> resultMaps are ideal. I often find I need to use a combination of both to
>>> get the best optimization. But if I often don't load lists of complete
>>> objects either. If I'm loading a large list, I'll use a lighter weight
>>> representation, and then only load the complete object graph for individual
>>> instances. I've never found a case where this wasn't a good idea anyway.
>>> Even when working with something like Rails, a rich domain ORM, I would
>>> often write optimized lightweight queries for large lists of flatter
>>> objects.
>>>
>>> The memory should not be significantly more than will ultimately be
>>> required to store your final result set. And any additional memory used
>>> should be released upon the closing of the SqlSession.
>>>
>>> If the memory isn't being released at the end of the session, that's a
>>> different story... but otherwise, this is normal behavior.
>>>
>>> Clinton
>>>
>>>
>>> On Thu, Jan 14, 2010 at 11:20 AM, Dave Rafkind <da...@gmail.com>wrote:
>>>
>>>> Thanks for the reply. What do you mean by nested result maps or selects?
>>>> Do you mean collections or associations with their own selects and result
>>>> maps? Why would ibatis exhibit this behavior in that case?
>>>>
>>>> And by flattening, you mean the same kind of stuff used to avoid the
>>>> n+1 select problem?
>>>>
>>>> On Thu, Jan 14, 2010 at 1:02 PM, Clinton Begin <clinton.begin@gmail.com
>>>> > wrote:
>>>>
>>>>> If it uses nested result maps or nested selects, I'm afraid you're out
>>>>> of luck. You'll need to reduce the query results, or flatten out the
>>>>> results.
>>>>>
>>>>> Clinton
>>>>>
>>>>> On 2010-01-14, Dave Rafkind <da...@gmail.com> wrote:
>>>>> > Hi ibatis list, I'm new to ibatis so perhaps this is a noob question.
>>>>> I'm
>>>>> > using Ibatis 3 (ibatis-3-core-3.0.0.216.jar) with a somewhat
>>>>> complicated
>>>>> > schema (plenty of circular links etc).
>>>>> >
>>>>> > I'm doing something like this:
>>>>> >
>>>>> > List<MyIdObject> ids = session.selectList("getAll");
>>>>> >
>>>>> > for (id : ids) {
>>>>> > MyObject o = session.select("getOne", id.getActualId());
>>>>> > }
>>>>> >
>>>>> > The first query returns a list about 2k big, and the second query in
>>>>> the for
>>>>> > loop returns objects that are somewhat large (have several
>>>>> collections in
>>>>> > them, a discriminator, etc).
>>>>> >
>>>>> > The problem I have is that as the for loop marches on it uses an
>>>>> > ever-increasing amount of memory. I would assuming that when the
>>>>> objects in
>>>>> > the body of the for loop go out of scope they can get garbage
>>>>> collected, but
>>>>> > apparently that never happens; is there some weird interaction with
>>>>> the
>>>>> > "first-level cache"? Should I be going about this a different way?
>>>>> >
>>>>> > Thanks!
>>>>> > Dave
>>>>> >
>>>>>
>>>>> --
>>>>> Sent from my mobile device
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
>>>>> For additional commands, e-mail: user-java-help@ibatis.apache.org
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: ibatis 3 memory leak
Posted by Clinton Begin <cl...@gmail.com>.
Hmm... it looks to me like your situation is a perfect storm.
* You're loading lots of objects.
* You're loading big objects.
* You are guaranteeing an N+1 selects problem with that for loop.
No one of these situations should create a major problem, but all three
together are a guaranteed disappointment.
That said, I'm surprised that 2000 rows would be that much of a problem...
how big are these objects really?
Also, memory wise, there should be almost no difference between using one
session or multiple. And no, I would not open multiple as in your second
example.
As for the serialization error, that might be a bug if LoadPair wasn't
serializable.
Clinton
On Mon, Jan 18, 2010 at 8:24 AM, Dave Rafkind <da...@gmail.com>wrote:
> Ok, that's some good information, I understand that you should marshall
> your objects with care. Unfortunately for me at some point I will have to
> marshall every single one of my database rows into an object graph. However
> what I took from your advice was not to be so stingy in re-using the session
> objects. So if I do something like this:
>
> for (id : ids) {
> SqlSession session2 = sessionMapper.openSession();
> MyObject o = session2.select("getOne", id.getActualId());
> session2.close();
> }
>
> Now memory usage is pretty good! However now performance is terrible. In
> attempting to rectify this I tried to put in a custom cache implementation,
> logging cache access to see how it was performing. I noticed it was never
> actually using the cache (never doing cache.putObject).
> I noticed that according to this discussion:
>
>
> http://markmail.org/search/?q=select+commit+cache+list%3Aorg.apache.ibatis.user-java#query:select%20commit%20cache%20list%3Aorg.apache.ibatis.user-java+page:1+mid:4mmwki3dnp57gweu+state:results
>
> ...commits don't occur (and thus cache fill doesn't occur) unless a
> transaction is committed. So for a pure read-only use case of the db the
> cache won't be very useful, right?
>
> Additionally, putting in a "session2.commit()" before the call to close()
> causes the following error (which I assume, but don't know is caused by
> committing when nothing actually needs to be comitted:)
>
> ### Error committing transaction. Cause:
> org.apache.ibatis.cache.CacheException: Error serializing object. Cause:
> java.io.NotSerializableException:
> org.apache.ibatis.executor.loader.ResultLoaderRegistry$LoadPair
>
>
>
> On Thu, Jan 14, 2010 at 2:44 PM, Clinton Begin <cl...@gmail.com>wrote:
>
>> By nested results, yes, I mean collections and associations. And by
>> "flattening" I mean avoiding the use of those.
>>
>> iBATIS exhibits this behavior, the same way any ORM would, because the
>> object instances need to be cached to preserve object identity. So as
>> you're result set is being read through, each unique object is stored.
>> iBATIS isn't quite as effcient as something like Hibernate can be with
>> circular references though, in that depending on how you map it out, you may
>> end up with multiple instances of the same data (parent/child relationships
>> mapped with resultMap).
>>
>> So with iBATIS, the most memory efficient approach is to use nested select
>> associations/collections. But for query performance, nested resultMaps are
>> ideal. I often find I need to use a combination of both to get the best
>> optimization. But if I often don't load lists of complete objects either.
>> If I'm loading a large list, I'll use a lighter weight representation, and
>> then only load the complete object graph for individual instances. I've
>> never found a case where this wasn't a good idea anyway. Even when working
>> with something like Rails, a rich domain ORM, I would often write optimized
>> lightweight queries for large lists of flatter objects.
>>
>> The memory should not be significantly more than will ultimately be
>> required to store your final result set. And any additional memory used
>> should be released upon the closing of the SqlSession.
>>
>> If the memory isn't being released at the end of the session, that's a
>> different story... but otherwise, this is normal behavior.
>>
>> Clinton
>>
>>
>> On Thu, Jan 14, 2010 at 11:20 AM, Dave Rafkind <da...@gmail.com>wrote:
>>
>>> Thanks for the reply. What do you mean by nested result maps or selects?
>>> Do you mean collections or associations with their own selects and result
>>> maps? Why would ibatis exhibit this behavior in that case?
>>>
>>> And by flattening, you mean the same kind of stuff used to avoid the n+1
>>> select problem?
>>>
>>> On Thu, Jan 14, 2010 at 1:02 PM, Clinton Begin <cl...@gmail.com>wrote:
>>>
>>>> If it uses nested result maps or nested selects, I'm afraid you're out
>>>> of luck. You'll need to reduce the query results, or flatten out the
>>>> results.
>>>>
>>>> Clinton
>>>>
>>>> On 2010-01-14, Dave Rafkind <da...@gmail.com> wrote:
>>>> > Hi ibatis list, I'm new to ibatis so perhaps this is a noob question.
>>>> I'm
>>>> > using Ibatis 3 (ibatis-3-core-3.0.0.216.jar) with a somewhat
>>>> complicated
>>>> > schema (plenty of circular links etc).
>>>> >
>>>> > I'm doing something like this:
>>>> >
>>>> > List<MyIdObject> ids = session.selectList("getAll");
>>>> >
>>>> > for (id : ids) {
>>>> > MyObject o = session.select("getOne", id.getActualId());
>>>> > }
>>>> >
>>>> > The first query returns a list about 2k big, and the second query in
>>>> the for
>>>> > loop returns objects that are somewhat large (have several collections
>>>> in
>>>> > them, a discriminator, etc).
>>>> >
>>>> > The problem I have is that as the for loop marches on it uses an
>>>> > ever-increasing amount of memory. I would assuming that when the
>>>> objects in
>>>> > the body of the for loop go out of scope they can get garbage
>>>> collected, but
>>>> > apparently that never happens; is there some weird interaction with
>>>> the
>>>> > "first-level cache"? Should I be going about this a different way?
>>>> >
>>>> > Thanks!
>>>> > Dave
>>>> >
>>>>
>>>> --
>>>> Sent from my mobile device
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
>>>> For additional commands, e-mail: user-java-help@ibatis.apache.org
>>>>
>>>>
>>>
>>
>
Re: ibatis 3 memory leak
Posted by Dave Rafkind <da...@gmail.com>.
Ok, that's some good information, I understand that you should marshall your
objects with care. Unfortunately for me at some point I will have to
marshall every single one of my database rows into an object graph. However
what I took from your advice was not to be so stingy in re-using the session
objects. So if I do something like this:
for (id : ids) {
SqlSession session2 = sessionMapper.openSession();
MyObject o = session2.select("getOne", id.getActualId());
session2.close();
}
Now memory usage is pretty good! However now performance is terrible. In
attempting to rectify this I tried to put in a custom cache implementation,
logging cache access to see how it was performing. I noticed it was never
actually using the cache (never doing cache.putObject).
I noticed that according to this discussion:
http://markmail.org/search/?q=select+commit+cache+list%3Aorg.apache.ibatis.user-java#query:select%20commit%20cache%20list%3Aorg.apache.ibatis.user-java+page:1+mid:4mmwki3dnp57gweu+state:results
...commits don't occur (and thus cache fill doesn't occur) unless a
transaction is committed. So for a pure read-only use case of the db the
cache won't be very useful, right?
Additionally, putting in a "session2.commit()" before the call to close()
causes the following error (which I assume, but don't know is caused by
committing when nothing actually needs to be comitted:)
### Error committing transaction. Cause:
org.apache.ibatis.cache.CacheException: Error serializing object. Cause:
java.io.NotSerializableException:
org.apache.ibatis.executor.loader.ResultLoaderRegistry$LoadPair
On Thu, Jan 14, 2010 at 2:44 PM, Clinton Begin <cl...@gmail.com>wrote:
> By nested results, yes, I mean collections and associations. And by
> "flattening" I mean avoiding the use of those.
>
> iBATIS exhibits this behavior, the same way any ORM would, because the
> object instances need to be cached to preserve object identity. So as
> you're result set is being read through, each unique object is stored.
> iBATIS isn't quite as effcient as something like Hibernate can be with
> circular references though, in that depending on how you map it out, you may
> end up with multiple instances of the same data (parent/child relationships
> mapped with resultMap).
>
> So with iBATIS, the most memory efficient approach is to use nested select
> associations/collections. But for query performance, nested resultMaps are
> ideal. I often find I need to use a combination of both to get the best
> optimization. But if I often don't load lists of complete objects either.
> If I'm loading a large list, I'll use a lighter weight representation, and
> then only load the complete object graph for individual instances. I've
> never found a case where this wasn't a good idea anyway. Even when working
> with something like Rails, a rich domain ORM, I would often write optimized
> lightweight queries for large lists of flatter objects.
>
> The memory should not be significantly more than will ultimately be
> required to store your final result set. And any additional memory used
> should be released upon the closing of the SqlSession.
>
> If the memory isn't being released at the end of the session, that's a
> different story... but otherwise, this is normal behavior.
>
> Clinton
>
>
> On Thu, Jan 14, 2010 at 11:20 AM, Dave Rafkind <da...@gmail.com>wrote:
>
>> Thanks for the reply. What do you mean by nested result maps or selects?
>> Do you mean collections or associations with their own selects and result
>> maps? Why would ibatis exhibit this behavior in that case?
>>
>> And by flattening, you mean the same kind of stuff used to avoid the n+1
>> select problem?
>>
>> On Thu, Jan 14, 2010 at 1:02 PM, Clinton Begin <cl...@gmail.com>wrote:
>>
>>> If it uses nested result maps or nested selects, I'm afraid you're out
>>> of luck. You'll need to reduce the query results, or flatten out the
>>> results.
>>>
>>> Clinton
>>>
>>> On 2010-01-14, Dave Rafkind <da...@gmail.com> wrote:
>>> > Hi ibatis list, I'm new to ibatis so perhaps this is a noob question.
>>> I'm
>>> > using Ibatis 3 (ibatis-3-core-3.0.0.216.jar) with a somewhat
>>> complicated
>>> > schema (plenty of circular links etc).
>>> >
>>> > I'm doing something like this:
>>> >
>>> > List<MyIdObject> ids = session.selectList("getAll");
>>> >
>>> > for (id : ids) {
>>> > MyObject o = session.select("getOne", id.getActualId());
>>> > }
>>> >
>>> > The first query returns a list about 2k big, and the second query in
>>> the for
>>> > loop returns objects that are somewhat large (have several collections
>>> in
>>> > them, a discriminator, etc).
>>> >
>>> > The problem I have is that as the for loop marches on it uses an
>>> > ever-increasing amount of memory. I would assuming that when the
>>> objects in
>>> > the body of the for loop go out of scope they can get garbage
>>> collected, but
>>> > apparently that never happens; is there some weird interaction with the
>>> > "first-level cache"? Should I be going about this a different way?
>>> >
>>> > Thanks!
>>> > Dave
>>> >
>>>
>>> --
>>> Sent from my mobile device
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
>>> For additional commands, e-mail: user-java-help@ibatis.apache.org
>>>
>>>
>>
>
Re: ibatis 3 memory leak
Posted by Clinton Begin <cl...@gmail.com>.
By nested results, yes, I mean collections and associations. And by
"flattening" I mean avoiding the use of those.
iBATIS exhibits this behavior, the same way any ORM would, because the
object instances need to be cached to preserve object identity. So as
you're result set is being read through, each unique object is stored.
iBATIS isn't quite as effcient as something like Hibernate can be with
circular references though, in that depending on how you map it out, you may
end up with multiple instances of the same data (parent/child relationships
mapped with resultMap).
So with iBATIS, the most memory efficient approach is to use nested select
associations/collections. But for query performance, nested resultMaps are
ideal. I often find I need to use a combination of both to get the best
optimization. But if I often don't load lists of complete objects either.
If I'm loading a large list, I'll use a lighter weight representation, and
then only load the complete object graph for individual instances. I've
never found a case where this wasn't a good idea anyway. Even when working
with something like Rails, a rich domain ORM, I would often write optimized
lightweight queries for large lists of flatter objects.
The memory should not be significantly more than will ultimately be required
to store your final result set. And any additional memory used should be
released upon the closing of the SqlSession.
If the memory isn't being released at the end of the session, that's a
different story... but otherwise, this is normal behavior.
Clinton
On Thu, Jan 14, 2010 at 11:20 AM, Dave Rafkind <da...@gmail.com>wrote:
> Thanks for the reply. What do you mean by nested result maps or selects? Do
> you mean collections or associations with their own selects and result maps?
> Why would ibatis exhibit this behavior in that case?
>
> And by flattening, you mean the same kind of stuff used to avoid the n+1
> select problem?
>
> On Thu, Jan 14, 2010 at 1:02 PM, Clinton Begin <cl...@gmail.com>wrote:
>
>> If it uses nested result maps or nested selects, I'm afraid you're out
>> of luck. You'll need to reduce the query results, or flatten out the
>> results.
>>
>> Clinton
>>
>> On 2010-01-14, Dave Rafkind <da...@gmail.com> wrote:
>> > Hi ibatis list, I'm new to ibatis so perhaps this is a noob question.
>> I'm
>> > using Ibatis 3 (ibatis-3-core-3.0.0.216.jar) with a somewhat complicated
>> > schema (plenty of circular links etc).
>> >
>> > I'm doing something like this:
>> >
>> > List<MyIdObject> ids = session.selectList("getAll");
>> >
>> > for (id : ids) {
>> > MyObject o = session.select("getOne", id.getActualId());
>> > }
>> >
>> > The first query returns a list about 2k big, and the second query in the
>> for
>> > loop returns objects that are somewhat large (have several collections
>> in
>> > them, a discriminator, etc).
>> >
>> > The problem I have is that as the for loop marches on it uses an
>> > ever-increasing amount of memory. I would assuming that when the objects
>> in
>> > the body of the for loop go out of scope they can get garbage collected,
>> but
>> > apparently that never happens; is there some weird interaction with the
>> > "first-level cache"? Should I be going about this a different way?
>> >
>> > Thanks!
>> > Dave
>> >
>>
>> --
>> Sent from my mobile device
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
>> For additional commands, e-mail: user-java-help@ibatis.apache.org
>>
>>
>
Re: ibatis 3 memory leak
Posted by Dave Rafkind <da...@gmail.com>.
Thanks for the reply. What do you mean by nested result maps or selects? Do
you mean collections or associations with their own selects and result maps?
Why would ibatis exhibit this behavior in that case?
And by flattening, you mean the same kind of stuff used to avoid the n+1
select problem?
On Thu, Jan 14, 2010 at 1:02 PM, Clinton Begin <cl...@gmail.com>wrote:
> If it uses nested result maps or nested selects, I'm afraid you're out
> of luck. You'll need to reduce the query results, or flatten out the
> results.
>
> Clinton
>
> On 2010-01-14, Dave Rafkind <da...@gmail.com> wrote:
> > Hi ibatis list, I'm new to ibatis so perhaps this is a noob question. I'm
> > using Ibatis 3 (ibatis-3-core-3.0.0.216.jar) with a somewhat complicated
> > schema (plenty of circular links etc).
> >
> > I'm doing something like this:
> >
> > List<MyIdObject> ids = session.selectList("getAll");
> >
> > for (id : ids) {
> > MyObject o = session.select("getOne", id.getActualId());
> > }
> >
> > The first query returns a list about 2k big, and the second query in the
> for
> > loop returns objects that are somewhat large (have several collections in
> > them, a discriminator, etc).
> >
> > The problem I have is that as the for loop marches on it uses an
> > ever-increasing amount of memory. I would assuming that when the objects
> in
> > the body of the for loop go out of scope they can get garbage collected,
> but
> > apparently that never happens; is there some weird interaction with the
> > "first-level cache"? Should I be going about this a different way?
> >
> > Thanks!
> > Dave
> >
>
> --
> Sent from my mobile device
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
> For additional commands, e-mail: user-java-help@ibatis.apache.org
>
>
Re: ibatis 3 memory leak
Posted by Clinton Begin <cl...@gmail.com>.
If it uses nested result maps or nested selects, I'm afraid you're out
of luck. You'll need to reduce the query results, or flatten out the
results.
Clinton
On 2010-01-14, Dave Rafkind <da...@gmail.com> wrote:
> Hi ibatis list, I'm new to ibatis so perhaps this is a noob question. I'm
> using Ibatis 3 (ibatis-3-core-3.0.0.216.jar) with a somewhat complicated
> schema (plenty of circular links etc).
>
> I'm doing something like this:
>
> List<MyIdObject> ids = session.selectList("getAll");
>
> for (id : ids) {
> MyObject o = session.select("getOne", id.getActualId());
> }
>
> The first query returns a list about 2k big, and the second query in the for
> loop returns objects that are somewhat large (have several collections in
> them, a discriminator, etc).
>
> The problem I have is that as the for loop marches on it uses an
> ever-increasing amount of memory. I would assuming that when the objects in
> the body of the for loop go out of scope they can get garbage collected, but
> apparently that never happens; is there some weird interaction with the
> "first-level cache"? Should I be going about this a different way?
>
> Thanks!
> Dave
>
--
Sent from my mobile device
---------------------------------------------------------------------
To unsubscribe, e-mail: user-java-unsubscribe@ibatis.apache.org
For additional commands, e-mail: user-java-help@ibatis.apache.org