You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@openjpa.apache.org by Jeff Oh <oh...@gmail.com> on 2014/05/03 00:43:12 UTC
Advice Requested: Instance Loading bug in 2.3.0
Hello All,
We've recently upgraded to 2.3.0 (from 1.2.2) and have encountered a nasty
bug where when loading a complicated cyclic object graph using a fetch
group, some relationships are not being populated under some circumstances.
Unfortunately, the graph really is quite complicated, and while I have a
set of test data where the problem can be reliably duplicated, I haven't
been able to make an integration test do the same yet. A simplified
version of the graph is that A is 1:M to B and A is 1:M to C, B is M:1 to A
(Bidirectional), and C is M:1 to A and C is also M:1 to B. The net effect
is a graph where cycles can exist, although in practice they generally
don't other than the bidirectional relationship between A and B.
Our failing case is interesting. When we load single instances of B (call
them B1 and B2) one at a time the load is always successful, with all of
the A, B, and C relationships populated - but only if loaded one at a time.
In other words, if we run entityManager.find(B1) or
entityManager.find(B2), then all is well.
An ascii art version of the graph traversals might look something like this:
B1 -> A1 -> C1 -> B3 -> A3 -> B3 -> A3
-> B3-1 -> A3
-> B3-2 -> A3
-> C1-2 -> ...
....
B2 -> A2 -> C2 -> A3 -> B3 -> A3
-> B3-1 -> A3
-> B3-2 -> A3
-> C2-2 -> ....
...
where B is M:1 to A, A is 1:M to C and A is 1:M to B.
Interestingly, while B1 and B2 are separate objects, they do share several
common objects in their graphs - call them A3 and B3 (as well as B3-1 and
B3-2). If we run a query that loads both B1 and B2 in the same query -
entityManager.find(B1 + B2), then one of the relationships from one of the
other B objects in the graph (call it B3) B3->A is null (not populated),
where B3->A should == A3. To clarify, B1.A1.C1.B3.getA() should equal A3,
and instead is null, and B2.A2.C2.A3.B3.getA() should also equal A3, but
instead is null.
Of course, the graph is being detached after load, so unfortunately lazy
loading the B3->A relationship is not possible.
When running through a debugger, it looks like what happens is that the A3
and B3-* instances are each being created more than one time. This seems
to make sense because the graph has cycles, relationships are loaded
recursively, and instances are not added to the transactional cache
(ManagedCache) until after they are fully initialized with all fields
loaded. Therefore new instances will not always be fully created when
they are needed again. As the call stack goes forward, the newest A and B
instances get fully initialized with all fields loaded. However, as the
call stack unwinds, the oldest A and B instances are the ones that
eventually win, and one of the B instances that wins is not fully
initialized and has a null field in it's A relationship (e.g. B3.getA() is
null, because B3's A was not ever set).
I'm currently tracing this through to try and determine exactly why the
outer (eldest) B3 isn't getting loaded with its A3, but while I do so, I
was wondering if anyone else has encountered a similar problem or has any
suggestions as to where I should focus my efforts.
My current thinking is that the multiple loading issue is OK and expected,
and that the problem is that the oldest B's aren't getting loaded with
their A's. But it is possible that the problem is that each individual
entity should only be initialized once, and that this is the root issue.
Comments would be welcome.
Thanks,
Jeff
Re: Advice Requested: Instance Loading bug in 2.3.0
Posted by Rick Curtis <cu...@gmail.com>.
> I¹m unsure as to why this code/data combo used to work in version 1.2.2 and
stopped working in 2.x
This sounds somewhat familiar, but I'm unable to find any posts on the
mailing lists. My best guess is that in 2.x we may have found a bug where
we weren't properly enforcing fetch depth... but that is just a gut
feeling. I'm glad you were able to figure it out, as I know these things
can be pretty painful to debug.
Thanks,
Rick
On Mon, May 5, 2014 at 11:46 AM, Jeff Oh <Je...@elasticpath.com> wrote:
> Hi Rick,
>
> Thanks for your suggestions. The good news is that I managed to determine
> what was causing the issue.
> The bad news is that it turns out, that despite my best intentions, that
> many of my initial diagnoses
> and thoughts were quite wrong.
>
> So - the parts that were incorrect:
>
> Initially, I thought that each instance was created multiple times, and
> later on a winner was selected.
> It turns out that each instance is only created once, as (I should have)
> expected. The multiple instances
> were an unfortunate side-effect of viewing multiple concurrent threads
> loading the same data in the debugger.
>
> Instead, the actual problem occurred because OpenJPA was halting the load
> because it hit the maximum recursion depth (defaulted to 1). I was able
> to fix the problem by adding ³recursionDepth = -1² to the @FetchAttribute
> annotations between A->B and B->A.
>
> I¹m unsure as to why this code/data combo used to work in version 1.2.2
> and stopped working in 2.x. My best theory is that perhaps OpenJPA 1.x
> might have ignored the recursionDepth because the recursive/cyclic
> relationship wasn¹t direct (e.g. A->C->B->A instead of A->B->A), but that
> 2.x might be a little bit smarter. Or perhaps something else, perhaps not
> related to OpenJPA at all, might have jiggled the starting conditions
> enough to cause the problem. In any case, the ultimate fix was simple,
> and I¹m happy about that.
>
>
> Thanks,
>
> Jeff
>
> On 2014-05-04, 6:24 AM, "Rick Curtis" <cu...@gmail.com> wrote:
>
> >A couple quick thoughts/debug suggestions for you :
> >
> >- Can we get a copy of your persistence.xml file?
> >- Can you try to run with 2.0.x to see if something was introduced in the
> >original 2.0 implementation effort, or if it came sometime after?
> >- Have you tried setting your fetch graph to be entirely loaded by the
> >default fetch group? I know this probably isn't ideal, but it might be an
> >interesting data point.
> >
> >If you are able to trim the test down into something consumable by others,
> >I'd be interested in taking a look at it.
> >
> >Good luck,
> >Rick
> >
> >
> >On Fri, May 2, 2014 at 5:43 PM, Jeff Oh <oh...@gmail.com> wrote:
> >
> >> Hello All,
> >>
> >> We've recently upgraded to 2.3.0 (from 1.2.2) and have encountered a
> >>nasty
> >> bug where when loading a complicated cyclic object graph using a fetch
> >> group, some relationships are not being populated under some
> >>circumstances.
> >>
> >> Unfortunately, the graph really is quite complicated, and while I have a
> >> set of test data where the problem can be reliably duplicated, I haven't
> >> been able to make an integration test do the same yet. A simplified
> >> version of the graph is that A is 1:M to B and A is 1:M to C, B is M:1
> >>to A
> >> (Bidirectional), and C is M:1 to A and C is also M:1 to B. The net
> >>effect
> >> is a graph where cycles can exist, although in practice they generally
> >> don't other than the bidirectional relationship between A and B.
> >>
> >> Our failing case is interesting. When we load single instances of B
> >>(call
> >> them B1 and B2) one at a time the load is always successful, with all of
> >> the A, B, and C relationships populated - but only if loaded one at a
> >>time.
> >> In other words, if we run entityManager.find(B1) or
> >> entityManager.find(B2), then all is well.
> >>
> >> An ascii art version of the graph traversals might look something like
> >> this:
> >>
> >> B1 -> A1 -> C1 -> B3 -> A3 -> B3 -> A3
> >> -> B3-1 -> A3
> >> -> B3-2 -> A3
> >> -> C1-2 -> ...
> >> ....
> >>
> >> B2 -> A2 -> C2 -> A3 -> B3 -> A3
> >> -> B3-1 -> A3
> >> -> B3-2 -> A3
> >> -> C2-2 -> ....
> >> ...
> >>
> >> where B is M:1 to A, A is 1:M to C and A is 1:M to B.
> >>
> >> Interestingly, while B1 and B2 are separate objects, they do share
> >>several
> >> common objects in their graphs - call them A3 and B3 (as well as B3-1
> >>and
> >> B3-2). If we run a query that loads both B1 and B2 in the same query -
> >> entityManager.find(B1 + B2), then one of the relationships from one of
> >>the
> >> other B objects in the graph (call it B3) B3->A is null (not populated),
> >> where B3->A should == A3. To clarify, B1.A1.C1.B3.getA() should equal
> >>A3,
> >> and instead is null, and B2.A2.C2.A3.B3.getA() should also equal A3, but
> >> instead is null.
> >>
> >> Of course, the graph is being detached after load, so unfortunately lazy
> >> loading the B3->A relationship is not possible.
> >>
> >> When running through a debugger, it looks like what happens is that the
> >>A3
> >> and B3-* instances are each being created more than one time. This
> >>seems
> >> to make sense because the graph has cycles, relationships are loaded
> >> recursively, and instances are not added to the transactional cache
> >> (ManagedCache) until after they are fully initialized with all fields
> >> loaded. Therefore new instances will not always be fully created when
> >> they are needed again. As the call stack goes forward, the newest A
> >>and B
> >> instances get fully initialized with all fields loaded. However, as the
> >> call stack unwinds, the oldest A and B instances are the ones that
> >> eventually win, and one of the B instances that wins is not fully
> >> initialized and has a null field in it's A relationship (e.g. B3.getA()
> >>is
> >> null, because B3's A was not ever set).
> >>
> >> I'm currently tracing this through to try and determine exactly why the
> >> outer (eldest) B3 isn't getting loaded with its A3, but while I do so, I
> >> was wondering if anyone else has encountered a similar problem or has
> >>any
> >> suggestions as to where I should focus my efforts.
> >>
> >> My current thinking is that the multiple loading issue is OK and
> >>expected,
> >> and that the problem is that the oldest B's aren't getting loaded with
> >> their A's. But it is possible that the problem is that each individual
> >> entity should only be initialized once, and that this is the root issue.
> >> Comments would be welcome.
> >>
> >>
> >> Thanks,
> >>
> >> Jeff
> >>
> >
> >
> >
> >--
> >*Rick Curtis*
>
>
>
>
> Jeff Oh, Sr. Software Engineer
> Phone: 604.408.8078 ext. 104
> Email: Jeff.Oh@elasticpath.com
>
> Elastic Path Software Inc.
> Web: www.elasticpath.com
> Blog: www.getelastic.com
> Community: grep.elasticpath.com
> Careers: www.elasticpath.com/jobs
>
> Confidentiality Notice: This message is intended only for the use of the
> designated addressee(s), and may contain information that is privileged,
> confidential and exempt from disclosure. Any unauthorized viewing,
> disclosure, copying, distribution or use of information contained in this
> e-mail is prohibited and may be unlawful. If you received this e-mail in
> error, please reply to the sender immediately to inform us you are not the
> intended recipient and delete the email from your computer system. Thank
> you.
>
>
>
--
*Rick Curtis*
Re: Advice Requested: Instance Loading bug in 2.3.0
Posted by Jeff Oh <Je...@elasticpath.com>.
Hi Rick,
Thanks for your suggestions. The good news is that I managed to determine
what was causing the issue.
The bad news is that it turns out, that despite my best intentions, that
many of my initial diagnoses
and thoughts were quite wrong.
So - the parts that were incorrect:
Initially, I thought that each instance was created multiple times, and
later on a winner was selected.
It turns out that each instance is only created once, as (I should have)
expected. The multiple instances
were an unfortunate side-effect of viewing multiple concurrent threads
loading the same data in the debugger.
Instead, the actual problem occurred because OpenJPA was halting the load
because it hit the maximum recursion depth (defaulted to 1). I was able
to fix the problem by adding ³recursionDepth = -1² to the @FetchAttribute
annotations between A->B and B->A.
I¹m unsure as to why this code/data combo used to work in version 1.2.2
and stopped working in 2.x. My best theory is that perhaps OpenJPA 1.x
might have ignored the recursionDepth because the recursive/cyclic
relationship wasn¹t direct (e.g. A->C->B->A instead of A->B->A), but that
2.x might be a little bit smarter. Or perhaps something else, perhaps not
related to OpenJPA at all, might have jiggled the starting conditions
enough to cause the problem. In any case, the ultimate fix was simple,
and I¹m happy about that.
Thanks,
Jeff
On 2014-05-04, 6:24 AM, "Rick Curtis" <cu...@gmail.com> wrote:
>A couple quick thoughts/debug suggestions for you :
>
>- Can we get a copy of your persistence.xml file?
>- Can you try to run with 2.0.x to see if something was introduced in the
>original 2.0 implementation effort, or if it came sometime after?
>- Have you tried setting your fetch graph to be entirely loaded by the
>default fetch group? I know this probably isn't ideal, but it might be an
>interesting data point.
>
>If you are able to trim the test down into something consumable by others,
>I'd be interested in taking a look at it.
>
>Good luck,
>Rick
>
>
>On Fri, May 2, 2014 at 5:43 PM, Jeff Oh <oh...@gmail.com> wrote:
>
>> Hello All,
>>
>> We've recently upgraded to 2.3.0 (from 1.2.2) and have encountered a
>>nasty
>> bug where when loading a complicated cyclic object graph using a fetch
>> group, some relationships are not being populated under some
>>circumstances.
>>
>> Unfortunately, the graph really is quite complicated, and while I have a
>> set of test data where the problem can be reliably duplicated, I haven't
>> been able to make an integration test do the same yet. A simplified
>> version of the graph is that A is 1:M to B and A is 1:M to C, B is M:1
>>to A
>> (Bidirectional), and C is M:1 to A and C is also M:1 to B. The net
>>effect
>> is a graph where cycles can exist, although in practice they generally
>> don't other than the bidirectional relationship between A and B.
>>
>> Our failing case is interesting. When we load single instances of B
>>(call
>> them B1 and B2) one at a time the load is always successful, with all of
>> the A, B, and C relationships populated - but only if loaded one at a
>>time.
>> In other words, if we run entityManager.find(B1) or
>> entityManager.find(B2), then all is well.
>>
>> An ascii art version of the graph traversals might look something like
>> this:
>>
>> B1 -> A1 -> C1 -> B3 -> A3 -> B3 -> A3
>> -> B3-1 -> A3
>> -> B3-2 -> A3
>> -> C1-2 -> ...
>> ....
>>
>> B2 -> A2 -> C2 -> A3 -> B3 -> A3
>> -> B3-1 -> A3
>> -> B3-2 -> A3
>> -> C2-2 -> ....
>> ...
>>
>> where B is M:1 to A, A is 1:M to C and A is 1:M to B.
>>
>> Interestingly, while B1 and B2 are separate objects, they do share
>>several
>> common objects in their graphs - call them A3 and B3 (as well as B3-1
>>and
>> B3-2). If we run a query that loads both B1 and B2 in the same query -
>> entityManager.find(B1 + B2), then one of the relationships from one of
>>the
>> other B objects in the graph (call it B3) B3->A is null (not populated),
>> where B3->A should == A3. To clarify, B1.A1.C1.B3.getA() should equal
>>A3,
>> and instead is null, and B2.A2.C2.A3.B3.getA() should also equal A3, but
>> instead is null.
>>
>> Of course, the graph is being detached after load, so unfortunately lazy
>> loading the B3->A relationship is not possible.
>>
>> When running through a debugger, it looks like what happens is that the
>>A3
>> and B3-* instances are each being created more than one time. This
>>seems
>> to make sense because the graph has cycles, relationships are loaded
>> recursively, and instances are not added to the transactional cache
>> (ManagedCache) until after they are fully initialized with all fields
>> loaded. Therefore new instances will not always be fully created when
>> they are needed again. As the call stack goes forward, the newest A
>>and B
>> instances get fully initialized with all fields loaded. However, as the
>> call stack unwinds, the oldest A and B instances are the ones that
>> eventually win, and one of the B instances that wins is not fully
>> initialized and has a null field in it's A relationship (e.g. B3.getA()
>>is
>> null, because B3's A was not ever set).
>>
>> I'm currently tracing this through to try and determine exactly why the
>> outer (eldest) B3 isn't getting loaded with its A3, but while I do so, I
>> was wondering if anyone else has encountered a similar problem or has
>>any
>> suggestions as to where I should focus my efforts.
>>
>> My current thinking is that the multiple loading issue is OK and
>>expected,
>> and that the problem is that the oldest B's aren't getting loaded with
>> their A's. But it is possible that the problem is that each individual
>> entity should only be initialized once, and that this is the root issue.
>> Comments would be welcome.
>>
>>
>> Thanks,
>>
>> Jeff
>>
>
>
>
>--
>*Rick Curtis*
Jeff Oh, Sr. Software Engineer
Phone: 604.408.8078 ext. 104
Email: Jeff.Oh@elasticpath.com
Elastic Path Software Inc.
Web: www.elasticpath.com
Blog: www.getelastic.com
Community: grep.elasticpath.com
Careers: www.elasticpath.com/jobs
Confidentiality Notice: This message is intended only for the use of the designated addressee(s), and may contain information that is privileged, confidential and exempt from disclosure. Any unauthorized viewing, disclosure, copying, distribution or use of information contained in this e-mail is prohibited and may be unlawful. If you received this e-mail in error, please reply to the sender immediately to inform us you are not the intended recipient and delete the email from your computer system. Thank you.
Re: Advice Requested: Instance Loading bug in 2.3.0
Posted by Rick Curtis <cu...@gmail.com>.
A couple quick thoughts/debug suggestions for you :
- Can we get a copy of your persistence.xml file?
- Can you try to run with 2.0.x to see if something was introduced in the
original 2.0 implementation effort, or if it came sometime after?
- Have you tried setting your fetch graph to be entirely loaded by the
default fetch group? I know this probably isn't ideal, but it might be an
interesting data point.
If you are able to trim the test down into something consumable by others,
I'd be interested in taking a look at it.
Good luck,
Rick
On Fri, May 2, 2014 at 5:43 PM, Jeff Oh <oh...@gmail.com> wrote:
> Hello All,
>
> We've recently upgraded to 2.3.0 (from 1.2.2) and have encountered a nasty
> bug where when loading a complicated cyclic object graph using a fetch
> group, some relationships are not being populated under some circumstances.
>
> Unfortunately, the graph really is quite complicated, and while I have a
> set of test data where the problem can be reliably duplicated, I haven't
> been able to make an integration test do the same yet. A simplified
> version of the graph is that A is 1:M to B and A is 1:M to C, B is M:1 to A
> (Bidirectional), and C is M:1 to A and C is also M:1 to B. The net effect
> is a graph where cycles can exist, although in practice they generally
> don't other than the bidirectional relationship between A and B.
>
> Our failing case is interesting. When we load single instances of B (call
> them B1 and B2) one at a time the load is always successful, with all of
> the A, B, and C relationships populated - but only if loaded one at a time.
> In other words, if we run entityManager.find(B1) or
> entityManager.find(B2), then all is well.
>
> An ascii art version of the graph traversals might look something like
> this:
>
> B1 -> A1 -> C1 -> B3 -> A3 -> B3 -> A3
> -> B3-1 -> A3
> -> B3-2 -> A3
> -> C1-2 -> ...
> ....
>
> B2 -> A2 -> C2 -> A3 -> B3 -> A3
> -> B3-1 -> A3
> -> B3-2 -> A3
> -> C2-2 -> ....
> ...
>
> where B is M:1 to A, A is 1:M to C and A is 1:M to B.
>
> Interestingly, while B1 and B2 are separate objects, they do share several
> common objects in their graphs - call them A3 and B3 (as well as B3-1 and
> B3-2). If we run a query that loads both B1 and B2 in the same query -
> entityManager.find(B1 + B2), then one of the relationships from one of the
> other B objects in the graph (call it B3) B3->A is null (not populated),
> where B3->A should == A3. To clarify, B1.A1.C1.B3.getA() should equal A3,
> and instead is null, and B2.A2.C2.A3.B3.getA() should also equal A3, but
> instead is null.
>
> Of course, the graph is being detached after load, so unfortunately lazy
> loading the B3->A relationship is not possible.
>
> When running through a debugger, it looks like what happens is that the A3
> and B3-* instances are each being created more than one time. This seems
> to make sense because the graph has cycles, relationships are loaded
> recursively, and instances are not added to the transactional cache
> (ManagedCache) until after they are fully initialized with all fields
> loaded. Therefore new instances will not always be fully created when
> they are needed again. As the call stack goes forward, the newest A and B
> instances get fully initialized with all fields loaded. However, as the
> call stack unwinds, the oldest A and B instances are the ones that
> eventually win, and one of the B instances that wins is not fully
> initialized and has a null field in it's A relationship (e.g. B3.getA() is
> null, because B3's A was not ever set).
>
> I'm currently tracing this through to try and determine exactly why the
> outer (eldest) B3 isn't getting loaded with its A3, but while I do so, I
> was wondering if anyone else has encountered a similar problem or has any
> suggestions as to where I should focus my efforts.
>
> My current thinking is that the multiple loading issue is OK and expected,
> and that the problem is that the oldest B's aren't getting loaded with
> their A's. But it is possible that the problem is that each individual
> entity should only be initialized once, and that this is the root issue.
> Comments would be welcome.
>
>
> Thanks,
>
> Jeff
>
--
*Rick Curtis*