You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cayenne.apache.org by Michael Gentry <mg...@masslight.net> on 2011/07/22 14:47:59 UTC
Ashwood Issues
This message will be a little dense ... just warning up front. :-)
We've had two main issues with AshwoodEntitySorter (and, I believe,
Ashwood) that have produced "Can't extract a master key" and "Cycles
found" errors.
1) Can't extract a master key ...
My guess is this stems from a glitch in IndegreeTopologicalSort. In
3.0, I modified _indexSorter() to have the following logging:
components = new HashMap(contractedReferentialDigraph.order());
int componentIndex = 0;
while (sorter.hasNext()) {
Collection component = (Collection) sorter.next();
log.info("looping over component");
ComponentRecord rec = new ComponentRecord(componentIndex++, component);
log.info("Creating new component record[" + componentIndex + "]: "
+ component);
for (Iterator i = component.iterator(); i.hasNext();) {
log.info("Adding {" + next + "," + rec + "}");
components.put(i.next(), rec);
}
}
This would produce logs like:
Creating new component record[57]: [proc_mod_item_install_cost]
looping over component
Adding {proc_mod_item_install_cost,index: 57; componet:
[proc_mod_item_install_cost]}
Creating new component record[58]: [proc_advance_bdgt_just]
looping over component
Adding {proc_advance_bdgt_just,index: 58; componet: [proc_advance_bdgt_just]}
Creating new component record[59]: [proc_rqmts_sched,
proc_bdgt_qtrs_data, proc_rqmts_sched_install_EIF,
proc_rqmts_sched_install_agent, proc_cost_element,
proc_history_and_planning, proc_manufacturer]
looping over component
Adding {proc_rqmts_sched,index: 59; componet: [proc_rqmts_sched,
proc_bdgt_qtrs_data, proc_rqmts_sched_install_EIF,
proc_rqmts_sched_install_agent, proc_cost_element,
proc_history_and_planning, proc_manufacturer]}
Adding {proc_bdgt_qtrs_data,index: 59; componet: [proc_rqmts_sched,
proc_bdgt_qtrs_data, proc_rqmts_sched_install_EIF,
proc_rqmts_sched_install_agent, proc_cost_element,
proc_history_and_planning, proc_manufacturer]}
Adding {proc_rqmts_sched_install_EIF,index: 59; componet:
[proc_rqmts_sched, proc_bdgt_qtrs_data, proc_rqmts_sched_install_EIF,
proc_rqmts_sched_install_agent, proc_cost_element,
proc_history_and_planning, proc_manufacturer]}
Adding {proc_rqmts_sched_install_agent,index: 59; componet:
[proc_rqmts_sched, proc_bdgt_qtrs_data, proc_rqmts_sched_install_EIF,
proc_rqmts_sched_install_agent, proc_cost_element,
proc_history_and_planning, proc_manufacturer]}
Adding {proc_cost_element,index: 59; componet: [proc_rqmts_sched,
proc_bdgt_qtrs_data, proc_rqmts_sched_install_EIF,
proc_rqmts_sched_install_agent, proc_cost_element,
proc_history_and_planning, proc_manufacturer]}
Adding {proc_history_and_planning,index: 59; componet:
[proc_rqmts_sched, proc_bdgt_qtrs_data, proc_rqmts_sched_install_EIF,
proc_rqmts_sched_install_agent, proc_cost_element,
proc_history_and_planning, proc_manufacturer]}
Adding {proc_manufacturer,index: 59; componet: [proc_rqmts_sched,
proc_bdgt_qtrs_data, proc_rqmts_sched_install_EIF,
proc_rqmts_sched_install_agent, proc_cost_element,
proc_history_and_planning, proc_manufacturer]}
Creating new component record[60]: [proc_ifp_funding]
looping over component
Adding {proc_ifp_funding,index: 60; componet: [proc_ifp_funding]}
Creating new component record[61]: [proc_pfp_related_proj]
looping over component
Adding {proc_pfp_related_proj,index: 61; componet: [proc_pfp_related_proj]}
Creating new component record[62]: [proc_compo_split]
The sorted order from IndegreeTopologicalSort would end up with
duplicates (#59 in this case). The output was 1..58, then duplicates
for 59, then 60..end. This later caused the TableComparator to fail
because the index in ComponentRecord was 59 and when it compared
proc_history_and_planning to proc_manufacturer, they were both 59
which returned 0 (equal to each other). Our proc_manufacturer has to
be inserted before proc_history_and_planning, so this caused the
"Can't extract a master key" error because proc_manufacturer was after
proc_history_and_planning even though they both had the same weight of
59.
I later modified the code to be:
IndegreeTopologicalSort sorter = new
IndegreeTopologicalSort(contractedReferentialDigraph);
components = new HashMap(contractedReferentialDigraph.order());
int componentIndex = 0;
log.info("AshwoodEntitySorter._indexSorter: looping over
IndegreeTopologicalSort digraph");
while (sorter.hasNext()) {
Collection component = (Collection) sorter.next();
log.info("AshwoodEntitySorter._indexSorter: looping over component");
for (Iterator i = component.iterator(); i.hasNext();) {
log.info("AshwoodEntitySorter._indexSorter: Creating new
component record[" + componentIndex + "]: " + component);
ComponentRecord rec = new ComponentRecord(componentIndex++, component);
Object next = i.next();
log.info("AshwoodEntitySorter._indexSorter: Adding {" + next +
"," + rec + "}");
components.put(next, rec);
}
}
I moved the componentIndex++ to be in the loop. This at least had the
advantage of no more duplicates (no repeated 59s), but the sort order
was still incorrect out of IndegreeTopologicalSort and the inserts
still failed. Although we seem to be alleviating this issues a bit
with @SortWeight, it appears the weighting is causing trickle-down
problems because we then have to go change the weighting of other
classes.
2) Cycles found ...
sortObjectsForEntity() appears to have issues if you override equals()
on Data Objects that are reflexive. If we comment out the
equals/hashcode that were added on the reflexive object, it works, if
we put it back, it fails. I haven't explored this one in depth, but
it appears that it is looking for identity and not equivalence on
reflexive relationships.
At this point, I'm considering re-writing AshwoodEntitySorter. At
least with 3.1 I can give it a different name and use the DI to patch
it for testing. :-)
mrg
Re: Ashwood Issues
Posted by Andrus Adamchik <an...@objectstyle.org>.
Very nice. Perhaps one area of Ashwood improvement here would be a better diagnostics of the loops. I.e. if the exception could have the exact relationships involved you could've solved your problem in 5 minutes.
Andrus
On Aug 1, 2011, at 4:13 PM, Michael Gentry wrote:
> BTW Andrus, the algorithm I was using in my entity sorter had a pretty
> significant flaw, so I'm going to have to restart it at some point.
> However, it did help us find a deep DbEntity cycle that was messing
> Ashwood up. We had something like A->B->C->D->E->F->G->H->A and
> Ashwood got confused and generated an incorrect insertion order
> instead of throwing an exception. We fixed the schema and model and
> Ashwood cleared right up, so it isn't as urgent for us now (we
> deployed a new release of our application on Thursday).
>
> Just thought I'd mention it,
>
> mrg
>
Re: Ashwood Issues
Posted by gilbertoca <gi...@gmail.com>.
Michael Gentry-2 wrote:
>
> BTW Andrus, the algorithm I was using in my entity sorter had a pretty
> significant flaw, so I'm going to have to restart it at some point.
> However, it did help us find a deep DbEntity cycle that was messing
> Ashwood up. We had something like A->B->C->D->E->F->G->H->A and
> Ashwood got confused and generated an incorrect insertion order
> instead of throwing an exception. We fixed the schema and model and
> Ashwood cleared right up, so it isn't as urgent for us now (we
> deployed a new release of our application on Thursday).
>
> Just thought I'd mention it,
>
> mrg
>
This framework[1] has some interesting algorithms. Maybe you can see some
useful information there.
Hth,
Gilberto
[1]http://jung.sourceforge.net/index.html
--
View this message in context: http://cayenne.195.n3.nabble.com/Ashwood-Issues-tp3191259p3520383.html
Sent from the Cayenne - Dev mailing list archive at Nabble.com.
Re: Ashwood Issues
Posted by Michael Gentry <mg...@masslight.net>.
BTW Andrus, the algorithm I was using in my entity sorter had a pretty
significant flaw, so I'm going to have to restart it at some point.
However, it did help us find a deep DbEntity cycle that was messing
Ashwood up. We had something like A->B->C->D->E->F->G->H->A and
Ashwood got confused and generated an incorrect insertion order
instead of throwing an exception. We fixed the schema and model and
Ashwood cleared right up, so it isn't as urgent for us now (we
deployed a new release of our application on Thursday).
Just thought I'd mention it,
mrg
Re: Ashwood Issues
Posted by Michael Gentry <mg...@masslight.net>.
On Mon, Jul 25, 2011 at 3:15 AM, Andrus Adamchik <an...@objectstyle.org> wrote:
> Is this going to be a general purpose sorter or something specifically targeting your model? I.e. can we use it in Cayenne as a second generation sorter that does all AshwoodEntitySorter does, plus handles extra scenarios?
That is my intention. There is nothing in it specific to the model
that is causing us grief with Ashwood, but I am trying to fix what
isn't working for us with Ashwood and hope it will work for all other
models, too. I'll have to get my customer to file a Corporate CLA,
though, to officially commit it as a second sorter. Even though I
worked on it on my own time over the weekend, they are also paying for
me to work on it as a fix to our current commit issues.
> Now overriding standard services in DI runtime for unit tests can be done by patching org.apache.cayenne.unit.di.server.ServerRuntimeProvider. See an inner class called ServerExtraModule. This is where you'd bind your sorter.
Awesome, thanks! It might be another day or two before I get to it,
but I definitely want to try it against all of the current Cayenne
tests.
mrg
> On Jul 25, 2011, at 3:10 AM, Michael Gentry wrote:
>> I'm working on a replacement for AshwoodEntitySorter. I currently
>> have it sorting the DbEntities correctly (at least in my test case).
>> I next need to work on sortObjectsForEntity(), which mainly handles
>> ordering in reflexive relationships. When I get this up and running
>> I'll test against our real model. Is there an easy way to plop this
>> into 3.1 and run all the test cases against my sorter instead of the
>> Ashwood one with some DI magic? (I'm wanting more test cases that the
>> ones I have.)
>>
>> Thanks,
>>
>> mrg
>>
>
>
Re: Ashwood Issues
Posted by Andrus Adamchik <an...@objectstyle.org>.
Is this going to be a general purpose sorter or something specifically targeting your model? I.e. can we use it in Cayenne as a second generation sorter that does all AshwoodEntitySorter does, plus handles extra scenarios?
Now overriding standard services in DI runtime for unit tests can be done by patching org.apache.cayenne.unit.di.server.ServerRuntimeProvider. See an inner class called ServerExtraModule. This is where you'd bind your sorter.
Andrus
On Jul 25, 2011, at 3:10 AM, Michael Gentry wrote:
> I'm working on a replacement for AshwoodEntitySorter. I currently
> have it sorting the DbEntities correctly (at least in my test case).
> I next need to work on sortObjectsForEntity(), which mainly handles
> ordering in reflexive relationships. When I get this up and running
> I'll test against our real model. Is there an easy way to plop this
> into 3.1 and run all the test cases against my sorter instead of the
> Ashwood one with some DI magic? (I'm wanting more test cases that the
> ones I have.)
>
> Thanks,
>
> mrg
>
Re: Ashwood Issues
Posted by Michael Gentry <mg...@masslight.net>.
I'm working on a replacement for AshwoodEntitySorter. I currently
have it sorting the DbEntities correctly (at least in my test case).
I next need to work on sortObjectsForEntity(), which mainly handles
ordering in reflexive relationships. When I get this up and running
I'll test against our real model. Is there an easy way to plop this
into 3.1 and run all the test cases against my sorter instead of the
Ashwood one with some DI magic? (I'm wanting more test cases that the
ones I have.)
Thanks,
mrg