You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cayenne.apache.org by "Musall, Maik" <ma...@selbstdenker.ag> on 2017/08/02 04:46:52 UTC
Re: PK conflicts when inserting complex graphs
Hi,
does nobody have any idea? Should I post this on the dev list because it could be a Cayenne bug?
Maik
> Am 28.07.2017 um 17:20 schrieb Musall, Maik <ma...@selbstdenker.ag>:
>
> Hi,
>
> I've been trying to resolve an unpleasant problem for a few days now. I'm trying to save a graph of new and changed objects related to each other, using a PostgreSQL 9.5 database. The number of objects is in the range between 10 and 100 usually. Several objects of the same class each, and numerous relationships involved. The model as a whole contains a few indirect cyclic relationships, but in these failing inserts, there aren't any cyclic relations involved.
>
> I get something like this:
>
> org.apache.cayenne.CayenneRuntimeException: [v.4.0.B2-SNAPSHOT Jul 12 2017 08:27:18] Commit Exception
> at org.apache.cayenne.access.DataContext.flushToParent(DataContext.java:774)
> at org.apache.cayenne.access.DataContext.commitChangesToParent(DataContext.java:682)
> at com.selbstdenker.canusa.powerd.PDObjectContext.saveChanges(PDObjectContext.java:171)
> (stuff ommitted)
> Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "pdcjourneyelementtexts_pk"
> Detail: Key (journeyelementtextskey)=(1528690) already exists.
> at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2198)
> at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1927)
> at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
> at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2892)
> at com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:125)
> at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java)
> at org.apache.cayenne.access.jdbc.BatchAction.runAsBatch(BatchAction.java:120)
> at org.apache.cayenne.access.jdbc.BatchAction.performAction(BatchAction.java:85)
> at org.apache.cayenne.access.DataNodeQueryAction.runQuery(DataNodeQueryAction.java:97)
> at org.apache.cayenne.access.DataNode.performQueries(DataNode.java:293)
> at org.apache.cayenne.access.DataDomainFlushAction.runQueries(DataDomainFlushAction.java:233)
> at org.apache.cayenne.access.DataDomainFlushAction.flush(DataDomainFlushAction.java:153)
> at org.apache.cayenne.access.DataDomain.onSyncFlush(DataDomain.java:633)
> at org.apache.cayenne.access.DataDomain.onSyncNoFilters(DataDomain.java:603)
> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
> at org.apache.cayenne.commitlog.CommitLogFilter.onSync(CommitLogFilter.java:85)
> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
> at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:73)
> at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:70)
> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:87)
> at org.apache.cayenne.tx.DefaultTransactionManager.performInLocalTransaction(DefaultTransactionManager.java:59)
> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:52)
> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:40)
> at org.apache.cayenne.tx.TransactionFilter.onSync(TransactionFilter.java:70)
> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
> at org.apache.cayenne.access.DataDomain.onSync(DataDomain.java:590)
> at org.apache.cayenne.access.DataContext.flushToParent(DataContext.java:742)
> ... 60 more
>
> A few notes:
>
> * Before this insert, the max primary key of this table was 1528658, last sequence value was 1528688.
> * It happens with one of the tables involved each time, and the table this happens with changes randomly.
> * I had all but the m:n cross tables set to database-generated pk, but also tried having everything Cayenne-generated (so that Cayenne would be fetching new sequence values before inserting, which I verified it did). Makes no difference.
> * I verified before each test run that all sequence values are between 0 and 32 greater than the current max primary key of the corresponding table (wrote a script for that).
>
> Then I did some research and found that maybe Cayenne fails to figure out a proper insert ordering. So I tried using WeightedAshwoodEntitySorter with @SortWeight (now in cayenne-server), following this mail thread:
> https://lists.apache.org/thread.html/0452cd5ca9de85f19882539710588f1b221da7627f845902ed10a7b6@1461051077@%3Cuser.cayenne.apache.org%3E <https://lists.apache.org/thread.html/0452cd5ca9de85f19882539710588f1b221da7627f845902ed10a7b6@1461051077@%3Cuser.cayenne.apache.org%3E>
> but with that the application won't even start up because one of the first DML statements already fails:
>
> Caused by: java.lang.NullPointerException: No record for DbEntity: org.apache.cayenne.map.DbEntity@a92be4f[name=PDCUserLeafletLog]
> at org.apache.cayenne.ashwood.AshwoodEntitySorter$DbEntityComparator.compare(AshwoodEntitySorter.java:368)
> at org.apache.cayenne.ashwood.AshwoodEntitySorter$DbEntityComparator.compare(AshwoodEntitySorter.java:352)
> at org.apache.cayenne.ashwood.WeightedAshwoodEntitySorter$WeightedDbEntityComparator.compare(WeightedAshwoodEntitySorter.java:103)
> at org.apache.cayenne.ashwood.WeightedAshwoodEntitySorter$WeightedDbEntityComparator.compare(WeightedAshwoodEntitySorter.java:95)
> at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
> at java.util.TimSort.sort(TimSort.java:220)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1454)
> at java.util.Collections.sort(Collections.java:175)
> at org.apache.cayenne.ashwood.AshwoodEntitySorter.sortDbEntities(AshwoodEntitySorter.java:182)
> at org.apache.cayenne.access.DataDomainInsertBucket.appendQueriesInternal(DataDomainInsertBucket.java:56)
> at org.apache.cayenne.access.DataDomainSyncBucket.appendQueries(DataDomainSyncBucket.java:78)
> at org.apache.cayenne.access.DataDomainFlushAction.preprocess(DataDomainFlushAction.java:185)
> at org.apache.cayenne.access.DataDomainFlushAction.flush(DataDomainFlushAction.java:143)
> at org.apache.cayenne.access.DataDomain.onSyncFlush(DataDomain.java:633)
> at org.apache.cayenne.access.DataDomain.onSyncNoFilters(DataDomain.java:603)
> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
> at org.apache.cayenne.commitlog.CommitLogFilter.onSync(CommitLogFilter.java:85)
> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
> at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:73)
> at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:70)
> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:87)
> at org.apache.cayenne.tx.DefaultTransactionManager.performInLocalTransaction(DefaultTransactionManager.java:59)
> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:52)
> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:40)
> at org.apache.cayenne.tx.TransactionFilter.onSync(TransactionFilter.java:70)
> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
> at org.apache.cayenne.access.DataDomain.onSync(DataDomain.java:590)
> at org.apache.cayenne.access.DataContext.flushToParent(DataContext.java:742)
> ... 18 more
>
> This again points to this mail thread: http://cayenne.195.n3.nabble.com/Erratic-AshwoodEntitySorter-NPE-on-Latest-Snapshot-td4027453.html#a4028225 <http://cayenne.195.n3.nabble.com/Erratic-AshwoodEntitySorter-NPE-on-Latest-Snapshot-td4027453.html#a4028225>
> and the NPE message is actually an output of the mentioned commit that Andrus added at the time, but the mail thread stopped there in December.
>
> The mentioned class PDCUserLeafletLog has @SortWeight(1000), inherited from the CayenneDataObject subclass that I use for everything. I assigned lower sort weights, starting at 1, for the main root classes that should be inserted first, incrementing for each relationship from there.
>
> So at this point, I'm at a loss how to solve this. Any help is greatly appreciated.
>
> Thanks
> Maik
>
Re: PK conflicts when inserting complex graphs
Posted by Mike Kienenberger <mk...@gmail.com>.
You do need to make sure that the database's sequence "Increment by"
value matches Cayenne's expected sequence increment (m).
You have to pick the same value to use in both places.
On Fri, Aug 4, 2017 at 11:26 AM, Musall, Maik <ma...@selbstdenker.ag> wrote:
> I think I've found the main reason why this isn't working with Cayenne-generated PKs. Cayenne will fetch the next sequence value n, and then use up to n+m values for inserts before requesting a new one, but the sequence will only be incremented by 1. So, when the app is shut down at this point and restarted, it will fetch a value from the sequence that is already used. I think Cayenne would have to set the new sequence value to whatever it's pk cache size is for that strategy, but it doesn't.
Re: PK conflicts when inserting complex graphs
Posted by "Musall, Maik" <ma...@selbstdenker.ag>.
Well,
I think I've found the main reason why this isn't working with Cayenne-generated PKs. Cayenne will fetch the next sequence value n, and then use up to n+m values for inserts before requesting a new one, but the sequence will only be incremented by 1. So, when the app is shut down at this point and restarted, it will fetch a value from the sequence that is already used. I think Cayenne would have to set the new sequence value to whatever it's pk cache size is for that strategy, but it doesn't.
I could probably set all tables to "custom sequence" pk generation strategy, with cached pk size 0 to avoid that, but at a big performance penalty because of an additional db roundtrip for every insert.
So I went back to the setup I wanted to stick with originally, which is Cayenne-based pks for all m:n link tables, and db generated pks for everything else. Using that, without Ashwood, Cayenne seems to be unable to figure out a proper insert order, although from the relationships it should be quite obvious that to have a foreign key from PDCJourneyElement, that new object has to be inserted first. The relationships here go like this:
PDCJourney >1:n> PDCJourneyElement >1:n> PDCBookingInfo
>1:1> PDCJourneyElementTexts
>1:1> PDCJourneyTexts
The insert order that Cayenne chooses with this is:
INSERT INTO PDCJourneyTexts
INSERT INTO PDCJourneyElementTexts
INSERT INTO PDCBookingInfo
rather than inserting PDCJourney first, then PDCJourneyElement, then PDCBookingInfo and the rest. So, this happens:
INSERT INTO PDCBookingInfo (<lots of attributes omitted>, journeyElementRef) VALUES (<lots of ? omitted>, ?)
2017-08-04 17:11:21,799 [WorkerThread6] (Slf4jJdbcEventLogger.java:466) INFO JdbcEventLogger - *** error.
org.apache.cayenne.CayenneRuntimeException: [v.4.0.B2-SNAPSHOT Jul 12 2017 08:27:18] Can't extract a master key. Missing key (journeyElementKey), master ID (<ObjectId:PDCJourneyElement, TEMP:000035CD0869D8F9>)
at org.apache.cayenne.access.DataDomainSyncBucket$PropagatedValueFactory.create(DataDomainSyncBucket.java:290)
at org.apache.cayenne.query.BatchQueryRow.getValue(BatchQueryRow.java:64)
at org.apache.cayenne.query.InsertBatchQuery$1.getValue(InsertBatchQuery.java:63)
at org.apache.cayenne.access.translator.batch.InsertBatchTranslator.doUpdateBindings(InsertBatchTranslator.java:121)
at org.apache.cayenne.access.translator.batch.DefaultBatchTranslator.updateBindings(DefaultBatchTranslator.java:78)
at org.apache.cayenne.access.jdbc.BatchAction.runAsIndividualQueries(BatchAction.java:163)
at org.apache.cayenne.access.jdbc.BatchAction.performAction(BatchAction.java:87)
at org.apache.cayenne.access.DataNodeQueryAction.runQuery(DataNodeQueryAction.java:97)
at org.apache.cayenne.access.DataNode.performQueries(DataNode.java:293)
at org.apache.cayenne.access.DataDomainFlushAction.runQueries(DataDomainFlushAction.java:233)
at org.apache.cayenne.access.DataDomainFlushAction.flush(DataDomainFlushAction.java:153)
at org.apache.cayenne.access.DataDomain.onSyncFlush(DataDomain.java:633)
at org.apache.cayenne.access.DataDomain.onSyncNoFilters(DataDomain.java:603)
at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
at org.apache.cayenne.commitlog.CommitLogFilter.onSync(CommitLogFilter.java:85)
at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:73)
at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:70)
at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:87)
at org.apache.cayenne.tx.DefaultTransactionManager.performInLocalTransaction(DefaultTransactionManager.java:59)
at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:52)
at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:40)
at org.apache.cayenne.tx.TransactionFilter.onSync(TransactionFilter.java:70)
at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
at org.apache.cayenne.access.DataDomain.onSync(DataDomain.java:590)
at org.apache.cayenne.access.DataContext.flushToParent(DataContext.java:742)
at org.apache.cayenne.access.DataContext.commitChangesToParent(DataContext.java:682)
Do I really have to use Ashwood and @SortWeight or similar facilities to tell Cayenne manually in which order inserts have to be done? EOF could figure it out itself every time… Besides, Ashwood is simply broken and throws NPE, as mentioned below. The weights that I would put into the annotations would basically describe the depth of the entity tree, but isn't that something that Cayenne is supposed to know on it's own from the model?
Maik
> Am 03.08.2017 um 15:14 schrieb Nikita Timofeev <nt...@objectstyle.com>:
>
> Are failing tables related directly to those x-tables you have? And
> did you review SQL insert/update logs, maybe you can note something
> strange there (e.g. wrong order of operations or incomplete data in
> duplicated inserts)?
> And it really will be great if you can test your code with fewer
> tables/records involved while still having failure.
>
> On Thu, Aug 3, 2017 at 12:22 PM, Musall, Maik <ma...@selbstdenker.ag> wrote:
>> Hi Nikita,
>>
>> the total number of entities is 130, and two of those have a meaningful string as primary key, while the rest has regular integers. No inheritance, but several flattened m:n relationships. I also have a few m:n relationships with the x-table being an entity of it's own, which contains additional attributes (like a sort key).
>>
>> Reducing the number of entities involved is not really an option, because having the entire commit going through as one transaction or fail as one transaction is the whole point of doing it in one transaction. I could do this as a test to narrow things down though if you think that could help.
>>
>> A test schema could be possible, but would be a lot of work obviously.
>>
>> Thanks
>> Maik
>>
>>
>>> Am 02.08.2017 um 09:14 schrieb Nikita Timofeev <nt...@objectstyle.com>:
>>>
>>> Hi Maik,
>>>
>>> D you have any meaningful PK or FK, inheritance or flattened
>>> attributes in your model?
>>> And maybe you can reduce number of entities involved or even better
>>> provide some test schema on which problem occurs?
>>>
>>> On Wed, Aug 2, 2017 at 7:46 AM, Musall, Maik <ma...@selbstdenker.ag> wrote:
>>>> Hi,
>>>>
>>>> does nobody have any idea? Should I post this on the dev list because it could be a Cayenne bug?
>>>>
>>>> Maik
>>>>
>>>>
>>>>> Am 28.07.2017 um 17:20 schrieb Musall, Maik <ma...@selbstdenker.ag>:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I've been trying to resolve an unpleasant problem for a few days now. I'm trying to save a graph of new and changed objects related to each other, using a PostgreSQL 9.5 database. The number of objects is in the range between 10 and 100 usually. Several objects of the same class each, and numerous relationships involved. The model as a whole contains a few indirect cyclic relationships, but in these failing inserts, there aren't any cyclic relations involved.
>>>>>
>>>>> I get something like this:
>>>>>
>>>>> org.apache.cayenne.CayenneRuntimeException: [v.4.0.B2-SNAPSHOT Jul 12 2017 08:27:18] Commit Exception
>>>>> at org.apache.cayenne.access.DataContext.flushToParent(DataContext.java:774)
>>>>> at org.apache.cayenne.access.DataContext.commitChangesToParent(DataContext.java:682)
>>>>> at com.selbstdenker.canusa.powerd.PDObjectContext.saveChanges(PDObjectContext.java:171)
>>>>> (stuff ommitted)
>>>>> Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "pdcjourneyelementtexts_pk"
>>>>> Detail: Key (journeyelementtextskey)=(1528690) already exists.
>>>>> at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2198)
>>>>> at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1927)
>>>>> at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
>>>>> at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2892)
>>>>> at com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:125)
>>>>> at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java)
>>>>> at org.apache.cayenne.access.jdbc.BatchAction.runAsBatch(BatchAction.java:120)
>>>>> at org.apache.cayenne.access.jdbc.BatchAction.performAction(BatchAction.java:85)
>>>>> at org.apache.cayenne.access.DataNodeQueryAction.runQuery(DataNodeQueryAction.java:97)
>>>>> at org.apache.cayenne.access.DataNode.performQueries(DataNode.java:293)
>>>>> at org.apache.cayenne.access.DataDomainFlushAction.runQueries(DataDomainFlushAction.java:233)
>>>>> at org.apache.cayenne.access.DataDomainFlushAction.flush(DataDomainFlushAction.java:153)
>>>>> at org.apache.cayenne.access.DataDomain.onSyncFlush(DataDomain.java:633)
>>>>> at org.apache.cayenne.access.DataDomain.onSyncNoFilters(DataDomain.java:603)
>>>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>>>> at org.apache.cayenne.commitlog.CommitLogFilter.onSync(CommitLogFilter.java:85)
>>>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>>>> at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:73)
>>>>> at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:70)
>>>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:87)
>>>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInLocalTransaction(DefaultTransactionManager.java:59)
>>>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:52)
>>>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:40)
>>>>> at org.apache.cayenne.tx.TransactionFilter.onSync(TransactionFilter.java:70)
>>>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>>>> at org.apache.cayenne.access.DataDomain.onSync(DataDomain.java:590)
>>>>> at org.apache.cayenne.access.DataContext.flushToParent(DataContext.java:742)
>>>>> ... 60 more
>>>>>
>>>>> A few notes:
>>>>>
>>>>> * Before this insert, the max primary key of this table was 1528658, last sequence value was 1528688.
>>>>> * It happens with one of the tables involved each time, and the table this happens with changes randomly.
>>>>> * I had all but the m:n cross tables set to database-generated pk, but also tried having everything Cayenne-generated (so that Cayenne would be fetching new sequence values before inserting, which I verified it did). Makes no difference.
>>>>> * I verified before each test run that all sequence values are between 0 and 32 greater than the current max primary key of the corresponding table (wrote a script for that).
>>>>>
>>>>> Then I did some research and found that maybe Cayenne fails to figure out a proper insert ordering. So I tried using WeightedAshwoodEntitySorter with @SortWeight (now in cayenne-server), following this mail thread:
>>>>> https://lists.apache.org/thread.html/0452cd5ca9de85f19882539710588f1b221da7627f845902ed10a7b6@1461051077@%3Cuser.cayenne.apache.org%3E <https://lists.apache.org/thread.html/0452cd5ca9de85f19882539710588f1b221da7627f845902ed10a7b6@1461051077@%3Cuser.cayenne.apache.org%3E>
>>>>> but with that the application won't even start up because one of the first DML statements already fails:
>>>>>
>>>>> Caused by: java.lang.NullPointerException: No record for DbEntity: org.apache.cayenne.map.DbEntity@a92be4f[name=PDCUserLeafletLog]
>>>>> at org.apache.cayenne.ashwood.AshwoodEntitySorter$DbEntityComparator.compare(AshwoodEntitySorter.java:368)
>>>>> at org.apache.cayenne.ashwood.AshwoodEntitySorter$DbEntityComparator.compare(AshwoodEntitySorter.java:352)
>>>>> at org.apache.cayenne.ashwood.WeightedAshwoodEntitySorter$WeightedDbEntityComparator.compare(WeightedAshwoodEntitySorter.java:103)
>>>>> at org.apache.cayenne.ashwood.WeightedAshwoodEntitySorter$WeightedDbEntityComparator.compare(WeightedAshwoodEntitySorter.java:95)
>>>>> at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
>>>>> at java.util.TimSort.sort(TimSort.java:220)
>>>>> at java.util.Arrays.sort(Arrays.java:1512)
>>>>> at java.util.ArrayList.sort(ArrayList.java:1454)
>>>>> at java.util.Collections.sort(Collections.java:175)
>>>>> at org.apache.cayenne.ashwood.AshwoodEntitySorter.sortDbEntities(AshwoodEntitySorter.java:182)
>>>>> at org.apache.cayenne.access.DataDomainInsertBucket.appendQueriesInternal(DataDomainInsertBucket.java:56)
>>>>> at org.apache.cayenne.access.DataDomainSyncBucket.appendQueries(DataDomainSyncBucket.java:78)
>>>>> at org.apache.cayenne.access.DataDomainFlushAction.preprocess(DataDomainFlushAction.java:185)
>>>>> at org.apache.cayenne.access.DataDomainFlushAction.flush(DataDomainFlushAction.java:143)
>>>>> at org.apache.cayenne.access.DataDomain.onSyncFlush(DataDomain.java:633)
>>>>> at org.apache.cayenne.access.DataDomain.onSyncNoFilters(DataDomain.java:603)
>>>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>>>> at org.apache.cayenne.commitlog.CommitLogFilter.onSync(CommitLogFilter.java:85)
>>>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>>>> at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:73)
>>>>> at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:70)
>>>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:87)
>>>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInLocalTransaction(DefaultTransactionManager.java:59)
>>>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:52)
>>>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:40)
>>>>> at org.apache.cayenne.tx.TransactionFilter.onSync(TransactionFilter.java:70)
>>>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>>>> at org.apache.cayenne.access.DataDomain.onSync(DataDomain.java:590)
>>>>> at org.apache.cayenne.access.DataContext.flushToParent(DataContext.java:742)
>>>>> ... 18 more
>>>>>
>>>>> This again points to this mail thread: http://cayenne.195.n3.nabble.com/Erratic-AshwoodEntitySorter-NPE-on-Latest-Snapshot-td4027453.html#a4028225 <http://cayenne.195.n3.nabble.com/Erratic-AshwoodEntitySorter-NPE-on-Latest-Snapshot-td4027453.html#a4028225>
>>>>> and the NPE message is actually an output of the mentioned commit that Andrus added at the time, but the mail thread stopped there in December.
>>>>>
>>>>> The mentioned class PDCUserLeafletLog has @SortWeight(1000), inherited from the CayenneDataObject subclass that I use for everything. I assigned lower sort weights, starting at 1, for the main root classes that should be inserted first, incrementing for each relationship from there.
>>>>>
>>>>> So at this point, I'm at a loss how to solve this. Any help is greatly appreciated.
>>>>>
>>>>> Thanks
>>>>> Maik
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Best regards,
>>> Nikita Timofeev
>>
>
>
>
> --
> Best regards,
> Nikita Timofeev
Re: PK conflicts when inserting complex graphs
Posted by Nikita Timofeev <nt...@objectstyle.com>.
Are failing tables related directly to those x-tables you have? And
did you review SQL insert/update logs, maybe you can note something
strange there (e.g. wrong order of operations or incomplete data in
duplicated inserts)?
And it really will be great if you can test your code with fewer
tables/records involved while still having failure.
On Thu, Aug 3, 2017 at 12:22 PM, Musall, Maik <ma...@selbstdenker.ag> wrote:
> Hi Nikita,
>
> the total number of entities is 130, and two of those have a meaningful string as primary key, while the rest has regular integers. No inheritance, but several flattened m:n relationships. I also have a few m:n relationships with the x-table being an entity of it's own, which contains additional attributes (like a sort key).
>
> Reducing the number of entities involved is not really an option, because having the entire commit going through as one transaction or fail as one transaction is the whole point of doing it in one transaction. I could do this as a test to narrow things down though if you think that could help.
>
> A test schema could be possible, but would be a lot of work obviously.
>
> Thanks
> Maik
>
>
>> Am 02.08.2017 um 09:14 schrieb Nikita Timofeev <nt...@objectstyle.com>:
>>
>> Hi Maik,
>>
>> D you have any meaningful PK or FK, inheritance or flattened
>> attributes in your model?
>> And maybe you can reduce number of entities involved or even better
>> provide some test schema on which problem occurs?
>>
>> On Wed, Aug 2, 2017 at 7:46 AM, Musall, Maik <ma...@selbstdenker.ag> wrote:
>>> Hi,
>>>
>>> does nobody have any idea? Should I post this on the dev list because it could be a Cayenne bug?
>>>
>>> Maik
>>>
>>>
>>>> Am 28.07.2017 um 17:20 schrieb Musall, Maik <ma...@selbstdenker.ag>:
>>>>
>>>> Hi,
>>>>
>>>> I've been trying to resolve an unpleasant problem for a few days now. I'm trying to save a graph of new and changed objects related to each other, using a PostgreSQL 9.5 database. The number of objects is in the range between 10 and 100 usually. Several objects of the same class each, and numerous relationships involved. The model as a whole contains a few indirect cyclic relationships, but in these failing inserts, there aren't any cyclic relations involved.
>>>>
>>>> I get something like this:
>>>>
>>>> org.apache.cayenne.CayenneRuntimeException: [v.4.0.B2-SNAPSHOT Jul 12 2017 08:27:18] Commit Exception
>>>> at org.apache.cayenne.access.DataContext.flushToParent(DataContext.java:774)
>>>> at org.apache.cayenne.access.DataContext.commitChangesToParent(DataContext.java:682)
>>>> at com.selbstdenker.canusa.powerd.PDObjectContext.saveChanges(PDObjectContext.java:171)
>>>> (stuff ommitted)
>>>> Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "pdcjourneyelementtexts_pk"
>>>> Detail: Key (journeyelementtextskey)=(1528690) already exists.
>>>> at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2198)
>>>> at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1927)
>>>> at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
>>>> at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2892)
>>>> at com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:125)
>>>> at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java)
>>>> at org.apache.cayenne.access.jdbc.BatchAction.runAsBatch(BatchAction.java:120)
>>>> at org.apache.cayenne.access.jdbc.BatchAction.performAction(BatchAction.java:85)
>>>> at org.apache.cayenne.access.DataNodeQueryAction.runQuery(DataNodeQueryAction.java:97)
>>>> at org.apache.cayenne.access.DataNode.performQueries(DataNode.java:293)
>>>> at org.apache.cayenne.access.DataDomainFlushAction.runQueries(DataDomainFlushAction.java:233)
>>>> at org.apache.cayenne.access.DataDomainFlushAction.flush(DataDomainFlushAction.java:153)
>>>> at org.apache.cayenne.access.DataDomain.onSyncFlush(DataDomain.java:633)
>>>> at org.apache.cayenne.access.DataDomain.onSyncNoFilters(DataDomain.java:603)
>>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>>> at org.apache.cayenne.commitlog.CommitLogFilter.onSync(CommitLogFilter.java:85)
>>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>>> at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:73)
>>>> at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:70)
>>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:87)
>>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInLocalTransaction(DefaultTransactionManager.java:59)
>>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:52)
>>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:40)
>>>> at org.apache.cayenne.tx.TransactionFilter.onSync(TransactionFilter.java:70)
>>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>>> at org.apache.cayenne.access.DataDomain.onSync(DataDomain.java:590)
>>>> at org.apache.cayenne.access.DataContext.flushToParent(DataContext.java:742)
>>>> ... 60 more
>>>>
>>>> A few notes:
>>>>
>>>> * Before this insert, the max primary key of this table was 1528658, last sequence value was 1528688.
>>>> * It happens with one of the tables involved each time, and the table this happens with changes randomly.
>>>> * I had all but the m:n cross tables set to database-generated pk, but also tried having everything Cayenne-generated (so that Cayenne would be fetching new sequence values before inserting, which I verified it did). Makes no difference.
>>>> * I verified before each test run that all sequence values are between 0 and 32 greater than the current max primary key of the corresponding table (wrote a script for that).
>>>>
>>>> Then I did some research and found that maybe Cayenne fails to figure out a proper insert ordering. So I tried using WeightedAshwoodEntitySorter with @SortWeight (now in cayenne-server), following this mail thread:
>>>> https://lists.apache.org/thread.html/0452cd5ca9de85f19882539710588f1b221da7627f845902ed10a7b6@1461051077@%3Cuser.cayenne.apache.org%3E <https://lists.apache.org/thread.html/0452cd5ca9de85f19882539710588f1b221da7627f845902ed10a7b6@1461051077@%3Cuser.cayenne.apache.org%3E>
>>>> but with that the application won't even start up because one of the first DML statements already fails:
>>>>
>>>> Caused by: java.lang.NullPointerException: No record for DbEntity: org.apache.cayenne.map.DbEntity@a92be4f[name=PDCUserLeafletLog]
>>>> at org.apache.cayenne.ashwood.AshwoodEntitySorter$DbEntityComparator.compare(AshwoodEntitySorter.java:368)
>>>> at org.apache.cayenne.ashwood.AshwoodEntitySorter$DbEntityComparator.compare(AshwoodEntitySorter.java:352)
>>>> at org.apache.cayenne.ashwood.WeightedAshwoodEntitySorter$WeightedDbEntityComparator.compare(WeightedAshwoodEntitySorter.java:103)
>>>> at org.apache.cayenne.ashwood.WeightedAshwoodEntitySorter$WeightedDbEntityComparator.compare(WeightedAshwoodEntitySorter.java:95)
>>>> at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
>>>> at java.util.TimSort.sort(TimSort.java:220)
>>>> at java.util.Arrays.sort(Arrays.java:1512)
>>>> at java.util.ArrayList.sort(ArrayList.java:1454)
>>>> at java.util.Collections.sort(Collections.java:175)
>>>> at org.apache.cayenne.ashwood.AshwoodEntitySorter.sortDbEntities(AshwoodEntitySorter.java:182)
>>>> at org.apache.cayenne.access.DataDomainInsertBucket.appendQueriesInternal(DataDomainInsertBucket.java:56)
>>>> at org.apache.cayenne.access.DataDomainSyncBucket.appendQueries(DataDomainSyncBucket.java:78)
>>>> at org.apache.cayenne.access.DataDomainFlushAction.preprocess(DataDomainFlushAction.java:185)
>>>> at org.apache.cayenne.access.DataDomainFlushAction.flush(DataDomainFlushAction.java:143)
>>>> at org.apache.cayenne.access.DataDomain.onSyncFlush(DataDomain.java:633)
>>>> at org.apache.cayenne.access.DataDomain.onSyncNoFilters(DataDomain.java:603)
>>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>>> at org.apache.cayenne.commitlog.CommitLogFilter.onSync(CommitLogFilter.java:85)
>>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>>> at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:73)
>>>> at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:70)
>>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:87)
>>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInLocalTransaction(DefaultTransactionManager.java:59)
>>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:52)
>>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:40)
>>>> at org.apache.cayenne.tx.TransactionFilter.onSync(TransactionFilter.java:70)
>>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>>> at org.apache.cayenne.access.DataDomain.onSync(DataDomain.java:590)
>>>> at org.apache.cayenne.access.DataContext.flushToParent(DataContext.java:742)
>>>> ... 18 more
>>>>
>>>> This again points to this mail thread: http://cayenne.195.n3.nabble.com/Erratic-AshwoodEntitySorter-NPE-on-Latest-Snapshot-td4027453.html#a4028225 <http://cayenne.195.n3.nabble.com/Erratic-AshwoodEntitySorter-NPE-on-Latest-Snapshot-td4027453.html#a4028225>
>>>> and the NPE message is actually an output of the mentioned commit that Andrus added at the time, but the mail thread stopped there in December.
>>>>
>>>> The mentioned class PDCUserLeafletLog has @SortWeight(1000), inherited from the CayenneDataObject subclass that I use for everything. I assigned lower sort weights, starting at 1, for the main root classes that should be inserted first, incrementing for each relationship from there.
>>>>
>>>> So at this point, I'm at a loss how to solve this. Any help is greatly appreciated.
>>>>
>>>> Thanks
>>>> Maik
>>>>
>>>
>>
>>
>>
>> --
>> Best regards,
>> Nikita Timofeev
>
--
Best regards,
Nikita Timofeev
Re: PK conflicts when inserting complex graphs
Posted by "Musall, Maik" <ma...@selbstdenker.ag>.
Hi Nikita,
the total number of entities is 130, and two of those have a meaningful string as primary key, while the rest has regular integers. No inheritance, but several flattened m:n relationships. I also have a few m:n relationships with the x-table being an entity of it's own, which contains additional attributes (like a sort key).
Reducing the number of entities involved is not really an option, because having the entire commit going through as one transaction or fail as one transaction is the whole point of doing it in one transaction. I could do this as a test to narrow things down though if you think that could help.
A test schema could be possible, but would be a lot of work obviously.
Thanks
Maik
> Am 02.08.2017 um 09:14 schrieb Nikita Timofeev <nt...@objectstyle.com>:
>
> Hi Maik,
>
> D you have any meaningful PK or FK, inheritance or flattened
> attributes in your model?
> And maybe you can reduce number of entities involved or even better
> provide some test schema on which problem occurs?
>
> On Wed, Aug 2, 2017 at 7:46 AM, Musall, Maik <ma...@selbstdenker.ag> wrote:
>> Hi,
>>
>> does nobody have any idea? Should I post this on the dev list because it could be a Cayenne bug?
>>
>> Maik
>>
>>
>>> Am 28.07.2017 um 17:20 schrieb Musall, Maik <ma...@selbstdenker.ag>:
>>>
>>> Hi,
>>>
>>> I've been trying to resolve an unpleasant problem for a few days now. I'm trying to save a graph of new and changed objects related to each other, using a PostgreSQL 9.5 database. The number of objects is in the range between 10 and 100 usually. Several objects of the same class each, and numerous relationships involved. The model as a whole contains a few indirect cyclic relationships, but in these failing inserts, there aren't any cyclic relations involved.
>>>
>>> I get something like this:
>>>
>>> org.apache.cayenne.CayenneRuntimeException: [v.4.0.B2-SNAPSHOT Jul 12 2017 08:27:18] Commit Exception
>>> at org.apache.cayenne.access.DataContext.flushToParent(DataContext.java:774)
>>> at org.apache.cayenne.access.DataContext.commitChangesToParent(DataContext.java:682)
>>> at com.selbstdenker.canusa.powerd.PDObjectContext.saveChanges(PDObjectContext.java:171)
>>> (stuff ommitted)
>>> Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "pdcjourneyelementtexts_pk"
>>> Detail: Key (journeyelementtextskey)=(1528690) already exists.
>>> at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2198)
>>> at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1927)
>>> at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
>>> at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2892)
>>> at com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:125)
>>> at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java)
>>> at org.apache.cayenne.access.jdbc.BatchAction.runAsBatch(BatchAction.java:120)
>>> at org.apache.cayenne.access.jdbc.BatchAction.performAction(BatchAction.java:85)
>>> at org.apache.cayenne.access.DataNodeQueryAction.runQuery(DataNodeQueryAction.java:97)
>>> at org.apache.cayenne.access.DataNode.performQueries(DataNode.java:293)
>>> at org.apache.cayenne.access.DataDomainFlushAction.runQueries(DataDomainFlushAction.java:233)
>>> at org.apache.cayenne.access.DataDomainFlushAction.flush(DataDomainFlushAction.java:153)
>>> at org.apache.cayenne.access.DataDomain.onSyncFlush(DataDomain.java:633)
>>> at org.apache.cayenne.access.DataDomain.onSyncNoFilters(DataDomain.java:603)
>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>> at org.apache.cayenne.commitlog.CommitLogFilter.onSync(CommitLogFilter.java:85)
>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>> at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:73)
>>> at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:70)
>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:87)
>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInLocalTransaction(DefaultTransactionManager.java:59)
>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:52)
>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:40)
>>> at org.apache.cayenne.tx.TransactionFilter.onSync(TransactionFilter.java:70)
>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>> at org.apache.cayenne.access.DataDomain.onSync(DataDomain.java:590)
>>> at org.apache.cayenne.access.DataContext.flushToParent(DataContext.java:742)
>>> ... 60 more
>>>
>>> A few notes:
>>>
>>> * Before this insert, the max primary key of this table was 1528658, last sequence value was 1528688.
>>> * It happens with one of the tables involved each time, and the table this happens with changes randomly.
>>> * I had all but the m:n cross tables set to database-generated pk, but also tried having everything Cayenne-generated (so that Cayenne would be fetching new sequence values before inserting, which I verified it did). Makes no difference.
>>> * I verified before each test run that all sequence values are between 0 and 32 greater than the current max primary key of the corresponding table (wrote a script for that).
>>>
>>> Then I did some research and found that maybe Cayenne fails to figure out a proper insert ordering. So I tried using WeightedAshwoodEntitySorter with @SortWeight (now in cayenne-server), following this mail thread:
>>> https://lists.apache.org/thread.html/0452cd5ca9de85f19882539710588f1b221da7627f845902ed10a7b6@1461051077@%3Cuser.cayenne.apache.org%3E <https://lists.apache.org/thread.html/0452cd5ca9de85f19882539710588f1b221da7627f845902ed10a7b6@1461051077@%3Cuser.cayenne.apache.org%3E>
>>> but with that the application won't even start up because one of the first DML statements already fails:
>>>
>>> Caused by: java.lang.NullPointerException: No record for DbEntity: org.apache.cayenne.map.DbEntity@a92be4f[name=PDCUserLeafletLog]
>>> at org.apache.cayenne.ashwood.AshwoodEntitySorter$DbEntityComparator.compare(AshwoodEntitySorter.java:368)
>>> at org.apache.cayenne.ashwood.AshwoodEntitySorter$DbEntityComparator.compare(AshwoodEntitySorter.java:352)
>>> at org.apache.cayenne.ashwood.WeightedAshwoodEntitySorter$WeightedDbEntityComparator.compare(WeightedAshwoodEntitySorter.java:103)
>>> at org.apache.cayenne.ashwood.WeightedAshwoodEntitySorter$WeightedDbEntityComparator.compare(WeightedAshwoodEntitySorter.java:95)
>>> at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
>>> at java.util.TimSort.sort(TimSort.java:220)
>>> at java.util.Arrays.sort(Arrays.java:1512)
>>> at java.util.ArrayList.sort(ArrayList.java:1454)
>>> at java.util.Collections.sort(Collections.java:175)
>>> at org.apache.cayenne.ashwood.AshwoodEntitySorter.sortDbEntities(AshwoodEntitySorter.java:182)
>>> at org.apache.cayenne.access.DataDomainInsertBucket.appendQueriesInternal(DataDomainInsertBucket.java:56)
>>> at org.apache.cayenne.access.DataDomainSyncBucket.appendQueries(DataDomainSyncBucket.java:78)
>>> at org.apache.cayenne.access.DataDomainFlushAction.preprocess(DataDomainFlushAction.java:185)
>>> at org.apache.cayenne.access.DataDomainFlushAction.flush(DataDomainFlushAction.java:143)
>>> at org.apache.cayenne.access.DataDomain.onSyncFlush(DataDomain.java:633)
>>> at org.apache.cayenne.access.DataDomain.onSyncNoFilters(DataDomain.java:603)
>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>> at org.apache.cayenne.commitlog.CommitLogFilter.onSync(CommitLogFilter.java:85)
>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>> at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:73)
>>> at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:70)
>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:87)
>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInLocalTransaction(DefaultTransactionManager.java:59)
>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:52)
>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:40)
>>> at org.apache.cayenne.tx.TransactionFilter.onSync(TransactionFilter.java:70)
>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>> at org.apache.cayenne.access.DataDomain.onSync(DataDomain.java:590)
>>> at org.apache.cayenne.access.DataContext.flushToParent(DataContext.java:742)
>>> ... 18 more
>>>
>>> This again points to this mail thread: http://cayenne.195.n3.nabble.com/Erratic-AshwoodEntitySorter-NPE-on-Latest-Snapshot-td4027453.html#a4028225 <http://cayenne.195.n3.nabble.com/Erratic-AshwoodEntitySorter-NPE-on-Latest-Snapshot-td4027453.html#a4028225>
>>> and the NPE message is actually an output of the mentioned commit that Andrus added at the time, but the mail thread stopped there in December.
>>>
>>> The mentioned class PDCUserLeafletLog has @SortWeight(1000), inherited from the CayenneDataObject subclass that I use for everything. I assigned lower sort weights, starting at 1, for the main root classes that should be inserted first, incrementing for each relationship from there.
>>>
>>> So at this point, I'm at a loss how to solve this. Any help is greatly appreciated.
>>>
>>> Thanks
>>> Maik
>>>
>>
>
>
>
> --
> Best regards,
> Nikita Timofeev
Re: PK conflicts when inserting complex graphs
Posted by Nikita Timofeev <nt...@objectstyle.com>.
It's highly unlikely. Your fix doesn't affect Postgres PK generator
and moreover problems with pkCacheSize would affect only custom
sequences.
On Wed, Aug 2, 2017 at 4:55 PM, Amedeo Mantica <am...@me.com> wrote:
> oops, this:
>
> https://github.com/apache/cayenne/pull/173/files
>
>> On 2 Aug 2017, at 15:54, Amedeo Mantica <am...@me.com> wrote:
>>
>> Hi,
>> may be related to this ?
>>
>> https://github.com/apache/cayenne/commit/814de6546f03e5276d2d100b4c79fe2d4062d5f6
>>
>> has been included in 4 ?
>>
>> Amedeo
>>
>>> On 2 Aug 2017, at 09:14, Nikita Timofeev <nt...@objectstyle.com> wrote:
>>>
>>> Hi Maik,
>>>
>>> D you have any meaningful PK or FK, inheritance or flattened
>>> attributes in your model?
>>> And maybe you can reduce number of entities involved or even better
>>> provide some test schema on which problem occurs?
>>>
>>> On Wed, Aug 2, 2017 at 7:46 AM, Musall, Maik <ma...@selbstdenker.ag> wrote:
>>>> Hi,
>>>>
>>>> does nobody have any idea? Should I post this on the dev list because it could be a Cayenne bug?
>>>>
>>>> Maik
>>>>
>>>>
>>>>> Am 28.07.2017 um 17:20 schrieb Musall, Maik <ma...@selbstdenker.ag>:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I've been trying to resolve an unpleasant problem for a few days now. I'm trying to save a graph of new and changed objects related to each other, using a PostgreSQL 9.5 database. The number of objects is in the range between 10 and 100 usually. Several objects of the same class each, and numerous relationships involved. The model as a whole contains a few indirect cyclic relationships, but in these failing inserts, there aren't any cyclic relations involved.
>>>>>
>>>>> I get something like this:
>>>>>
>>>>> org.apache.cayenne.CayenneRuntimeException: [v.4.0.B2-SNAPSHOT Jul 12 2017 08:27:18] Commit Exception
>>>>> at org.apache.cayenne.access.DataContext.flushToParent(DataContext.java:774)
>>>>> at org.apache.cayenne.access.DataContext.commitChangesToParent(DataContext.java:682)
>>>>> at com.selbstdenker.canusa.powerd.PDObjectContext.saveChanges(PDObjectContext.java:171)
>>>>> (stuff ommitted)
>>>>> Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "pdcjourneyelementtexts_pk"
>>>>> Detail: Key (journeyelementtextskey)=(1528690) already exists.
>>>>> at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2198)
>>>>> at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1927)
>>>>> at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
>>>>> at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2892)
>>>>> at com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:125)
>>>>> at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java)
>>>>> at org.apache.cayenne.access.jdbc.BatchAction.runAsBatch(BatchAction.java:120)
>>>>> at org.apache.cayenne.access.jdbc.BatchAction.performAction(BatchAction.java:85)
>>>>> at org.apache.cayenne.access.DataNodeQueryAction.runQuery(DataNodeQueryAction.java:97)
>>>>> at org.apache.cayenne.access.DataNode.performQueries(DataNode.java:293)
>>>>> at org.apache.cayenne.access.DataDomainFlushAction.runQueries(DataDomainFlushAction.java:233)
>>>>> at org.apache.cayenne.access.DataDomainFlushAction.flush(DataDomainFlushAction.java:153)
>>>>> at org.apache.cayenne.access.DataDomain.onSyncFlush(DataDomain.java:633)
>>>>> at org.apache.cayenne.access.DataDomain.onSyncNoFilters(DataDomain.java:603)
>>>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>>>> at org.apache.cayenne.commitlog.CommitLogFilter.onSync(CommitLogFilter.java:85)
>>>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>>>> at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:73)
>>>>> at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:70)
>>>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:87)
>>>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInLocalTransaction(DefaultTransactionManager.java:59)
>>>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:52)
>>>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:40)
>>>>> at org.apache.cayenne.tx.TransactionFilter.onSync(TransactionFilter.java:70)
>>>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>>>> at org.apache.cayenne.access.DataDomain.onSync(DataDomain.java:590)
>>>>> at org.apache.cayenne.access.DataContext.flushToParent(DataContext.java:742)
>>>>> ... 60 more
>>>>>
>>>>> A few notes:
>>>>>
>>>>> * Before this insert, the max primary key of this table was 1528658, last sequence value was 1528688.
>>>>> * It happens with one of the tables involved each time, and the table this happens with changes randomly.
>>>>> * I had all but the m:n cross tables set to database-generated pk, but also tried having everything Cayenne-generated (so that Cayenne would be fetching new sequence values before inserting, which I verified it did). Makes no difference.
>>>>> * I verified before each test run that all sequence values are between 0 and 32 greater than the current max primary key of the corresponding table (wrote a script for that).
>>>>>
>>>>> Then I did some research and found that maybe Cayenne fails to figure out a proper insert ordering. So I tried using WeightedAshwoodEntitySorter with @SortWeight (now in cayenne-server), following this mail thread:
>>>>> https://lists.apache.org/thread.html/0452cd5ca9de85f19882539710588f1b221da7627f845902ed10a7b6@1461051077@%3Cuser.cayenne.apache.org%3E <https://lists.apache.org/thread.html/0452cd5ca9de85f19882539710588f1b221da7627f845902ed10a7b6@1461051077@%3Cuser.cayenne.apache.org%3E>
>>>>> but with that the application won't even start up because one of the first DML statements already fails:
>>>>>
>>>>> Caused by: java.lang.NullPointerException: No record for DbEntity: org.apache.cayenne.map.DbEntity@a92be4f[name=PDCUserLeafletLog]
>>>>> at org.apache.cayenne.ashwood.AshwoodEntitySorter$DbEntityComparator.compare(AshwoodEntitySorter.java:368)
>>>>> at org.apache.cayenne.ashwood.AshwoodEntitySorter$DbEntityComparator.compare(AshwoodEntitySorter.java:352)
>>>>> at org.apache.cayenne.ashwood.WeightedAshwoodEntitySorter$WeightedDbEntityComparator.compare(WeightedAshwoodEntitySorter.java:103)
>>>>> at org.apache.cayenne.ashwood.WeightedAshwoodEntitySorter$WeightedDbEntityComparator.compare(WeightedAshwoodEntitySorter.java:95)
>>>>> at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
>>>>> at java.util.TimSort.sort(TimSort.java:220)
>>>>> at java.util.Arrays.sort(Arrays.java:1512)
>>>>> at java.util.ArrayList.sort(ArrayList.java:1454)
>>>>> at java.util.Collections.sort(Collections.java:175)
>>>>> at org.apache.cayenne.ashwood.AshwoodEntitySorter.sortDbEntities(AshwoodEntitySorter.java:182)
>>>>> at org.apache.cayenne.access.DataDomainInsertBucket.appendQueriesInternal(DataDomainInsertBucket.java:56)
>>>>> at org.apache.cayenne.access.DataDomainSyncBucket.appendQueries(DataDomainSyncBucket.java:78)
>>>>> at org.apache.cayenne.access.DataDomainFlushAction.preprocess(DataDomainFlushAction.java:185)
>>>>> at org.apache.cayenne.access.DataDomainFlushAction.flush(DataDomainFlushAction.java:143)
>>>>> at org.apache.cayenne.access.DataDomain.onSyncFlush(DataDomain.java:633)
>>>>> at org.apache.cayenne.access.DataDomain.onSyncNoFilters(DataDomain.java:603)
>>>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>>>> at org.apache.cayenne.commitlog.CommitLogFilter.onSync(CommitLogFilter.java:85)
>>>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>>>> at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:73)
>>>>> at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:70)
>>>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:87)
>>>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInLocalTransaction(DefaultTransactionManager.java:59)
>>>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:52)
>>>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:40)
>>>>> at org.apache.cayenne.tx.TransactionFilter.onSync(TransactionFilter.java:70)
>>>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>>>> at org.apache.cayenne.access.DataDomain.onSync(DataDomain.java:590)
>>>>> at org.apache.cayenne.access.DataContext.flushToParent(DataContext.java:742)
>>>>> ... 18 more
>>>>>
>>>>> This again points to this mail thread: http://cayenne.195.n3.nabble.com/Erratic-AshwoodEntitySorter-NPE-on-Latest-Snapshot-td4027453.html#a4028225 <http://cayenne.195.n3.nabble.com/Erratic-AshwoodEntitySorter-NPE-on-Latest-Snapshot-td4027453.html#a4028225>
>>>>> and the NPE message is actually an output of the mentioned commit that Andrus added at the time, but the mail thread stopped there in December.
>>>>>
>>>>> The mentioned class PDCUserLeafletLog has @SortWeight(1000), inherited from the CayenneDataObject subclass that I use for everything. I assigned lower sort weights, starting at 1, for the main root classes that should be inserted first, incrementing for each relationship from there.
>>>>>
>>>>> So at this point, I'm at a loss how to solve this. Any help is greatly appreciated.
>>>>>
>>>>> Thanks
>>>>> Maik
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Best regards,
>>> Nikita Timofeev
>>
>
--
Best regards,
Nikita Timofeev
Re: PK conflicts when inserting complex graphs
Posted by Amedeo Mantica <am...@me.com>.
oops, this:
https://github.com/apache/cayenne/pull/173/files
> On 2 Aug 2017, at 15:54, Amedeo Mantica <am...@me.com> wrote:
>
> Hi,
> may be related to this ?
>
> https://github.com/apache/cayenne/commit/814de6546f03e5276d2d100b4c79fe2d4062d5f6
>
> has been included in 4 ?
>
> Amedeo
>
>> On 2 Aug 2017, at 09:14, Nikita Timofeev <nt...@objectstyle.com> wrote:
>>
>> Hi Maik,
>>
>> D you have any meaningful PK or FK, inheritance or flattened
>> attributes in your model?
>> And maybe you can reduce number of entities involved or even better
>> provide some test schema on which problem occurs?
>>
>> On Wed, Aug 2, 2017 at 7:46 AM, Musall, Maik <ma...@selbstdenker.ag> wrote:
>>> Hi,
>>>
>>> does nobody have any idea? Should I post this on the dev list because it could be a Cayenne bug?
>>>
>>> Maik
>>>
>>>
>>>> Am 28.07.2017 um 17:20 schrieb Musall, Maik <ma...@selbstdenker.ag>:
>>>>
>>>> Hi,
>>>>
>>>> I've been trying to resolve an unpleasant problem for a few days now. I'm trying to save a graph of new and changed objects related to each other, using a PostgreSQL 9.5 database. The number of objects is in the range between 10 and 100 usually. Several objects of the same class each, and numerous relationships involved. The model as a whole contains a few indirect cyclic relationships, but in these failing inserts, there aren't any cyclic relations involved.
>>>>
>>>> I get something like this:
>>>>
>>>> org.apache.cayenne.CayenneRuntimeException: [v.4.0.B2-SNAPSHOT Jul 12 2017 08:27:18] Commit Exception
>>>> at org.apache.cayenne.access.DataContext.flushToParent(DataContext.java:774)
>>>> at org.apache.cayenne.access.DataContext.commitChangesToParent(DataContext.java:682)
>>>> at com.selbstdenker.canusa.powerd.PDObjectContext.saveChanges(PDObjectContext.java:171)
>>>> (stuff ommitted)
>>>> Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "pdcjourneyelementtexts_pk"
>>>> Detail: Key (journeyelementtextskey)=(1528690) already exists.
>>>> at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2198)
>>>> at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1927)
>>>> at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
>>>> at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2892)
>>>> at com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:125)
>>>> at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java)
>>>> at org.apache.cayenne.access.jdbc.BatchAction.runAsBatch(BatchAction.java:120)
>>>> at org.apache.cayenne.access.jdbc.BatchAction.performAction(BatchAction.java:85)
>>>> at org.apache.cayenne.access.DataNodeQueryAction.runQuery(DataNodeQueryAction.java:97)
>>>> at org.apache.cayenne.access.DataNode.performQueries(DataNode.java:293)
>>>> at org.apache.cayenne.access.DataDomainFlushAction.runQueries(DataDomainFlushAction.java:233)
>>>> at org.apache.cayenne.access.DataDomainFlushAction.flush(DataDomainFlushAction.java:153)
>>>> at org.apache.cayenne.access.DataDomain.onSyncFlush(DataDomain.java:633)
>>>> at org.apache.cayenne.access.DataDomain.onSyncNoFilters(DataDomain.java:603)
>>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>>> at org.apache.cayenne.commitlog.CommitLogFilter.onSync(CommitLogFilter.java:85)
>>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>>> at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:73)
>>>> at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:70)
>>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:87)
>>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInLocalTransaction(DefaultTransactionManager.java:59)
>>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:52)
>>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:40)
>>>> at org.apache.cayenne.tx.TransactionFilter.onSync(TransactionFilter.java:70)
>>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>>> at org.apache.cayenne.access.DataDomain.onSync(DataDomain.java:590)
>>>> at org.apache.cayenne.access.DataContext.flushToParent(DataContext.java:742)
>>>> ... 60 more
>>>>
>>>> A few notes:
>>>>
>>>> * Before this insert, the max primary key of this table was 1528658, last sequence value was 1528688.
>>>> * It happens with one of the tables involved each time, and the table this happens with changes randomly.
>>>> * I had all but the m:n cross tables set to database-generated pk, but also tried having everything Cayenne-generated (so that Cayenne would be fetching new sequence values before inserting, which I verified it did). Makes no difference.
>>>> * I verified before each test run that all sequence values are between 0 and 32 greater than the current max primary key of the corresponding table (wrote a script for that).
>>>>
>>>> Then I did some research and found that maybe Cayenne fails to figure out a proper insert ordering. So I tried using WeightedAshwoodEntitySorter with @SortWeight (now in cayenne-server), following this mail thread:
>>>> https://lists.apache.org/thread.html/0452cd5ca9de85f19882539710588f1b221da7627f845902ed10a7b6@1461051077@%3Cuser.cayenne.apache.org%3E <https://lists.apache.org/thread.html/0452cd5ca9de85f19882539710588f1b221da7627f845902ed10a7b6@1461051077@%3Cuser.cayenne.apache.org%3E>
>>>> but with that the application won't even start up because one of the first DML statements already fails:
>>>>
>>>> Caused by: java.lang.NullPointerException: No record for DbEntity: org.apache.cayenne.map.DbEntity@a92be4f[name=PDCUserLeafletLog]
>>>> at org.apache.cayenne.ashwood.AshwoodEntitySorter$DbEntityComparator.compare(AshwoodEntitySorter.java:368)
>>>> at org.apache.cayenne.ashwood.AshwoodEntitySorter$DbEntityComparator.compare(AshwoodEntitySorter.java:352)
>>>> at org.apache.cayenne.ashwood.WeightedAshwoodEntitySorter$WeightedDbEntityComparator.compare(WeightedAshwoodEntitySorter.java:103)
>>>> at org.apache.cayenne.ashwood.WeightedAshwoodEntitySorter$WeightedDbEntityComparator.compare(WeightedAshwoodEntitySorter.java:95)
>>>> at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
>>>> at java.util.TimSort.sort(TimSort.java:220)
>>>> at java.util.Arrays.sort(Arrays.java:1512)
>>>> at java.util.ArrayList.sort(ArrayList.java:1454)
>>>> at java.util.Collections.sort(Collections.java:175)
>>>> at org.apache.cayenne.ashwood.AshwoodEntitySorter.sortDbEntities(AshwoodEntitySorter.java:182)
>>>> at org.apache.cayenne.access.DataDomainInsertBucket.appendQueriesInternal(DataDomainInsertBucket.java:56)
>>>> at org.apache.cayenne.access.DataDomainSyncBucket.appendQueries(DataDomainSyncBucket.java:78)
>>>> at org.apache.cayenne.access.DataDomainFlushAction.preprocess(DataDomainFlushAction.java:185)
>>>> at org.apache.cayenne.access.DataDomainFlushAction.flush(DataDomainFlushAction.java:143)
>>>> at org.apache.cayenne.access.DataDomain.onSyncFlush(DataDomain.java:633)
>>>> at org.apache.cayenne.access.DataDomain.onSyncNoFilters(DataDomain.java:603)
>>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>>> at org.apache.cayenne.commitlog.CommitLogFilter.onSync(CommitLogFilter.java:85)
>>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>>> at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:73)
>>>> at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:70)
>>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:87)
>>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInLocalTransaction(DefaultTransactionManager.java:59)
>>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:52)
>>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:40)
>>>> at org.apache.cayenne.tx.TransactionFilter.onSync(TransactionFilter.java:70)
>>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>>> at org.apache.cayenne.access.DataDomain.onSync(DataDomain.java:590)
>>>> at org.apache.cayenne.access.DataContext.flushToParent(DataContext.java:742)
>>>> ... 18 more
>>>>
>>>> This again points to this mail thread: http://cayenne.195.n3.nabble.com/Erratic-AshwoodEntitySorter-NPE-on-Latest-Snapshot-td4027453.html#a4028225 <http://cayenne.195.n3.nabble.com/Erratic-AshwoodEntitySorter-NPE-on-Latest-Snapshot-td4027453.html#a4028225>
>>>> and the NPE message is actually an output of the mentioned commit that Andrus added at the time, but the mail thread stopped there in December.
>>>>
>>>> The mentioned class PDCUserLeafletLog has @SortWeight(1000), inherited from the CayenneDataObject subclass that I use for everything. I assigned lower sort weights, starting at 1, for the main root classes that should be inserted first, incrementing for each relationship from there.
>>>>
>>>> So at this point, I'm at a loss how to solve this. Any help is greatly appreciated.
>>>>
>>>> Thanks
>>>> Maik
>>>>
>>>
>>
>>
>>
>> --
>> Best regards,
>> Nikita Timofeev
>
Re: PK conflicts when inserting complex graphs
Posted by Amedeo Mantica <am...@me.com>.
Hi,
may be related to this ?
https://github.com/apache/cayenne/commit/814de6546f03e5276d2d100b4c79fe2d4062d5f6
has been included in 4 ?
Amedeo
> On 2 Aug 2017, at 09:14, Nikita Timofeev <nt...@objectstyle.com> wrote:
>
> Hi Maik,
>
> D you have any meaningful PK or FK, inheritance or flattened
> attributes in your model?
> And maybe you can reduce number of entities involved or even better
> provide some test schema on which problem occurs?
>
> On Wed, Aug 2, 2017 at 7:46 AM, Musall, Maik <ma...@selbstdenker.ag> wrote:
>> Hi,
>>
>> does nobody have any idea? Should I post this on the dev list because it could be a Cayenne bug?
>>
>> Maik
>>
>>
>>> Am 28.07.2017 um 17:20 schrieb Musall, Maik <ma...@selbstdenker.ag>:
>>>
>>> Hi,
>>>
>>> I've been trying to resolve an unpleasant problem for a few days now. I'm trying to save a graph of new and changed objects related to each other, using a PostgreSQL 9.5 database. The number of objects is in the range between 10 and 100 usually. Several objects of the same class each, and numerous relationships involved. The model as a whole contains a few indirect cyclic relationships, but in these failing inserts, there aren't any cyclic relations involved.
>>>
>>> I get something like this:
>>>
>>> org.apache.cayenne.CayenneRuntimeException: [v.4.0.B2-SNAPSHOT Jul 12 2017 08:27:18] Commit Exception
>>> at org.apache.cayenne.access.DataContext.flushToParent(DataContext.java:774)
>>> at org.apache.cayenne.access.DataContext.commitChangesToParent(DataContext.java:682)
>>> at com.selbstdenker.canusa.powerd.PDObjectContext.saveChanges(PDObjectContext.java:171)
>>> (stuff ommitted)
>>> Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "pdcjourneyelementtexts_pk"
>>> Detail: Key (journeyelementtextskey)=(1528690) already exists.
>>> at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2198)
>>> at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1927)
>>> at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
>>> at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2892)
>>> at com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:125)
>>> at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java)
>>> at org.apache.cayenne.access.jdbc.BatchAction.runAsBatch(BatchAction.java:120)
>>> at org.apache.cayenne.access.jdbc.BatchAction.performAction(BatchAction.java:85)
>>> at org.apache.cayenne.access.DataNodeQueryAction.runQuery(DataNodeQueryAction.java:97)
>>> at org.apache.cayenne.access.DataNode.performQueries(DataNode.java:293)
>>> at org.apache.cayenne.access.DataDomainFlushAction.runQueries(DataDomainFlushAction.java:233)
>>> at org.apache.cayenne.access.DataDomainFlushAction.flush(DataDomainFlushAction.java:153)
>>> at org.apache.cayenne.access.DataDomain.onSyncFlush(DataDomain.java:633)
>>> at org.apache.cayenne.access.DataDomain.onSyncNoFilters(DataDomain.java:603)
>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>> at org.apache.cayenne.commitlog.CommitLogFilter.onSync(CommitLogFilter.java:85)
>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>> at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:73)
>>> at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:70)
>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:87)
>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInLocalTransaction(DefaultTransactionManager.java:59)
>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:52)
>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:40)
>>> at org.apache.cayenne.tx.TransactionFilter.onSync(TransactionFilter.java:70)
>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>> at org.apache.cayenne.access.DataDomain.onSync(DataDomain.java:590)
>>> at org.apache.cayenne.access.DataContext.flushToParent(DataContext.java:742)
>>> ... 60 more
>>>
>>> A few notes:
>>>
>>> * Before this insert, the max primary key of this table was 1528658, last sequence value was 1528688.
>>> * It happens with one of the tables involved each time, and the table this happens with changes randomly.
>>> * I had all but the m:n cross tables set to database-generated pk, but also tried having everything Cayenne-generated (so that Cayenne would be fetching new sequence values before inserting, which I verified it did). Makes no difference.
>>> * I verified before each test run that all sequence values are between 0 and 32 greater than the current max primary key of the corresponding table (wrote a script for that).
>>>
>>> Then I did some research and found that maybe Cayenne fails to figure out a proper insert ordering. So I tried using WeightedAshwoodEntitySorter with @SortWeight (now in cayenne-server), following this mail thread:
>>> https://lists.apache.org/thread.html/0452cd5ca9de85f19882539710588f1b221da7627f845902ed10a7b6@1461051077@%3Cuser.cayenne.apache.org%3E <https://lists.apache.org/thread.html/0452cd5ca9de85f19882539710588f1b221da7627f845902ed10a7b6@1461051077@%3Cuser.cayenne.apache.org%3E>
>>> but with that the application won't even start up because one of the first DML statements already fails:
>>>
>>> Caused by: java.lang.NullPointerException: No record for DbEntity: org.apache.cayenne.map.DbEntity@a92be4f[name=PDCUserLeafletLog]
>>> at org.apache.cayenne.ashwood.AshwoodEntitySorter$DbEntityComparator.compare(AshwoodEntitySorter.java:368)
>>> at org.apache.cayenne.ashwood.AshwoodEntitySorter$DbEntityComparator.compare(AshwoodEntitySorter.java:352)
>>> at org.apache.cayenne.ashwood.WeightedAshwoodEntitySorter$WeightedDbEntityComparator.compare(WeightedAshwoodEntitySorter.java:103)
>>> at org.apache.cayenne.ashwood.WeightedAshwoodEntitySorter$WeightedDbEntityComparator.compare(WeightedAshwoodEntitySorter.java:95)
>>> at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
>>> at java.util.TimSort.sort(TimSort.java:220)
>>> at java.util.Arrays.sort(Arrays.java:1512)
>>> at java.util.ArrayList.sort(ArrayList.java:1454)
>>> at java.util.Collections.sort(Collections.java:175)
>>> at org.apache.cayenne.ashwood.AshwoodEntitySorter.sortDbEntities(AshwoodEntitySorter.java:182)
>>> at org.apache.cayenne.access.DataDomainInsertBucket.appendQueriesInternal(DataDomainInsertBucket.java:56)
>>> at org.apache.cayenne.access.DataDomainSyncBucket.appendQueries(DataDomainSyncBucket.java:78)
>>> at org.apache.cayenne.access.DataDomainFlushAction.preprocess(DataDomainFlushAction.java:185)
>>> at org.apache.cayenne.access.DataDomainFlushAction.flush(DataDomainFlushAction.java:143)
>>> at org.apache.cayenne.access.DataDomain.onSyncFlush(DataDomain.java:633)
>>> at org.apache.cayenne.access.DataDomain.onSyncNoFilters(DataDomain.java:603)
>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>> at org.apache.cayenne.commitlog.CommitLogFilter.onSync(CommitLogFilter.java:85)
>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>> at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:73)
>>> at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:70)
>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:87)
>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInLocalTransaction(DefaultTransactionManager.java:59)
>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:52)
>>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:40)
>>> at org.apache.cayenne.tx.TransactionFilter.onSync(TransactionFilter.java:70)
>>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>>> at org.apache.cayenne.access.DataDomain.onSync(DataDomain.java:590)
>>> at org.apache.cayenne.access.DataContext.flushToParent(DataContext.java:742)
>>> ... 18 more
>>>
>>> This again points to this mail thread: http://cayenne.195.n3.nabble.com/Erratic-AshwoodEntitySorter-NPE-on-Latest-Snapshot-td4027453.html#a4028225 <http://cayenne.195.n3.nabble.com/Erratic-AshwoodEntitySorter-NPE-on-Latest-Snapshot-td4027453.html#a4028225>
>>> and the NPE message is actually an output of the mentioned commit that Andrus added at the time, but the mail thread stopped there in December.
>>>
>>> The mentioned class PDCUserLeafletLog has @SortWeight(1000), inherited from the CayenneDataObject subclass that I use for everything. I assigned lower sort weights, starting at 1, for the main root classes that should be inserted first, incrementing for each relationship from there.
>>>
>>> So at this point, I'm at a loss how to solve this. Any help is greatly appreciated.
>>>
>>> Thanks
>>> Maik
>>>
>>
>
>
>
> --
> Best regards,
> Nikita Timofeev
Re: PK conflicts when inserting complex graphs
Posted by Nikita Timofeev <nt...@objectstyle.com>.
Hi Maik,
D you have any meaningful PK or FK, inheritance or flattened
attributes in your model?
And maybe you can reduce number of entities involved or even better
provide some test schema on which problem occurs?
On Wed, Aug 2, 2017 at 7:46 AM, Musall, Maik <ma...@selbstdenker.ag> wrote:
> Hi,
>
> does nobody have any idea? Should I post this on the dev list because it could be a Cayenne bug?
>
> Maik
>
>
>> Am 28.07.2017 um 17:20 schrieb Musall, Maik <ma...@selbstdenker.ag>:
>>
>> Hi,
>>
>> I've been trying to resolve an unpleasant problem for a few days now. I'm trying to save a graph of new and changed objects related to each other, using a PostgreSQL 9.5 database. The number of objects is in the range between 10 and 100 usually. Several objects of the same class each, and numerous relationships involved. The model as a whole contains a few indirect cyclic relationships, but in these failing inserts, there aren't any cyclic relations involved.
>>
>> I get something like this:
>>
>> org.apache.cayenne.CayenneRuntimeException: [v.4.0.B2-SNAPSHOT Jul 12 2017 08:27:18] Commit Exception
>> at org.apache.cayenne.access.DataContext.flushToParent(DataContext.java:774)
>> at org.apache.cayenne.access.DataContext.commitChangesToParent(DataContext.java:682)
>> at com.selbstdenker.canusa.powerd.PDObjectContext.saveChanges(PDObjectContext.java:171)
>> (stuff ommitted)
>> Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "pdcjourneyelementtexts_pk"
>> Detail: Key (journeyelementtextskey)=(1528690) already exists.
>> at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2198)
>> at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1927)
>> at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
>> at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2892)
>> at com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:125)
>> at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java)
>> at org.apache.cayenne.access.jdbc.BatchAction.runAsBatch(BatchAction.java:120)
>> at org.apache.cayenne.access.jdbc.BatchAction.performAction(BatchAction.java:85)
>> at org.apache.cayenne.access.DataNodeQueryAction.runQuery(DataNodeQueryAction.java:97)
>> at org.apache.cayenne.access.DataNode.performQueries(DataNode.java:293)
>> at org.apache.cayenne.access.DataDomainFlushAction.runQueries(DataDomainFlushAction.java:233)
>> at org.apache.cayenne.access.DataDomainFlushAction.flush(DataDomainFlushAction.java:153)
>> at org.apache.cayenne.access.DataDomain.onSyncFlush(DataDomain.java:633)
>> at org.apache.cayenne.access.DataDomain.onSyncNoFilters(DataDomain.java:603)
>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>> at org.apache.cayenne.commitlog.CommitLogFilter.onSync(CommitLogFilter.java:85)
>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>> at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:73)
>> at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:70)
>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:87)
>> at org.apache.cayenne.tx.DefaultTransactionManager.performInLocalTransaction(DefaultTransactionManager.java:59)
>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:52)
>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:40)
>> at org.apache.cayenne.tx.TransactionFilter.onSync(TransactionFilter.java:70)
>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>> at org.apache.cayenne.access.DataDomain.onSync(DataDomain.java:590)
>> at org.apache.cayenne.access.DataContext.flushToParent(DataContext.java:742)
>> ... 60 more
>>
>> A few notes:
>>
>> * Before this insert, the max primary key of this table was 1528658, last sequence value was 1528688.
>> * It happens with one of the tables involved each time, and the table this happens with changes randomly.
>> * I had all but the m:n cross tables set to database-generated pk, but also tried having everything Cayenne-generated (so that Cayenne would be fetching new sequence values before inserting, which I verified it did). Makes no difference.
>> * I verified before each test run that all sequence values are between 0 and 32 greater than the current max primary key of the corresponding table (wrote a script for that).
>>
>> Then I did some research and found that maybe Cayenne fails to figure out a proper insert ordering. So I tried using WeightedAshwoodEntitySorter with @SortWeight (now in cayenne-server), following this mail thread:
>> https://lists.apache.org/thread.html/0452cd5ca9de85f19882539710588f1b221da7627f845902ed10a7b6@1461051077@%3Cuser.cayenne.apache.org%3E <https://lists.apache.org/thread.html/0452cd5ca9de85f19882539710588f1b221da7627f845902ed10a7b6@1461051077@%3Cuser.cayenne.apache.org%3E>
>> but with that the application won't even start up because one of the first DML statements already fails:
>>
>> Caused by: java.lang.NullPointerException: No record for DbEntity: org.apache.cayenne.map.DbEntity@a92be4f[name=PDCUserLeafletLog]
>> at org.apache.cayenne.ashwood.AshwoodEntitySorter$DbEntityComparator.compare(AshwoodEntitySorter.java:368)
>> at org.apache.cayenne.ashwood.AshwoodEntitySorter$DbEntityComparator.compare(AshwoodEntitySorter.java:352)
>> at org.apache.cayenne.ashwood.WeightedAshwoodEntitySorter$WeightedDbEntityComparator.compare(WeightedAshwoodEntitySorter.java:103)
>> at org.apache.cayenne.ashwood.WeightedAshwoodEntitySorter$WeightedDbEntityComparator.compare(WeightedAshwoodEntitySorter.java:95)
>> at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
>> at java.util.TimSort.sort(TimSort.java:220)
>> at java.util.Arrays.sort(Arrays.java:1512)
>> at java.util.ArrayList.sort(ArrayList.java:1454)
>> at java.util.Collections.sort(Collections.java:175)
>> at org.apache.cayenne.ashwood.AshwoodEntitySorter.sortDbEntities(AshwoodEntitySorter.java:182)
>> at org.apache.cayenne.access.DataDomainInsertBucket.appendQueriesInternal(DataDomainInsertBucket.java:56)
>> at org.apache.cayenne.access.DataDomainSyncBucket.appendQueries(DataDomainSyncBucket.java:78)
>> at org.apache.cayenne.access.DataDomainFlushAction.preprocess(DataDomainFlushAction.java:185)
>> at org.apache.cayenne.access.DataDomainFlushAction.flush(DataDomainFlushAction.java:143)
>> at org.apache.cayenne.access.DataDomain.onSyncFlush(DataDomain.java:633)
>> at org.apache.cayenne.access.DataDomain.onSyncNoFilters(DataDomain.java:603)
>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>> at org.apache.cayenne.commitlog.CommitLogFilter.onSync(CommitLogFilter.java:85)
>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>> at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:73)
>> at org.apache.cayenne.tx.TransactionFilter$1.perform(TransactionFilter.java:70)
>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:87)
>> at org.apache.cayenne.tx.DefaultTransactionManager.performInLocalTransaction(DefaultTransactionManager.java:59)
>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:52)
>> at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:40)
>> at org.apache.cayenne.tx.TransactionFilter.onSync(TransactionFilter.java:70)
>> at org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(DataDomain.java:764)
>> at org.apache.cayenne.access.DataDomain.onSync(DataDomain.java:590)
>> at org.apache.cayenne.access.DataContext.flushToParent(DataContext.java:742)
>> ... 18 more
>>
>> This again points to this mail thread: http://cayenne.195.n3.nabble.com/Erratic-AshwoodEntitySorter-NPE-on-Latest-Snapshot-td4027453.html#a4028225 <http://cayenne.195.n3.nabble.com/Erratic-AshwoodEntitySorter-NPE-on-Latest-Snapshot-td4027453.html#a4028225>
>> and the NPE message is actually an output of the mentioned commit that Andrus added at the time, but the mail thread stopped there in December.
>>
>> The mentioned class PDCUserLeafletLog has @SortWeight(1000), inherited from the CayenneDataObject subclass that I use for everything. I assigned lower sort weights, starting at 1, for the main root classes that should be inserted first, incrementing for each relationship from there.
>>
>> So at this point, I'm at a loss how to solve this. Any help is greatly appreciated.
>>
>> Thanks
>> Maik
>>
>
--
Best regards,
Nikita Timofeev