You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cayenne.apache.org by Martin Thelian <ma...@lovo.cc> on 2008/05/15 17:02:37 UTC
Deadlock between commitChanges and snapshotsUpdatedForObjects
Hi,
in our application we are using cayenne 1.2.4 and have the problem that
we run into a deadlock if two users try to update their data in
parallel. I've analyzed the thread-dumps and this are the threads that
are involved:
------------------------------------------
"userDataBeanMessageListener-5" prio=1 tid=0x08f85f58 nid=0x53d3
runnable [0x83cf5000..0x83cf6e30]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
- locked <0x9303e150> (a java.io.BufferedInputStream)
at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:256)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1163)
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:347)
- locked <0x93027df8> (a org.postgresql.core.v3.QueryExecutorImpl)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2574)
at
org.objectstyle.cayenne.access.jdbc.BatchAction.runAsBatch(BatchAction.java:164)
at
org.objectstyle.cayenne.access.jdbc.BatchAction.performAction(BatchAction.java:114)
at
org.objectstyle.cayenne.access.DataNodeQueryAction.runQuery(DataNodeQueryAction.java:95)
at
org.objectstyle.cayenne.access.DataNode.performQueries(DataNode.java:309)
at
org.objectstyle.cayenne.access.DataDomainFlushAction.runQueries(DataDomainFlushAction.java:255)
at
org.objectstyle.cayenne.access.DataDomainFlushAction.flush(DataDomainFlushAction.java:177)
- locked <0x92f78110> (a
org.objectstyle.cayenne.access.DataRowStore)
at
org.objectstyle.cayenne.access.DataDomain.onSyncFlush(DataDomain.java:846)
at
org.objectstyle.cayenne.access.DataDomain$2.transform(DataDomain.java:817)
at
org.objectstyle.cayenne.access.DataDomain.runInTransaction(DataDomain.java:862)
at
org.objectstyle.cayenne.access.DataDomain.onSync(DataDomain.java:814)
at
org.objectstyle.cayenne.access.DataContext.flushToParent(DataContext.java:1270)
- locked <0x8eff78d8> (a org.objectstyle.cayenne.access.ObjectStore)
at
org.objectstyle.cayenne.access.DataContext.commitChanges(DataContext.java:1174)
at
cc.lovo.data.cayenne.CayenneUserDataBean.setUserData(CayenneUserDataBean.java:851)
at
cc.lovo.data.cayenne.CayenneUserDataBean.createUser(CayenneUserDataBean.java:627)
[...]
"userDataBeanMessageListener-1" prio=1 tid=0x08f81cd0 nid=0x53cf waiting
for monitor entry [0x83ef9000..0x83efb030]
at
org.objectstyle.cayenne.access.DataRowStore.snapshotsUpdatedForObjects(DataRowStore.java:272)
- waiting to lock <0x92f78110> (a
org.objectstyle.cayenne.access.DataRowStore)
at
org.objectstyle.cayenne.access.ObjectResolver.objectsFromDataRows(ObjectResolver.java:157)
at
org.objectstyle.cayenne.access.ObjectResolver.synchronizedObjectsFromDataRows(ObjectResolver.java:133)
- locked <0x8eff7dc8> (a org.objectstyle.cayenne.access.ObjectStore)
at
org.objectstyle.cayenne.access.DataDomainQueryAction.interceptObjectConversion(DataDomainQueryAction.java:355)
at
org.objectstyle.cayenne.access.DataDomainQueryAction.execute(DataDomainQueryAction.java:152)
at
org.objectstyle.cayenne.access.DataDomain.onQuery(DataDomain.java:782)
at
org.objectstyle.cayenne.util.ObjectContextQueryAction.runQuery(ObjectContextQueryAction.java:253)
at
org.objectstyle.cayenne.access.DataContextQueryAction.execute(DataContextQueryAction.java:90)
at
org.objectstyle.cayenne.access.DataContext.onQuery(DataContext.java:1431)
at
org.objectstyle.cayenne.access.DataContext.performQuery(DataContext.java:1420)
at
cc.lovo.data.cayenne.common.CayenneDataBean.getMBForLanCou(CayenneDataBean.java:156)
at
cc.lovo.data.cayenne.common.CayenneDataBean.getCurrLocale(CayenneDataBean.java:177)
at
cc.lovo.data.cayenne.CayenneUserDataBean.getProfileData(CayenneUserDataBean.java:199)
at
cc.lovo.data.cayenne.CayenneUserDataBean.getUser(CayenneUserDataBean.java:147)
[...]
"EventDispatchThread-4" daemon prio=1 tid=0x85b9e280 nid=0x5488 waiting
for monitor entry [0x7d428000..0x7d429030]
at
org.objectstyle.cayenne.access.ObjectStore.processSnapshotEvent(ObjectStore.java:813)
- waiting to lock <0x8eff78d8> (a
org.objectstyle.cayenne.access.ObjectStore)
at
org.objectstyle.cayenne.access.ObjectStore.snapshotsChanged(ObjectStore.java:804)
at sun.reflect.GeneratedMethodAccessor113.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.objectstyle.cayenne.util.Invocation.fire(Invocation.java:240)
at
org.objectstyle.cayenne.event.EventManager$InvocationDispatch.fire(EventManager.java:452)
at
org.objectstyle.cayenne.event.EventManager$DispatchThread.run(EventManager.java:499)
------------------------------------------
The thread "userDataBeanMessageListener-5" seem to lock the whole
DataRowCache via a call to function
org.objectstyle.cayenne.access.DataDomainFlushAction.flush(DataDomainFlushAction.java:177):
|> synchronized (context.getObjectStore().getDataRowCache()) {
|> [...]
|> runQueries();
|> [...]
|> }
but itself seems to wait via Socket-read for the thread
userDataBeanMessageListener-1 to finishes its DB-query. The Problem is
that this thread also tries to enter a sync block in the DataRowStore
but blocks.
The relevant code in
org.objectstyle.cayenne.access.DataRowStore.snapshotsUpdatedForObjects(DataRowStore.java:272):
|> void snapshotsUpdatedForObjects(List objects, List snapshots,
boolean refresh) {
|> [...]
|>
|> synchronized (this) {
|> [...]
A third thread also seems to try to enter a synchronized block in
|> synchronized void processSnapshotEvent(SnapshotEvent event) {
|> [...]
|> }
Any hints how we can get around this problem?
Thanks,
Martin
Re: Deadlock between commitChanges and snapshotsUpdatedForObjects
Posted by Andrus Adamchik <an...@objectstyle.org>.
On May 16, 2008, at 7:45 AM, Andrus Adamchik wrote:
> it is still not clear how exactly thread 5 locks that PostgreSQL table
I mean thread 1 of course. Thread 5 is the one getting stuck.
Andrus
Re: Deadlock between commitChanges and snapshotsUpdatedForObjects
Posted by Tore Halset <ha...@pvv.ntnu.no>.
On 19. mai. 2008, at 09.20, Martin Thelian wrote:
> thank you for the hints. I'll try the patch. Are there any pitfalls
> when
> using this patch?
No pitfalls that I have found. It solved the problem for us. We used
it in production for some time before we moved to a 3.0 milestone that
had the patch built in.
After applying the patch our slow MS SQL Server got beaten ever more
by our application as the database had to handle several commits in
parallel. After moving over to PostgreSQL, things are a lot smoother.
Regards,
- Tore.
Re: Deadlock between commitChanges and snapshotsUpdatedForObjects
Posted by Martin Thelian <ma...@lovo.cc>.
Hi,
thank you for the hints. I'll try the patch. Are there any pitfalls when
using this patch?
Thanks,
Martin
Andrus Adamchik schrieb:
> Ok, it is more clear now (it is still not clear how exactly thread 5
> locks that PostgreSQL table, but the overall explanation makes sense).
> One suggestion is to upgrade to Cayenne 3.0 (3.0M3 for now, and 3.0M4
> once that becomes available). It reduces the scope of the shared
> DataRowStore lock per this Jira:
>
> https://issues.apache.org/cayenne/browse/CAY-722
>
> Alternatively you may attempt to patch 1.2 based on CAY-722 diff:
>
> http://tinyurl.com/56ja5r
>
> Andrus
Re: Deadlock between commitChanges and snapshotsUpdatedForObjects
Posted by Andrus Adamchik <an...@objectstyle.org>.
Ok, it is more clear now (it is still not clear how exactly thread 5
locks that PostgreSQL table, but the overall explanation makes sense).
One suggestion is to upgrade to Cayenne 3.0 (3.0M3 for now, and 3.0M4
once that becomes available). It reduces the scope of the shared
DataRowStore lock per this Jira:
https://issues.apache.org/cayenne/browse/CAY-722
Alternatively you may attempt to patch 1.2 based on CAY-722 diff:
http://tinyurl.com/56ja5r
Andrus
On May 16, 2008, at 3:08 AM, Martin Thelian wrote:
> Hi!
>
> Andrus Adamchik schrieb:
>> Hmm... I don't see a deadlock, just a contention with other threads
>> waiting for "userDataBeanMessageListener-5" to finish commit. So does
>> it result in slowness or a complete deadlock?
> It's not just slow, these threads completely bock each other. The
> first
> thread userDataBeanMessageListener-5 can not finish it's commit
> because
> it seems to be waiting for the DB to release a lock to a table the
> other
> thread is accessing. And the second thread
> userDataBeanMessageListener-1
> can not finish its query because of the synchronized block it can
> not enter.
>
> The only thing I can do if this problem occurs is to kill the runtime
> and start it again.
>
>> (there are known issues with nested contexts [1], but there weren't
>> any with the top-level contexts in a while).
> No we don't use nested context. But we are using transaction.
>
> Regards,
> Martin
>
Re: Deadlock between commitChanges and snapshotsUpdatedForObjects
Posted by Martin Thelian <ma...@lovo.cc>.
Hi!
Andrus Adamchik schrieb:
> Hmm... I don't see a deadlock, just a contention with other threads
> waiting for "userDataBeanMessageListener-5" to finish commit. So does
> it result in slowness or a complete deadlock?
It's not just slow, these threads completely bock each other. The first
thread userDataBeanMessageListener-5 can not finish it's commit because
it seems to be waiting for the DB to release a lock to a table the other
thread is accessing. And the second thread userDataBeanMessageListener-1
can not finish its query because of the synchronized block it can not enter.
The only thing I can do if this problem occurs is to kill the runtime
and start it again.
> (there are known issues with nested contexts [1], but there weren't
> any with the top-level contexts in a while).
No we don't use nested context. But we are using transaction.
Regards,
Martin
Re: Deadlock between commitChanges and snapshotsUpdatedForObjects
Posted by Andrus Adamchik <an...@objectstyle.org>.
Hmm... I don't see a deadlock, just a contention with other threads
waiting for "userDataBeanMessageListener-5" to finish commit. So does
it result in slowness or a complete deadlock?
(there are known issues with nested contexts [1], but there weren't
any with the top-level contexts in a while).
Andrus
[1] https://issues.apache.org/cayenne/browse/CAY-957
On May 15, 2008, at 11:02 AM, Martin Thelian wrote:
> Hi,
>
> in our application we are using cayenne 1.2.4 and have the problem
> that
> we run into a deadlock if two users try to update their data in
> parallel. I've analyzed the thread-dumps and this are the threads that
> are involved:
>
> ------------------------------------------
> "userDataBeanMessageListener-5" prio=1 tid=0x08f85f58 nid=0x53d3
> runnable [0x83cf5000..0x83cf6e30]
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:129)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:
> 218)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:
> 235)
> - locked <0x9303e150> (a java.io.BufferedInputStream)
> at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:256)
> at
> org
> .postgresql
> .core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1163)
> at
> org
> .postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:
> 347)
> - locked <0x93027df8> (a
> org.postgresql.core.v3.QueryExecutorImpl)
> at
> org
> .postgresql
> .jdbc2
> .AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2574)
> at
> org
> .objectstyle
> .cayenne.access.jdbc.BatchAction.runAsBatch(BatchAction.java:164)
> at
> org
> .objectstyle
> .cayenne.access.jdbc.BatchAction.performAction(BatchAction.java:114)
> at
> org
> .objectstyle
> .cayenne
> .access.DataNodeQueryAction.runQuery(DataNodeQueryAction.java:95)
> at
> org.objectstyle.cayenne.access.DataNode.performQueries(DataNode.java:
> 309)
> at
> org
> .objectstyle
> .cayenne
> .access.DataDomainFlushAction.runQueries(DataDomainFlushAction.java:
> 255)
> at
> org
> .objectstyle
> .cayenne
> .access.DataDomainFlushAction.flush(DataDomainFlushAction.java:177)
> - locked <0x92f78110> (a
> org.objectstyle.cayenne.access.DataRowStore)
> at
> org
> .objectstyle.cayenne.access.DataDomain.onSyncFlush(DataDomain.java:
> 846)
> at
> org.objectstyle.cayenne.access.DataDomain
> $2.transform(DataDomain.java:817)
> at
> org
> .objectstyle
> .cayenne.access.DataDomain.runInTransaction(DataDomain.java:862)
> at
> org.objectstyle.cayenne.access.DataDomain.onSync(DataDomain.java:814)
> at
> org
> .objectstyle
> .cayenne.access.DataContext.flushToParent(DataContext.java:1270)
> - locked <0x8eff78d8> (a
> org.objectstyle.cayenne.access.ObjectStore)
> at
> org
> .objectstyle
> .cayenne.access.DataContext.commitChanges(DataContext.java:1174)
> at
> cc
> .lovo
> .data
> .cayenne.CayenneUserDataBean.setUserData(CayenneUserDataBean.java:851)
> at
> cc
> .lovo
> .data
> .cayenne.CayenneUserDataBean.createUser(CayenneUserDataBean.java:627)
> [...]
>
> "userDataBeanMessageListener-1" prio=1 tid=0x08f81cd0 nid=0x53cf
> waiting
> for monitor entry [0x83ef9000..0x83efb030]
> at
> org
> .objectstyle
> .cayenne
> .access.DataRowStore.snapshotsUpdatedForObjects(DataRowStore.java:272)
> - waiting to lock <0x92f78110> (a
> org.objectstyle.cayenne.access.DataRowStore)
> at
> org
> .objectstyle
> .cayenne
> .access.ObjectResolver.objectsFromDataRows(ObjectResolver.java:157)
> at
> org
> .objectstyle
> .cayenne
> .access
> .ObjectResolver.synchronizedObjectsFromDataRows(ObjectResolver.java:
> 133)
> - locked <0x8eff7dc8> (a
> org.objectstyle.cayenne.access.ObjectStore)
> at
> org
> .objectstyle
> .cayenne
> .access
> .DataDomainQueryAction
> .interceptObjectConversion(DataDomainQueryAction.java:355)
> at
> org
> .objectstyle
> .cayenne
> .access.DataDomainQueryAction.execute(DataDomainQueryAction.java:152)
> at
> org.objectstyle.cayenne.access.DataDomain.onQuery(DataDomain.java:782)
> at
> org
> .objectstyle
> .cayenne
> .util
> .ObjectContextQueryAction.runQuery(ObjectContextQueryAction.java:253)
> at
> org
> .objectstyle
> .cayenne
> .access.DataContextQueryAction.execute(DataContextQueryAction.java:90)
> at
> org.objectstyle.cayenne.access.DataContext.onQuery(DataContext.java:
> 1431)
> at
> org
> .objectstyle
> .cayenne.access.DataContext.performQuery(DataContext.java:1420)
> at
> cc
> .lovo
> .data
> .cayenne.common.CayenneDataBean.getMBForLanCou(CayenneDataBean.java:
> 156)
> at
> cc
> .lovo
> .data
> .cayenne.common.CayenneDataBean.getCurrLocale(CayenneDataBean.java:
> 177)
> at
> cc
> .lovo
> .data
> .cayenne.CayenneUserDataBean.getProfileData(CayenneUserDataBean.java:
> 199)
> at
> cc
> .lovo
> .data.cayenne.CayenneUserDataBean.getUser(CayenneUserDataBean.java:
> 147)
> [...]
>
> "EventDispatchThread-4" daemon prio=1 tid=0x85b9e280 nid=0x5488
> waiting
> for monitor entry [0x7d428000..0x7d429030]
> at
> org
> .objectstyle
> .cayenne.access.ObjectStore.processSnapshotEvent(ObjectStore.java:813)
> - waiting to lock <0x8eff78d8> (a
> org.objectstyle.cayenne.access.ObjectStore)
> at
> org
> .objectstyle
> .cayenne.access.ObjectStore.snapshotsChanged(ObjectStore.java:804)
> at sun.reflect.GeneratedMethodAccessor113.invoke(Unknown
> Source)
> at
> sun
> .reflect
> .DelegatingMethodAccessorImpl
> .invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at
> org.objectstyle.cayenne.util.Invocation.fire(Invocation.java:240)
> at
> org.objectstyle.cayenne.event.EventManager
> $InvocationDispatch.fire(EventManager.java:452)
> at
> org.objectstyle.cayenne.event.EventManager
> $DispatchThread.run(EventManager.java:499)
>
> ------------------------------------------
>
> The thread "userDataBeanMessageListener-5" seem to lock the whole
> DataRowCache via a call to function
> org
> .objectstyle
> .cayenne
> .access.DataDomainFlushAction.flush(DataDomainFlushAction.java:177):
>
> |> synchronized (context.getObjectStore().getDataRowCache()) {
> |> [...]
> |> runQueries();
> |> [...]
> |> }
>
> but itself seems to wait via Socket-read for the thread
> userDataBeanMessageListener-1 to finishes its DB-query. The Problem is
> that this thread also tries to enter a sync block in the DataRowStore
> but blocks.
>
> The relevant code in
> org
> .objectstyle
> .cayenne
> .access.DataRowStore.snapshotsUpdatedForObjects(DataRowStore.java:
> 272):
>
> |> void snapshotsUpdatedForObjects(List objects, List snapshots,
> boolean refresh) {
> |> [...]
> |>
> |> synchronized (this) {
> |> [...]
>
>
> A third thread also seems to try to enter a synchronized block in
> |> synchronized void processSnapshotEvent(SnapshotEvent event) {
> |> [...]
> |> }
>
>
> Any hints how we can get around this problem?
>
> Thanks,
> Martin
>