Deadlock.

[John Gunning <js...@yahoo.co.uk>]

Hi there,


I am experiencing a deadlock issue which looks similar
to CAY-297. I understand CAY-297 was fixed in release
1.1 but I am using version 1.2.1 and the deadalock
issue arises. When I set
"cayenne.DataRowStore.remote.notify" to false the
issue does not arise. However, I need this set to true
so that remote web services can be notified. I'm also
extending the Cayenne EventBridge class. I attach the
stack traces of the deadlocking threads as well as the
processMessage and sendExternalEvent methods of the
EventBridge extension. Any ideas anyone?...


Thanks in advance,

John Gunning.

Send instant messages to your online friends http://uk.messenger.yahoo.com 

Re: Deadlock.

[Andrus Adamchik <an...@objectstyle.org>]

John,

After staring at the stack trace a bit longer, I think I have a  
better idea of what's going on (although JVM thread dumps are still  
better as they show all the locks owned by each thread :-)). You are  
using nested DataContexts contexts and while a child commits to  
parent, parent is being refreshed via an event, resulting in a  
deadlock. Could you possibly open a bug report? We may implement a  
similar solution - sending an event in a separate thread.

http://issues.apache.org/cayenne/

Thanks
Andrus


On Dec 11, 2006, at 2:39 PM, Andrus Adamchik wrote:

> Do you have a full thread dump printed by JVM? From these two  
> threads I can't figure out the nature of a deadlock.
>
> Thanks
> Andrus
>
> On Dec 8, 2006, at 2:22 PM, John Gunning wrote:
>
>> Hi there,
>>
>>
>> I am experiencing a deadlock issue which looks similar
>> to CAY-297. I understand CAY-297 was fixed in release
>> 1.1 but I am using version 1.2.1 and the deadalock
>> issue arises. When I set
>> "cayenne.DataRowStore.remote.notify" to false the
>> issue does not arise. However, I need this set to true
>> so that remote web services can be notified. I'm also
>> extending the Cayenne EventBridge class. I attach the
>> stack traces of the deadlocking threads as well as the
>> processMessage and sendExternalEvent methods of the
>> EventBridge extension. Any ideas anyone?...
>>
>>
>> Thanks in advance,
>>
>> John Gunning.
>>
>> Send instant messages to your online friends http:// 
>> uk.messenger.yahoo.com Deadlock found :-
>> "JMS Session Delivery Thread" Id=31 in BLOCKED on  
>> lock=org.objectstyle.cayenne.access.ObjectStore@cc7f9e
>>      owned by EventDispatchThread-3 Id=15
>>     at org.objectstyle.cayenne.access.ObjectStore.registerDiff 
>> (ObjectStore.java:200)
>>     at  
>> org.objectstyle.cayenne.access.ObjectStore.recordObjectCreated 
>> (ObjectStore.java:152)
>>     at org.objectstyle.cayenne.access.DataContext.registerNewObject 
>> (DataContext.java:922)
>>     at org.objectstyle.cayenne.access.ChildDiffLoader.nodeCreated 
>> (ChildDiffLoader.java:110)
>>     at org.objectstyle.cayenne.graph.NodeCreateOperation.apply 
>> (NodeCreateOperation.java:73)
>>     at org.objectstyle.cayenne.graph.CompoundDiff.apply 
>> (CompoundDiff.java:133)
>>     at org.objectstyle.cayenne.access.ObjectStoreGraphDiff.apply 
>> (ObjectStoreGraphDiff.java:155)
>>     at org.objectstyle.cayenne.access.DataContext.onContextFlush 
>> (DataContext.java:1215)
>>     at org.objectstyle.cayenne.access.DataContext.onSync 
>> (DataContext.java:1194)
>>     at org.objectstyle.cayenne.access.DataContext.flushToParent 
>> (DataContext.java:1261)
>>     at org.objectstyle.cayenne.access.DataContext.commitChanges 
>> (DataContext.java:1165)
>>     at  
>> com.bbcnews.mps.transcode.TranscodeProcessor.triggerTranscodes 
>> (TranscodeProcessor.java:265)
>>     at  
>> com.bbcnews.mps.transcode.TranscodeProcessor.processRequestMessage 
>> (TranscodeProcessor.java:237)
>>     at com.bbcnews.mps.transcode.TranscodeProcessor.processMessage 
>> (TranscodeProcessor.java:148)
>>     at com.bbcnews.jms.MessageListenerProxy.onMessage 
>> (MessageListenerProxy.java:134)
>>     at progress.message.jimpl.Session.rbB_(Unknown Source)
>>     at progress.message.jimpl.QueueSession.run(Unknown Source)
>>     at progress.message.jimpl.ep.run(Unknown Source)
>>
>> "EventDispatchThread-3" Id=15 in BLOCKED on  
>> lock=org.objectstyle.cayenne.access.ObjectStore@1ce5e7a
>>      owned by JMS Session Delivery Thread Id=31
>>     at  
>> org.objectstyle.cayenne.access.ObjectStore.processSnapshotEvent 
>> (ObjectStore.java:813)
>>     at  
>> org.objectstyle.cayenne.access.DataContextMergeHandler.graphChanged 
>> (DataContextMergeHandler.java:131)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown  
>> Source)
>>     at java.lang.reflect.Method.invoke(Unknown Source)
>>     at org.objectstyle.cayenne.util.Invocation.fire 
>> (Invocation.java:240)
>>     at org.objectstyle.cayenne.event.EventManager$Dispatch.fire 
>> (EventManager.java:433)
>>     at org.objectstyle.cayenne.event.DispatchQueue.dispatchEvent 
>> (DispatchQueue.java:182)
>>     at org.objectstyle.cayenne.event.DispatchQueue.dispatchEvent 
>> (DispatchQueue.java:94)
>>     at org.objectstyle.cayenne.event.EventManager.dispatchEvent 
>> (EventManager.java:372)
>>     at org.objectstyle.cayenne.event.EventManager.postEvent 
>> (EventManager.java:343)
>>     at  
>> org.objectstyle.cayenne.access.DataContext.fireDataChannelChanged 
>> (DataContext.java:1731)
>>     at  
>> org.objectstyle.cayenne.access.ObjectStore.processSnapshotEvent 
>> (ObjectStore.java:842)
>>     at org.objectstyle.cayenne.access.ObjectStore.snapshotsChanged 
>> (ObjectStore.java:804)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown  
>> Source)
>>     at java.lang.reflect.Method.invoke(Unknown Source)
>>     at org.objectstyle.cayenne.util.Invocation.fire 
>> (Invocation.java:240)
>>     at org.objectstyle.cayenne.event.EventManager 
>> $InvocationDispatch.fire(EventManager.java:452)
>>     at org.objectstyle.cayenne.event.EventManager 
>> $DispatchThread.run(EventManager.java:499)	public void  
>> processMessage(final WrappedMessage message) {
>> 		if (!(message instanceof CayenneNotificationMessage)) {
>> 			LOG.error("Not a valid notication message class: "
>> 					+ message.getClass().getName());
>> 		}
>>
>> 		LOG.debug("Received a notification message");
>> 		try {
>> 			final String vmID = ((CayenneNotificationMessage) message)
>> 					.getRuntimeId();
>>
>> 			LOG.debug("My VM: " + VM_ID);
>> 			LOG.debug("Event VM: " + vmID);
>> 			if (VM_ID.equals(vmID)) {
>> 				LOG.debug("... from the same VM!");
>> 				return;
>> 			}
>>
>> 			final CayenneEvent event = (CayenneEvent)  
>> ((CayenneNotificationMessage) message)
>> 					.getEvent();
>> 			if (event != null) {
>> 				LOG.debug("... processing the event");
>> 				onExternalEvent(event);
>> 			}
>> 		} catch (Exception e) {
>> 			LOG.error("Random notification exception: "
>> 					+ e.getLocalizedMessage());
>> 		}
>> 	}
>>
>>
>> 	protected void sendExternalEvent(final CayenneEvent localEvent)  
>> throws JMSException {
>> 		LOG.debug("Sending an event");
>> 		final CayenneNotificationMessage message = new  
>> CayenneNotificationMessage();
>> 		message.setRuntimeId(VM_ID);
>> 		message.setEvent(localEvent);
>> 		sendProxy.sendMessageToTopic(message, topicName);
>> 		LOG.debug("Sent the event");
>> 	}
>
>

Re: Deadlock.

[Andrus Adamchik <an...@objectstyle.org>]

Do you have a full thread dump printed by JVM? From these two threads  
I can't figure out the nature of a deadlock.

Thanks
Andrus

On Dec 8, 2006, at 2:22 PM, John Gunning wrote:

> Hi there,
>
>
> I am experiencing a deadlock issue which looks similar
> to CAY-297. I understand CAY-297 was fixed in release
> 1.1 but I am using version 1.2.1 and the deadalock
> issue arises. When I set
> "cayenne.DataRowStore.remote.notify" to false the
> issue does not arise. However, I need this set to true
> so that remote web services can be notified. I'm also
> extending the Cayenne EventBridge class. I attach the
> stack traces of the deadlocking threads as well as the
> processMessage and sendExternalEvent methods of the
> EventBridge extension. Any ideas anyone?...
>
>
> Thanks in advance,
>
> John Gunning.
>
> Send instant messages to your online friends http:// 
> uk.messenger.yahoo.com Deadlock found :-
> "JMS Session Delivery Thread" Id=31 in BLOCKED on  
> lock=org.objectstyle.cayenne.access.ObjectStore@cc7f9e
>      owned by EventDispatchThread-3 Id=15
>     at org.objectstyle.cayenne.access.ObjectStore.registerDiff 
> (ObjectStore.java:200)
>     at  
> org.objectstyle.cayenne.access.ObjectStore.recordObjectCreated 
> (ObjectStore.java:152)
>     at org.objectstyle.cayenne.access.DataContext.registerNewObject 
> (DataContext.java:922)
>     at org.objectstyle.cayenne.access.ChildDiffLoader.nodeCreated 
> (ChildDiffLoader.java:110)
>     at org.objectstyle.cayenne.graph.NodeCreateOperation.apply 
> (NodeCreateOperation.java:73)
>     at org.objectstyle.cayenne.graph.CompoundDiff.apply 
> (CompoundDiff.java:133)
>     at org.objectstyle.cayenne.access.ObjectStoreGraphDiff.apply 
> (ObjectStoreGraphDiff.java:155)
>     at org.objectstyle.cayenne.access.DataContext.onContextFlush 
> (DataContext.java:1215)
>     at org.objectstyle.cayenne.access.DataContext.onSync 
> (DataContext.java:1194)
>     at org.objectstyle.cayenne.access.DataContext.flushToParent 
> (DataContext.java:1261)
>     at org.objectstyle.cayenne.access.DataContext.commitChanges 
> (DataContext.java:1165)
>     at  
> com.bbcnews.mps.transcode.TranscodeProcessor.triggerTranscodes 
> (TranscodeProcessor.java:265)
>     at  
> com.bbcnews.mps.transcode.TranscodeProcessor.processRequestMessage 
> (TranscodeProcessor.java:237)
>     at com.bbcnews.mps.transcode.TranscodeProcessor.processMessage 
> (TranscodeProcessor.java:148)
>     at com.bbcnews.jms.MessageListenerProxy.onMessage 
> (MessageListenerProxy.java:134)
>     at progress.message.jimpl.Session.rbB_(Unknown Source)
>     at progress.message.jimpl.QueueSession.run(Unknown Source)
>     at progress.message.jimpl.ep.run(Unknown Source)
>
> "EventDispatchThread-3" Id=15 in BLOCKED on  
> lock=org.objectstyle.cayenne.access.ObjectStore@1ce5e7a
>      owned by JMS Session Delivery Thread Id=31
>     at  
> org.objectstyle.cayenne.access.ObjectStore.processSnapshotEvent 
> (ObjectStore.java:813)
>     at  
> org.objectstyle.cayenne.access.DataContextMergeHandler.graphChanged 
> (DataContextMergeHandler.java:131)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>     at java.lang.reflect.Method.invoke(Unknown Source)
>     at org.objectstyle.cayenne.util.Invocation.fire(Invocation.java: 
> 240)
>     at org.objectstyle.cayenne.event.EventManager$Dispatch.fire 
> (EventManager.java:433)
>     at org.objectstyle.cayenne.event.DispatchQueue.dispatchEvent 
> (DispatchQueue.java:182)
>     at org.objectstyle.cayenne.event.DispatchQueue.dispatchEvent 
> (DispatchQueue.java:94)
>     at org.objectstyle.cayenne.event.EventManager.dispatchEvent 
> (EventManager.java:372)
>     at org.objectstyle.cayenne.event.EventManager.postEvent 
> (EventManager.java:343)
>     at  
> org.objectstyle.cayenne.access.DataContext.fireDataChannelChanged 
> (DataContext.java:1731)
>     at  
> org.objectstyle.cayenne.access.ObjectStore.processSnapshotEvent 
> (ObjectStore.java:842)
>     at org.objectstyle.cayenne.access.ObjectStore.snapshotsChanged 
> (ObjectStore.java:804)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>     at java.lang.reflect.Method.invoke(Unknown Source)
>     at org.objectstyle.cayenne.util.Invocation.fire(Invocation.java: 
> 240)
>     at org.objectstyle.cayenne.event.EventManager 
> $InvocationDispatch.fire(EventManager.java:452)
>     at org.objectstyle.cayenne.event.EventManager$DispatchThread.run 
> (EventManager.java:499)	public void processMessage(final  
> WrappedMessage message) {
> 		if (!(message instanceof CayenneNotificationMessage)) {
> 			LOG.error("Not a valid notication message class: "
> 					+ message.getClass().getName());
> 		}
>
> 		LOG.debug("Received a notification message");
> 		try {
> 			final String vmID = ((CayenneNotificationMessage) message)
> 					.getRuntimeId();
>
> 			LOG.debug("My VM: " + VM_ID);
> 			LOG.debug("Event VM: " + vmID);
> 			if (VM_ID.equals(vmID)) {
> 				LOG.debug("... from the same VM!");
> 				return;
> 			}
>
> 			final CayenneEvent event = (CayenneEvent)  
> ((CayenneNotificationMessage) message)
> 					.getEvent();
> 			if (event != null) {
> 				LOG.debug("... processing the event");
> 				onExternalEvent(event);
> 			}
> 		} catch (Exception e) {
> 			LOG.error("Random notification exception: "
> 					+ e.getLocalizedMessage());
> 		}
> 	}
>
>
> 	protected void sendExternalEvent(final CayenneEvent localEvent)  
> throws JMSException {
> 		LOG.debug("Sending an event");
> 		final CayenneNotificationMessage message = new  
> CayenneNotificationMessage();
> 		message.setRuntimeId(VM_ID);
> 		message.setEvent(localEvent);
> 		sendProxy.sendMessageToTopic(message, topicName);
> 		LOG.debug("Sent the event");
> 	}