You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@xmlbeans.apache.org by Cezar Andrei <ce...@oracle.com> on 2008/11/06 17:53:37 UTC
RE: RE: XmlBeans and Synchronization
You're right Paul, using setXXX methods like you describer can get to a deadlock. We're working on a fix for this.
Cezar
________________________________
From: Paul Hepworth [mailto:Paul.hepworth@detica.com]
Sent: Thursday, October 30, 2008 3:42 AM
To: dev@xmlbeans.apache.org; cezar.andrei@oracle.com
Subject: RE: RE: XmlBeans and Synchronization
Hi Cezar
The deadlock isn't happening because of code in XmlObjectBase, but rather because of generated code from the schema.
So when I compile our schema, if I look at any setXXX(.) method in the generated sources, it synchronizes on its monitor first.
There is then a call to check_orphaned(), followed by a call to get_store().find_element_user(.) (or add_element_user(.) if it's not in the store), and finally a call to set(.) which ends up in the XmlObjectBase set method, which will then do as you've described.
The issue that is causing the deadlock is that the generated code has already synchronized on the monitor (step 1a in my mail).
Thanks
Paul
________________________________
From: Cezar Andrei [mailto:cezar.andrei@oracle.com]
Sent: 29 October 2008 21:08
To: dev@xmlbeans.apache.org
Subject: RE: RE: XmlBeans and Synchronization
Paul,
Actually the code that does the set() synchronization is the following (see XmlObjectBase:2048):
// about to grab two locks: don't deadlock ourselves
GlobalLock.acquire();
acquired = true;
synchronized (monitor())
{
synchronized (obj.monitor())
{
GlobalLock.release();
acquired = false;
newObj = setterHelper( obj );
}
}
There are 3 places where a thread can get stuck:
1) before GlobalLock.acquire(): Since it's a GlobalLock, only one thread can run between GlobalLock.acquire(); and GlobalLock.release(); at one time.
2) Before synchronized (monitor()): which means that another thread is doing something on this object, but since the other thread doesn't have the global lock, it means that the other thread should have everything needed to finish its operation.
3) Before synchronized (obj.monitor()): same as before but on the second object.
Assuming that the application that is using XMLBeans doesn't use GlobalLock, or uses some other external synchronization, I don't see how a deadlock could happen in your scenario.
Cezar
________________________________
From: Paul Hepworth [mailto:Paul.hepworth@detica.com]
Sent: Wednesday, October 29, 2008 11:37 AM
To: dev@xmlbeans.apache.org
Subject: RE: RE: XmlBeans and Synchronization
All
I believe I have found a defect with XmlBeans which can cause a deadlock to occur under a specific situation.
The situation can occur when calling a setter on an Object which synchronizes on Locale A, to set an Object which synchronizes on Locale B at the same time as calling a setter on an Object which synchronizes on Locale B to set an Object which synchronizes on Locale A.
The code which XmlBeans generates follows this flow:
1. MyObjectUnderLocaleA.setMyElement(MyObjectUnderLocaleB)
a) Synchronize on monitor()
b) Delegates to XmlObjectBase.set(XmlObject src)
i. Identifies that MyObjectUnderLocaleA.monitor() != MyObjectUnderLocaleB.monitor()
ii. GlobalLock.acquire
iii. Synchronize on monitor()
iv. Synchronize on MyObjectUnderLocaleB.monitor()
So if the reverse is happening concurrently, Thread 1 will have a lock on Locale A and the GlobalLock, and Thread 2 will have a lock on Locale B and wait to get the GlobalLock - hence the deadlock situation.
It may be that this is arising from a strange/undesired use of XmlBeans - we have a HashMap which caches template XmlBean objects, and when we need an instance of the template, we were calling the copy() method and returning that. I believe this means the template and our instance share the same locale. Once the number of users ramps up, there are a lot of users sharing the same locale and consequently introducing a lot of contention.
We have worked around the problem for now by performing an XmlObject.Factory.parse(...) and so each returned object is then in its own Locale.
If someone can confirm this, can it be raised as a defect? If it's not a defect, then is there a way to do a deep copy that will create the new object in a new Locale?
Thanks
Paul
-----Original Message-----
From: Paul Hepworth
Sent: 27 October 2008 14:28
To: 'dev@xmlbeans.apache.org'
Subject: RE: XmlBeans and Synchronization
Hi Radu
Thanks for the explanation. In theory it all seems sound. Only our application is getting itself locked somehow.
I've done a thread-dump from Weblogic, and have multiple threads waiting on XmlObjectBase.java:1944 which is fine and to be expected as per your explanation.
I then have another thread which is stuck on XmlObjectBase.java:1949 - so this is the one that has the GlobalLock, but is in turn waiting on another lock which isn't getting released at all.
I have several other threads that are blocked on XmlObjectBase.java:116 performing a copy which I believe are fine and just waiting for a lock.
I then have a few threads performing simple gets that are blocked waiting for locks on Locale.
I have no other threads that are doing any processing (I've left the app in this state for 10 or so minutes and all the threads are just locked up)
As I can't find much info on the net, could you explain a bit more about the Local please?
What I'd like to know is the following:
- Is there a new Local per instance of a document or a shared one for all instances of the same type? i.e. if I call MyDocument.Factory.newInstance(), does this get a new Local each time, or 1 shared one for all MyDocument instances?
- We have a template service that reads and parses XML stored in the DB, this is cached for performance reasons, and when we request a template, we call the copy() method. Does this mean that each copied instance shares the same Locale? If that's the case, this will greatly impact the performance of our app.
- Basically... what determines the Locale?
Additionally, you mention turning off synchronisation... I've attempted this in the code using XmlBeans 2.3.0 and run into the problems mentioned here: https://issues.apache.org/jira/browse/XMLBEANS-328 it should also be noted that we were having these errors in our performance test environment originally and applied the patch mentioned. This stopped the errors described, however we were still running into the deadlock situation.
Is this something which has been fixed in 2.4? Is it a safe option to turn off synchronisation?
Any help greatly appreciated!
Thanks
Paul
Hi Paul,
I can tell you what the synchronization strategy is. Each XmlObject belongs to an xml document,
and each xml document belongs to what XmlBeans calls a Locale object (nothing to do with the
Java Locale). This is the synchronization domain for XmlBeans. When doing a set, there are
two XmlObjects involved: the source (the parameter of the "set" method) and the target (the
object you're calling "set" on). XMLBeans requires that they both be synchronized. Now, if
the synchronization domain for the two is the same (the same Locale object), then there's
no problem (see check at XmlObjectBase.java:1934 if you have the 2.3.0 source). But if the
synchronization domain is not the same, then it means that two locks must be acquired and
so there is a possibility for deadlock. In order to avoid the deadlock, XMLBeans has the "GlobalLock".
This global lock is held only for as long as it is necessary to acquire the two XmlObject
locks and then it is immediately released. This guards against the deadlock, but doesn't guard
against the possibility that one thread grabs the global lock and then blocks on waiting for
another thread to finish with one of the XmlObject locks it is trying to acquire. Eventually,
it will grab the lock it is waiting for and then release the Global lock, but while it is
waiting, all the other threads in the whole system are prevented from initiating a set that
requires two locks, which can be a problem.
So I think first of all, it would be interesting for you to look at the stack traces for all
the threads to confirm my scenario above. Then, as far as solution, the only reliable one
that I know of is not use the "setMyValue" method and replace it with "addNewMyValue" whenever
possible because it doesn't require synchronization (and the overall performance is better).
In XMLBeans 2.4.0, I think, removing synchronization altogether from the XMLBeans layer would
also solve the problem.
Radu
________________________________
From: Paul Hepworth [mailto:Paul.hepworth@detica.com]
Sent: Friday, October 24, 2008 5:00 AM
To: user@xmlbeans.apache.org; dev@xmlbeans.apache.org
Subject: XmlBeans and Synchronization
Hi (included dev list as this may be too in depth for the user list)
I'm using XmlBeans 2.3.0 in a Weblogic 9.2 J2EE web application (JVM 1.5.0_10). We use XmlBeans
to compile our schema and these effectively become our domain objects. These are looked up
from the DB, stored in the Http Session, manipulated from the UI and passed to the DB again.
The application appears to run fine with a single user, however when we test it with multiple
users using LoadRunner, we hit issues.
What we are seeing is that we're getting stuck threads with the following stack:
at java.lang.Object.wait(Native Method)
- waiting on <0xd4a816e8> (a org.apache.xmlbeans.impl.common.Mutex)
at java.lang.Object.wait(Object.java:474)
at org.apache.xmlbeans.impl.common.Mutex.acquire(Mutex.java:33)
- locked <0xd4a816e8> (a org.apache.xmlbeans.impl.common.Mutex)
at org.apache.xmlbeans.impl.common.GlobalLock.acquire(GlobalLock.java:27)
at org.apache.xmlbeans.impl.values.XmlObjectBase.set(XmlObjectBase.java:1944)
at ...MyClassImpl.setMyValue(...)
Can anyone shed any light on why this may happen?
More specifically, does anyone know what the synchronisation strategy is in XmlBeans. From
my investigation, the GlobalLock uses a static Mutex, which means that if an instance needs
to take out more than 1 lock, it attempts to acquire a GlobalLock and will subsequently lock
out all other threads in the JVM from making any changes to an XmlObject.
If that's the case, it seems very strange that making an update to 1 instance of an XmlObject
would lock out all other threads form updating any other instance of an XmlObject. I would
have thought that the synchronisation would have solely been around the instance itself.
We have also seen the same issues that have been recorded here: https://issues.apache.org/jira/browse/XMLBEANS-328
Any help on this would be much appreciated as it's causing major issues!!
Thanks
Paul
This message should be regarded as confidential. If you have received this email in error
please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard copy by an authorised
signatory. The contents of this email may relate to dealings with other companies within the
Detica Group plc group of companies.
Detica Limited is registered in England under No: 1337451.
Registered offices: Surrey Research Park, Guildford, Surrey, GU2 7YP, England.