You are viewing a plain text version of this content. The canonical link for it is here.
Posted to muse-dev@ws.apache.org by rl...@xecu.net on 2007/08/24 22:37:21 UTC
EMPTY_DOC thread stability issues
Use of EMPTY_DOC (an instance of an empty DOM Document element within
class XmlUtils) among multiple threads causes unpredictable results.
While testing a messaging service that uses the Apache Muse WS-N
implementation, I noticed that 4-6 messages in a 1000 were being dropped.
Notification messages were being sent at a rate of 20 per second. Where
the messages were dropped, the following stack trace occurred:
2007-08-24 15:27:58,031 ERROR [STDERR] Aug 24, 2007 3:27:58 PM
org.apache.muse.util.LoggingUtils logError
INFO: There was an error while processing a request:
[ID = 'NoMessageContent'] The NotificationMessage XML does not have a
Message element. All messages must have a message payload associated with
them.
org.apache.muse.ws.notification.impl.SimpleNotificationMessage.<init>(SimpleNotificationMessage.java:117)
org.apache.muse.ws.notification.impl.NotificationMessageSerializer.fromXML(NotificationMessageSerializer.java:46)
org.apache.muse.core.serializer.ArraySerializer.fromXML(ArraySerializer.java:126)
org.apache.muse.ws.notification.impl.NotifyHandler.fromXML(NotifyHandler.java:62)
org.apache.muse.core.SimpleResource.invoke(SimpleResource.java:368)
org.apache.muse.core.routing.SimpleResourceRouter.invoke(SimpleResourceRouter.java:290)
org.apache.muse.core.platform.axis2.AxisIsolationLayer.invoke(AxisIsolationLayer.java:136)
org.apache.muse.core.platform.axis2.AxisIsolationLayer.handleRequest(AxisIsolationLayer.java:88)
sun.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:324)
org.apache.axis2.receivers.RawXMLINOutMessageReceiver.invokeBusinessLogic(RawXMLINOutMessageReceiver.java:88)
org.apache.axis2.receivers.AbstractInOutSyncMessageReceiver.receive(AbstractInOutSyncMessageReceiver.java:39)
org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:493)
org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:319)
22 more...
All of the messages did in fact have a message payload associated with
them. After further investigation I noticed that
AxisEnvironment.convertToDOM is calling XmlUtils.createElement(QName)
which uses the common EMPTY_DOC Document instance. Sharing EMPTY_DOC
among multiple threads is not safe.
I've also encountered this issue while attempting to set the producer
reference on a NotificationMessage. The code actually performs a deep
copy of the EndpointReference class which also relies on EMPTY_DOC. I
hacked a work-around within my code that avoided use of EMPTY_DOC and
resolved the issue.
A grep for EMPTY_DOC against the code base turned up over 70 instances of
its use. That does not count methods that use it that are called by other
methods, so the actual usage is much higher. I'm going to open a JIRA bug
against this issue and submit patches for it.
Related to that, I've submitted patches for other issues (MUSE-240,
MUSE-241) over a month ago that have not been committed. The patch for
MUSE-240 also resolves MUSE-225. Please let me know the appropriate
process for non-committers to have patches applied to the code base. When
I submitted the JIRA issues they showed up in this mailing list, but is an
explicit email more appropriate? Let me know how best to proceed.
Thanks!
Rich Lucente
---------------------------------------------------------------------
To unsubscribe, e-mail: muse-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: muse-dev-help@ws.apache.org
RE: EMPTY_DOC thread stability issues
Posted by "Vinh Nguyen (vinguye2)" <vi...@cisco.com>.
Interesting...
"Is Xerces DOM implementation thread-safe? No."
http://xerces.apache.org/xerces2-j/faq-dom.html
________________________________
From: Vinh Nguyen (vinguye2)
Sent: Friday, August 31, 2007 10:31 AM
To: muse-dev@ws.apache.org
Subject: RE: EMPTY_DOC thread stability issues
I should add...I locally updated the EndpointReference class and changed
all XmlUtils.EMPTY_DOC references to XmlUtils.createDocument(). I no
longer get errors related to EndpointReference, but now get other errors
in other classes like SimpleNotificationMessage which attempts to
serialize the message. I'm mostly concerned with serialization at this
point, since it affects both notifications and operation responses.
Will test further...
________________________________
From: Vinh Nguyen (vinguye2)
Sent: Friday, August 31, 2007 10:06 AM
To: muse-dev@ws.apache.org
Subject: RE: EMPTY_DOC thread stability issues
Hi Dan,
You're probably be right. I'll continue to test further. I think I
heard of issues with Xerces and will check to see if this is related.
-Vinh
________________________________
From: Daniel Jemiolo [mailto:danjemiolo@us.ibm.com]
Sent: Friday, August 31, 2007 7:30 AM
To: muse-dev@ws.apache.org
Subject: RE: EMPTY_DOC thread stability issues
The problem isn't that EMPTY_DOC is being modified - that would only
happen if we appended to the Document itself. Since Document is a
factory, we can use it to create fragments that, while their "parent" is
the Document, they are not actually attached to it (otherwise you'd get
RuntimeExceptions about the Document having more than one root element).
For reasons that I do not understand, the Xerces parser/factory is not
stateless, and that is causing the concurrency bugs. The simple act of
creating new elements at the same time generates the exception. Either
way, we'll have to remove it.
Dan
"Vinh Nguyen \(vinguye2\)" <vi...@cisco.com> wrote on 08/31/2007
04:34:28 AM:
> Hi Dan,
> I've also done some testing just now and am finding this to be a
> very serious issue. It really affects the usability of Muse as a
whole.
>
> To test, I created three resources. Each had a loop to generate 100
> notifications with no pause between notifications. So out of 300
> total notifications generated, I had these results:
> Test A = 9 notifications dropped
> Test B = 24 notifications dropped
>
> Attached is a file containing the various exceptions which caused
> SimpleSubscriptionManager.publish() to fail. As Rich pointed out,
> the culprit is using EMPTY_DOC from multiple threads (i.e. each
> client request is one thread).
>
> The XmlUtils.EMPTY_DOC javadocs has this:
> "This should NOT be used by callers that want to build new fragments
> and attach them to a Document...you should never append children."
>
> But in XmlUtils itself and many other classes, the following code
> patterns are used:
>
> Document doc = XmlUtils.EMPTY_DOC;
> Element xml = XmlUtils.createElement(doc, qname);
> xml.appendChild(node);
>
> OR:
> doc.importNode(node, true);
> The doc is a shared object but is being modified, so errors will
> occur. So just about all the serializers are affected, including
> the EndpointReference class. This means errors will most likely
> occur when Muse handles requests/responses from multiple clients, or
> when notifications are sent from multiple resources. The latter is
> easier to test.
>
> So far, we've been testing using just one client and one producer
> instance, so the problem doesn't occur. But now that we are testing
> by using multiple producers, the exceptions are occuring frequently.
>
> To begin the fix, all serializers and the EndpointReference class
> needs to be patched. So instead of:
> doc = XmlUtils.EMPTY_DOC;
>
> We should do:
> doc = XmlUtils.createDocument();
>
> The overhead of creating a new document is small when compared to
> the multi thread issue, which cannot be avoided. But, we can
> optimize creating new documents by updating XmlUtils to create the
> DocumentBuilderFactory only once. This way, XmlUtils.
> createBuilder() doesn't have to create a new factory everytime.
> -Vinh
> From: Daniel Jemiolo [mailto:danjemiolo@us.ibm.com]
> Sent: Monday, August 27, 2007 7:31 AM
> To: muse-dev@ws.apache.org
> Subject: Re: EMPTY_DOC thread stability issues
> The use of EMPTY_DOC was an attempt to avoid creating a new Document
> every time we wanted to copy or create small fragments of XML. This
> happened a lot during the request/response process, so the creation
> of these factory objects was not insignificant. Of course, you are
> right about the threading issue, so I guess we're out of luck there.
>
> I am setting aside Friday of this week to go through current JIRA
> items and apply all patches that have been submitted, close any
> issues that are recommended for closure, and sort unscheduled items
> into 2.3 if necessary. Sorry for the delay.
>
> Dan
>
>
> [image removed] rlucente@xecu.net
>
>
> rlucente@xecu.net
> 08/24/2007 04:37 PM
>
> Please respond to
> muse-dev@ws.apache.org
>
> [image removed]
> To
>
> [image removed]
> muse-dev@ws.apache.org
>
> [image removed]
> cc
>
> [image removed]
>
> [image removed]
> Subject
>
> [image removed]
> EMPTY_DOC thread stability issues
>
> [image removed]
>
> [image removed]
>
>
> Use of EMPTY_DOC (an instance of an empty DOM Document element within
> class XmlUtils) among multiple threads causes unpredictable results.
> While testing a messaging service that uses the Apache Muse WS-N
> implementation, I noticed that 4-6 messages in a 1000 were being
dropped.
> Notification messages were being sent at a rate of 20 per second.
Where
> the messages were dropped, the following stack trace occurred:
>
> 2007-08-24 15:27:58,031 ERROR [STDERR] Aug 24, 2007 3:27:58 PM
> org.apache.muse.util.LoggingUtils logError
> INFO: There was an error while processing a request:
>
> [ID = 'NoMessageContent'] The NotificationMessage XML does not have a
> Message element. All messages must have a message payload associated
with
> them.
>
> org.apache.muse.ws.notification.impl.
> SimpleNotificationMessage.<init>(SimpleNotificationMessage.java:117)
> org.apache.muse.ws.notification.impl.
>
NotificationMessageSerializer.fromXML(NotificationMessageSerializer.java
:46)
> org.apache.muse.core.serializer.ArraySerializer.
> fromXML(ArraySerializer.java:126)
> org.apache.muse.ws.notification.impl.NotifyHandler.
> fromXML(NotifyHandler.java:62)
>
org.apache.muse.core.SimpleResource.invoke(SimpleResource.java:368)
> org.apache.muse.core.routing.SimpleResourceRouter.
> invoke(SimpleResourceRouter.java:290)
> org.apache.muse.core.platform.axis2.AxisIsolationLayer.
> invoke(AxisIsolationLayer.java:136)
> org.apache.muse.core.platform.axis2.AxisIsolationLayer.
> handleRequest(AxisIsolationLayer.java:88)
> sun.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)
> sun.reflect.DelegatingMethodAccessorImpl.
> invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:324)
> org.apache.axis2.receivers.RawXMLINOutMessageReceiver.
> invokeBusinessLogic(RawXMLINOutMessageReceiver.java:88)
> org.apache.axis2.receivers.AbstractInOutSyncMessageReceiver.
> receive(AbstractInOutSyncMessageReceiver.java:39)
> org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:493)
> org.apache.axis2.transport.http.HTTPTransportUtils.
> processHTTPPostRequest(HTTPTransportUtils.java:319)
> 22 more...
>
> All of the messages did in fact have a message payload associated with
> them. After further investigation I noticed that
> AxisEnvironment.convertToDOM is calling XmlUtils.createElement(QName)
> which uses the common EMPTY_DOC Document instance. Sharing EMPTY_DOC
> among multiple threads is not safe.
>
> I've also encountered this issue while attempting to set the producer
> reference on a NotificationMessage. The code actually performs a deep
> copy of the EndpointReference class which also relies on EMPTY_DOC. I
> hacked a work-around within my code that avoided use of EMPTY_DOC and
> resolved the issue.
>
> A grep for EMPTY_DOC against the code base turned up over 70 instances
of
> its use. That does not count methods that use it that are called by
other
> methods, so the actual usage is much higher. I'm going to open a JIRA
bug
> against this issue and submit patches for it.
>
> Related to that, I've submitted patches for other issues (MUSE-240,
> MUSE-241) over a month ago that have not been committed. The patch
for
> MUSE-240 also resolves MUSE-225. Please let me know the appropriate
> process for non-committers to have patches applied to the code base.
When
> I submitted the JIRA issues they showed up in this mailing list, but
is an
> explicit email more appropriate? Let me know how best to proceed.
>
> Thanks!
>
> Rich Lucente
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: muse-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: muse-dev-help@ws.apache.org
>
> [attachment "EMPTY_DOC_errors.txt" deleted by Daniel
Jemiolo/Durham/IBM]
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: muse-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: muse-dev-help@ws.apache.org
RE: EMPTY_DOC thread stability issues
Posted by rl...@xecu.net.
I like this solution, but care still needs to be taken to ensure that
original documents are created as needed. There's 70+ locations in the
code where this needs to be examined. If I understand the software
correctly, when a WS-N Notify request is received, a convertToDOM occurs
and then the notify request is forwarded to the NotificationConsumer
implementation. The default implementation creates a thread for
processing the notify request, and then immediately releases the thread
that called it. The same worker thread may try to share a document that
is also being processed by another newly created thread, and we're back to
the same problem. I'm not sure ThreadLocal helps since the worker threads
(managed by the web container) and newly created threads could share the
same object.
> Hi,
>
> due to the discussion on the mailing list I did some experiments with muse
> related to the thread stability issues. As a result I would just come up
> with a small fix that should solve the majority of the problems and also
> takes the performance issues into account that lead to the creation of
> EMPTY_DOC. I would suggest to extend XmlUtils with the following piece of
> code:
>
> private static final ThreadLocal local = new ThreadLocal();
>
> public static synchronized Document getLocalEmptyDoc() {
> Document doc = (Document)local.get();
> if (doc == null) {
> doc = createDocument();
> local.set(doc);
> }
> return doc;
> }
>
> The usage of EMPTY_DOC should be replaced by a call to getLocalEmptyDoc().
> Existing functionality can remain untouched.
>
> However, I don't think that this solves all the problems. One other
> problem is that for the creation of permanent objects like EPR's of
> resources, separate XML documents have to be used. One of the problems was
> that multiple threads (resources) accessed one Resource EPR (or the _xml
> object inside the EPR) at the same time when a NotificationMessage was
> created. Hereby the setProducerReference() and setSubscriptionReference()
> of SimpleNotificatioMessage were invoked transparently.
>
> Furthermore, I would suggest to use a new Xml Document within
>
> public EndpointReference(EndpointReference copy, QName typeName) {
> ...
> Document doc = XmlUtils.createDocument();
> ...
> }
>
> This also contibutes to stability, since this constructor is used within
> the implementation to create the EPR instances of new resources.
>
> With the mentioned changes I obtained quite good results in my test setup
> (notification provider+consumer sample with 3 to 10 resources each) with
> MUSE. This means I did not experience exceptions anymore, but I think this
> has to be tested in more detail.
>
> Oliver
>
> -------- Original-Nachricht --------
>> Datum: Fri, 31 Aug 2007 15:09:21 -0400
>> Von: Daniel Jemiolo <da...@us.ibm.com>
>> An: muse-dev@ws.apache.org
>> Betreff: RE: EMPTY_DOC thread stability issues
>
>>
>>
>> Yeah... this is what led to the creation of EMPTY_DOC. The exceptions
>> you're seeing now are caused by you creating an Element with one factory
>> (Document) and then trying to append it to another one. For example,
>> NotificationMessage serialization will create a Document and start
>> appending Elements... one of those will be an EPR, which serializes
>> itself
>> with its own Document. In order to prevent the exception, the
>> NotificationMessage code needs to call Document.importNode() on the EPR
>> Element (or have toXML() somehow return the Document it used along with
>> the
>> serialized XML, which would make the API really ugly).
>>
>> The importNode() method does a deep copy of the Element; there is no way
>> to
>> just switch an Element's parent Document. As you start to replace
>> EMPTY_DOC
>> with calls to createDocument() *and* importNode(), you will see lots of
>> XML
>> fragments being copied lots of times on the way to a complete SOAP
>> message.
>> So, if you think there's too much object creation now, you ain't seen
>> nothing yet. ;)
>>
>> Dan
>>
>>
>>
>> "Vinh Nguyen \(vinguye2\)" <vi...@cisco.com> wrote on 08/31/2007
>> 01:30:57 PM:
>>
>> > I should add...I locally updated the EndpointReference class and
>> > changed all XmlUtils.EMPTY_DOC references to XmlUtils.
>> > createDocument(). I no longer get errors related to
>> > EndpointReference, but now get other errors in other classes like
>> > SimpleNotificationMessage which attempts to serialize the message.
>> > I'm mostly concerned with serialization at this point, since it
>> > affects both notifications and operation responses. Will test
>> further...
>> >
>> > From: Vinh Nguyen (vinguye2)
>> > Sent: Friday, August 31, 2007 10:06 AM
>> > To: muse-dev@ws.apache.org
>> > Subject: RE: EMPTY_DOC thread stability issues
>>
>> > Hi Dan,:
>> > You're probably be right. I'll continue to test further. I think I
>> > heard of issues with Xerces and will check to see if this is related.
>> > -Vinh
>> >
>> > From: Daniel Jemiolo [mailto:danjemiolo@us.ibm.com]
>> > Sent: Friday, August 31, 2007 7:30 AM
>> > To: muse-dev@ws.apache.org
>> > Subject: RE: EMPTY_DOC thread stability issues
>>
>> > The problem isn't that EMPTY_DOC is being modified - that would only
>> > happen if we appended to the Document itself. Since Document is a
>> > factory, we can use it to create fragments that, while their
>> > "parent" is the Document, they are not actually attached to it
>> > (otherwise you'd get RuntimeExceptions about the Document having
>> > more than one root element).
>> >
>> > For reasons that I do not understand, the Xerces parser/factory is
>> > not stateless, and that is causing the concurrency bugs. The simple
>> > act of creating new elements at the same time generates the
>> > exception. Either way, we'll have to remove it.
>> >
>> > Dan
>> >
>> >
>> >
>> > "Vinh Nguyen \(vinguye2\)" <vi...@cisco.com> wrote on 08/31/2007
>> > 04:34:28 AM:
>> >
>> > > Hi Dan,
>> > > I've also done some testing just now and am finding this to be a
>> > > very serious issue. It really affects the usability of Muse as a
>> whole.
>> > >
>> > > To test, I created three resources. Each had a loop to generate 100
>> > > notifications with no pause between notifications. So out of 300
>> > > total notifications generated, I had these results:
>> > > Test A = 9 notifications dropped
>> > > Test B = 24 notifications dropped
>> > >
>> > > Attached is a file containing the various exceptions which caused
>> > > SimpleSubscriptionManager.publish() to fail. As Rich pointed out,
>> > > the culprit is using EMPTY_DOC from multiple threads (i.e. each
>> > > client request is one thread).
>> > >
>> > > The XmlUtils.EMPTY_DOC javadocs has this:
>> > > "This should NOT be used by callers that want to build new fragments
>> > > and attach them to a Document...you should never append children."
>> > >
>> > > But in XmlUtils itself and many other classes, the following code
>> > > patterns are used:
>> > >
>> > > Document doc = XmlUtils.EMPTY_DOC;
>> > > Element xml = XmlUtils.createElement(doc, qname);
>> > > xml.appendChild(node);
>> > >
>> > > OR:
>> > > doc.importNode(node, true);
>> > > The doc is a shared object but is being modified, so errors will
>> > > occur. So just about all the serializers are affected, including
>> > > the EndpointReference class. This means errors will most likely
>> > > occur when Muse handles requests/responses from multiple clients, or
>> > > when notifications are sent from multiple resources. The latter is
>> > > easier to test.
>> > >
>> > > So far, we've been testing using just one client and one producer
>> > > instance, so the problem doesn't occur. But now that we are testing
>> > > by using multiple producers, the exceptions are occuring frequently.
>> > >
>> > > To begin the fix, all serializers and the EndpointReference class
>> > > needs to be patched. So instead of:
>> > > doc = XmlUtils.EMPTY_DOC;
>> > >
>> > > We should do:
>> > > doc = XmlUtils.createDocument();
>> > >
>> > > The overhead of creating a new document is small when compared to
>> > > the multi thread issue, which cannot be avoided. But, we can
>> > > optimize creating new documents by updating XmlUtils to create the
>> > > DocumentBuilderFactory only once. This way, XmlUtils.
>> > > createBuilder() doesn't have to create a new factory everytime.
>> > > -Vinh
>> > > From: Daniel Jemiolo [mailto:danjemiolo@us.ibm.com]
>> > > Sent: Monday, August 27, 2007 7:31 AM
>> > > To: muse-dev@ws.apache.org
>> > > Subject: Re: EMPTY_DOC thread stability issues
>> >
>> > > The use of EMPTY_DOC was an attempt to avoid creating a new Document
>> > > every time we wanted to copy or create small fragments of XML. This
>> > > happened a lot during the request/response process, so the creation
>> > > of these factory objects was not insignificant. Of course, you are
>> > > right about the threading issue, so I guess we're out of luck there.
>> > >
>> > > I am setting aside Friday of this week to go through current JIRA
>> > > items and apply all patches that have been submitted, close any
>> > > issues that are recommended for closure, and sort unscheduled items
>> > > into 2.3 if necessary. Sorry for the delay.
>> > >
>> > > Dan
>> > >
>> > >
>> > > [image removed] rlucente@xecu.net
>> > >
>> >
>> > >
>> > > rlucente@xecu.net
>> > > 08/24/2007 04:37 PM
>> > >
>> > > Please respond to
>> > > muse-dev@ws.apache.org
>> > >
>> > > [image removed]
>> > > To
>> > >
>> > > [image removed]
>> > > muse-dev@ws.apache.org
>> > >
>> > > [image removed]
>> > > cc
>> > >
>> > > [image removed]
>> > >
>> > > [image removed]
>> > > Subject
>> > >
>> > > [image removed]
>> > > EMPTY_DOC thread stability issues
>> > >
>> > > [image removed]
>> > >
>> > > [image removed]
>> > >
>> > >
>> > > Use of EMPTY_DOC (an instance of an empty DOM Document element
>> within
>> > > class XmlUtils) among multiple threads causes unpredictable results.
>> > > While testing a messaging service that uses the Apache Muse WS-N
>> > > implementation, I noticed that 4-6 messages in a 1000 were being
>> dropped.
>> > > Notification messages were being sent at a rate of 20 per second.
>> Where
>> > > the messages were dropped, the following stack trace occurred:
>> > >
>> > > 2007-08-24 15:27:58,031 ERROR [STDERR] Aug 24, 2007 3:27:58 PM
>> > > org.apache.muse.util.LoggingUtils logError
>> > > INFO: There was an error while processing a request:
>> > >
>> > > [ID = 'NoMessageContent'] The NotificationMessage XML does not have
>> a
>> > > Message element. All messages must have a message payload associated
>> with
>> > > them.
>> > >
>> > > org.apache.muse.ws.notification.impl.
>> > > SimpleNotificationMessage.<init>(SimpleNotificationMessage.java:117)
>> > > org.apache.muse.ws.notification.impl.
>> > > NotificationMessageSerializer.fromXML
>> (NotificationMessageSerializer.java:46)
>> > > org.apache.muse.core.serializer.ArraySerializer.
>> > > fromXML(ArraySerializer.java:126)
>> > > org.apache.muse.ws.notification.impl.NotifyHandler.
>> > > fromXML(NotifyHandler.java:62)
>> > > org.apache.muse.core.SimpleResource.invoke
>> (SimpleResource.java:368)
>> > > org.apache.muse.core.routing.SimpleResourceRouter.
>> > > invoke(SimpleResourceRouter.java:290)
>> > > org.apache.muse.core.platform.axis2.AxisIsolationLayer.
>> > > invoke(AxisIsolationLayer.java:136)
>> > > org.apache.muse.core.platform.axis2.AxisIsolationLayer.
>> > > handleRequest(AxisIsolationLayer.java:88)
>> > > sun.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)
>> > > sun.reflect.DelegatingMethodAccessorImpl.
>> > > invoke(DelegatingMethodAccessorImpl.java:25)
>> > > java.lang.reflect.Method.invoke(Method.java:324)
>> > > org.apache.axis2.receivers.RawXMLINOutMessageReceiver.
>> > > invokeBusinessLogic(RawXMLINOutMessageReceiver.java:88)
>> > > org.apache.axis2.receivers.AbstractInOutSyncMessageReceiver.
>> > > receive(AbstractInOutSyncMessageReceiver.java:39)
>> > > org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:493)
>> > > org.apache.axis2.transport.http.HTTPTransportUtils.
>> > > processHTTPPostRequest(HTTPTransportUtils.java:319)
>> > > 22 more...
>> > >
>> > > All of the messages did in fact have a message payload associated
>> with
>> > > them. After further investigation I noticed that
>> > > AxisEnvironment.convertToDOM is calling
>> XmlUtils.createElement(QName)
>> > > which uses the common EMPTY_DOC Document instance. Sharing
>> EMPTY_DOC
>> > > among multiple threads is not safe.
>> > >
>> > > I've also encountered this issue while attempting to set the
>> producer
>> > > reference on a NotificationMessage. The code actually performs a
>> deep
>> > > copy of the EndpointReference class which also relies on EMPTY_DOC.
>> I
>> > > hacked a work-around within my code that avoided use of EMPTY_DOC
>> and
>> > > resolved the issue.
>> > >
>> > > A grep for EMPTY_DOC against the code base turned up over 70
>> instances
>> of
>> > > its use. That does not count methods that use it that are called by
>> other
>> > > methods, so the actual usage is much higher. I'm going to open a
>> JIRA
>> bug
>> > > against this issue and submit patches for it.
>> > >
>> > > Related to that, I've submitted patches for other issues (MUSE-240,
>> > > MUSE-241) over a month ago that have not been committed. The patch
>> for
>> > > MUSE-240 also resolves MUSE-225. Please let me know the appropriate
>> > > process for non-committers to have patches applied to the code base.
>> When
>> > > I submitted the JIRA issues they showed up in this mailing list, but
>> is
>> an
>> > > explicit email more appropriate? Let me know how best to proceed.
>> > >
>> > > Thanks!
>> > >
>> > > Rich Lucente
>> > >
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: muse-dev-unsubscribe@ws.apache.org
>> > > For additional commands, e-mail: muse-dev-help@ws.apache.org
>> > >
>> > > [attachment "EMPTY_DOC_errors.txt" deleted by Daniel
>> Jemiolo/Durham/IBM]
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: muse-dev-unsubscribe@ws.apache.org
>> > > For additional commands, e-mail: muse-dev-help@ws.apache.org
>
> --
> Ist Ihr Browser Vista-kompatibel? Jetzt die neuesten
> Browser-Versionen downloaden: http://www.gmx.net/de/go/browser
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: muse-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: muse-dev-help@ws.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: muse-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: muse-dev-help@ws.apache.org
RE: EMPTY_DOC thread stability issues
Posted by Oliver Waeldrich <ol...@gmx.de>.
Hi,
due to the discussion on the mailing list I did some experiments with muse related to the thread stability issues. As a result I would just come up with a small fix that should solve the majority of the problems and also takes the performance issues into account that lead to the creation of EMPTY_DOC. I would suggest to extend XmlUtils with the following piece of code:
private static final ThreadLocal local = new ThreadLocal();
public static synchronized Document getLocalEmptyDoc() {
Document doc = (Document)local.get();
if (doc == null) {
doc = createDocument();
local.set(doc);
}
return doc;
}
The usage of EMPTY_DOC should be replaced by a call to getLocalEmptyDoc(). Existing functionality can remain untouched.
However, I don't think that this solves all the problems. One other problem is that for the creation of permanent objects like EPR's of resources, separate XML documents have to be used. One of the problems was that multiple threads (resources) accessed one Resource EPR (or the _xml object inside the EPR) at the same time when a NotificationMessage was created. Hereby the setProducerReference() and setSubscriptionReference() of SimpleNotificatioMessage were invoked transparently.
Furthermore, I would suggest to use a new Xml Document within
public EndpointReference(EndpointReference copy, QName typeName) {
...
Document doc = XmlUtils.createDocument();
...
}
This also contibutes to stability, since this constructor is used within the implementation to create the EPR instances of new resources.
With the mentioned changes I obtained quite good results in my test setup (notification provider+consumer sample with 3 to 10 resources each) with MUSE. This means I did not experience exceptions anymore, but I think this has to be tested in more detail.
Oliver
-------- Original-Nachricht --------
> Datum: Fri, 31 Aug 2007 15:09:21 -0400
> Von: Daniel Jemiolo <da...@us.ibm.com>
> An: muse-dev@ws.apache.org
> Betreff: RE: EMPTY_DOC thread stability issues
>
>
> Yeah... this is what led to the creation of EMPTY_DOC. The exceptions
> you're seeing now are caused by you creating an Element with one factory
> (Document) and then trying to append it to another one. For example,
> NotificationMessage serialization will create a Document and start
> appending Elements... one of those will be an EPR, which serializes itself
> with its own Document. In order to prevent the exception, the
> NotificationMessage code needs to call Document.importNode() on the EPR
> Element (or have toXML() somehow return the Document it used along with
> the
> serialized XML, which would make the API really ugly).
>
> The importNode() method does a deep copy of the Element; there is no way
> to
> just switch an Element's parent Document. As you start to replace
> EMPTY_DOC
> with calls to createDocument() *and* importNode(), you will see lots of
> XML
> fragments being copied lots of times on the way to a complete SOAP
> message.
> So, if you think there's too much object creation now, you ain't seen
> nothing yet. ;)
>
> Dan
>
>
>
> "Vinh Nguyen \(vinguye2\)" <vi...@cisco.com> wrote on 08/31/2007
> 01:30:57 PM:
>
> > I should add...I locally updated the EndpointReference class and
> > changed all XmlUtils.EMPTY_DOC references to XmlUtils.
> > createDocument(). I no longer get errors related to
> > EndpointReference, but now get other errors in other classes like
> > SimpleNotificationMessage which attempts to serialize the message.
> > I'm mostly concerned with serialization at this point, since it
> > affects both notifications and operation responses. Will test
> further...
> >
> > From: Vinh Nguyen (vinguye2)
> > Sent: Friday, August 31, 2007 10:06 AM
> > To: muse-dev@ws.apache.org
> > Subject: RE: EMPTY_DOC thread stability issues
>
> > Hi Dan,:
> > You're probably be right. I'll continue to test further. I think I
> > heard of issues with Xerces and will check to see if this is related.
> > -Vinh
> >
> > From: Daniel Jemiolo [mailto:danjemiolo@us.ibm.com]
> > Sent: Friday, August 31, 2007 7:30 AM
> > To: muse-dev@ws.apache.org
> > Subject: RE: EMPTY_DOC thread stability issues
>
> > The problem isn't that EMPTY_DOC is being modified - that would only
> > happen if we appended to the Document itself. Since Document is a
> > factory, we can use it to create fragments that, while their
> > "parent" is the Document, they are not actually attached to it
> > (otherwise you'd get RuntimeExceptions about the Document having
> > more than one root element).
> >
> > For reasons that I do not understand, the Xerces parser/factory is
> > not stateless, and that is causing the concurrency bugs. The simple
> > act of creating new elements at the same time generates the
> > exception. Either way, we'll have to remove it.
> >
> > Dan
> >
> >
> >
> > "Vinh Nguyen \(vinguye2\)" <vi...@cisco.com> wrote on 08/31/2007
> > 04:34:28 AM:
> >
> > > Hi Dan,
> > > I've also done some testing just now and am finding this to be a
> > > very serious issue. It really affects the usability of Muse as a
> whole.
> > >
> > > To test, I created three resources. Each had a loop to generate 100
> > > notifications with no pause between notifications. So out of 300
> > > total notifications generated, I had these results:
> > > Test A = 9 notifications dropped
> > > Test B = 24 notifications dropped
> > >
> > > Attached is a file containing the various exceptions which caused
> > > SimpleSubscriptionManager.publish() to fail. As Rich pointed out,
> > > the culprit is using EMPTY_DOC from multiple threads (i.e. each
> > > client request is one thread).
> > >
> > > The XmlUtils.EMPTY_DOC javadocs has this:
> > > "This should NOT be used by callers that want to build new fragments
> > > and attach them to a Document...you should never append children."
> > >
> > > But in XmlUtils itself and many other classes, the following code
> > > patterns are used:
> > >
> > > Document doc = XmlUtils.EMPTY_DOC;
> > > Element xml = XmlUtils.createElement(doc, qname);
> > > xml.appendChild(node);
> > >
> > > OR:
> > > doc.importNode(node, true);
> > > The doc is a shared object but is being modified, so errors will
> > > occur. So just about all the serializers are affected, including
> > > the EndpointReference class. This means errors will most likely
> > > occur when Muse handles requests/responses from multiple clients, or
> > > when notifications are sent from multiple resources. The latter is
> > > easier to test.
> > >
> > > So far, we've been testing using just one client and one producer
> > > instance, so the problem doesn't occur. But now that we are testing
> > > by using multiple producers, the exceptions are occuring frequently.
> > >
> > > To begin the fix, all serializers and the EndpointReference class
> > > needs to be patched. So instead of:
> > > doc = XmlUtils.EMPTY_DOC;
> > >
> > > We should do:
> > > doc = XmlUtils.createDocument();
> > >
> > > The overhead of creating a new document is small when compared to
> > > the multi thread issue, which cannot be avoided. But, we can
> > > optimize creating new documents by updating XmlUtils to create the
> > > DocumentBuilderFactory only once. This way, XmlUtils.
> > > createBuilder() doesn't have to create a new factory everytime.
> > > -Vinh
> > > From: Daniel Jemiolo [mailto:danjemiolo@us.ibm.com]
> > > Sent: Monday, August 27, 2007 7:31 AM
> > > To: muse-dev@ws.apache.org
> > > Subject: Re: EMPTY_DOC thread stability issues
> >
> > > The use of EMPTY_DOC was an attempt to avoid creating a new Document
> > > every time we wanted to copy or create small fragments of XML. This
> > > happened a lot during the request/response process, so the creation
> > > of these factory objects was not insignificant. Of course, you are
> > > right about the threading issue, so I guess we're out of luck there.
> > >
> > > I am setting aside Friday of this week to go through current JIRA
> > > items and apply all patches that have been submitted, close any
> > > issues that are recommended for closure, and sort unscheduled items
> > > into 2.3 if necessary. Sorry for the delay.
> > >
> > > Dan
> > >
> > >
> > > [image removed] rlucente@xecu.net
> > >
> >
> > >
> > > rlucente@xecu.net
> > > 08/24/2007 04:37 PM
> > >
> > > Please respond to
> > > muse-dev@ws.apache.org
> > >
> > > [image removed]
> > > To
> > >
> > > [image removed]
> > > muse-dev@ws.apache.org
> > >
> > > [image removed]
> > > cc
> > >
> > > [image removed]
> > >
> > > [image removed]
> > > Subject
> > >
> > > [image removed]
> > > EMPTY_DOC thread stability issues
> > >
> > > [image removed]
> > >
> > > [image removed]
> > >
> > >
> > > Use of EMPTY_DOC (an instance of an empty DOM Document element within
> > > class XmlUtils) among multiple threads causes unpredictable results.
> > > While testing a messaging service that uses the Apache Muse WS-N
> > > implementation, I noticed that 4-6 messages in a 1000 were being
> dropped.
> > > Notification messages were being sent at a rate of 20 per second.
> Where
> > > the messages were dropped, the following stack trace occurred:
> > >
> > > 2007-08-24 15:27:58,031 ERROR [STDERR] Aug 24, 2007 3:27:58 PM
> > > org.apache.muse.util.LoggingUtils logError
> > > INFO: There was an error while processing a request:
> > >
> > > [ID = 'NoMessageContent'] The NotificationMessage XML does not have a
> > > Message element. All messages must have a message payload associated
> with
> > > them.
> > >
> > > org.apache.muse.ws.notification.impl.
> > > SimpleNotificationMessage.<init>(SimpleNotificationMessage.java:117)
> > > org.apache.muse.ws.notification.impl.
> > > NotificationMessageSerializer.fromXML
> (NotificationMessageSerializer.java:46)
> > > org.apache.muse.core.serializer.ArraySerializer.
> > > fromXML(ArraySerializer.java:126)
> > > org.apache.muse.ws.notification.impl.NotifyHandler.
> > > fromXML(NotifyHandler.java:62)
> > > org.apache.muse.core.SimpleResource.invoke
> (SimpleResource.java:368)
> > > org.apache.muse.core.routing.SimpleResourceRouter.
> > > invoke(SimpleResourceRouter.java:290)
> > > org.apache.muse.core.platform.axis2.AxisIsolationLayer.
> > > invoke(AxisIsolationLayer.java:136)
> > > org.apache.muse.core.platform.axis2.AxisIsolationLayer.
> > > handleRequest(AxisIsolationLayer.java:88)
> > > sun.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)
> > > sun.reflect.DelegatingMethodAccessorImpl.
> > > invoke(DelegatingMethodAccessorImpl.java:25)
> > > java.lang.reflect.Method.invoke(Method.java:324)
> > > org.apache.axis2.receivers.RawXMLINOutMessageReceiver.
> > > invokeBusinessLogic(RawXMLINOutMessageReceiver.java:88)
> > > org.apache.axis2.receivers.AbstractInOutSyncMessageReceiver.
> > > receive(AbstractInOutSyncMessageReceiver.java:39)
> > > org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:493)
> > > org.apache.axis2.transport.http.HTTPTransportUtils.
> > > processHTTPPostRequest(HTTPTransportUtils.java:319)
> > > 22 more...
> > >
> > > All of the messages did in fact have a message payload associated with
> > > them. After further investigation I noticed that
> > > AxisEnvironment.convertToDOM is calling XmlUtils.createElement(QName)
> > > which uses the common EMPTY_DOC Document instance. Sharing EMPTY_DOC
> > > among multiple threads is not safe.
> > >
> > > I've also encountered this issue while attempting to set the producer
> > > reference on a NotificationMessage. The code actually performs a deep
> > > copy of the EndpointReference class which also relies on EMPTY_DOC. I
> > > hacked a work-around within my code that avoided use of EMPTY_DOC and
> > > resolved the issue.
> > >
> > > A grep for EMPTY_DOC against the code base turned up over 70 instances
> of
> > > its use. That does not count methods that use it that are called by
> other
> > > methods, so the actual usage is much higher. I'm going to open a JIRA
> bug
> > > against this issue and submit patches for it.
> > >
> > > Related to that, I've submitted patches for other issues (MUSE-240,
> > > MUSE-241) over a month ago that have not been committed. The patch
> for
> > > MUSE-240 also resolves MUSE-225. Please let me know the appropriate
> > > process for non-committers to have patches applied to the code base.
> When
> > > I submitted the JIRA issues they showed up in this mailing list, but
> is
> an
> > > explicit email more appropriate? Let me know how best to proceed.
> > >
> > > Thanks!
> > >
> > > Rich Lucente
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: muse-dev-unsubscribe@ws.apache.org
> > > For additional commands, e-mail: muse-dev-help@ws.apache.org
> > >
> > > [attachment "EMPTY_DOC_errors.txt" deleted by Daniel
> Jemiolo/Durham/IBM]
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: muse-dev-unsubscribe@ws.apache.org
> > > For additional commands, e-mail: muse-dev-help@ws.apache.org
--
Ist Ihr Browser Vista-kompatibel? Jetzt die neuesten
Browser-Versionen downloaden: http://www.gmx.net/de/go/browser
---------------------------------------------------------------------
To unsubscribe, e-mail: muse-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: muse-dev-help@ws.apache.org
RE: EMPTY_DOC thread stability issues
Posted by Daniel Jemiolo <da...@us.ibm.com>.
Yeah... this is what led to the creation of EMPTY_DOC. The exceptions
you're seeing now are caused by you creating an Element with one factory
(Document) and then trying to append it to another one. For example,
NotificationMessage serialization will create a Document and start
appending Elements... one of those will be an EPR, which serializes itself
with its own Document. In order to prevent the exception, the
NotificationMessage code needs to call Document.importNode() on the EPR
Element (or have toXML() somehow return the Document it used along with the
serialized XML, which would make the API really ugly).
The importNode() method does a deep copy of the Element; there is no way to
just switch an Element's parent Document. As you start to replace EMPTY_DOC
with calls to createDocument() *and* importNode(), you will see lots of XML
fragments being copied lots of times on the way to a complete SOAP message.
So, if you think there's too much object creation now, you ain't seen
nothing yet. ;)
Dan
"Vinh Nguyen \(vinguye2\)" <vi...@cisco.com> wrote on 08/31/2007
01:30:57 PM:
> I should add...I locally updated the EndpointReference class and
> changed all XmlUtils.EMPTY_DOC references to XmlUtils.
> createDocument(). I no longer get errors related to
> EndpointReference, but now get other errors in other classes like
> SimpleNotificationMessage which attempts to serialize the message.
> I'm mostly concerned with serialization at this point, since it
> affects both notifications and operation responses. Will test further...
>
> From: Vinh Nguyen (vinguye2)
> Sent: Friday, August 31, 2007 10:06 AM
> To: muse-dev@ws.apache.org
> Subject: RE: EMPTY_DOC thread stability issues
> Hi Dan,:
> You're probably be right. I'll continue to test further. I think I
> heard of issues with Xerces and will check to see if this is related.
> -Vinh
>
> From: Daniel Jemiolo [mailto:danjemiolo@us.ibm.com]
> Sent: Friday, August 31, 2007 7:30 AM
> To: muse-dev@ws.apache.org
> Subject: RE: EMPTY_DOC thread stability issues
> The problem isn't that EMPTY_DOC is being modified - that would only
> happen if we appended to the Document itself. Since Document is a
> factory, we can use it to create fragments that, while their
> "parent" is the Document, they are not actually attached to it
> (otherwise you'd get RuntimeExceptions about the Document having
> more than one root element).
>
> For reasons that I do not understand, the Xerces parser/factory is
> not stateless, and that is causing the concurrency bugs. The simple
> act of creating new elements at the same time generates the
> exception. Either way, we'll have to remove it.
>
> Dan
>
>
>
> "Vinh Nguyen \(vinguye2\)" <vi...@cisco.com> wrote on 08/31/2007
> 04:34:28 AM:
>
> > Hi Dan,
> > I've also done some testing just now and am finding this to be a
> > very serious issue. It really affects the usability of Muse as a
whole.
> >
> > To test, I created three resources. Each had a loop to generate 100
> > notifications with no pause between notifications. So out of 300
> > total notifications generated, I had these results:
> > Test A = 9 notifications dropped
> > Test B = 24 notifications dropped
> >
> > Attached is a file containing the various exceptions which caused
> > SimpleSubscriptionManager.publish() to fail. As Rich pointed out,
> > the culprit is using EMPTY_DOC from multiple threads (i.e. each
> > client request is one thread).
> >
> > The XmlUtils.EMPTY_DOC javadocs has this:
> > "This should NOT be used by callers that want to build new fragments
> > and attach them to a Document...you should never append children."
> >
> > But in XmlUtils itself and many other classes, the following code
> > patterns are used:
> >
> > Document doc = XmlUtils.EMPTY_DOC;
> > Element xml = XmlUtils.createElement(doc, qname);
> > xml.appendChild(node);
> >
> > OR:
> > doc.importNode(node, true);
> > The doc is a shared object but is being modified, so errors will
> > occur. So just about all the serializers are affected, including
> > the EndpointReference class. This means errors will most likely
> > occur when Muse handles requests/responses from multiple clients, or
> > when notifications are sent from multiple resources. The latter is
> > easier to test.
> >
> > So far, we've been testing using just one client and one producer
> > instance, so the problem doesn't occur. But now that we are testing
> > by using multiple producers, the exceptions are occuring frequently.
> >
> > To begin the fix, all serializers and the EndpointReference class
> > needs to be patched. So instead of:
> > doc = XmlUtils.EMPTY_DOC;
> >
> > We should do:
> > doc = XmlUtils.createDocument();
> >
> > The overhead of creating a new document is small when compared to
> > the multi thread issue, which cannot be avoided. But, we can
> > optimize creating new documents by updating XmlUtils to create the
> > DocumentBuilderFactory only once. This way, XmlUtils.
> > createBuilder() doesn't have to create a new factory everytime.
> > -Vinh
> > From: Daniel Jemiolo [mailto:danjemiolo@us.ibm.com]
> > Sent: Monday, August 27, 2007 7:31 AM
> > To: muse-dev@ws.apache.org
> > Subject: Re: EMPTY_DOC thread stability issues
>
> > The use of EMPTY_DOC was an attempt to avoid creating a new Document
> > every time we wanted to copy or create small fragments of XML. This
> > happened a lot during the request/response process, so the creation
> > of these factory objects was not insignificant. Of course, you are
> > right about the threading issue, so I guess we're out of luck there.
> >
> > I am setting aside Friday of this week to go through current JIRA
> > items and apply all patches that have been submitted, close any
> > issues that are recommended for closure, and sort unscheduled items
> > into 2.3 if necessary. Sorry for the delay.
> >
> > Dan
> >
> >
> > [image removed] rlucente@xecu.net
> >
>
> >
> > rlucente@xecu.net
> > 08/24/2007 04:37 PM
> >
> > Please respond to
> > muse-dev@ws.apache.org
> >
> > [image removed]
> > To
> >
> > [image removed]
> > muse-dev@ws.apache.org
> >
> > [image removed]
> > cc
> >
> > [image removed]
> >
> > [image removed]
> > Subject
> >
> > [image removed]
> > EMPTY_DOC thread stability issues
> >
> > [image removed]
> >
> > [image removed]
> >
> >
> > Use of EMPTY_DOC (an instance of an empty DOM Document element within
> > class XmlUtils) among multiple threads causes unpredictable results.
> > While testing a messaging service that uses the Apache Muse WS-N
> > implementation, I noticed that 4-6 messages in a 1000 were being
dropped.
> > Notification messages were being sent at a rate of 20 per second.
Where
> > the messages were dropped, the following stack trace occurred:
> >
> > 2007-08-24 15:27:58,031 ERROR [STDERR] Aug 24, 2007 3:27:58 PM
> > org.apache.muse.util.LoggingUtils logError
> > INFO: There was an error while processing a request:
> >
> > [ID = 'NoMessageContent'] The NotificationMessage XML does not have a
> > Message element. All messages must have a message payload associated
with
> > them.
> >
> > org.apache.muse.ws.notification.impl.
> > SimpleNotificationMessage.<init>(SimpleNotificationMessage.java:117)
> > org.apache.muse.ws.notification.impl.
> > NotificationMessageSerializer.fromXML
(NotificationMessageSerializer.java:46)
> > org.apache.muse.core.serializer.ArraySerializer.
> > fromXML(ArraySerializer.java:126)
> > org.apache.muse.ws.notification.impl.NotifyHandler.
> > fromXML(NotifyHandler.java:62)
> > org.apache.muse.core.SimpleResource.invoke
(SimpleResource.java:368)
> > org.apache.muse.core.routing.SimpleResourceRouter.
> > invoke(SimpleResourceRouter.java:290)
> > org.apache.muse.core.platform.axis2.AxisIsolationLayer.
> > invoke(AxisIsolationLayer.java:136)
> > org.apache.muse.core.platform.axis2.AxisIsolationLayer.
> > handleRequest(AxisIsolationLayer.java:88)
> > sun.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)
> > sun.reflect.DelegatingMethodAccessorImpl.
> > invoke(DelegatingMethodAccessorImpl.java:25)
> > java.lang.reflect.Method.invoke(Method.java:324)
> > org.apache.axis2.receivers.RawXMLINOutMessageReceiver.
> > invokeBusinessLogic(RawXMLINOutMessageReceiver.java:88)
> > org.apache.axis2.receivers.AbstractInOutSyncMessageReceiver.
> > receive(AbstractInOutSyncMessageReceiver.java:39)
> > org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:493)
> > org.apache.axis2.transport.http.HTTPTransportUtils.
> > processHTTPPostRequest(HTTPTransportUtils.java:319)
> > 22 more...
> >
> > All of the messages did in fact have a message payload associated with
> > them. After further investigation I noticed that
> > AxisEnvironment.convertToDOM is calling XmlUtils.createElement(QName)
> > which uses the common EMPTY_DOC Document instance. Sharing EMPTY_DOC
> > among multiple threads is not safe.
> >
> > I've also encountered this issue while attempting to set the producer
> > reference on a NotificationMessage. The code actually performs a deep
> > copy of the EndpointReference class which also relies on EMPTY_DOC. I
> > hacked a work-around within my code that avoided use of EMPTY_DOC and
> > resolved the issue.
> >
> > A grep for EMPTY_DOC against the code base turned up over 70 instances
of
> > its use. That does not count methods that use it that are called by
other
> > methods, so the actual usage is much higher. I'm going to open a JIRA
bug
> > against this issue and submit patches for it.
> >
> > Related to that, I've submitted patches for other issues (MUSE-240,
> > MUSE-241) over a month ago that have not been committed. The patch for
> > MUSE-240 also resolves MUSE-225. Please let me know the appropriate
> > process for non-committers to have patches applied to the code base.
When
> > I submitted the JIRA issues they showed up in this mailing list, but is
an
> > explicit email more appropriate? Let me know how best to proceed.
> >
> > Thanks!
> >
> > Rich Lucente
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: muse-dev-unsubscribe@ws.apache.org
> > For additional commands, e-mail: muse-dev-help@ws.apache.org
> >
> > [attachment "EMPTY_DOC_errors.txt" deleted by Daniel
Jemiolo/Durham/IBM]
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: muse-dev-unsubscribe@ws.apache.org
> > For additional commands, e-mail: muse-dev-help@ws.apache.org
RE: EMPTY_DOC thread stability issues
Posted by "Vinh Nguyen (vinguye2)" <vi...@cisco.com>.
I should add...I locally updated the EndpointReference class and changed
all XmlUtils.EMPTY_DOC references to XmlUtils.createDocument(). I no
longer get errors related to EndpointReference, but now get other errors
in other classes like SimpleNotificationMessage which attempts to
serialize the message. I'm mostly concerned with serialization at this
point, since it affects both notifications and operation responses.
Will test further...
________________________________
From: Vinh Nguyen (vinguye2)
Sent: Friday, August 31, 2007 10:06 AM
To: muse-dev@ws.apache.org
Subject: RE: EMPTY_DOC thread stability issues
Hi Dan,
You're probably be right. I'll continue to test further. I think I
heard of issues with Xerces and will check to see if this is related.
-Vinh
________________________________
From: Daniel Jemiolo [mailto:danjemiolo@us.ibm.com]
Sent: Friday, August 31, 2007 7:30 AM
To: muse-dev@ws.apache.org
Subject: RE: EMPTY_DOC thread stability issues
The problem isn't that EMPTY_DOC is being modified - that would only
happen if we appended to the Document itself. Since Document is a
factory, we can use it to create fragments that, while their "parent" is
the Document, they are not actually attached to it (otherwise you'd get
RuntimeExceptions about the Document having more than one root element).
For reasons that I do not understand, the Xerces parser/factory is not
stateless, and that is causing the concurrency bugs. The simple act of
creating new elements at the same time generates the exception. Either
way, we'll have to remove it.
Dan
"Vinh Nguyen \(vinguye2\)" <vi...@cisco.com> wrote on 08/31/2007
04:34:28 AM:
> Hi Dan,
> I've also done some testing just now and am finding this to be a
> very serious issue. It really affects the usability of Muse as a
whole.
>
> To test, I created three resources. Each had a loop to generate 100
> notifications with no pause between notifications. So out of 300
> total notifications generated, I had these results:
> Test A = 9 notifications dropped
> Test B = 24 notifications dropped
>
> Attached is a file containing the various exceptions which caused
> SimpleSubscriptionManager.publish() to fail. As Rich pointed out,
> the culprit is using EMPTY_DOC from multiple threads (i.e. each
> client request is one thread).
>
> The XmlUtils.EMPTY_DOC javadocs has this:
> "This should NOT be used by callers that want to build new fragments
> and attach them to a Document...you should never append children."
>
> But in XmlUtils itself and many other classes, the following code
> patterns are used:
>
> Document doc = XmlUtils.EMPTY_DOC;
> Element xml = XmlUtils.createElement(doc, qname);
> xml.appendChild(node);
>
> OR:
> doc.importNode(node, true);
> The doc is a shared object but is being modified, so errors will
> occur. So just about all the serializers are affected, including
> the EndpointReference class. This means errors will most likely
> occur when Muse handles requests/responses from multiple clients, or
> when notifications are sent from multiple resources. The latter is
> easier to test.
>
> So far, we've been testing using just one client and one producer
> instance, so the problem doesn't occur. But now that we are testing
> by using multiple producers, the exceptions are occuring frequently.
>
> To begin the fix, all serializers and the EndpointReference class
> needs to be patched. So instead of:
> doc = XmlUtils.EMPTY_DOC;
>
> We should do:
> doc = XmlUtils.createDocument();
>
> The overhead of creating a new document is small when compared to
> the multi thread issue, which cannot be avoided. But, we can
> optimize creating new documents by updating XmlUtils to create the
> DocumentBuilderFactory only once. This way, XmlUtils.
> createBuilder() doesn't have to create a new factory everytime.
> -Vinh
> From: Daniel Jemiolo [mailto:danjemiolo@us.ibm.com]
> Sent: Monday, August 27, 2007 7:31 AM
> To: muse-dev@ws.apache.org
> Subject: Re: EMPTY_DOC thread stability issues
> The use of EMPTY_DOC was an attempt to avoid creating a new Document
> every time we wanted to copy or create small fragments of XML. This
> happened a lot during the request/response process, so the creation
> of these factory objects was not insignificant. Of course, you are
> right about the threading issue, so I guess we're out of luck there.
>
> I am setting aside Friday of this week to go through current JIRA
> items and apply all patches that have been submitted, close any
> issues that are recommended for closure, and sort unscheduled items
> into 2.3 if necessary. Sorry for the delay.
>
> Dan
>
>
> [image removed] rlucente@xecu.net
>
>
> rlucente@xecu.net
> 08/24/2007 04:37 PM
>
> Please respond to
> muse-dev@ws.apache.org
>
> [image removed]
> To
>
> [image removed]
> muse-dev@ws.apache.org
>
> [image removed]
> cc
>
> [image removed]
>
> [image removed]
> Subject
>
> [image removed]
> EMPTY_DOC thread stability issues
>
> [image removed]
>
> [image removed]
>
>
> Use of EMPTY_DOC (an instance of an empty DOM Document element within
> class XmlUtils) among multiple threads causes unpredictable results.
> While testing a messaging service that uses the Apache Muse WS-N
> implementation, I noticed that 4-6 messages in a 1000 were being
dropped.
> Notification messages were being sent at a rate of 20 per second.
Where
> the messages were dropped, the following stack trace occurred:
>
> 2007-08-24 15:27:58,031 ERROR [STDERR] Aug 24, 2007 3:27:58 PM
> org.apache.muse.util.LoggingUtils logError
> INFO: There was an error while processing a request:
>
> [ID = 'NoMessageContent'] The NotificationMessage XML does not have a
> Message element. All messages must have a message payload associated
with
> them.
>
> org.apache.muse.ws.notification.impl.
> SimpleNotificationMessage.<init>(SimpleNotificationMessage.java:117)
> org.apache.muse.ws.notification.impl.
>
NotificationMessageSerializer.fromXML(NotificationMessageSerializer.java
:46)
> org.apache.muse.core.serializer.ArraySerializer.
> fromXML(ArraySerializer.java:126)
> org.apache.muse.ws.notification.impl.NotifyHandler.
> fromXML(NotifyHandler.java:62)
>
org.apache.muse.core.SimpleResource.invoke(SimpleResource.java:368)
> org.apache.muse.core.routing.SimpleResourceRouter.
> invoke(SimpleResourceRouter.java:290)
> org.apache.muse.core.platform.axis2.AxisIsolationLayer.
> invoke(AxisIsolationLayer.java:136)
> org.apache.muse.core.platform.axis2.AxisIsolationLayer.
> handleRequest(AxisIsolationLayer.java:88)
> sun.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)
> sun.reflect.DelegatingMethodAccessorImpl.
> invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:324)
> org.apache.axis2.receivers.RawXMLINOutMessageReceiver.
> invokeBusinessLogic(RawXMLINOutMessageReceiver.java:88)
> org.apache.axis2.receivers.AbstractInOutSyncMessageReceiver.
> receive(AbstractInOutSyncMessageReceiver.java:39)
> org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:493)
> org.apache.axis2.transport.http.HTTPTransportUtils.
> processHTTPPostRequest(HTTPTransportUtils.java:319)
> 22 more...
>
> All of the messages did in fact have a message payload associated with
> them. After further investigation I noticed that
> AxisEnvironment.convertToDOM is calling XmlUtils.createElement(QName)
> which uses the common EMPTY_DOC Document instance. Sharing EMPTY_DOC
> among multiple threads is not safe.
>
> I've also encountered this issue while attempting to set the producer
> reference on a NotificationMessage. The code actually performs a deep
> copy of the EndpointReference class which also relies on EMPTY_DOC. I
> hacked a work-around within my code that avoided use of EMPTY_DOC and
> resolved the issue.
>
> A grep for EMPTY_DOC against the code base turned up over 70 instances
of
> its use. That does not count methods that use it that are called by
other
> methods, so the actual usage is much higher. I'm going to open a JIRA
bug
> against this issue and submit patches for it.
>
> Related to that, I've submitted patches for other issues (MUSE-240,
> MUSE-241) over a month ago that have not been committed. The patch
for
> MUSE-240 also resolves MUSE-225. Please let me know the appropriate
> process for non-committers to have patches applied to the code base.
When
> I submitted the JIRA issues they showed up in this mailing list, but
is an
> explicit email more appropriate? Let me know how best to proceed.
>
> Thanks!
>
> Rich Lucente
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: muse-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: muse-dev-help@ws.apache.org
>
> [attachment "EMPTY_DOC_errors.txt" deleted by Daniel
Jemiolo/Durham/IBM]
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: muse-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: muse-dev-help@ws.apache.org
RE: EMPTY_DOC thread stability issues
Posted by "Vinh Nguyen (vinguye2)" <vi...@cisco.com>.
Hi Dan,
You're probably be right. I'll continue to test further. I think I
heard of issues with Xerces and will check to see if this is related.
-Vinh
________________________________
From: Daniel Jemiolo [mailto:danjemiolo@us.ibm.com]
Sent: Friday, August 31, 2007 7:30 AM
To: muse-dev@ws.apache.org
Subject: RE: EMPTY_DOC thread stability issues
The problem isn't that EMPTY_DOC is being modified - that would only
happen if we appended to the Document itself. Since Document is a
factory, we can use it to create fragments that, while their "parent" is
the Document, they are not actually attached to it (otherwise you'd get
RuntimeExceptions about the Document having more than one root element).
For reasons that I do not understand, the Xerces parser/factory is not
stateless, and that is causing the concurrency bugs. The simple act of
creating new elements at the same time generates the exception. Either
way, we'll have to remove it.
Dan
"Vinh Nguyen \(vinguye2\)" <vi...@cisco.com> wrote on 08/31/2007
04:34:28 AM:
> Hi Dan,
> I've also done some testing just now and am finding this to be a
> very serious issue. It really affects the usability of Muse as a
whole.
>
> To test, I created three resources. Each had a loop to generate 100
> notifications with no pause between notifications. So out of 300
> total notifications generated, I had these results:
> Test A = 9 notifications dropped
> Test B = 24 notifications dropped
>
> Attached is a file containing the various exceptions which caused
> SimpleSubscriptionManager.publish() to fail. As Rich pointed out,
> the culprit is using EMPTY_DOC from multiple threads (i.e. each
> client request is one thread).
>
> The XmlUtils.EMPTY_DOC javadocs has this:
> "This should NOT be used by callers that want to build new fragments
> and attach them to a Document...you should never append children."
>
> But in XmlUtils itself and many other classes, the following code
> patterns are used:
>
> Document doc = XmlUtils.EMPTY_DOC;
> Element xml = XmlUtils.createElement(doc, qname);
> xml.appendChild(node);
>
> OR:
> doc.importNode(node, true);
> The doc is a shared object but is being modified, so errors will
> occur. So just about all the serializers are affected, including
> the EndpointReference class. This means errors will most likely
> occur when Muse handles requests/responses from multiple clients, or
> when notifications are sent from multiple resources. The latter is
> easier to test.
>
> So far, we've been testing using just one client and one producer
> instance, so the problem doesn't occur. But now that we are testing
> by using multiple producers, the exceptions are occuring frequently.
>
> To begin the fix, all serializers and the EndpointReference class
> needs to be patched. So instead of:
> doc = XmlUtils.EMPTY_DOC;
>
> We should do:
> doc = XmlUtils.createDocument();
>
> The overhead of creating a new document is small when compared to
> the multi thread issue, which cannot be avoided. But, we can
> optimize creating new documents by updating XmlUtils to create the
> DocumentBuilderFactory only once. This way, XmlUtils.
> createBuilder() doesn't have to create a new factory everytime.
> -Vinh
> From: Daniel Jemiolo [mailto:danjemiolo@us.ibm.com]
> Sent: Monday, August 27, 2007 7:31 AM
> To: muse-dev@ws.apache.org
> Subject: Re: EMPTY_DOC thread stability issues
> The use of EMPTY_DOC was an attempt to avoid creating a new Document
> every time we wanted to copy or create small fragments of XML. This
> happened a lot during the request/response process, so the creation
> of these factory objects was not insignificant. Of course, you are
> right about the threading issue, so I guess we're out of luck there.
>
> I am setting aside Friday of this week to go through current JIRA
> items and apply all patches that have been submitted, close any
> issues that are recommended for closure, and sort unscheduled items
> into 2.3 if necessary. Sorry for the delay.
>
> Dan
>
>
> [image removed] rlucente@xecu.net
>
>
> rlucente@xecu.net
> 08/24/2007 04:37 PM
>
> Please respond to
> muse-dev@ws.apache.org
>
> [image removed]
> To
>
> [image removed]
> muse-dev@ws.apache.org
>
> [image removed]
> cc
>
> [image removed]
>
> [image removed]
> Subject
>
> [image removed]
> EMPTY_DOC thread stability issues
>
> [image removed]
>
> [image removed]
>
>
> Use of EMPTY_DOC (an instance of an empty DOM Document element within
> class XmlUtils) among multiple threads causes unpredictable results.
> While testing a messaging service that uses the Apache Muse WS-N
> implementation, I noticed that 4-6 messages in a 1000 were being
dropped.
> Notification messages were being sent at a rate of 20 per second.
Where
> the messages were dropped, the following stack trace occurred:
>
> 2007-08-24 15:27:58,031 ERROR [STDERR] Aug 24, 2007 3:27:58 PM
> org.apache.muse.util.LoggingUtils logError
> INFO: There was an error while processing a request:
>
> [ID = 'NoMessageContent'] The NotificationMessage XML does not have a
> Message element. All messages must have a message payload associated
with
> them.
>
> org.apache.muse.ws.notification.impl.
> SimpleNotificationMessage.<init>(SimpleNotificationMessage.java:117)
> org.apache.muse.ws.notification.impl.
>
NotificationMessageSerializer.fromXML(NotificationMessageSerializer.java
:46)
> org.apache.muse.core.serializer.ArraySerializer.
> fromXML(ArraySerializer.java:126)
> org.apache.muse.ws.notification.impl.NotifyHandler.
> fromXML(NotifyHandler.java:62)
>
org.apache.muse.core.SimpleResource.invoke(SimpleResource.java:368)
> org.apache.muse.core.routing.SimpleResourceRouter.
> invoke(SimpleResourceRouter.java:290)
> org.apache.muse.core.platform.axis2.AxisIsolationLayer.
> invoke(AxisIsolationLayer.java:136)
> org.apache.muse.core.platform.axis2.AxisIsolationLayer.
> handleRequest(AxisIsolationLayer.java:88)
> sun.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)
> sun.reflect.DelegatingMethodAccessorImpl.
> invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:324)
> org.apache.axis2.receivers.RawXMLINOutMessageReceiver.
> invokeBusinessLogic(RawXMLINOutMessageReceiver.java:88)
> org.apache.axis2.receivers.AbstractInOutSyncMessageReceiver.
> receive(AbstractInOutSyncMessageReceiver.java:39)
> org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:493)
> org.apache.axis2.transport.http.HTTPTransportUtils.
> processHTTPPostRequest(HTTPTransportUtils.java:319)
> 22 more...
>
> All of the messages did in fact have a message payload associated with
> them. After further investigation I noticed that
> AxisEnvironment.convertToDOM is calling XmlUtils.createElement(QName)
> which uses the common EMPTY_DOC Document instance. Sharing EMPTY_DOC
> among multiple threads is not safe.
>
> I've also encountered this issue while attempting to set the producer
> reference on a NotificationMessage. The code actually performs a deep
> copy of the EndpointReference class which also relies on EMPTY_DOC. I
> hacked a work-around within my code that avoided use of EMPTY_DOC and
> resolved the issue.
>
> A grep for EMPTY_DOC against the code base turned up over 70 instances
of
> its use. That does not count methods that use it that are called by
other
> methods, so the actual usage is much higher. I'm going to open a JIRA
bug
> against this issue and submit patches for it.
>
> Related to that, I've submitted patches for other issues (MUSE-240,
> MUSE-241) over a month ago that have not been committed. The patch
for
> MUSE-240 also resolves MUSE-225. Please let me know the appropriate
> process for non-committers to have patches applied to the code base.
When
> I submitted the JIRA issues they showed up in this mailing list, but
is an
> explicit email more appropriate? Let me know how best to proceed.
>
> Thanks!
>
> Rich Lucente
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: muse-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: muse-dev-help@ws.apache.org
>
> [attachment "EMPTY_DOC_errors.txt" deleted by Daniel
Jemiolo/Durham/IBM]
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: muse-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: muse-dev-help@ws.apache.org
RE: EMPTY_DOC thread stability issues
Posted by Daniel Jemiolo <da...@us.ibm.com>.
The problem isn't that EMPTY_DOC is being modified - that would only happen
if we appended to the Document itself. Since Document is a factory, we can
use it to create fragments that, while their "parent" is the Document, they
are not actually attached to it (otherwise you'd get RuntimeExceptions
about the Document having more than one root element).
For reasons that I do not understand, the Xerces parser/factory is not
stateless, and that is causing the concurrency bugs. The simple act of
creating new elements at the same time generates the exception. Either way,
we'll have to remove it.
Dan
"Vinh Nguyen \(vinguye2\)" <vi...@cisco.com> wrote on 08/31/2007
04:34:28 AM:
> Hi Dan,
> I've also done some testing just now and am finding this to be a
> very serious issue. It really affects the usability of Muse as a whole.
>
> To test, I created three resources. Each had a loop to generate 100
> notifications with no pause between notifications. So out of 300
> total notifications generated, I had these results:
> Test A = 9 notifications dropped
> Test B = 24 notifications dropped
>
> Attached is a file containing the various exceptions which caused
> SimpleSubscriptionManager.publish() to fail. As Rich pointed out,
> the culprit is using EMPTY_DOC from multiple threads (i.e. each
> client request is one thread).
>
> The XmlUtils.EMPTY_DOC javadocs has this:
> "This should NOT be used by callers that want to build new fragments
> and attach them to a Document...you should never append children.">
>
> But in XmlUtils itself and many other classes, the following code
> patterns are used:
>
> Document doc = XmlUtils.EMPTY_DOC;
> Element xml = XmlUtils.createElement(doc, qname);
> xml.appendChild(node);
>
> OR:
> doc.importNode(node, true);
> The doc is a shared object but is being modified, so errors will
> occur. So just about all the serializers are affected, including
> the EndpointReference class. This means errors will most likely
> occur when Muse handles requests/responses from multiple clients, or
> when notifications are sent from multiple resources. The latter is
> easier to test.
>
> So far, we've been testing using just one client and one producer
> instance, so the problem doesn't occur. But now that we are testing
> by using multiple producers, the exceptions are occuring frequently.
>
> To begin the fix, all serializers and the EndpointReference class
> needs to be patched. So instead of:
> doc = XmlUtils.EMPTY_DOC;
>
> We should do:
> doc = XmlUtils.createDocument();
>
> The overhead of creating a new document is small when compared to
> the multi thread issue, which cannot be avoided. But, we can
> optimize creating new documents by updating XmlUtils to create the
> DocumentBuilderFactory only once. This way, XmlUtils.
> createBuilder() doesn't have to create a new factory everytime.
> -Vinh
> From: Daniel Jemiolo [mailto:danjemiolo@us.ibm.com]
> Sent: Monday, August 27, 2007 7:31 AM
> To: muse-dev@ws.apache.org
> Subject: Re: EMPTY_DOC thread stability issues
> The use of EMPTY_DOC was an attempt to avoid creating a new Document
> every time we wanted to copy or create small fragments of XML. This
> happened a lot during the request/response process, so the creation
> of these factory objects was not insignificant. Of course, you are
> right about the threading issue, so I guess we're out of luck there.
>
> I am setting aside Friday of this week to go through current JIRA
> items and apply all patches that have been submitted, close any
> issues that are recommended for closure, and sort unscheduled items
> into 2.3 if necessary. Sorry for the delay.
>
> Dan
>
>
> [image removed] rlucente@xecu.net
>
>
> rlucente@xecu.net
> 08/24/2007 04:37 PM
>
> Please respond to
> muse-dev@ws.apache.org
>
> [image removed]
> To
>
> [image removed]
> muse-dev@ws.apache.org
>
> [image removed]
> cc
>
> [image removed]
>
> [image removed]
> Subject
>
> [image removed]
> EMPTY_DOC thread stability issues
>
> [image removed]
>
> [image removed]
>
>
> Use of EMPTY_DOC (an instance of an empty DOM Document element within
> class XmlUtils) among multiple threads causes unpredictable results.
> While testing a messaging service that uses the Apache Muse WS-N
> implementation, I noticed that 4-6 messages in a 1000 were being dropped.
> Notification messages were being sent at a rate of 20 per second. Where
> the messages were dropped, the following stack trace occurred:
>
> 2007-08-24 15:27:58,031 ERROR [STDERR] Aug 24, 2007 3:27:58 PM
> org.apache.muse.util.LoggingUtils logError
> INFO: There was an error while processing a request:
>
> [ID = 'NoMessageContent'] The NotificationMessage XML does not have a
> Message element. All messages must have a message payload associated with
> them.
>
> org.apache.muse.ws.notification.impl.
> SimpleNotificationMessage.<init>(SimpleNotificationMessage.java:117)
> org.apache.muse.ws.notification.impl.
> NotificationMessageSerializer.fromXML
(NotificationMessageSerializer.java:46)
> org.apache.muse.core.serializer.ArraySerializer.
> fromXML(ArraySerializer.java:126)
> org.apache.muse.ws.notification.impl.NotifyHandler.
> fromXML(NotifyHandler.java:62)
> org.apache.muse.core.SimpleResource.invoke
(SimpleResource.java:368)
> org.apache.muse.core.routing.SimpleResourceRouter.
> invoke(SimpleResourceRouter.java:290)
> org.apache.muse.core.platform.axis2.AxisIsolationLayer.
> invoke(AxisIsolationLayer.java:136)
> org.apache.muse.core.platform.axis2.AxisIsolationLayer.
> handleRequest(AxisIsolationLayer.java:88)
> sun.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)
> sun.reflect.DelegatingMethodAccessorImpl.
> invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:324)
> org.apache.axis2.receivers.RawXMLINOutMessageReceiver.
> invokeBusinessLogic(RawXMLINOutMessageReceiver.java:88)
> org.apache.axis2.receivers.AbstractInOutSyncMessageReceiver.
> receive(AbstractInOutSyncMessageReceiver.java:39)
> org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:493)
> org.apache.axis2.transport.http.HTTPTransportUtils.
> processHTTPPostRequest(HTTPTransportUtils.java:319)
> 22 more...
>
> All of the messages did in fact have a message payload associated with
> them. After further investigation I noticed that
> AxisEnvironment.convertToDOM is calling XmlUtils.createElement(QName)
> which uses the common EMPTY_DOC Document instance. Sharing EMPTY_DOC
> among multiple threads is not safe.
>
> I've also encountered this issue while attempting to set the producer
> reference on a NotificationMessage. The code actually performs a deep
> copy of the EndpointReference class which also relies on EMPTY_DOC. I
> hacked a work-around within my code that avoided use of EMPTY_DOC and
> resolved the issue.
>
> A grep for EMPTY_DOC against the code base turned up over 70 instances of
> its use. That does not count methods that use it that are called by
other
> methods, so the actual usage is much higher. I'm going to open a JIRA
bug
> against this issue and submit patches for it.
>
> Related to that, I've submitted patches for other issues (MUSE-240,
> MUSE-241) over a month ago that have not been committed. The patch for
> MUSE-240 also resolves MUSE-225. Please let me know the appropriate
> process for non-committers to have patches applied to the code base.
When
> I submitted the JIRA issues they showed up in this mailing list, but is
an
> explicit email more appropriate? Let me know how best to proceed.
>
> Thanks!
>
> Rich Lucente
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: muse-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: muse-dev-help@ws.apache.org
>
> [attachment "EMPTY_DOC_errors.txt" deleted by Daniel Jemiolo/Durham/IBM]
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: muse-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: muse-dev-help@ws.apache.org
RE: EMPTY_DOC thread stability issues
Posted by "Vinh Nguyen (vinguye2)" <vi...@cisco.com>.
Hi Dan,
I've also done some testing just now and am finding this to be a very
serious issue. It really affects the usability of Muse as a whole.
To test, I created three resources. Each had a loop to generate 100
notifications with no pause between notifications. So out of 300 total
notifications generated, I had these results:
Test A = 9 notifications dropped
Test B = 24 notifications dropped
Attached is a file containing the various exceptions which caused
SimpleSubscriptionManager.publish() to fail. As Rich pointed out, the
culprit is using EMPTY_DOC from multiple threads (i.e. each client
request is one thread).
The XmlUtils.EMPTY_DOC javadocs has this:
"This should NOT be used by callers that want to build new fragments and
attach them to a Document...you should never append children."
But in XmlUtils itself and many other classes, the following code
patterns are used:
Document doc = XmlUtils.EMPTY_DOC;
Element xml = XmlUtils.createElement(doc, qname);
xml.appendChild(node);
OR:
doc.importNode(node, true);
The doc is a shared object but is being modified, so errors will occur.
So just about all the serializers are affected, including the
EndpointReference class. This means errors will most likely occur when
Muse handles requests/responses from multiple clients, or when
notifications are sent from multiple resources. The latter is easier to
test.
So far, we've been testing using just one client and one producer
instance, so the problem doesn't occur. But now that we are testing by
using multiple producers, the exceptions are occuring frequently.
To begin the fix, all serializers and the EndpointReference class needs
to be patched. So instead of:
doc = XmlUtils.EMPTY_DOC;
We should do:
doc = XmlUtils.createDocument();
The overhead of creating a new document is small when compared to the
multi thread issue, which cannot be avoided. But, we can optimize
creating new documents by updating XmlUtils to create the
DocumentBuilderFactory only once. This way, XmlUtils.createBuilder()
doesn't have to create a new factory everytime.
-Vinh
________________________________
From: Daniel Jemiolo [mailto:danjemiolo@us.ibm.com]
Sent: Monday, August 27, 2007 7:31 AM
To: muse-dev@ws.apache.org
Subject: Re: EMPTY_DOC thread stability issues
The use of EMPTY_DOC was an attempt to avoid creating a new Document
every time we wanted to copy or create small fragments of XML. This
happened a lot during the request/response process, so the creation of
these factory objects was not insignificant. Of course, you are right
about the threading issue, so I guess we're out of luck there.
I am setting aside Friday of this week to go through current JIRA items
and apply all patches that have been submitted, close any issues that
are recommended for closure, and sort unscheduled items into 2.3 if
necessary. Sorry for the delay.
Dan
rlucente@xecu.net
rlucente@xecu.net
08/24/2007 04:37 PM
Please respond to
muse-dev@ws.apache.org
To
muse-dev@ws.apache.org
cc
Subject
EMPTY_DOC thread stability issues
Use of EMPTY_DOC (an instance of an empty DOM Document element within
class XmlUtils) among multiple threads causes unpredictable results.
While testing a messaging service that uses the Apache Muse WS-N
implementation, I noticed that 4-6 messages in a 1000 were being
dropped.
Notification messages were being sent at a rate of 20 per second. Where
the messages were dropped, the following stack trace occurred:
2007-08-24 15:27:58,031 ERROR [STDERR] Aug 24, 2007 3:27:58 PM
org.apache.muse.util.LoggingUtils logError
INFO: There was an error while processing a request:
[ID = 'NoMessageContent'] The NotificationMessage XML does not have a
Message element. All messages must have a message payload associated
with
them.
org.apache.muse.ws.notification.impl.SimpleNotificationMessage.<init>(Si
mpleNotificationMessage.java:117)
org.apache.muse.ws.notification.impl.NotificationMessageSerializer.fromX
ML(NotificationMessageSerializer.java:46)
org.apache.muse.core.serializer.ArraySerializer.fromXML(ArraySerializer.
java:126)
org.apache.muse.ws.notification.impl.NotifyHandler.fromXML(NotifyHandler
.java:62)
org.apache.muse.core.SimpleResource.invoke(SimpleResource.java:368)
org.apache.muse.core.routing.SimpleResourceRouter.invoke(SimpleResourceR
outer.java:290)
org.apache.muse.core.platform.axis2.AxisIsolationLayer.invoke(AxisIsolat
ionLayer.java:136)
org.apache.muse.core.platform.axis2.AxisIsolationLayer.handleRequest(Axi
sIsolationLayer.java:88)
sun.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
java.lang.reflect.Method.invoke(Method.java:324)
org.apache.axis2.receivers.RawXMLINOutMessageReceiver.invokeBusinessLogi
c(RawXMLINOutMessageReceiver.java:88)
org.apache.axis2.receivers.AbstractInOutSyncMessageReceiver.receive(Abst
ractInOutSyncMessageReceiver.java:39)
org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:493)
org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostReques
t(HTTPTransportUtils.java:319)
22 more...
All of the messages did in fact have a message payload associated with
them. After further investigation I noticed that
AxisEnvironment.convertToDOM is calling XmlUtils.createElement(QName)
which uses the common EMPTY_DOC Document instance. Sharing EMPTY_DOC
among multiple threads is not safe.
I've also encountered this issue while attempting to set the producer
reference on a NotificationMessage. The code actually performs a deep
copy of the EndpointReference class which also relies on EMPTY_DOC. I
hacked a work-around within my code that avoided use of EMPTY_DOC and
resolved the issue.
A grep for EMPTY_DOC against the code base turned up over 70 instances
of
its use. That does not count methods that use it that are called by
other
methods, so the actual usage is much higher. I'm going to open a JIRA
bug
against this issue and submit patches for it.
Related to that, I've submitted patches for other issues (MUSE-240,
MUSE-241) over a month ago that have not been committed. The patch for
MUSE-240 also resolves MUSE-225. Please let me know the appropriate
process for non-committers to have patches applied to the code base.
When
I submitted the JIRA issues they showed up in this mailing list, but is
an
explicit email more appropriate? Let me know how best to proceed.
Thanks!
Rich Lucente
---------------------------------------------------------------------
To unsubscribe, e-mail: muse-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: muse-dev-help@ws.apache.org
Re: EMPTY_DOC thread stability issues
Posted by Daniel Jemiolo <da...@us.ibm.com>.
The use of EMPTY_DOC was an attempt to avoid creating a new Document every
time we wanted to copy or create small fragments of XML. This happened a
lot during the request/response process, so the creation of these factory
objects was not insignificant. Of course, you are right about the threading
issue, so I guess we're out of luck there.
I am setting aside Friday of this week to go through current JIRA items and
apply all patches that have been submitted, close any issues that are
recommended for closure, and sort unscheduled items into 2.3 if necessary.
Sorry for the delay.
Dan
rlucente@xecu.net
08/24/2007 04:37 To
PM muse-dev@ws.apache.org
cc
Please respond to Subject
muse-dev@ws.apach EMPTY_DOC thread stability issues
e.org
Use of EMPTY_DOC (an instance of an empty DOM Document element within
class XmlUtils) among multiple threads causes unpredictable results.
While testing a messaging service that uses the Apache Muse WS-N
implementation, I noticed that 4-6 messages in a 1000 were being dropped.
Notification messages were being sent at a rate of 20 per second. Where
the messages were dropped, the following stack trace occurred:
2007-08-24 15:27:58,031 ERROR [STDERR] Aug 24, 2007 3:27:58 PM
org.apache.muse.util.LoggingUtils logError
INFO: There was an error while processing a request:
[ID = 'NoMessageContent'] The NotificationMessage XML does not have a
Message element. All messages must have a message payload associated with
them.
org.apache.muse.ws.notification.impl.SimpleNotificationMessage.<init>
(SimpleNotificationMessage.java:117)
org.apache.muse.ws.notification.impl.NotificationMessageSerializer.fromXML
(NotificationMessageSerializer.java:46)
org.apache.muse.core.serializer.ArraySerializer.fromXML
(ArraySerializer.java:126)
org.apache.muse.ws.notification.impl.NotifyHandler.fromXML
(NotifyHandler.java:62)
org.apache.muse.core.SimpleResource.invoke(SimpleResource.java:368)
org.apache.muse.core.routing.SimpleResourceRouter.invoke
(SimpleResourceRouter.java:290)
org.apache.muse.core.platform.axis2.AxisIsolationLayer.invoke
(AxisIsolationLayer.java:136)
org.apache.muse.core.platform.axis2.AxisIsolationLayer.handleRequest
(AxisIsolationLayer.java:88)
sun.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:324)
org.apache.axis2.receivers.RawXMLINOutMessageReceiver.invokeBusinessLogic
(RawXMLINOutMessageReceiver.java:88)
org.apache.axis2.receivers.AbstractInOutSyncMessageReceiver.receive
(AbstractInOutSyncMessageReceiver.java:39)
org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:493)
org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest
(HTTPTransportUtils.java:319)
22 more...
All of the messages did in fact have a message payload associated with
them. After further investigation I noticed that
AxisEnvironment.convertToDOM is calling XmlUtils.createElement(QName)
which uses the common EMPTY_DOC Document instance. Sharing EMPTY_DOC
among multiple threads is not safe.
I've also encountered this issue while attempting to set the producer
reference on a NotificationMessage. The code actually performs a deep
copy of the EndpointReference class which also relies on EMPTY_DOC. I
hacked a work-around within my code that avoided use of EMPTY_DOC and
resolved the issue.
A grep for EMPTY_DOC against the code base turned up over 70 instances of
its use. That does not count methods that use it that are called by other
methods, so the actual usage is much higher. I'm going to open a JIRA bug
against this issue and submit patches for it.
Related to that, I've submitted patches for other issues (MUSE-240,
MUSE-241) over a month ago that have not been committed. The patch for
MUSE-240 also resolves MUSE-225. Please let me know the appropriate
process for non-committers to have patches applied to the code base. When
I submitted the JIRA issues they showed up in this mailing list, but is an
explicit email more appropriate? Let me know how best to proceed.
Thanks!
Rich Lucente
---------------------------------------------------------------------
To unsubscribe, e-mail: muse-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: muse-dev-help@ws.apache.org