You are viewing a plain text version of this content. The canonical link for it is here.
Posted to muse-user@ws.apache.org by Ch...@swisscom.com on 2007/09/02 19:28:39 UTC
RE: EndpointReference thread-safe?
Hi All,
I don't think synchronizing on any set of functions will solve the
problem. EndpointReference is using XmlUtils.EMPTY_DOC i.e. a single
document instance for every thread. As Dong said, according the the
javadocs ( and XmlUtils.createBuilder() of course ^_^) you must use a
DocumentBuilder per thread, the documents used within it cannot be used
safely across threads (or the document builder itself) when involving
writes.
So unless I'm missing something (always possible) EndpointReference
should be using XmlUtils.createDocument() instead (lines 124, 175 and
239). I think its a bug, and if its agreed to be one, I'll have a look
for others (72 references to EMPTY_DOC in the codebase) as this is also
very important to me, and raise Jiras with patches off the 2.2 code.
Vinh/Dong is it possible to verify this works as a fix?
cheers,
Chris
PS(If any one wants me to make patches against latest HEAD instead, let
me know)
-----Original Message-----
From: Vinh Nguyen (vinguye2) [mailto:vinguye2@cisco.com]
Sent: Friday, August 31, 2007 8:02 PM
To: muse-user@ws.apache.org
Cc: dong.nguyen@l-3comcept.com
Subject: RE: EndpointReference thread-safe?
Hi Dong,
Have you found a resolution on this yet? Looks like you are
encountering a problem we're seeing now, too. Synchronize on setting
the producer EPR is probably not the best solution because this really
slows things down, especially if you have to generate large numbers of
notifications.
-Vinh
-----Original Message-----
From: dnguyen [mailto:dong.nguyen@l-3comcept.com]
Sent: Wednesday, April 04, 2007 9:45 AM
To: muse-user@ws.apache.org
Subject: Re: EndpointReference thread-safe?
The only thing that works is to synchronize on the producer EPR, since
making copies of it does not seem to work. Does this mean that the EPR
deep copy or Xerces deep copy is broken? Anyway, this effectively
serializes all notifies to any consumers, which impacts scalability to a
certain degree.
--
View this message in context:
http://www.nabble.com/EndpointReference-thread-safe--tf3506487.html#a983
9159
Sent from the Muse User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: muse-user-unsubscribe@ws.apache.org
For additional commands, e-mail: muse-user-help@ws.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: muse-user-unsubscribe@ws.apache.org
For additional commands, e-mail: muse-user-help@ws.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: muse-user-unsubscribe@ws.apache.org
For additional commands, e-mail: muse-user-help@ws.apache.org
RE: EndpointReference thread-safe?
Posted by "Vinh Nguyen (vinguye2)" <vi...@cisco.com>.
Hi Dong and Chris,
Thanks for your replies. Let's merge onto the thread "EMPTY_DOC thread
stability issues". This EPR problem stems from this.
-Vinh
-----Original Message-----
From: dnguyen [mailto:dong.nguyen@l-3comcept.com]
Sent: Tuesday, September 04, 2007 1:36 PM
To: muse-user@ws.apache.org
Subject: RE: EndpointReference thread-safe?
Hi Chris,
I have not been able to create a testcase with EPR by itself being
accessed by multiple threads that will cause the issue. It is only when
we send subscribers NotificationMessage using separate threads for each
subscriber.
I am willing to try the patch if available.
Dong
Chris.Twiner wrote:
>
> Hi,
>
> From what I could work out, from within the list comments and the
> code, the state is stored in the Document itself, and as cloneNode
> uses Object.clone and then sets the doc it won't work. Using
> importNode helps a little (as it uses
> getFirstChild()/getNextSibling()), but it just puts the problem to a
later stage.
>
> getAllElements just does the same, calls getChildNodes and then forces
> the cache to be used. Deleting the cache just stops the null for the
> parent, it doesn't stop incorrect nodes being returned or race
> conditions with other nulls.
>
> The simple thing is to stop using getChildNodes, from what I can see
> in the code there isn't a need for it. The only place I've seen that
> doesn't require all of the nodes anyway is in EndpointReference's
> getNumberOfParameters, but that behaviour can be safely cached (its
> not used directly in the project anyway).
>
> Looking further at the use cases in Muse only the IsolationLayer
> (because of the DeferredImpl) needs to call hasChildNodes() on the
> document node, for it to force that synchronizeChildren be called (its
> cached from then on in each node). Then every other piece of code can
> simply pointer chase with the getFirstChild()/getNextSibling()
approach.
> No synchronization required.
>
> re using other jaxp's, the DOM itself makes no statement about even
> read thread safety. All of the jaxp impls suffer some form of
> threading problem. Considering all of the problems with fighting
> against namespace problems (much worse IMO) it makes sense to stick
> with the devil you know :-<.
>
> Again for most of the xerces releases using the
> getFirstChild()/getNextSibling() is a seamless dropin for the
> getChildNodes problem. Its a shame that the xerces guys are very much
> against any form of thread safety (except application enforced).
> Going with the standard approach the only safe thing is to always
> serialize to objects / keep the strings around, which would overly
> complicate the code.
>
> I'm willing to give it a try and send you patched libs to try out (I
> don't have a test case for this yet) if its quick to reproduce, just
> let me know. If it works out I can raise a jira with the patches.
>
> cheers,
> Chris
>
> PS(this problem is a very unpleasent suprise for me, I've been caching
> Documents in another application which can then be used in jaxp
> transformations and xalan uses the getChildNodes approach in a few
> places. Now I see its only been working by luck alone :-< ).
>
> -----Original Message-----
> From: dnguyen [mailto:dong.nguyen@l-3comcept.com]
> Sent: Tuesday, September 04, 2007 6:46 PM
> To: muse-user@ws.apache.org
> Subject: RE: EndpointReference thread-safe?
>
>
> Hi,
>
> After some digging a good while back, we found that Xerces optimized
> the processing by using caching. Tracing through the Xerces source,
> we discovered that the cache was being deleted after the thread
> finished processing or parsing the DOM object, thus invalidating the
> current "state"
> of the DOM object for other threads that may be in the middle of
> processing.
> It also seemed that even if you make a "deep" copy of the DOM object,
> they would share the same cache (not sure)? We hacked
> org.apache.xerces.dom.ParentNode to instead not delete this cache.
> This got us around the NullPointerException due to invoking toXML() on
> the producer EPR in SimpleNotificationMessage.toXML(), but we ran
> across another NullPointerException further down the source at
> XmlUtils.getAllNamespaces(root). The stacktrace ends at
> XmlUtils.getAllElements(). At that point, we have given up.
>
> The question now is could another JAXP library be easily substituted
> for Xerces?
>
> Dong
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: muse-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: muse-user-help@ws.apache.org
>
>
>
--
View this message in context:
http://www.nabble.com/EndpointReference-thread-safe--tf3506487.html#a124
86133
Sent from the Muse User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: muse-user-unsubscribe@ws.apache.org
For additional commands, e-mail: muse-user-help@ws.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: muse-user-unsubscribe@ws.apache.org
For additional commands, e-mail: muse-user-help@ws.apache.org
RE: EndpointReference thread-safe?
Posted by dnguyen <do...@l-3comcept.com>.
Hi Chris,
I have not been able to create a testcase with EPR by itself being accessed
by multiple threads that will cause the issue. It is only when we send
subscribers NotificationMessage using separate threads for each subscriber.
I am willing to try the patch if available.
Dong
Chris.Twiner wrote:
>
> Hi,
>
> From what I could work out, from within the list comments and the code,
> the state is stored in the Document itself, and as cloneNode uses
> Object.clone and then sets the doc it won't work. Using importNode
> helps a little (as it uses getFirstChild()/getNextSibling()), but it
> just puts the problem to a later stage.
>
> getAllElements just does the same, calls getChildNodes and then forces
> the cache to be used. Deleting the cache just stops the null for the
> parent, it doesn't stop incorrect nodes being returned or race
> conditions with other nulls.
>
> The simple thing is to stop using getChildNodes, from what I can see in
> the code there isn't a need for it. The only place I've seen that
> doesn't require all of the nodes anyway is in EndpointReference's
> getNumberOfParameters, but that behaviour can be safely cached (its not
> used directly in the project anyway).
>
> Looking further at the use cases in Muse only the IsolationLayer
> (because of the DeferredImpl) needs to call hasChildNodes() on the
> document node, for it to force that synchronizeChildren be called (its
> cached from then on in each node). Then every other piece of code can
> simply pointer chase with the getFirstChild()/getNextSibling() approach.
> No synchronization required.
>
> re using other jaxp's, the DOM itself makes no statement about even read
> thread safety. All of the jaxp impls suffer some form of threading
> problem. Considering all of the problems with fighting against
> namespace problems (much worse IMO) it makes sense to stick with the
> devil you know :-<.
>
> Again for most of the xerces releases using the
> getFirstChild()/getNextSibling() is a seamless dropin for the
> getChildNodes problem. Its a shame that the xerces guys are very much
> against any form of thread safety (except application enforced). Going
> with the standard approach the only safe thing is to always serialize to
> objects / keep the strings around, which would overly complicate the
> code.
>
> I'm willing to give it a try and send you patched libs to try out (I
> don't have a test case for this yet) if its quick to reproduce, just let
> me know. If it works out I can raise a jira with the patches.
>
> cheers,
> Chris
>
> PS(this problem is a very unpleasent suprise for me, I've been caching
> Documents in another application which can then be used in jaxp
> transformations and xalan uses the getChildNodes approach in a few
> places. Now I see its only been working by luck alone :-< ).
>
> -----Original Message-----
> From: dnguyen [mailto:dong.nguyen@l-3comcept.com]
> Sent: Tuesday, September 04, 2007 6:46 PM
> To: muse-user@ws.apache.org
> Subject: RE: EndpointReference thread-safe?
>
>
> Hi,
>
> After some digging a good while back, we found that Xerces optimized the
> processing by using caching. Tracing through the Xerces source, we
> discovered that the cache was being deleted after the thread finished
> processing or parsing the DOM object, thus invalidating the current
> "state"
> of the DOM object for other threads that may be in the middle of
> processing.
> It also seemed that even if you make a "deep" copy of the DOM object,
> they would share the same cache (not sure)? We hacked
> org.apache.xerces.dom.ParentNode to instead not delete this cache. This
> got us around the NullPointerException due to invoking toXML() on the
> producer EPR in SimpleNotificationMessage.toXML(), but we ran across
> another NullPointerException further down the source at
> XmlUtils.getAllNamespaces(root). The stacktrace ends at
> XmlUtils.getAllElements(). At that point, we have given up.
>
> The question now is could another JAXP library be easily substituted for
> Xerces?
>
> Dong
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: muse-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: muse-user-help@ws.apache.org
>
>
>
--
View this message in context: http://www.nabble.com/EndpointReference-thread-safe--tf3506487.html#a12486133
Sent from the Muse User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: muse-user-unsubscribe@ws.apache.org
For additional commands, e-mail: muse-user-help@ws.apache.org
RE: EndpointReference thread-safe?
Posted by Ch...@swisscom.com.
Hi,
>From what I could work out, from within the list comments and the code,
the state is stored in the Document itself, and as cloneNode uses
Object.clone and then sets the doc it won't work. Using importNode
helps a little (as it uses getFirstChild()/getNextSibling()), but it
just puts the problem to a later stage.
getAllElements just does the same, calls getChildNodes and then forces
the cache to be used. Deleting the cache just stops the null for the
parent, it doesn't stop incorrect nodes being returned or race
conditions with other nulls.
The simple thing is to stop using getChildNodes, from what I can see in
the code there isn't a need for it. The only place I've seen that
doesn't require all of the nodes anyway is in EndpointReference's
getNumberOfParameters, but that behaviour can be safely cached (its not
used directly in the project anyway).
Looking further at the use cases in Muse only the IsolationLayer
(because of the DeferredImpl) needs to call hasChildNodes() on the
document node, for it to force that synchronizeChildren be called (its
cached from then on in each node). Then every other piece of code can
simply pointer chase with the getFirstChild()/getNextSibling() approach.
No synchronization required.
re using other jaxp's, the DOM itself makes no statement about even read
thread safety. All of the jaxp impls suffer some form of threading
problem. Considering all of the problems with fighting against
namespace problems (much worse IMO) it makes sense to stick with the
devil you know :-<.
Again for most of the xerces releases using the
getFirstChild()/getNextSibling() is a seamless dropin for the
getChildNodes problem. Its a shame that the xerces guys are very much
against any form of thread safety (except application enforced). Going
with the standard approach the only safe thing is to always serialize to
objects / keep the strings around, which would overly complicate the
code.
I'm willing to give it a try and send you patched libs to try out (I
don't have a test case for this yet) if its quick to reproduce, just let
me know. If it works out I can raise a jira with the patches.
cheers,
Chris
PS(this problem is a very unpleasent suprise for me, I've been caching
Documents in another application which can then be used in jaxp
transformations and xalan uses the getChildNodes approach in a few
places. Now I see its only been working by luck alone :-< ).
-----Original Message-----
From: dnguyen [mailto:dong.nguyen@l-3comcept.com]
Sent: Tuesday, September 04, 2007 6:46 PM
To: muse-user@ws.apache.org
Subject: RE: EndpointReference thread-safe?
Hi,
After some digging a good while back, we found that Xerces optimized the
processing by using caching. Tracing through the Xerces source, we
discovered that the cache was being deleted after the thread finished
processing or parsing the DOM object, thus invalidating the current
"state"
of the DOM object for other threads that may be in the middle of
processing.
It also seemed that even if you make a "deep" copy of the DOM object,
they would share the same cache (not sure)? We hacked
org.apache.xerces.dom.ParentNode to instead not delete this cache. This
got us around the NullPointerException due to invoking toXML() on the
producer EPR in SimpleNotificationMessage.toXML(), but we ran across
another NullPointerException further down the source at
XmlUtils.getAllNamespaces(root). The stacktrace ends at
XmlUtils.getAllElements(). At that point, we have given up.
The question now is could another JAXP library be easily substituted for
Xerces?
Dong
---------------------------------------------------------------------
To unsubscribe, e-mail: muse-user-unsubscribe@ws.apache.org
For additional commands, e-mail: muse-user-help@ws.apache.org
RE: EndpointReference thread-safe?
Posted by dnguyen <do...@l-3comcept.com>.
Hi,
After some digging a good while back, we found that Xerces optimized the
processing by using caching. Tracing through the Xerces source, we
discovered that the cache was being deleted after the thread finished
processing or parsing the DOM object, thus invalidating the current "state"
of the DOM object for other threads that may be in the middle of processing.
It also seemed that even if you make a "deep" copy of the DOM object, they
would share the same cache (not sure)? We hacked
org.apache.xerces.dom.ParentNode to instead not delete this cache. This got
us around the NullPointerException due to invoking toXML() on the producer
EPR in SimpleNotificationMessage.toXML(), but we ran across another
NullPointerException further down the source at
XmlUtils.getAllNamespaces(root). The stacktrace ends at
XmlUtils.getAllElements(). At that point, we have given up.
The question now is could another JAXP library be easily substituted for
Xerces?
Dong
Chris.Twiner wrote:
>
> Sorry a little further research and I found this issue:
>
> https://issues.apache.org/jira/browse/XERCESJ-911 (nodeListItem unsafe)
>
> i.e. it doesn't matter about having seperate builders, its just that the
> entire xml tree cannot be represented in the xerces dom and then read
> with nodelists.
>
> xerces wise this is due to an optimisation. Since EPRS are essential to
> Muse and there is alot of code that uses the getChildNodes / NodeList
> combination it seems to be that only a hack (using hasChildren in a
> synchronized block in the EPR then getFirstChild / getNextSibling when
> reading items from it) will remove the problem and reduce contention for
> the lock.
>
> In short using nodelists in xerces are 100% thread unsafe, but the rest
> of it doesn't claim to be thread safe even for reading in multiple
> threads :-<.
>
> chris
>
> NB (the hack is to trigger synchronizeChildren to be called for
> defferedimpls etc, but I'm not sure it really covers all the needs given
> the dom heavy interface in eprs and the rest of muse).
>
> -----Original Message-----
> From: Twiner Chris, IT-TBU-DL2-EAI-EDE
> Sent: Sunday, September 02, 2007 7:29 PM
> To: muse-user@ws.apache.org
> Cc: dong.nguyen@l-3comcept.com
> Subject: RE: EndpointReference thread-safe?
>
> Hi All,
>
> I don't think synchronizing on any set of functions will solve the
> problem. EndpointReference is using XmlUtils.EMPTY_DOC i.e. a single
> document instance for every thread. As Dong said, according the the
> javadocs ( and XmlUtils.createBuilder() of course ^_^) you must use a
> DocumentBuilder per thread, the documents used within it cannot be used
> safely across threads (or the document builder itself) when involving
> writes.
>
> So unless I'm missing something (always possible) EndpointReference
> should be using XmlUtils.createDocument() instead (lines 124, 175 and
> 239). I think its a bug, and if its agreed to be one, I'll have a look
> for others (72 references to EMPTY_DOC in the codebase) as this is also
> very important to me, and raise Jiras with patches off the 2.2 code.
>
> Vinh/Dong is it possible to verify this works as a fix?
>
> cheers,
> Chris
>
> PS(If any one wants me to make patches against latest HEAD instead, let
> me know)
>
> -----Original Message-----
> From: Vinh Nguyen (vinguye2) [mailto:vinguye2@cisco.com]
> Sent: Friday, August 31, 2007 8:02 PM
> To: muse-user@ws.apache.org
> Cc: dong.nguyen@l-3comcept.com
> Subject: RE: EndpointReference thread-safe?
>
> Hi Dong,
>
> Have you found a resolution on this yet? Looks like you are
> encountering a problem we're seeing now, too. Synchronize on setting
> the producer EPR is probably not the best solution because this really
> slows things down, especially if you have to generate large numbers of
> notifications.
>
> -Vinh
>
>
> -----Original Message-----
> From: dnguyen [mailto:dong.nguyen@l-3comcept.com]
> Sent: Wednesday, April 04, 2007 9:45 AM
> To: muse-user@ws.apache.org
> Subject: Re: EndpointReference thread-safe?
>
>
> The only thing that works is to synchronize on the producer EPR, since
> making copies of it does not seem to work. Does this mean that the EPR
> deep copy or Xerces deep copy is broken? Anyway, this effectively
> serializes all notifies to any consumers, which impacts scalability to a
> certain degree.
> --
> View this message in context:
> http://www.nabble.com/EndpointReference-thread-safe--tf3506487.html#a983
> 9159
> Sent from the Muse User mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: muse-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: muse-user-help@ws.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: muse-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: muse-user-help@ws.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: muse-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: muse-user-help@ws.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: muse-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: muse-user-help@ws.apache.org
>
>
>
--
View this message in context: http://www.nabble.com/EndpointReference-thread-safe--tf3506487.html#a12481946
Sent from the Muse User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: muse-user-unsubscribe@ws.apache.org
For additional commands, e-mail: muse-user-help@ws.apache.org
RE: EndpointReference thread-safe?
Posted by Ch...@swisscom.com.
Sorry a little further research and I found this issue:
https://issues.apache.org/jira/browse/XERCESJ-911 (nodeListItem unsafe)
i.e. it doesn't matter about having seperate builders, its just that the
entire xml tree cannot be represented in the xerces dom and then read
with nodelists.
xerces wise this is due to an optimisation. Since EPRS are essential to
Muse and there is alot of code that uses the getChildNodes / NodeList
combination it seems to be that only a hack (using hasChildren in a
synchronized block in the EPR then getFirstChild / getNextSibling when
reading items from it) will remove the problem and reduce contention for
the lock.
In short using nodelists in xerces are 100% thread unsafe, but the rest
of it doesn't claim to be thread safe even for reading in multiple
threads :-<.
chris
NB (the hack is to trigger synchronizeChildren to be called for
defferedimpls etc, but I'm not sure it really covers all the needs given
the dom heavy interface in eprs and the rest of muse).
-----Original Message-----
From: Twiner Chris, IT-TBU-DL2-EAI-EDE
Sent: Sunday, September 02, 2007 7:29 PM
To: muse-user@ws.apache.org
Cc: dong.nguyen@l-3comcept.com
Subject: RE: EndpointReference thread-safe?
Hi All,
I don't think synchronizing on any set of functions will solve the
problem. EndpointReference is using XmlUtils.EMPTY_DOC i.e. a single
document instance for every thread. As Dong said, according the the
javadocs ( and XmlUtils.createBuilder() of course ^_^) you must use a
DocumentBuilder per thread, the documents used within it cannot be used
safely across threads (or the document builder itself) when involving
writes.
So unless I'm missing something (always possible) EndpointReference
should be using XmlUtils.createDocument() instead (lines 124, 175 and
239). I think its a bug, and if its agreed to be one, I'll have a look
for others (72 references to EMPTY_DOC in the codebase) as this is also
very important to me, and raise Jiras with patches off the 2.2 code.
Vinh/Dong is it possible to verify this works as a fix?
cheers,
Chris
PS(If any one wants me to make patches against latest HEAD instead, let
me know)
-----Original Message-----
From: Vinh Nguyen (vinguye2) [mailto:vinguye2@cisco.com]
Sent: Friday, August 31, 2007 8:02 PM
To: muse-user@ws.apache.org
Cc: dong.nguyen@l-3comcept.com
Subject: RE: EndpointReference thread-safe?
Hi Dong,
Have you found a resolution on this yet? Looks like you are
encountering a problem we're seeing now, too. Synchronize on setting
the producer EPR is probably not the best solution because this really
slows things down, especially if you have to generate large numbers of
notifications.
-Vinh
-----Original Message-----
From: dnguyen [mailto:dong.nguyen@l-3comcept.com]
Sent: Wednesday, April 04, 2007 9:45 AM
To: muse-user@ws.apache.org
Subject: Re: EndpointReference thread-safe?
The only thing that works is to synchronize on the producer EPR, since
making copies of it does not seem to work. Does this mean that the EPR
deep copy or Xerces deep copy is broken? Anyway, this effectively
serializes all notifies to any consumers, which impacts scalability to a
certain degree.
--
View this message in context:
http://www.nabble.com/EndpointReference-thread-safe--tf3506487.html#a983
9159
Sent from the Muse User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: muse-user-unsubscribe@ws.apache.org
For additional commands, e-mail: muse-user-help@ws.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: muse-user-unsubscribe@ws.apache.org
For additional commands, e-mail: muse-user-help@ws.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: muse-user-unsubscribe@ws.apache.org
For additional commands, e-mail: muse-user-help@ws.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: muse-user-unsubscribe@ws.apache.org
For additional commands, e-mail: muse-user-help@ws.apache.org