You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by "Hendley, Sam" <Sa...@sensus.com> on 2014/10/14 03:45:00 UTC
XSLT Thread Safety Issue
We are having a very strange (and intermittent) issue with some of our camel routes.
We are in the process of upgrading to a more up-to-date version but are still on camel 2.10.7 and running on java 1.6. I have looked through the forums and tickets but haven't found anything that looks like it could cause this issue.
What appears to be occurring is this:
* An XML document is received from a JMS queue (as a text message)
o The xml is correctly formatted and is what we expect to see
* It is then routed through an XSLT transformation (to strip off excess namespacing)
* 99% of the time this works, but very occasionally we get Xpath errors when we go onto the next step
* We were able to do a heap dump of a system that was having this problem, when we dug through the dump we were able to analyze an instance of the failure (same exchangeId)
o It turns out that when the issue occurs the enclosing tags (document root) have disappeared rendering the document un-xpathable
Since this works nearly all of the time and there is nothing special about the failing documents we suspect it could be a threading or race condition in the XSLT processing. Are there any known issues that might be related? This route is running in a relatively large and busy camel application that makes heavy use of XSLT transforms. We believe similar issues have occurred in other instances but haven't debugged to this level so that is just a hunch.
I have included some details below, let me know if there is more I should include. I still have the heap/thread dump if there is anything else that might be worth digging into.
Thanks for any help or advise,
Sam Hendley
Jms Timeout message (notice it has xml header and wrapping of EncryptionKeyFlow)
[cid:image001.png@01CFD7FC.D4F23FD0]
<?xml version="1.0" encoding="UTF-8"?>
<EncryptionKeyFlow xmlns="http://www.example.com/messages">
<transactionId>2193-49984781-keymanager-1411163761368-126275</transactionId>
<responseQueue>orch.encryptionkey.encryptionkeyTimeout.queue</responseQueue>
....
<status>FAIL</status>
<substatus>timeout</substatus>
<statusMessages>Request timed out while waiting for response(s).</statusMessages>
</EncryptionKeyFlow>
Camel Body (stripped off top and bottom elements)
[cid:image002.png@01CFD7FC.D4F23FD0]
<transactionId>2193-49984781-keymanager-1411163761368-126275</transactionId>
<responseQueue>orch.encryptionkey.encryptionkeyTimeout.queue</responseQueue>
...
<status>FAIL</status>
<substatus>timeout</substatus>
<statusMessages>Request timed out while waiting for response(s).</statusMessages>
Route:
from(JMS_encryptionkeyTimeout).routeId(JMS_encryptionkeyTimeout)
.log(LoggingLevel.INFO, "Timed out waiting for encryptionkey response")
.choice()
.when(isInBodyNullOrEmpty)
.to(SEDA_encryptionkeyOrchDead)
.otherwise()
.to(DIRECT_processBackToClient)
.end();
// Called from several places - when a response needs to be sent back to the client
from(DIRECT_processBackToClient).routeId(DIRECT_processBackToClient)
.setHeader(HDR_JMS_CORRELATION_ID).xpath("//*[local-name()='transactionId']", String.class)
.to(XSLT_toAPIResponsePreprocessing)
.choice()
// New response type may be added later
.when().xpath("not(/*[local-name()='responseQueue'])")
.log(LoggingLevel.ERROR, "Unable to format encryptionkey response")
.to(SEDA_encryptionkeyOrchDead)
.otherwise()
.recipientList().simple(JMS_RESPONSE_QUEUE)
.end();
XSLT that is probably at fault:
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output standalone="yes" omit-xml-declaration="yes" />
<xsl:template match="processing-instruction()">
<xsl:copy><xsl:apply-templates/></xsl:copy>
</xsl:template>
<xsl:template match="*">
<xsl:element name="{local-name()}"><xsl:apply-templates select="@*|node()"/></xsl:element>
</xsl:template>
<xsl:template match="@*">
<xsl:attribute name="{local-name()}"><xsl:value-of select="."/></xsl:attribute>
</xsl:template>
</xsl:stylesheet>
Re: XSLT Thread Safety Issue
Posted by Christian Müller <ch...@gmail.com>.
Hello Sam!
I don't have so much experience with XSLT, but maybe my suggestions are
nevertheless helpful:
- if possible, use the Saxon transformer factory as described here [1]. The
JDK build in is slow, buggy, ...
- if possible, use the Saxon XPath factory as described here [2]. It's much
faster than the JDK build in as I measured [3].
- if it's not possible to use Saxon, may try to update the JDK to get a
newer version of the transformer/XPath factories (I believe it's an issue
there)
- and Willem was referring to the XSLT component options
transformerCacheSize and contentCache, but it looks like you are using the
default values. So, there is no issue here.
[1] http://camel.apache.org/xslt
[2] http://camel.apache.org/xpath.html
[3] http://www.slideshare.net/muellerc/apache-con-na-2013
Best,
Christian
On Tue, Oct 14, 2014 at 3:44 PM, Hendley, Sam <Sa...@sensus.com>
wrote:
> The route endpoint was
> "xslt:classpath:xslt/orch-encryptionkey-flowRemoveNamespace.xsl". You may
> have been asking for the actual XSLT which I included but it looks like it
> got pruned by the listserver. I will include it here xml encoded:
>
> <xsl:stylesheet version="1.0" xmlns:xsl="
> http://www.w3.org/1999/XSL/Transform">
> <xsl:output standalone="yes"
> omit-xml-declaration="yes" />
>
> <xsl:template match="processing-instruction()">
> <xsl:copy><xsl:apply-templates/></xsl:copy>
> </xsl:template>
>
> <xsl:template match="*">
> <xsl:element
> name="{local-name()}"><xsl:apply-templates select="@
> *|node()"/></xsl:element>
> </xsl:template>
>
> <xsl:template match="@*">
> <xsl:attribute
> name="{local-name()}"><xsl:value-of
> select="."/></xsl:attribute>
> </xsl:template>
> </xsl:stylesheet>
>
>
> I don't believe we are using either of those options, a full text search
> of our entire codebase didn't yield any references.
>
> Sam
>
> -----Original Message-----
> From: Willem Jiang [mailto:willem.jiang@gmail.com]
> Sent: Monday, October 13, 2014 11:07 PM
> To: users@camel.apache.org
> Subject: Re: XSLT Thread Safety Issue
>
> Can I have a look at you XSLT endpoint uri?
> Did you use the options of transformerCacheSize or contentCache?
>
> --
> Willem Jiang
>
> Red Hat, Inc.
> Web: http://www.redhat.com
> Blog: http://willemjiang.blogspot.com (English) http://jnn.iteye.com
> (Chinese)
> Twitter: willemjiang
> Weibo: 姜宁willem
>
>
>
> On October 14, 2014 at 9:45:39 AM, Hendley, Sam (sam.hendley@sensus.com)
> wrote:
> > We are having a very strange (and intermittent) issue with some of our
> camel routes.
> >
> > We are in the process of upgrading to a more up-to-date version but
> > are still on camel 2.10.7 and running on java 1.6. I have looked
> > through the forums and tickets but haven't found anything that looks
> like it could cause this issue.
> >
> > What appears to be occurring is this:
> >
> > * An XML document is received from a JMS queue (as a text message)
> >
> > o The xml is correctly formatted and is what we expect to see
> >
> > * It is then routed through an XSLT transformation (to strip off
> > excess namespacing)
> >
> > * 99% of the time this works, but very occasionally we get Xpath
> > errors when we go onto the next step
> >
> > * We were able to do a heap dump of a system that was having this
> > problem, when we dug through the dump we were able to analyze an
> > instance of the failure (same exchangeId)
> >
> > o It turns out that when the issue occurs the enclosing tags (document
> > root) have disappeared rendering the document un-xpathable
> >
> > Since this works nearly all of the time and there is nothing special
> > about the failing documents we suspect it could be a threading or race
> condition in the XSLT processing.
> > Are there any known issues that might be related? This route is
> > running in a relatively large and busy camel application that makes
> > heavy use of XSLT transforms. We believe similar issues have occurred
> > in other instances but haven't debugged to this level so that is just a
> hunch.
> >
> > I have included some details below, let me know if there is more I
> > should include. I still have the heap/thread dump if there is anything
> else that might be worth digging into.
> >
> > Thanks for any help or advise,
> >
> > Sam Hendley
> >
> > Jms Timeout message (notice it has xml header and wrapping of
> > EncryptionKeyFlow) [cid:image001.png@01CFD7FC.D4F23FD0]
> >
> >
> >
> > 2193-49984781-keymanager-1411163761368-126275
> > orch.encryptionkey.encryptionkeyTimeout.queue
> > ....
> > FAIL
> > timeout
> > Request timed out while waiting for response(s).
> >
> >
> > Camel Body (stripped off top and bottom elements)
> > [cid:image002.png@01CFD7FC.D4F23FD0]
> >
> > 2193-49984781-keymanager-1411163761368-126275
> > orch.encryptionkey.encryptionkeyTimeout.queue
> > ...
> > FAIL
> > timeout
> > Request timed out while waiting for response(s).
> >
> > Route:
> > from(JMS_encryptionkeyTimeout).routeId(JMS_encryptionkeyTimeout)
> > .log(LoggingLevel.INFO, "Timed out waiting for encryptionkey response")
> > .choice()
> > .when(isInBodyNullOrEmpty)
> > .to(SEDA_encryptionkeyOrchDead)
> > .otherwise()
> > .to(DIRECT_processBackToClient)
> > .end();
> >
> > // Called from several places - when a response needs to be sent back to
> the client
> > from(DIRECT_processBackToClient).routeId(DIRECT_processBackToClient)
> >
> .setHeader(HDR_JMS_CORRELATION_ID).xpath("//*[local-name()='transactionId']",
> > String.class)
> > .to(XSLT_toAPIResponsePreprocessing)
> > .choice()
> > // New response type may be added later
> > .when().xpath("not(/*[local-name()='responseQueue'])")
> > .log(LoggingLevel.ERROR, "Unable to format encryptionkey response")
> > .to(SEDA_encryptionkeyOrchDead)
> > .otherwise()
> > .recipientList().simple(JMS_RESPONSE_QUEUE)
> > .end();
> >
> > XSLT that is probably at fault:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
>
RE: XSLT Thread Safety Issue
Posted by "Hendley, Sam" <Sa...@sensus.com>.
The route endpoint was "xslt:classpath:xslt/orch-encryptionkey-flowRemoveNamespace.xsl". You may have been asking for the actual XSLT which I included but it looks like it got pruned by the listserver. I will include it here xml encoded:
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output standalone="yes" omit-xml-declaration="yes" />
<xsl:template match="processing-instruction()">
<xsl:copy><xsl:apply-templates/></xsl:copy>
</xsl:template>
<xsl:template match="*">
<xsl:element name="{local-name()}"><xsl:apply-templates select="@*|node()"/></xsl:element>
</xsl:template>
<xsl:template match="@*">
<xsl:attribute name="{local-name()}"><xsl:value-of select="."/></xsl:attribute>
</xsl:template>
</xsl:stylesheet>
I don't believe we are using either of those options, a full text search of our entire codebase didn't yield any references.
Sam
-----Original Message-----
From: Willem Jiang [mailto:willem.jiang@gmail.com]
Sent: Monday, October 13, 2014 11:07 PM
To: users@camel.apache.org
Subject: Re: XSLT Thread Safety Issue
Can I have a look at you XSLT endpoint uri?
Did you use the options of transformerCacheSize or contentCache?
--
Willem Jiang
Red Hat, Inc.
Web: http://www.redhat.com
Blog: http://willemjiang.blogspot.com (English) http://jnn.iteye.com (Chinese)
Twitter: willemjiang
Weibo: 姜宁willem
On October 14, 2014 at 9:45:39 AM, Hendley, Sam (sam.hendley@sensus.com) wrote:
> We are having a very strange (and intermittent) issue with some of our camel routes.
>
> We are in the process of upgrading to a more up-to-date version but
> are still on camel 2.10.7 and running on java 1.6. I have looked
> through the forums and tickets but haven't found anything that looks like it could cause this issue.
>
> What appears to be occurring is this:
>
> * An XML document is received from a JMS queue (as a text message)
>
> o The xml is correctly formatted and is what we expect to see
>
> * It is then routed through an XSLT transformation (to strip off
> excess namespacing)
>
> * 99% of the time this works, but very occasionally we get Xpath
> errors when we go onto the next step
>
> * We were able to do a heap dump of a system that was having this
> problem, when we dug through the dump we were able to analyze an
> instance of the failure (same exchangeId)
>
> o It turns out that when the issue occurs the enclosing tags (document
> root) have disappeared rendering the document un-xpathable
>
> Since this works nearly all of the time and there is nothing special
> about the failing documents we suspect it could be a threading or race condition in the XSLT processing.
> Are there any known issues that might be related? This route is
> running in a relatively large and busy camel application that makes
> heavy use of XSLT transforms. We believe similar issues have occurred
> in other instances but haven't debugged to this level so that is just a hunch.
>
> I have included some details below, let me know if there is more I
> should include. I still have the heap/thread dump if there is anything else that might be worth digging into.
>
> Thanks for any help or advise,
>
> Sam Hendley
>
> Jms Timeout message (notice it has xml header and wrapping of
> EncryptionKeyFlow) [cid:image001.png@01CFD7FC.D4F23FD0]
>
>
>
> 2193-49984781-keymanager-1411163761368-126275
> orch.encryptionkey.encryptionkeyTimeout.queue
> ....
> FAIL
> timeout
> Request timed out while waiting for response(s).
>
>
> Camel Body (stripped off top and bottom elements)
> [cid:image002.png@01CFD7FC.D4F23FD0]
>
> 2193-49984781-keymanager-1411163761368-126275
> orch.encryptionkey.encryptionkeyTimeout.queue
> ...
> FAIL
> timeout
> Request timed out while waiting for response(s).
>
> Route:
> from(JMS_encryptionkeyTimeout).routeId(JMS_encryptionkeyTimeout)
> .log(LoggingLevel.INFO, "Timed out waiting for encryptionkey response")
> .choice()
> .when(isInBodyNullOrEmpty)
> .to(SEDA_encryptionkeyOrchDead)
> .otherwise()
> .to(DIRECT_processBackToClient)
> .end();
>
> // Called from several places - when a response needs to be sent back to the client
> from(DIRECT_processBackToClient).routeId(DIRECT_processBackToClient)
> .setHeader(HDR_JMS_CORRELATION_ID).xpath("//*[local-name()='transactionId']",
> String.class)
> .to(XSLT_toAPIResponsePreprocessing)
> .choice()
> // New response type may be added later
> .when().xpath("not(/*[local-name()='responseQueue'])")
> .log(LoggingLevel.ERROR, "Unable to format encryptionkey response")
> .to(SEDA_encryptionkeyOrchDead)
> .otherwise()
> .recipientList().simple(JMS_RESPONSE_QUEUE)
> .end();
>
> XSLT that is probably at fault:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
Re: XSLT Thread Safety Issue
Posted by Willem Jiang <wi...@gmail.com>.
Can I have a look at you XSLT endpoint uri?
Did you use the options of transformerCacheSize or contentCache?
--
Willem Jiang
Red Hat, Inc.
Web: http://www.redhat.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.iteye.com (Chinese)
Twitter: willemjiang
Weibo: 姜宁willem
On October 14, 2014 at 9:45:39 AM, Hendley, Sam (sam.hendley@sensus.com) wrote:
> We are having a very strange (and intermittent) issue with some of our camel routes.
>
> We are in the process of upgrading to a more up-to-date version but are still on camel 2.10.7
> and running on java 1.6. I have looked through the forums and tickets but haven't found
> anything that looks like it could cause this issue.
>
> What appears to be occurring is this:
>
> * An XML document is received from a JMS queue (as a text message)
>
> o The xml is correctly formatted and is what we expect to see
>
> * It is then routed through an XSLT transformation (to strip off excess namespacing)
>
> * 99% of the time this works, but very occasionally we get Xpath errors when we go onto the
> next step
>
> * We were able to do a heap dump of a system that was having this problem, when we dug through
> the dump we were able to analyze an instance of the failure (same exchangeId)
>
> o It turns out that when the issue occurs the enclosing tags (document root) have disappeared
> rendering the document un-xpathable
>
> Since this works nearly all of the time and there is nothing special about the failing
> documents we suspect it could be a threading or race condition in the XSLT processing.
> Are there any known issues that might be related? This route is running in a relatively
> large and busy camel application that makes heavy use of XSLT transforms. We believe
> similar issues have occurred in other instances but haven't debugged to this level so
> that is just a hunch.
>
> I have included some details below, let me know if there is more I should include. I still
> have the heap/thread dump if there is anything else that might be worth digging into.
>
> Thanks for any help or advise,
>
> Sam Hendley
>
> Jms Timeout message (notice it has xml header and wrapping of EncryptionKeyFlow)
> [cid:image001.png@01CFD7FC.D4F23FD0]
>
>
>
> 2193-49984781-keymanager-1411163761368-126275
> orch.encryptionkey.encryptionkeyTimeout.queue
> ....
> FAIL
> timeout
> Request timed out while waiting for response(s).
>
>
> Camel Body (stripped off top and bottom elements)
> [cid:image002.png@01CFD7FC.D4F23FD0]
>
> 2193-49984781-keymanager-1411163761368-126275
> orch.encryptionkey.encryptionkeyTimeout.queue
> ...
> FAIL
> timeout
> Request timed out while waiting for response(s).
>
> Route:
> from(JMS_encryptionkeyTimeout).routeId(JMS_encryptionkeyTimeout)
> .log(LoggingLevel.INFO, "Timed out waiting for encryptionkey response")
> .choice()
> .when(isInBodyNullOrEmpty)
> .to(SEDA_encryptionkeyOrchDead)
> .otherwise()
> .to(DIRECT_processBackToClient)
> .end();
>
> // Called from several places - when a response needs to be sent back to the client
> from(DIRECT_processBackToClient).routeId(DIRECT_processBackToClient)
> .setHeader(HDR_JMS_CORRELATION_ID).xpath("//*[local-name()='transactionId']",
> String.class)
> .to(XSLT_toAPIResponsePreprocessing)
> .choice()
> // New response type may be added later
> .when().xpath("not(/*[local-name()='responseQueue'])")
> .log(LoggingLevel.ERROR, "Unable to format encryptionkey response")
> .to(SEDA_encryptionkeyOrchDead)
> .otherwise()
> .recipientList().simple(JMS_RESPONSE_QUEUE)
> .end();
>
> XSLT that is probably at fault:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>