You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cxf.apache.org by Alessio Soldano <as...@redhat.com> on 2011/06/01 10:52:24 UTC
WS-Security concurrency issue with CXF 2.2.12 + WSS4J 1.5.10
Hi,
we're running some WS-Security load tests with Apache CXF 2.2.12 + WSS4J
1.5.10 and might have found a concurrency issue.
We're getting the following exception:
[runTest(bash)] OUT> Caused by: java.util.ConcurrentModificationException
[runTest(bash)] OUT> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
[runTest(bash)] OUT> at java.util.AbstractList$Itr.next(AbstractList.java:343)
[runTest(bash)] OUT> at org.apache.ws.security.WSDocInfo.getSecurityTokenReference(WSDocInfo.java:86)
[runTest(bash)] OUT> at org.apache.ws.security.message.EnvelopeIdResolver.engineResolve(EnvelopeIdResolver.java:114)
[runTest(bash)] OUT> at org.apache.xml.security.utils.resolver.ResourceResolver.resolve(Unknown Source)
[runTest(bash)] OUT> at org.apache.xml.security.signature.Reference.getContentsBeforeTransformation(Unknown Source)
[runTest(bash)] OUT> at org.apache.xml.security.signature.Reference.dereferenceURIandPerformTransforms(Unknown Source)
[runTest(bash)] OUT> at org.apache.xml.security.signature.Reference.calculateDigest(Unknown Source)
[runTest(bash)] OUT> at org.apache.xml.security.signature.Reference.verify(Unknown Source)
[runTest(bash)] OUT> at org.apache.xml.security.signature.Manifest.verifyReferences(Unknown Source)
[runTest(bash)] OUT> at org.apache.xml.security.signature.SignedInfo.verify(Unknown Source)
[runTest(bash)] OUT> at org.apache.xml.security.signature.XMLSignature.checkSignatureValue(Unknown Source)
[runTest(bash)] OUT> at org.apache.xml.security.signature.XMLSignature.checkSignatureValue(Unknown Source)
[runTest(bash)] OUT> at org.apache.ws.security.processor.SignatureProcessor.verifyXMLSignature(SignatureProcessor.java:470)
[runTest(bash)] OUT> at org.apache.ws.security.processor.SignatureProcessor.handleToken(SignatureProcessor.java:114)
[runTest(bash)] OUT> at org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:328)
[runTest(bash)] OUT> at org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:245)
[runTest(bash)] OUT> at org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:208)
[runTest(bash)] OUT> at org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:78)
[runTest(bash)] OUT> at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:243)
[runTest(bash)] OUT> at org.apache.cxf.endpoint.ClientImpl.onMessage(ClientImpl.java:755)
[runTest(bash)] OUT> at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:2408)
[runTest(bash)] OUT> at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:2278)
[runTest(bash)] OUT> at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:2121)
[runTest(bash)] OUT> at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:66)
[runTest(bash)] OUT> at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:695)
[runTest(bash)] OUT> at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
[runTest(bash)] OUT> at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:243)
[runTest(bash)] OUT> at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:516)
[runTest(bash)] OUT> at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:313)
[runTest(bash)] OUT> at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:265)
[runTest(bash)] OUT> at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:73)
[runTest(bash)] OUT> at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:124)
[runTest(bash)] OUT> ... 9 more
Interestingly, forcing the use of WSS4J 1.5.8 instead (we discovered
this by accident ;-) ) the problem goes away.
The application basically builds up a JAXWS client using WS-Security
(Sign only) through programmatically configured WSS4J interceptors:
http://fpaste.org/pYHM/
Once the proxy is ready on client side, it's called concurrently using
multiple threads (performIteration() method in the code).
Does the problem/exception above ring any bell? I'm going to investigate
this a bit more in any case.
Thanks
Alessio
--
Alessio Soldano
Web Service Lead, JBoss
Re: WS-Security concurrency issue with CXF 2.2.12 + WSS4J 1.5.10
Posted by Alessio Soldano <as...@redhat.com>.
Hi Dan,
On 06/01/2011 05:01 PM, Daniel Kulp wrote:
> Is this something that you can create a small maven testcase for to
> demonstrate? I'm not really sure what would be going on. :-(
currently this is reproduced using an application that hits a ws
endpoint on a candidate for the next JBoss Enterprise Application
Platform release, however this seems to be a problem reproducible on
client side regardless of what is serving the endpoint on server side. I
can probably try that and then clean the load app from any Red Hat
product stuff so that I can send it to you, I need a bit of time
unfortunately.
Cheers
Alessio
--
Alessio Soldano
Web Service Lead, JBoss
Re: WS-Security concurrency issue with CXF 2.2.12 + WSS4J 1.5.10
Posted by Daniel Kulp <dk...@apache.org>.
On Wednesday, June 01, 2011 4:52:24 AM Alessio Soldano wrote:
> Hi,
> we're running some WS-Security load tests with Apache CXF 2.2.12 + WSS4J
> 1.5.10 and might have found a concurrency issue.
I know this was an issue with 1.5.6 or so that I fixed (or thought I fixed) a
long time ago. It was way more apparent with 64bit JDK's than 32bit as well
due to the way WSS4J was hashing doc objects and such. However, I really
thought this was fixed.
Is this something that you can create a small maven testcase for to
demonstrate? I'm not really sure what would be going on. :-(
Dan
> We're getting the following exception:
>
> [runTest(bash)] OUT> Caused by: java.util.ConcurrentModificationException
> [runTest(bash)] OUT> at
> java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> [runTest(bash)] OUT> at
> java.util.AbstractList$Itr.next(AbstractList.java:343) [runTest(bash)]
> OUT> at
> org.apache.ws.security.WSDocInfo.getSecurityTokenReference(WSDocInfo.java:
> 86) [runTest(bash)] OUT> at
> org.apache.ws.security.message.EnvelopeIdResolver.engineResolve(EnvelopeId
> Resolver.java:114) [runTest(bash)] OUT> at
> org.apache.xml.security.utils.resolver.ResourceResolver.resolve(Unknown
> Source) [runTest(bash)] OUT> at
> org.apache.xml.security.signature.Reference.getContentsBeforeTransformatio
> n(Unknown Source) [runTest(bash)] OUT> at
> org.apache.xml.security.signature.Reference.dereferenceURIandPerformTransf
> orms(Unknown Source) [runTest(bash)] OUT> at
> org.apache.xml.security.signature.Reference.calculateDigest(Unknown
> Source) [runTest(bash)] OUT> at
> org.apache.xml.security.signature.Reference.verify(Unknown Source)
> [runTest(bash)] OUT> at
> org.apache.xml.security.signature.Manifest.verifyReferences(Unknown
> Source) [runTest(bash)] OUT> at
> org.apache.xml.security.signature.SignedInfo.verify(Unknown Source)
> [runTest(bash)] OUT> at
> org.apache.xml.security.signature.XMLSignature.checkSignatureValue(Unknown
> Source) [runTest(bash)] OUT> at
> org.apache.xml.security.signature.XMLSignature.checkSignatureValue(Unknown
> Source) [runTest(bash)] OUT> at
> org.apache.ws.security.processor.SignatureProcessor.verifyXMLSignature(Sig
> natureProcessor.java:470) [runTest(bash)] OUT> at
> org.apache.ws.security.processor.SignatureProcessor.handleToken(SignatureP
> rocessor.java:114) [runTest(bash)] OUT> at
> org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEn
> gine.java:328) [runTest(bash)] OUT> at
> org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEn
> gine.java:245) [runTest(bash)] OUT> at
> org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInI
> nterceptor.java:208) [runTest(bash)] OUT> at
> org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInI
> nterceptor.java:78) [runTest(bash)] OUT> at
> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorCha
> in.java:243) [runTest(bash)] OUT> at
> org.apache.cxf.endpoint.ClientImpl.onMessage(ClientImpl.java:755)
> [runTest(bash)] OUT> at
> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleRespon
> seInternal(HTTPConduit.java:2408) [runTest(bash)] OUT> at
> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleRespon
> se(HTTPConduit.java:2278) [runTest(bash)] OUT> at
> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPCo
> nduit.java:2121) [runTest(bash)] OUT> at
> org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:66)
> [runTest(bash)] OUT> at
> org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:695)
> [runTest(bash)] OUT> at
> org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInt
> erceptor.handleMessage(MessageSenderInterceptor.java:62) [runTest(bash)]
> OUT> at
> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorCha
> in.java:243) [runTest(bash)] OUT> at
> org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:516)
> [runTest(bash)] OUT> at
> org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:313)
> [runTest(bash)] OUT> at
> org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:265)
> [runTest(bash)] OUT> at
> org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:73)
> [runTest(bash)] OUT> at
> org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:124)
> [runTest(bash)] OUT> ... 9 more
>
>
> Interestingly, forcing the use of WSS4J 1.5.8 instead (we discovered
> this by accident ;-) ) the problem goes away.
> The application basically builds up a JAXWS client using WS-Security
> (Sign only) through programmatically configured WSS4J interceptors:
> http://fpaste.org/pYHM/
> Once the proxy is ready on client side, it's called concurrently using
> multiple threads (performIteration() method in the code).
> Does the problem/exception above ring any bell? I'm going to investigate
> this a bit more in any case.
> Thanks
> Alessio
--
Daniel Kulp
dkulp@apache.org
http://dankulp.com/blog
Talend - http://www.talend.com
Re: WS-Security concurrency issue with CXF 2.2.12 + WSS4J 1.5.10
Posted by Colm O hEigeartaigh <co...@apache.org>.
Cool. Any chance of contributing the test to CXF in some form? It
would be useful to make sure that these kinds of issues don't creep
back in at any stage.
Colm.
On Thu, Jun 2, 2011 at 10:39 AM, Alessio Soldano <as...@redhat.com> wrote:
> On 06/01/2011 07:57 PM, Alessio Soldano wrote:
>>
>> On 06/01/2011 05:57 PM, Daniel Kulp wrote:
>>>
>>> https://issues.apache.org/jira/browse/WSS-292
>>>
>>> and will be committing a fix soon. It would be great if you could test a
>>> snapshot with it.
>>
>> Sure, that's not a problem at all, thanks!
>>
> OK, I've tried a snapshot built from rev. 1130199 (the one of your fix) and
> I can't reproduce the concurrency issue anymore. Cool, thanks.
> Alessio
>
> --
> Alessio Soldano
> Web Service Lead, JBoss
>
>
--
Colm O hEigeartaigh
http://coheigea.blogspot.com/
Talend - http://www.talend.com
Re: WS-Security concurrency issue with CXF 2.2.12 + WSS4J 1.5.10
Posted by Alessio Soldano <as...@redhat.com>.
On 06/01/2011 07:57 PM, Alessio Soldano wrote:
> On 06/01/2011 05:57 PM, Daniel Kulp wrote:
>> https://issues.apache.org/jira/browse/WSS-292
>>
>> and will be committing a fix soon. It would be great if you could
>> test a
>> snapshot with it.
>
> Sure, that's not a problem at all, thanks!
>
OK, I've tried a snapshot built from rev. 1130199 (the one of your fix)
and I can't reproduce the concurrency issue anymore. Cool, thanks.
Alessio
--
Alessio Soldano
Web Service Lead, JBoss
Re: WS-Security concurrency issue with CXF 2.2.12 + WSS4J 1.5.10
Posted by Alessio Soldano <as...@redhat.com>.
On 06/01/2011 05:57 PM, Daniel Kulp wrote:
> No, it's slightly different. Up until that commit, the EnvelopIdResolver was
> pretty much just used as a singleton. It didn't have any state and thus a
> singleton was created and reused. With that commit, it was given state, but
> all the places that used it as a singleton were not updated. I've created a
> JIRA:
> https://issues.apache.org/jira/browse/WSS-292
>
> and will be committing a fix soon. It would be great if you could test a
> snapshot with it.
Sure, that's not a problem at all, thanks!
Alessio
--
Alessio Soldano
Web Service Lead, JBoss
Re: WS-Security concurrency issue with CXF 2.2.12 + WSS4J 1.5.10
Posted by Daniel Kulp <dk...@apache.org>.
On Wednesday, June 01, 2011 11:29:22 AM Alessio Soldano wrote:
> OK, here I am with some data.
> I've been able to reproduce the problem locally (had to tune the load
> test application to run on my laptop and still reproduce the problem
> ;-)) and it seems CXF 2.4.1-SNAPSHOT + WSS4J 1.6 is not affected.
> So I had a look at the commits that went in to the 1.5.x fix branch of
> WSS4J and I think I've found the commit that introduces the problem,
> http://svn.apache.org/viewvc?view=revision&revision=947604 . I've not
> enough knowledge of the project (at least atm) to really understand the
> reason, however the binary obtained by building the 947603 revision is
> not showing the concurrency issue below, while 947604 reproduces the
> problem (both versions tested with CXF 2.2.12).
> At first sight the only way I see the WSDocInfo could be used
> concurrently is because of some issues with getting it from the
> WSDocInfoStore... but again, I'm not an expert at all here. Dan, could
> this perhaps be related to the hashing issue you mentioned in the other
> reply? I'm indeed running a 64bit JDK.
No, it's slightly different. Up until that commit, the EnvelopIdResolver was
pretty much just used as a singleton. It didn't have any state and thus a
singleton was created and reused. With that commit, it was given state, but
all the places that used it as a singleton were not updated. I've created a
JIRA:
https://issues.apache.org/jira/browse/WSS-292
and will be committing a fix soon. It would be great if you could test a
snapshot with it.
Dan
>
> Thanks
> Alessio
>
> On 06/01/2011 11:47 AM, Alessio Soldano wrote:
> > Hi Colm,
> > yes, I'll try reproducing this locally (the load test is actually run
> > on a bench environment that can't be moved quickly to cxf 2.4.0 /
> > wss4j 1.6) and let you know asap.
> > Thanks
> > Alessio
> >
> > On 06/01/2011 11:26 AM, Colm O hEigeartaigh wrote:
> >> Hi Alessio,
> >>
> >> Very interesting, I haven't seen this problem before. It looks like a
> >> problem in WSS4J - I'm not sure how multiple threads are altering the
> >> same WSDocInfo object.
> >>
> >> Would it be possible to check whether it exists with CXF 2.4.0 and
> >> WSS4J 1.6.0? I want to get a WSS4J 1.6.1 release out asap, so it would
> >> be cool to fix this problem if it's there.
> >>
> >> Colm.
> >>
> >> On Wed, Jun 1, 2011 at 9:52 AM, Alessio Soldano<as...@redhat.com>
> >>
> >> wrote:
> >>> Hi,
> >>> we're running some WS-Security load tests with Apache CXF 2.2.12 +
> >>> WSS4J
> >>> 1.5.10 and might have found a concurrency issue.
> >>> We're getting the following exception:
> >>>
> >>> [runTest(bash)] OUT> Caused by:
> >>> java.util.ConcurrentModificationException
> >>> [runTest(bash)] OUT> at
> >>> java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372
> >>> )
> >>>
> >>> [runTest(bash)] OUT> at
> >>> java.util.AbstractList$Itr.next(AbstractList.java:343)
> >>> [runTest(bash)] OUT> at
> >>> org.apache.ws.security.WSDocInfo.getSecurityTokenReference(WSDocInfo.ja
> >>> va:86)
> >>>
> >>> [runTest(bash)] OUT> at
> >>> org.apache.ws.security.message.EnvelopeIdResolver.engineResolve(Envelop
> >>> eIdResolver.java:114)
> >>>
> >>> [runTest(bash)] OUT> at
> >>> org.apache.xml.security.utils.resolver.ResourceResolver.resolve(Unknown
> >>> Source)
> >>> [runTest(bash)] OUT> at
> >>> org.apache.xml.security.signature.Reference.getContentsBeforeTransforma
> >>> tion(Unknown
> >>>
> >>> Source)
> >>> [runTest(bash)] OUT> at
> >>> org.apache.xml.security.signature.Reference.dereferenceURIandPerformTra
> >>> nsforms(Unknown
> >>>
> >>> Source)
> >>> [runTest(bash)] OUT> at
> >>> org.apache.xml.security.signature.Reference.calculateDigest(Unknown
> >>> Source)
> >>> [runTest(bash)] OUT> at
> >>> org.apache.xml.security.signature.Reference.verify(Unknown Source)
> >>> [runTest(bash)] OUT> at
> >>> org.apache.xml.security.signature.Manifest.verifyReferences(Unknown
> >>> Source)
> >>> [runTest(bash)] OUT> at
> >>> org.apache.xml.security.signature.SignedInfo.verify(Unknown Source)
> >>> [runTest(bash)] OUT> at
> >>> org.apache.xml.security.signature.XMLSignature.checkSignatureValue(Unkn
> >>> own
> >>>
> >>> Source)
> >>> [runTest(bash)] OUT> at
> >>> org.apache.xml.security.signature.XMLSignature.checkSignatureValue(Unkn
> >>> own
> >>>
> >>> Source)
> >>> [runTest(bash)] OUT> at
> >>> org.apache.ws.security.processor.SignatureProcessor.verifyXMLSignature(
> >>> SignatureProcessor.java:470)
> >>>
> >>> [runTest(bash)] OUT> at
> >>> org.apache.ws.security.processor.SignatureProcessor.handleToken(Signatu
> >>> reProcessor.java:114)
> >>>
> >>> [runTest(bash)] OUT> at
> >>> org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurit
> >>> yEngine.java:328)
> >>>
> >>> [runTest(bash)] OUT> at
> >>> org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurit
> >>> yEngine.java:245)
> >>>
> >>> [runTest(bash)] OUT> at
> >>> org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4J
> >>> InInterceptor.java:208)
> >>>
> >>> [runTest(bash)] OUT> at
> >>> org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4J
> >>> InInterceptor.java:78)
> >>>
> >>> [runTest(bash)] OUT> at
> >>> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptor
> >>> Chain.java:243)
> >>>
> >>> [runTest(bash)] OUT> at
> >>> org.apache.cxf.endpoint.ClientImpl.onMessage(ClientImpl.java:755)
> >>> [runTest(bash)] OUT> at
> >>> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleRes
> >>> ponseInternal(HTTPConduit.java:2408)
> >>>
> >>> [runTest(bash)] OUT> at
> >>> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleRes
> >>> ponse(HTTPConduit.java:2278)
> >>>
> >>> [runTest(bash)] OUT> at
> >>> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTT
> >>> PConduit.java:2121)
> >>>
> >>> [runTest(bash)] OUT> at
> >>> org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:66)
> >>> [runTest(bash)] OUT> at
> >>> org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:695)
> >>> [runTest(bash)] OUT> at
> >>> org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEnding
> >>> Interceptor.handleMessage(MessageSenderInterceptor.java:62)
> >>>
> >>> [runTest(bash)] OUT> at
> >>> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptor
> >>> Chain.java:243)
> >>>
> >>> [runTest(bash)] OUT> at
> >>> org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:516)
> >>> [runTest(bash)] OUT> at
> >>> org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:313)
> >>> [runTest(bash)] OUT> at
> >>> org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:265)
> >>> [runTest(bash)] OUT> at
> >>> org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:73)
> >>> [runTest(bash)] OUT> at
> >>> org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:124)
> >>> [runTest(bash)] OUT> ... 9 more
> >>>
> >>>
> >>> Interestingly, forcing the use of WSS4J 1.5.8 instead (we discovered
> >>> this by
> >>> accident ;-) ) the problem goes away.
> >>> The application basically builds up a JAXWS client using WS-Security
> >>> (Sign
> >>> only) through programmatically configured WSS4J interceptors:
> >>> http://fpaste.org/pYHM/
> >>> Once the proxy is ready on client side, it's called concurrently using
> >>> multiple threads (performIteration() method in the code).
> >>> Does the problem/exception above ring any bell? I'm going to
> >>> investigate
> >>> this a bit more in any case.
> >>> Thanks
> >>> Alessio
--
Daniel Kulp
dkulp@apache.org
http://dankulp.com/blog
Talend - http://www.talend.com
Re: WS-Security concurrency issue with CXF 2.2.12 + WSS4J 1.5.10
Posted by Alessio Soldano <as...@redhat.com>.
OK, here I am with some data.
I've been able to reproduce the problem locally (had to tune the load
test application to run on my laptop and still reproduce the problem
;-)) and it seems CXF 2.4.1-SNAPSHOT + WSS4J 1.6 is not affected.
So I had a look at the commits that went in to the 1.5.x fix branch of
WSS4J and I think I've found the commit that introduces the problem,
http://svn.apache.org/viewvc?view=revision&revision=947604 . I've not
enough knowledge of the project (at least atm) to really understand the
reason, however the binary obtained by building the 947603 revision is
not showing the concurrency issue below, while 947604 reproduces the
problem (both versions tested with CXF 2.2.12).
At first sight the only way I see the WSDocInfo could be used
concurrently is because of some issues with getting it from the
WSDocInfoStore... but again, I'm not an expert at all here. Dan, could
this perhaps be related to the hashing issue you mentioned in the other
reply? I'm indeed running a 64bit JDK.
Thanks
Alessio
On 06/01/2011 11:47 AM, Alessio Soldano wrote:
> Hi Colm,
> yes, I'll try reproducing this locally (the load test is actually run
> on a bench environment that can't be moved quickly to cxf 2.4.0 /
> wss4j 1.6) and let you know asap.
> Thanks
> Alessio
>
> On 06/01/2011 11:26 AM, Colm O hEigeartaigh wrote:
>> Hi Alessio,
>>
>> Very interesting, I haven't seen this problem before. It looks like a
>> problem in WSS4J - I'm not sure how multiple threads are altering the
>> same WSDocInfo object.
>>
>> Would it be possible to check whether it exists with CXF 2.4.0 and
>> WSS4J 1.6.0? I want to get a WSS4J 1.6.1 release out asap, so it would
>> be cool to fix this problem if it's there.
>>
>> Colm.
>>
>> On Wed, Jun 1, 2011 at 9:52 AM, Alessio Soldano<as...@redhat.com>
>> wrote:
>>> Hi,
>>> we're running some WS-Security load tests with Apache CXF 2.2.12 +
>>> WSS4J
>>> 1.5.10 and might have found a concurrency issue.
>>> We're getting the following exception:
>>>
>>> [runTest(bash)] OUT> Caused by:
>>> java.util.ConcurrentModificationException
>>> [runTest(bash)] OUT> at
>>> java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
>>>
>>> [runTest(bash)] OUT> at
>>> java.util.AbstractList$Itr.next(AbstractList.java:343)
>>> [runTest(bash)] OUT> at
>>> org.apache.ws.security.WSDocInfo.getSecurityTokenReference(WSDocInfo.java:86)
>>>
>>> [runTest(bash)] OUT> at
>>> org.apache.ws.security.message.EnvelopeIdResolver.engineResolve(EnvelopeIdResolver.java:114)
>>>
>>> [runTest(bash)] OUT> at
>>> org.apache.xml.security.utils.resolver.ResourceResolver.resolve(Unknown
>>> Source)
>>> [runTest(bash)] OUT> at
>>> org.apache.xml.security.signature.Reference.getContentsBeforeTransformation(Unknown
>>>
>>> Source)
>>> [runTest(bash)] OUT> at
>>> org.apache.xml.security.signature.Reference.dereferenceURIandPerformTransforms(Unknown
>>>
>>> Source)
>>> [runTest(bash)] OUT> at
>>> org.apache.xml.security.signature.Reference.calculateDigest(Unknown
>>> Source)
>>> [runTest(bash)] OUT> at
>>> org.apache.xml.security.signature.Reference.verify(Unknown Source)
>>> [runTest(bash)] OUT> at
>>> org.apache.xml.security.signature.Manifest.verifyReferences(Unknown
>>> Source)
>>> [runTest(bash)] OUT> at
>>> org.apache.xml.security.signature.SignedInfo.verify(Unknown Source)
>>> [runTest(bash)] OUT> at
>>> org.apache.xml.security.signature.XMLSignature.checkSignatureValue(Unknown
>>>
>>> Source)
>>> [runTest(bash)] OUT> at
>>> org.apache.xml.security.signature.XMLSignature.checkSignatureValue(Unknown
>>>
>>> Source)
>>> [runTest(bash)] OUT> at
>>> org.apache.ws.security.processor.SignatureProcessor.verifyXMLSignature(SignatureProcessor.java:470)
>>>
>>> [runTest(bash)] OUT> at
>>> org.apache.ws.security.processor.SignatureProcessor.handleToken(SignatureProcessor.java:114)
>>>
>>> [runTest(bash)] OUT> at
>>> org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:328)
>>>
>>> [runTest(bash)] OUT> at
>>> org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:245)
>>>
>>> [runTest(bash)] OUT> at
>>> org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:208)
>>>
>>> [runTest(bash)] OUT> at
>>> org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:78)
>>>
>>> [runTest(bash)] OUT> at
>>> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:243)
>>>
>>> [runTest(bash)] OUT> at
>>> org.apache.cxf.endpoint.ClientImpl.onMessage(ClientImpl.java:755)
>>> [runTest(bash)] OUT> at
>>> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:2408)
>>>
>>> [runTest(bash)] OUT> at
>>> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:2278)
>>>
>>> [runTest(bash)] OUT> at
>>> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:2121)
>>>
>>> [runTest(bash)] OUT> at
>>> org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:66)
>>> [runTest(bash)] OUT> at
>>> org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:695)
>>> [runTest(bash)] OUT> at
>>> org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
>>>
>>> [runTest(bash)] OUT> at
>>> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:243)
>>>
>>> [runTest(bash)] OUT> at
>>> org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:516)
>>> [runTest(bash)] OUT> at
>>> org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:313)
>>> [runTest(bash)] OUT> at
>>> org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:265)
>>> [runTest(bash)] OUT> at
>>> org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:73)
>>> [runTest(bash)] OUT> at
>>> org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:124)
>>> [runTest(bash)] OUT> ... 9 more
>>>
>>>
>>> Interestingly, forcing the use of WSS4J 1.5.8 instead (we discovered
>>> this by
>>> accident ;-) ) the problem goes away.
>>> The application basically builds up a JAXWS client using WS-Security
>>> (Sign
>>> only) through programmatically configured WSS4J interceptors:
>>> http://fpaste.org/pYHM/
>>> Once the proxy is ready on client side, it's called concurrently using
>>> multiple threads (performIteration() method in the code).
>>> Does the problem/exception above ring any bell? I'm going to
>>> investigate
>>> this a bit more in any case.
>>> Thanks
>>> Alessio
>>>
>>> --
>>> Alessio Soldano
>>> Web Service Lead, JBoss
>>>
>>>
>>
>>
>
>
--
Alessio Soldano
Web Service Lead, JBoss
Re: WS-Security concurrency issue with CXF 2.2.12 + WSS4J 1.5.10
Posted by Alessio Soldano <as...@redhat.com>.
Hi Colm,
yes, I'll try reproducing this locally (the load test is actually run on
a bench environment that can't be moved quickly to cxf 2.4.0 / wss4j
1.6) and let you know asap.
Thanks
Alessio
On 06/01/2011 11:26 AM, Colm O hEigeartaigh wrote:
> Hi Alessio,
>
> Very interesting, I haven't seen this problem before. It looks like a
> problem in WSS4J - I'm not sure how multiple threads are altering the
> same WSDocInfo object.
>
> Would it be possible to check whether it exists with CXF 2.4.0 and
> WSS4J 1.6.0? I want to get a WSS4J 1.6.1 release out asap, so it would
> be cool to fix this problem if it's there.
>
> Colm.
>
> On Wed, Jun 1, 2011 at 9:52 AM, Alessio Soldano<as...@redhat.com> wrote:
>> Hi,
>> we're running some WS-Security load tests with Apache CXF 2.2.12 + WSS4J
>> 1.5.10 and might have found a concurrency issue.
>> We're getting the following exception:
>>
>> [runTest(bash)] OUT> Caused by: java.util.ConcurrentModificationException
>> [runTest(bash)] OUT> at
>> java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
>> [runTest(bash)] OUT> at
>> java.util.AbstractList$Itr.next(AbstractList.java:343)
>> [runTest(bash)] OUT> at
>> org.apache.ws.security.WSDocInfo.getSecurityTokenReference(WSDocInfo.java:86)
>> [runTest(bash)] OUT> at
>> org.apache.ws.security.message.EnvelopeIdResolver.engineResolve(EnvelopeIdResolver.java:114)
>> [runTest(bash)] OUT> at
>> org.apache.xml.security.utils.resolver.ResourceResolver.resolve(Unknown
>> Source)
>> [runTest(bash)] OUT> at
>> org.apache.xml.security.signature.Reference.getContentsBeforeTransformation(Unknown
>> Source)
>> [runTest(bash)] OUT> at
>> org.apache.xml.security.signature.Reference.dereferenceURIandPerformTransforms(Unknown
>> Source)
>> [runTest(bash)] OUT> at
>> org.apache.xml.security.signature.Reference.calculateDigest(Unknown Source)
>> [runTest(bash)] OUT> at
>> org.apache.xml.security.signature.Reference.verify(Unknown Source)
>> [runTest(bash)] OUT> at
>> org.apache.xml.security.signature.Manifest.verifyReferences(Unknown Source)
>> [runTest(bash)] OUT> at
>> org.apache.xml.security.signature.SignedInfo.verify(Unknown Source)
>> [runTest(bash)] OUT> at
>> org.apache.xml.security.signature.XMLSignature.checkSignatureValue(Unknown
>> Source)
>> [runTest(bash)] OUT> at
>> org.apache.xml.security.signature.XMLSignature.checkSignatureValue(Unknown
>> Source)
>> [runTest(bash)] OUT> at
>> org.apache.ws.security.processor.SignatureProcessor.verifyXMLSignature(SignatureProcessor.java:470)
>> [runTest(bash)] OUT> at
>> org.apache.ws.security.processor.SignatureProcessor.handleToken(SignatureProcessor.java:114)
>> [runTest(bash)] OUT> at
>> org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:328)
>> [runTest(bash)] OUT> at
>> org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:245)
>> [runTest(bash)] OUT> at
>> org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:208)
>> [runTest(bash)] OUT> at
>> org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:78)
>> [runTest(bash)] OUT> at
>> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:243)
>> [runTest(bash)] OUT> at
>> org.apache.cxf.endpoint.ClientImpl.onMessage(ClientImpl.java:755)
>> [runTest(bash)] OUT> at
>> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:2408)
>> [runTest(bash)] OUT> at
>> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:2278)
>> [runTest(bash)] OUT> at
>> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:2121)
>> [runTest(bash)] OUT> at
>> org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:66)
>> [runTest(bash)] OUT> at
>> org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:695)
>> [runTest(bash)] OUT> at
>> org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
>> [runTest(bash)] OUT> at
>> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:243)
>> [runTest(bash)] OUT> at
>> org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:516)
>> [runTest(bash)] OUT> at
>> org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:313)
>> [runTest(bash)] OUT> at
>> org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:265)
>> [runTest(bash)] OUT> at
>> org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:73)
>> [runTest(bash)] OUT> at
>> org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:124)
>> [runTest(bash)] OUT> ... 9 more
>>
>>
>> Interestingly, forcing the use of WSS4J 1.5.8 instead (we discovered this by
>> accident ;-) ) the problem goes away.
>> The application basically builds up a JAXWS client using WS-Security (Sign
>> only) through programmatically configured WSS4J interceptors:
>> http://fpaste.org/pYHM/
>> Once the proxy is ready on client side, it's called concurrently using
>> multiple threads (performIteration() method in the code).
>> Does the problem/exception above ring any bell? I'm going to investigate
>> this a bit more in any case.
>> Thanks
>> Alessio
>>
>> --
>> Alessio Soldano
>> Web Service Lead, JBoss
>>
>>
>
>
--
Alessio Soldano
Web Service Lead, JBoss
Re: WS-Security concurrency issue with CXF 2.2.12 + WSS4J 1.5.10
Posted by Colm O hEigeartaigh <co...@apache.org>.
Hi Alessio,
Very interesting, I haven't seen this problem before. It looks like a
problem in WSS4J - I'm not sure how multiple threads are altering the
same WSDocInfo object.
Would it be possible to check whether it exists with CXF 2.4.0 and
WSS4J 1.6.0? I want to get a WSS4J 1.6.1 release out asap, so it would
be cool to fix this problem if it's there.
Colm.
On Wed, Jun 1, 2011 at 9:52 AM, Alessio Soldano <as...@redhat.com> wrote:
> Hi,
> we're running some WS-Security load tests with Apache CXF 2.2.12 + WSS4J
> 1.5.10 and might have found a concurrency issue.
> We're getting the following exception:
>
> [runTest(bash)] OUT> Caused by: java.util.ConcurrentModificationException
> [runTest(bash)] OUT> at
> java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> [runTest(bash)] OUT> at
> java.util.AbstractList$Itr.next(AbstractList.java:343)
> [runTest(bash)] OUT> at
> org.apache.ws.security.WSDocInfo.getSecurityTokenReference(WSDocInfo.java:86)
> [runTest(bash)] OUT> at
> org.apache.ws.security.message.EnvelopeIdResolver.engineResolve(EnvelopeIdResolver.java:114)
> [runTest(bash)] OUT> at
> org.apache.xml.security.utils.resolver.ResourceResolver.resolve(Unknown
> Source)
> [runTest(bash)] OUT> at
> org.apache.xml.security.signature.Reference.getContentsBeforeTransformation(Unknown
> Source)
> [runTest(bash)] OUT> at
> org.apache.xml.security.signature.Reference.dereferenceURIandPerformTransforms(Unknown
> Source)
> [runTest(bash)] OUT> at
> org.apache.xml.security.signature.Reference.calculateDigest(Unknown Source)
> [runTest(bash)] OUT> at
> org.apache.xml.security.signature.Reference.verify(Unknown Source)
> [runTest(bash)] OUT> at
> org.apache.xml.security.signature.Manifest.verifyReferences(Unknown Source)
> [runTest(bash)] OUT> at
> org.apache.xml.security.signature.SignedInfo.verify(Unknown Source)
> [runTest(bash)] OUT> at
> org.apache.xml.security.signature.XMLSignature.checkSignatureValue(Unknown
> Source)
> [runTest(bash)] OUT> at
> org.apache.xml.security.signature.XMLSignature.checkSignatureValue(Unknown
> Source)
> [runTest(bash)] OUT> at
> org.apache.ws.security.processor.SignatureProcessor.verifyXMLSignature(SignatureProcessor.java:470)
> [runTest(bash)] OUT> at
> org.apache.ws.security.processor.SignatureProcessor.handleToken(SignatureProcessor.java:114)
> [runTest(bash)] OUT> at
> org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:328)
> [runTest(bash)] OUT> at
> org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:245)
> [runTest(bash)] OUT> at
> org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:208)
> [runTest(bash)] OUT> at
> org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:78)
> [runTest(bash)] OUT> at
> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:243)
> [runTest(bash)] OUT> at
> org.apache.cxf.endpoint.ClientImpl.onMessage(ClientImpl.java:755)
> [runTest(bash)] OUT> at
> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:2408)
> [runTest(bash)] OUT> at
> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:2278)
> [runTest(bash)] OUT> at
> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:2121)
> [runTest(bash)] OUT> at
> org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:66)
> [runTest(bash)] OUT> at
> org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:695)
> [runTest(bash)] OUT> at
> org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
> [runTest(bash)] OUT> at
> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:243)
> [runTest(bash)] OUT> at
> org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:516)
> [runTest(bash)] OUT> at
> org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:313)
> [runTest(bash)] OUT> at
> org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:265)
> [runTest(bash)] OUT> at
> org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:73)
> [runTest(bash)] OUT> at
> org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:124)
> [runTest(bash)] OUT> ... 9 more
>
>
> Interestingly, forcing the use of WSS4J 1.5.8 instead (we discovered this by
> accident ;-) ) the problem goes away.
> The application basically builds up a JAXWS client using WS-Security (Sign
> only) through programmatically configured WSS4J interceptors:
> http://fpaste.org/pYHM/
> Once the proxy is ready on client side, it's called concurrently using
> multiple threads (performIteration() method in the code).
> Does the problem/exception above ring any bell? I'm going to investigate
> this a bit more in any case.
> Thanks
> Alessio
>
> --
> Alessio Soldano
> Web Service Lead, JBoss
>
>
--
Colm O hEigeartaigh
http://coheigea.blogspot.com/
Talend - http://www.talend.com