You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cxf.apache.org by dhogan <di...@forgerock.com> on 2013/04/08 23:37:19 UTC
WebSphere 8.5 hanging CXF STS implementation?
Hi-
I am working on getting an STS (CXF 2.7.2) implementation working on a
variety of containers.
Thus far, I've been successful on Tomcat, JBoss EAP 4, EAP 5, and AS 7,
WebLogic 11g
and 12c, all on Ubuntu 10.04. Now I'm working on WebSphere 8.5 (last in the
list!!), and can't seem to get things working.
Perhaps others have run into similar issues, and can shed some light?
I've followed the directions for WebSphere from here:
http://cxf.apache.org/docs/application-server-specific-configuration-guide.html
I publish an STS with e.g. UsernameTokens over the Symmetric binding, and
attempt to
do an issue operation. The invocation hangs. Eventually, WebSphere produces
the following log:
[4/8/13 13:54:47:845 PDT] 00000025 ThreadMonitor W WSVR0605W: Thread
"WebContainer : 0" (000000c7) has been active for 661856 milliseconds and
may be hung. There is/are 1 thread(s) in total in the server that may be
hung.
at
com.ibm.ws.webservices.engine.xmlsoap.SOAPElement.wrapChildDOMNode(SOAPElement.java:3223)
at
com.ibm.ws.webservices.engine.xmlsoap.SOAPElement.insertBefore(SOAPElement.java:1896)
at org.apache.xerces.dom.ParentNode.internalInsertBefore(Unknown
Source)
at org.apache.xerces.dom.ParentNode.replaceChild(Unknown Source)
at
org.apache.xml.security.encryption.XMLCipher.decryptElement(XMLCipher.java:1627)
at
org.apache.xml.security.encryption.XMLCipher.doFinal(XMLCipher.java:980)
at
org.apache.ws.security.processor.ReferenceListProcessor.decryptEncryptedData(ReferenceListProcessor.java:337)
at
org.apache.ws.security.processor.ReferenceListProcessor.decryptDataRefEmbedded(ReferenceListProcessor.java:197)
at
org.apache.ws.security.processor.ReferenceListProcessor.handleReferenceList(ReferenceListProcessor.java:104)
at
org.apache.ws.security.processor.ReferenceListProcessor.handleToken(ReferenceListProcessor.java:64)
at
org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:396)
at
org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:274)
at
org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:93)
at
org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271)
at
org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at
org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:239)
at
org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:218)
at
org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:198)
at
org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:137)
at
org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:158)
at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:243)
at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:163)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:595)
at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:219)
at
com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1224)
at
com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:774)
at
com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:456)
at
com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at
com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)
at
com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:97)
at
org.forgerock.openam.validation.ResponseValidationFilter.doFilter(ResponseValidationFilter.java:44)
at
com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
at
com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at
com.sun.identity.setup.AMSetupFilter.doFilter(AMSetupFilter.java:95)
at
com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
at
com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at
com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:928)
at
com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1025)
at
com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:87)
at
com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:895)
at
com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)
at
com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:195)
at
com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:459)
at
com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:526)
at
com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:312)
at
com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:87)
at
com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
at
com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at
com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at
com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1814)
Triggering a system dump from the WebSphere console generates the following
information:
3XMTHREADINFO "WebContainer : 0" J9VMThread:0x0BC1E400,
j9thread_t:0x0CA5CF78, java/lang/Thread:0x7A926E18, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x20C0, native priority:0x5,
native policy:UNKNOWN)
3XMTHREADINFO2 (native stack address range from:0x02697000,
to:0x026D8000, size:0x41000)
3XMTHREADINFO3 Java callstack:
4XESTACKTRACE at
org/apache/xerces/dom/ElementNSImpl.<init>(Bytecode PC:9(Compiled Code))
4XESTACKTRACE at
com/ibm/ws/webservices/engine/xmlsoap/dom/impl/DOMElementImpl.<init>(DOMElementImpl.java:97(Compiled
Code))
4XESTACKTRACE at
com/ibm/ws/webservices/engine/xmlsoap/dom/impl/DOMElementImpl_MS.<init>(DOMElementImpl_MS.java:133(Compiled
Code))
4XESTACKTRACE at
com/ibm/ws/webservices/engine/xmlsoap/SOAPElement.<init>(SOAPElement.java:465(Compiled
Code))
4XESTACKTRACE at
com/ibm/ws/webservices/engine/xmlsoap/SOAPElement.createChildElement(SOAPElement.java:3152(Compiled
Code))
4XESTACKTRACE at
com/ibm/ws/webservices/engine/xmlsoap/SOAPElement.wrapChildDOMNode(SOAPElement.java:3218(Compiled
Code))
4XESTACKTRACE at
com/ibm/ws/webservices/engine/xmlsoap/SOAPElement.insertBefore(SOAPElement.java:1896(Compiled
Code))
4XESTACKTRACE at
org/apache/xerces/dom/ParentNode.internalInsertBefore(Bytecode
PC:91(Compiled Code))
4XESTACKTRACE at
org/apache/xerces/dom/ParentNode.replaceChild(Bytecode PC:12)
4XESTACKTRACE at
org/apache/xml/security/encryption/XMLCipher.decryptElement(XMLCipher.java:1627)
4XESTACKTRACE at
org/apache/xml/security/encryption/XMLCipher.doFinal(XMLCipher.java:980)
4XESTACKTRACE at
org/apache/ws/security/processor/ReferenceListProcessor.decryptEncryptedData(ReferenceListProcessor.java:337)
4XESTACKTRACE at
org/apache/ws/security/processor/ReferenceListProcessor.decryptDataRefEmbedded(ReferenceListProcessor.java:197)
4XESTACKTRACE at
org/apache/ws/security/processor/ReferenceListProcessor.handleReferenceList(ReferenceListProcessor.java:104)
4XESTACKTRACE at
org/apache/ws/security/processor/ReferenceListProcessor.handleToken(ReferenceListProcessor.java:64)
4XESTACKTRACE at
org/apache/ws/security/WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:396)
4XESTACKTRACE at
org/apache/cxf/ws/security/wss4j/WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:274)
4XESTACKTRACE at
org/apache/cxf/ws/security/wss4j/WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:93)
4XESTACKTRACE at
org/apache/cxf/phase/PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271)
5XESTACKTRACE (entered lock:
org/apache/cxf/phase/PhaseInterceptorChain@0x7D08A1D8, entry count: 1)
4XESTACKTRACE at
org/apache/cxf/transport/ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
4XESTACKTRACE at
org/apache/cxf/transport/http/AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:239)
4XESTACKTRACE at
org/apache/cxf/transport/servlet/ServletController.invokeDestination(ServletController.java:218)
4XESTACKTRACE at
org/apache/cxf/transport/servlet/ServletController.invoke(ServletController.java:198)
4XESTACKTRACE at
org/apache/cxf/transport/servlet/ServletController.invoke(ServletController.java:137)
4XESTACKTRACE at
org/apache/cxf/transport/servlet/CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:158)
4XESTACKTRACE at
org/apache/cxf/transport/servlet/AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:243)
4XESTACKTRACE at
org/apache/cxf/transport/servlet/AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:163)
4XESTACKTRACE at
javax/servlet/http/HttpServlet.service(HttpServlet.java:595)
4XESTACKTRACE at
org/apache/cxf/transport/servlet/AbstractHTTPServlet.service(AbstractHTTPServlet.java:219)
4XESTACKTRACE at
com/ibm/ws/webcontainer/servlet/ServletWrapper.service(ServletWrapper.java:1224)
4XESTACKTRACE at
com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:774)
4XESTACKTRACE at
com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:456)
4XESTACKTRACE at
com/ibm/ws/webcontainer/servlet/ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
4XESTACKTRACE at
com/ibm/ws/webcontainer/filter/WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)
4XESTACKTRACE at
com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:97)
4XESTACKTRACE at
org/forgerock/openam/validation/ResponseValidationFilter.doFilter(ResponseValidationFilter.java:44)
4XESTACKTRACE at
com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
4XESTACKTRACE at
com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
4XESTACKTRACE at
com/sun/identity/setup/AMSetupFilter.doFilter(AMSetupFilter.java:95)
4XESTACKTRACE at
com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
4XESTACKTRACE at
com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
4XESTACKTRACE at
com/ibm/ws/webcontainer/filter/WebAppFilterManager.doFilter(WebAppFilterManager.java:928)
4XESTACKTRACE at
com/ibm/ws/webcontainer/filter/WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1025)
4XESTACKTRACE at
com/ibm/ws/webcontainer/servlet/CacheServletWrapper.handleRequest(CacheServletWrapper.java:87)
4XESTACKTRACE at
com/ibm/ws/webcontainer/WebContainer.handleRequest(WebContainer.java:895)
4XESTACKTRACE at
com/ibm/ws/webcontainer/WSWebContainer.handleRequest(WSWebContainer.java:1662)
4XESTACKTRACE at
com/ibm/ws/webcontainer/channel/WCChannelLink.ready(WCChannelLink.java:195)
4XESTACKTRACE at
com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleDiscrimination(HttpInboundLink.java:459)
4XESTACKTRACE at
com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleNewRequest(HttpInboundLink.java:526)
4XESTACKTRACE at
com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.processRequest(HttpInboundLink.java:312)
4XESTACKTRACE at
com/ibm/ws/http/channel/inbound/impl/HttpICLReadCallback.complete(HttpICLReadCallback.java:87)
4XESTACKTRACE at
com/ibm/ws/tcp/channel/impl/AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
4XESTACKTRACE at
com/ibm/io/async/AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
4XESTACKTRACE at
com/ibm/io/async/AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
4XESTACKTRACE at
com/ibm/io/async/AsyncFuture.completed(AsyncFuture.java:138)
4XESTACKTRACE at
com/ibm/io/async/ResultHandler.complete(ResultHandler.java:204)
4XESTACKTRACE at
com/ibm/io/async/ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
4XESTACKTRACE at
com/ibm/io/async/ResultHandler$2.run(ResultHandler.java:905)
4XESTACKTRACE at
com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1814(Compiled Code))
3XMTHREADINFO3 Native callstack:
4XENATIVESTACK (0x003085C0 [libj9prt26.so+0xf5c0])
4XENATIVESTACK (0x00317FD2 [libj9prt26.so+0x1efd2])
4XENATIVESTACK (0x003082B1 [libj9prt26.so+0xf2b1])
4XENATIVESTACK (0x003083A5 [libj9prt26.so+0xf3a5])
4XENATIVESTACK (0x00307EB4 [libj9prt26.so+0xeeb4])
4XENATIVESTACK (0x00317FD2 [libj9prt26.so+0x1efd2])
4XENATIVESTACK (0x00307E23 [libj9prt26.so+0xee23])
4XENATIVESTACK (0x00302A5D [libj9prt26.so+0x9a5d])
4XENATIVESTACK __kernel_rt_sigreturn+0x0 (0x00AE7410)
4XENATIVESTACK j9thread_monitor_wait+0x85 (0x00F84B21
[libj9thr26.so+0x8b21])
4XENATIVESTACK (0x002C9019 [libj9vm26.so+0x5c019])
4XENATIVESTACK (0x002C9354 [libj9vm26.so+0x5c354])
4XENATIVESTACK (0x00272D01 [libj9vm26.so+0x5d01])
4XENATIVESTACK (0x01593386 [libj9jit26.so+0x5cf386])
4XENATIVESTACK (0x002D0924 [libj9vm26.so+0x63924])
4XENATIVESTACK (0x00317FD2 [libj9prt26.so+0x1efd2])
4XENATIVESTACK (0x002D0876 [libj9vm26.so+0x63876])
4XENATIVESTACK (0x00F851E5 [libj9thr26.so+0x91e5])
4XENATIVESTACK (0x0083496E [libpthread.so.0+0x596e])
4XENATIVESTACK clone+0x5e (0x001DDA4E [libc.so.6+0xcda4e])
Note that the dump log indicates that the invocation progressed a bit
further. This is not a mistake, and I've noticed the same behavior several
times. The box is pegged (CPU at >99%) once this occurs, and it seems like
the disparity is a function of time differences when the log entry and dumps
are generated. Additional dump state(see below) seems to indicate that the
invoking thread has entered a monitor's wait-set, but is never notified out
of the waiting state. I'm not sure if this is a red-herring, or if it is a
function of the invocation simply no longer making progress (i.e. starving
because CPU is pegged - though this too, seems unlikely).
...
2LKREGMON &newThread->threadNameMutex lock (0x0C04EC98): <unowned>
2LKREGMON JVM_RawMonitor lock (0x0C04ECF0): <unowned>
2LKREGMON Thread public flags mutex lock (0x0C04ED48): <unowned>
3LKNOTIFYQ Waiting to be notified:
3LKWAITNOTIFY "WebContainer : 0" (0x0BC1E400)
2LKREGMON &(classLoader->mutex) lock (0x0C04EDA0): <unowned>
2LKREGMON JIT-QueueSlotMonitor-1103 lock (0x0C04F0B8): <unowned>
2LKREGMON &(classLoader->mutex) lock (0x0C04F2C8): <unowned>
...
Anyone have any suggestions or similar experiences?
Thanks
Dirk
--
View this message in context: http://cxf.547215.n5.nabble.com/WebSphere-8-5-hanging-CXF-STS-implementation-tp5725997.html
Sent from the cxf-user mailing list archive at Nabble.com.
RE: WebSphere 8.5 hanging CXF STS implementation?
Posted by dhogan <di...@forgerock.com>.
An update (with more information), but still no solution. Again, I have a
web-service that publishes STS instances. I've:
1. disabled the WebSphere web-services engine
2. set the single classloader policy which looks locally first.
Publishing this initial web-service fails, with the following stack trace
(other CXF users have experienced this - e.g.
http://cxf.547215.n5.nabble.com/Getting-Caused-by-java-lang-IncompatibleClassChangeError-org-apache-neethi-AssertionBuilderFactory-e5-td5716804.html.)
[4/15/13 15:05:05:246 PDT] 00000057 webapp E
com.ibm.ws.webcontainer.webapp.WebApp logServletError SRVE0293E: [Servlet
Error]-[new_sts]: org.apache.cxf.bus.extension.ExtensionException: Could not
load extension class org.apache.cxf.ws.policy.AssertionBuilderRegistryImpl.
at
org.apache.cxf.bus.extension.Extension.tryClass(Extension.java:173)
at
org.apache.cxf.bus.extension.Extension.getClassObject(Extension.java:185)
at
org.apache.cxf.bus.extension.ExtensionManagerImpl.activateAllByType(ExtensionManagerImpl.java:138)
at
org.apache.cxf.bus.extension.ExtensionManagerBus.<init>(ExtensionManagerBus.java:126)
at
org.apache.cxf.bus.extension.ExtensionManagerBus.<init>(ExtensionManagerBus.java:134)
at org.apache.cxf.bus.CXFBusFactory.createBus(CXFBusFactory.java:40)
at org.apache.cxf.bus.CXFBusFactory.createBus(CXFBusFactory.java:36)
at org.apache.cxf.bus.CXFBusFactory.createBus(CXFBusFactory.java:32)
at
org.apache.cxf.transport.servlet.CXFNonSpringServlet.loadBus(CXFNonSpringServlet.java:101)
at
org.forgerock.openam.sts.publish.web.STSInstancePublisher.loadBus(STSInstancePublisher.java:19)
at
org.apache.cxf.transport.servlet.CXFNonSpringServlet.init(CXFNonSpringServlet.java:71)
at
com.ibm.ws.webcontainer.servlet.ServletWrapper.init(ServletWrapper.java:337)
at
com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.init(ServletWrapperImpl.java:168)
at
com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:630)
at
com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:457)
at
com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at
com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)
at
com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:97)
at
org.forgerock.openam.validation.ResponseValidationFilter.doFilter(ResponseValidationFilter.java:44)
at
com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
at
com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at
com.sun.identity.setup.AMSetupFilter.doFilter(AMSetupFilter.java:95)
at
com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
at
com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at
com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:928)
at
com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1025)
at
com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3761)
at
com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:304)
at
com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:975)
at
com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)
at
com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:195)
at
com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:459)
at
com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:526)
at
com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:312)
at
com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:87)
at
com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
at
com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at
com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at
com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1815)
Caused by: java.lang.IncompatibleClassChangeError:
org.apache.neethi.AssertionBuilderFactory
at java.lang.ClassLoader.defineClassImpl(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:264)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:74)
at
com.ibm.ws.classloader.CompoundClassLoader._defineClass(CompoundClassLoader.java:853)
at
com.ibm.ws.classloader.CompoundClassLoader.localFindClass(CompoundClassLoader.java:763)
at
com.ibm.ws.classloader.CompoundClassLoader.loadClass(CompoundClassLoader.java:586)
at java.lang.ClassLoader.loadClass(ClassLoader.java:650)
at java.lang.ClassLoader.defineClassImpl(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:264)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:74)
at
com.ibm.ws.classloader.CompoundClassLoader._defineClass(CompoundClassLoader.java:853)
at
com.ibm.ws.classloader.CompoundClassLoader.localFindClass(CompoundClassLoader.java:763)
at
com.ibm.ws.classloader.CompoundClassLoader.loadClass(CompoundClassLoader.java:586)
at java.lang.ClassLoader.loadClass(ClassLoader.java:650)
at
org.apache.cxf.bus.extension.Extension.tryClass(Extension.java:169)
... 42 more
Triggering the verbose classloading shows:
class load: org.apache.neethi.AssertionBuilderFactory from:
file:/home/dhogan/IBM/WebSphere/AppServer/plugins/org.apache.axis2.jar.
Attempting to address this problem via the endorsed mechanism won't work (as
suggested in responses to this problem) , as the class in question is not
part of the package-set that can be overridden by the endorsed mechanism
(besides the fact that the class is being loaded by WebSphere, not by java).
So I expanded the org.apache.axis2.jar, and replaced the org.apache.neethi
classes with those in neethi-3.0.2.jar. This got me a bit further. Next
error:
java.lang.NoSuchMethodError:
org/apache/ws/commons/schema/XmlSchemaCollection.read(Lorg/w3c/dom/Document;Ljava/lang/String;)Lorg/apache/ws/commons/schema/XmlSchema;
at
org.apache.cxf.common.xmlschema.SchemaCollection.read(SchemaCollection.java:133)
at
org.apache.cxf.databinding.AbstractDataBinding.addSchemaDocument(AbstractDataBinding.java:192)
at
org.apache.cxf.databinding.AbstractDataBinding.addSchemaDocument(AbstractDataBinding.java:95)
at
org.apache.cxf.jaxb.JAXBDataBinding.initialize(JAXBDataBinding.java:388)
at
org.apache.cxf.service.factory.AbstractServiceFactoryBean.initializeDataBindings(AbstractServiceFactoryBean.java:86)
at
org.apache.cxf.service.factory.ReflectionServiceFactoryBean.buildServiceFromClass(ReflectionServiceFactoryBean.java:475)
at
org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean.buildServiceFromClass(JaxWsServiceFactoryBean.java:690)
at
org.apache.cxf.service.factory.ReflectionServiceFactoryBean.initializeServiceModel(ReflectionServiceFactoryBean.java:537)
at
org.apache.cxf.service.factory.ReflectionServiceFactoryBean.create(ReflectionServiceFactoryBean.java:249)
at
org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean.create(JaxWsServiceFactoryBean.java:205)
at
org.apache.cxf.frontend.AbstractWSDLBasedEndpointFactory.createEndpoint(AbstractWSDLBasedEndpointFactory.java:101)
at
org.apache.cxf.frontend.ServerFactoryBean.create(ServerFactoryBean.java:159)
at
org.apache.cxf.jaxws.JaxWsServerFactoryBean.create(JaxWsServerFactoryBean.java:211)
at
org.forgerock.openam.sts.publish.web.STSInstancePublisher.loadBus(STSInstancePublisher.java:33)
at
org.apache.cxf.transport.servlet.CXFNonSpringServlet.init(CXFNonSpringServlet.java:71)
at
com.ibm.ws.webcontainer.servlet.ServletWrapper.init(ServletWrapper.java:336)
at
com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.init(ServletWrapperImpl.java:168)
at
com.ibm.ws.webcontainer.servlet.ServletWrapper.load(ServletWrapper.java:1360)
at
com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:982)
at
com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3748)
at
com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:304)
at
com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:962)
at
com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)
at
com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:195)
at
com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:459)
at
com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:526)
at
com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:312)
at
com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:283)
at
com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
at
com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
at
com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
at
com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at
com.ibm.io.async.AsyncChannelFuture$1.run(AsyncChannelFuture.java:205)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1814)
>From the classloader logs:
class load: org.apache.ws.commons.schema.XmlSchemaCollection from:
file:/home/dhogan/IBM/WebSphere/AppServer/plugins/org.apache.axis2.jar
Again, we are not dealing with a package that can be over-ridden by the
endorsed mechanism.
I replaced the org.apache.ws.commons.schema classes in the
org.apache.axis2.jar with those in xmlschema-core-2.0.3.jar. That allowed me
to successfully publish an STS instance (callers authenticated via
username-tokens over the symmetric binding).
As soon as I try to get my STS to issue a SAML token, the box CPU pegs with
the stack traces mentioned in my first post. I get the same issue set on
Ubuntu 10.04 desktop (not an officially supported platform for WebSphere
8.5), and on SUSE Linux Enterprise Desktop 11 (officially supported
WebSphere 8.5 platform).
I get this problem regardless of whether I package the servicemix .jar in my
.war file.
I'm hoping that someone in CXF-land has run into this problem and can
provide some insight...
Thanks
Dirk
--
View this message in context: http://cxf.547215.n5.nabble.com/WebSphere-8-5-hanging-CXF-STS-implementation-tp5725997p5726390.html
Sent from the cxf-user mailing list archive at Nabble.com.
RE: WebSphere 8.5 hanging CXF STS implementation?
Posted by dhogan <di...@forgerock.com>.
Thanks for the response. Yes, I have read (and re-read) the thaker pdf, and
have deployed the servicemix .jar in my .war, as specified in your link, and
neatly summarized here:
http://cxf.apache.org/docs/application-server-specific-configuration-guide.html
I still have the same problem - pegged CPU, and no response to my
invocation. The fact that the stack trace seems slightly different every
time I trigger a java dump may indicate that the invocation is being
dispatched again and again, in a tight loop. I will poke some more, and
update if I find anything.
Thanks
Dirk
--
View this message in context: http://cxf.547215.n5.nabble.com/WebSphere-8-5-hanging-CXF-STS-implementation-tp5725997p5726055.html
Sent from the cxf-user mailing list archive at Nabble.com.
RE: WebSphere 8.5 hanging CXF STS implementation?
Posted by Oliver Wulff <ow...@talend.com>.
Maybe the following thread helps:
http://mail-archives.apache.org/mod_mbox/cxf-users/201201.mbox/%3C79AB4452999C844D9920E0363533273110B499@S10BE002.SH10.lan%3E
Oli
________________________________________
From: dhogan [dirk.hogan@forgerock.com]
Sent: 08 April 2013 23:37
To: users@cxf.apache.org
Subject: WebSphere 8.5 hanging CXF STS implementation?
Hi-
I am working on getting an STS (CXF 2.7.2) implementation working on a
variety of containers.
Thus far, I've been successful on Tomcat, JBoss EAP 4, EAP 5, and AS 7,
WebLogic 11g
and 12c, all on Ubuntu 10.04. Now I'm working on WebSphere 8.5 (last in the
list!!), and can't seem to get things working.
Perhaps others have run into similar issues, and can shed some light?
I've followed the directions for WebSphere from here:
http://cxf.apache.org/docs/application-server-specific-configuration-guide.html
I publish an STS with e.g. UsernameTokens over the Symmetric binding, and
attempt to
do an issue operation. The invocation hangs. Eventually, WebSphere produces
the following log:
[4/8/13 13:54:47:845 PDT] 00000025 ThreadMonitor W WSVR0605W: Thread
"WebContainer : 0" (000000c7) has been active for 661856 milliseconds and
may be hung. There is/are 1 thread(s) in total in the server that may be
hung.
at
com.ibm.ws.webservices.engine.xmlsoap.SOAPElement.wrapChildDOMNode(SOAPElement.java:3223)
at
com.ibm.ws.webservices.engine.xmlsoap.SOAPElement.insertBefore(SOAPElement.java:1896)
at org.apache.xerces.dom.ParentNode.internalInsertBefore(Unknown
Source)
at org.apache.xerces.dom.ParentNode.replaceChild(Unknown Source)
at
org.apache.xml.security.encryption.XMLCipher.decryptElement(XMLCipher.java:1627)
at
org.apache.xml.security.encryption.XMLCipher.doFinal(XMLCipher.java:980)
at
org.apache.ws.security.processor.ReferenceListProcessor.decryptEncryptedData(ReferenceListProcessor.java:337)
at
org.apache.ws.security.processor.ReferenceListProcessor.decryptDataRefEmbedded(ReferenceListProcessor.java:197)
at
org.apache.ws.security.processor.ReferenceListProcessor.handleReferenceList(ReferenceListProcessor.java:104)
at
org.apache.ws.security.processor.ReferenceListProcessor.handleToken(ReferenceListProcessor.java:64)
at
org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:396)
at
org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:274)
at
org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:93)
at
org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271)
at
org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at
org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:239)
at
org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:218)
at
org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:198)
at
org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:137)
at
org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:158)
at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:243)
at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:163)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:595)
at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:219)
at
com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1224)
at
com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:774)
at
com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:456)
at
com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at
com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)
at
com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:97)
at
org.forgerock.openam.validation.ResponseValidationFilter.doFilter(ResponseValidationFilter.java:44)
at
com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
at
com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at
com.sun.identity.setup.AMSetupFilter.doFilter(AMSetupFilter.java:95)
at
com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
at
com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at
com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:928)
at
com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1025)
at
com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:87)
at
com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:895)
at
com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)
at
com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:195)
at
com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:459)
at
com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:526)
at
com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:312)
at
com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:87)
at
com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
at
com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at
com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at
com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1814)
Triggering a system dump from the WebSphere console generates the following
information:
3XMTHREADINFO "WebContainer : 0" J9VMThread:0x0BC1E400,
j9thread_t:0x0CA5CF78, java/lang/Thread:0x7A926E18, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x20C0, native priority:0x5,
native policy:UNKNOWN)
3XMTHREADINFO2 (native stack address range from:0x02697000,
to:0x026D8000, size:0x41000)
3XMTHREADINFO3 Java callstack:
4XESTACKTRACE at
org/apache/xerces/dom/ElementNSImpl.<init>(Bytecode PC:9(Compiled Code))
4XESTACKTRACE at
com/ibm/ws/webservices/engine/xmlsoap/dom/impl/DOMElementImpl.<init>(DOMElementImpl.java:97(Compiled
Code))
4XESTACKTRACE at
com/ibm/ws/webservices/engine/xmlsoap/dom/impl/DOMElementImpl_MS.<init>(DOMElementImpl_MS.java:133(Compiled
Code))
4XESTACKTRACE at
com/ibm/ws/webservices/engine/xmlsoap/SOAPElement.<init>(SOAPElement.java:465(Compiled
Code))
4XESTACKTRACE at
com/ibm/ws/webservices/engine/xmlsoap/SOAPElement.createChildElement(SOAPElement.java:3152(Compiled
Code))
4XESTACKTRACE at
com/ibm/ws/webservices/engine/xmlsoap/SOAPElement.wrapChildDOMNode(SOAPElement.java:3218(Compiled
Code))
4XESTACKTRACE at
com/ibm/ws/webservices/engine/xmlsoap/SOAPElement.insertBefore(SOAPElement.java:1896(Compiled
Code))
4XESTACKTRACE at
org/apache/xerces/dom/ParentNode.internalInsertBefore(Bytecode
PC:91(Compiled Code))
4XESTACKTRACE at
org/apache/xerces/dom/ParentNode.replaceChild(Bytecode PC:12)
4XESTACKTRACE at
org/apache/xml/security/encryption/XMLCipher.decryptElement(XMLCipher.java:1627)
4XESTACKTRACE at
org/apache/xml/security/encryption/XMLCipher.doFinal(XMLCipher.java:980)
4XESTACKTRACE at
org/apache/ws/security/processor/ReferenceListProcessor.decryptEncryptedData(ReferenceListProcessor.java:337)
4XESTACKTRACE at
org/apache/ws/security/processor/ReferenceListProcessor.decryptDataRefEmbedded(ReferenceListProcessor.java:197)
4XESTACKTRACE at
org/apache/ws/security/processor/ReferenceListProcessor.handleReferenceList(ReferenceListProcessor.java:104)
4XESTACKTRACE at
org/apache/ws/security/processor/ReferenceListProcessor.handleToken(ReferenceListProcessor.java:64)
4XESTACKTRACE at
org/apache/ws/security/WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:396)
4XESTACKTRACE at
org/apache/cxf/ws/security/wss4j/WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:274)
4XESTACKTRACE at
org/apache/cxf/ws/security/wss4j/WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:93)
4XESTACKTRACE at
org/apache/cxf/phase/PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271)
5XESTACKTRACE (entered lock:
org/apache/cxf/phase/PhaseInterceptorChain@0x7D08A1D8, entry count: 1)
4XESTACKTRACE at
org/apache/cxf/transport/ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
4XESTACKTRACE at
org/apache/cxf/transport/http/AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:239)
4XESTACKTRACE at
org/apache/cxf/transport/servlet/ServletController.invokeDestination(ServletController.java:218)
4XESTACKTRACE at
org/apache/cxf/transport/servlet/ServletController.invoke(ServletController.java:198)
4XESTACKTRACE at
org/apache/cxf/transport/servlet/ServletController.invoke(ServletController.java:137)
4XESTACKTRACE at
org/apache/cxf/transport/servlet/CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:158)
4XESTACKTRACE at
org/apache/cxf/transport/servlet/AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:243)
4XESTACKTRACE at
org/apache/cxf/transport/servlet/AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:163)
4XESTACKTRACE at
javax/servlet/http/HttpServlet.service(HttpServlet.java:595)
4XESTACKTRACE at
org/apache/cxf/transport/servlet/AbstractHTTPServlet.service(AbstractHTTPServlet.java:219)
4XESTACKTRACE at
com/ibm/ws/webcontainer/servlet/ServletWrapper.service(ServletWrapper.java:1224)
4XESTACKTRACE at
com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:774)
4XESTACKTRACE at
com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:456)
4XESTACKTRACE at
com/ibm/ws/webcontainer/servlet/ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
4XESTACKTRACE at
com/ibm/ws/webcontainer/filter/WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)
4XESTACKTRACE at
com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:97)
4XESTACKTRACE at
org/forgerock/openam/validation/ResponseValidationFilter.doFilter(ResponseValidationFilter.java:44)
4XESTACKTRACE at
com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
4XESTACKTRACE at
com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
4XESTACKTRACE at
com/sun/identity/setup/AMSetupFilter.doFilter(AMSetupFilter.java:95)
4XESTACKTRACE at
com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
4XESTACKTRACE at
com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
4XESTACKTRACE at
com/ibm/ws/webcontainer/filter/WebAppFilterManager.doFilter(WebAppFilterManager.java:928)
4XESTACKTRACE at
com/ibm/ws/webcontainer/filter/WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1025)
4XESTACKTRACE at
com/ibm/ws/webcontainer/servlet/CacheServletWrapper.handleRequest(CacheServletWrapper.java:87)
4XESTACKTRACE at
com/ibm/ws/webcontainer/WebContainer.handleRequest(WebContainer.java:895)
4XESTACKTRACE at
com/ibm/ws/webcontainer/WSWebContainer.handleRequest(WSWebContainer.java:1662)
4XESTACKTRACE at
com/ibm/ws/webcontainer/channel/WCChannelLink.ready(WCChannelLink.java:195)
4XESTACKTRACE at
com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleDiscrimination(HttpInboundLink.java:459)
4XESTACKTRACE at
com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleNewRequest(HttpInboundLink.java:526)
4XESTACKTRACE at
com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.processRequest(HttpInboundLink.java:312)
4XESTACKTRACE at
com/ibm/ws/http/channel/inbound/impl/HttpICLReadCallback.complete(HttpICLReadCallback.java:87)
4XESTACKTRACE at
com/ibm/ws/tcp/channel/impl/AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
4XESTACKTRACE at
com/ibm/io/async/AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
4XESTACKTRACE at
com/ibm/io/async/AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
4XESTACKTRACE at
com/ibm/io/async/AsyncFuture.completed(AsyncFuture.java:138)
4XESTACKTRACE at
com/ibm/io/async/ResultHandler.complete(ResultHandler.java:204)
4XESTACKTRACE at
com/ibm/io/async/ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
4XESTACKTRACE at
com/ibm/io/async/ResultHandler$2.run(ResultHandler.java:905)
4XESTACKTRACE at
com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1814(Compiled Code))
3XMTHREADINFO3 Native callstack:
4XENATIVESTACK (0x003085C0 [libj9prt26.so+0xf5c0])
4XENATIVESTACK (0x00317FD2 [libj9prt26.so+0x1efd2])
4XENATIVESTACK (0x003082B1 [libj9prt26.so+0xf2b1])
4XENATIVESTACK (0x003083A5 [libj9prt26.so+0xf3a5])
4XENATIVESTACK (0x00307EB4 [libj9prt26.so+0xeeb4])
4XENATIVESTACK (0x00317FD2 [libj9prt26.so+0x1efd2])
4XENATIVESTACK (0x00307E23 [libj9prt26.so+0xee23])
4XENATIVESTACK (0x00302A5D [libj9prt26.so+0x9a5d])
4XENATIVESTACK __kernel_rt_sigreturn+0x0 (0x00AE7410)
4XENATIVESTACK j9thread_monitor_wait+0x85 (0x00F84B21
[libj9thr26.so+0x8b21])
4XENATIVESTACK (0x002C9019 [libj9vm26.so+0x5c019])
4XENATIVESTACK (0x002C9354 [libj9vm26.so+0x5c354])
4XENATIVESTACK (0x00272D01 [libj9vm26.so+0x5d01])
4XENATIVESTACK (0x01593386 [libj9jit26.so+0x5cf386])
4XENATIVESTACK (0x002D0924 [libj9vm26.so+0x63924])
4XENATIVESTACK (0x00317FD2 [libj9prt26.so+0x1efd2])
4XENATIVESTACK (0x002D0876 [libj9vm26.so+0x63876])
4XENATIVESTACK (0x00F851E5 [libj9thr26.so+0x91e5])
4XENATIVESTACK (0x0083496E [libpthread.so.0+0x596e])
4XENATIVESTACK clone+0x5e (0x001DDA4E [libc.so.6+0xcda4e])
Note that the dump log indicates that the invocation progressed a bit
further. This is not a mistake, and I've noticed the same behavior several
times. The box is pegged (CPU at >99%) once this occurs, and it seems like
the disparity is a function of time differences when the log entry and dumps
are generated. Additional dump state(see below) seems to indicate that the
invoking thread has entered a monitor's wait-set, but is never notified out
of the waiting state. I'm not sure if this is a red-herring, or if it is a
function of the invocation simply no longer making progress (i.e. starving
because CPU is pegged - though this too, seems unlikely).
...
2LKREGMON &newThread->threadNameMutex lock (0x0C04EC98): <unowned>
2LKREGMON JVM_RawMonitor lock (0x0C04ECF0): <unowned>
2LKREGMON Thread public flags mutex lock (0x0C04ED48): <unowned>
3LKNOTIFYQ Waiting to be notified:
3LKWAITNOTIFY "WebContainer : 0" (0x0BC1E400)
2LKREGMON &(classLoader->mutex) lock (0x0C04EDA0): <unowned>
2LKREGMON JIT-QueueSlotMonitor-1103 lock (0x0C04F0B8): <unowned>
2LKREGMON &(classLoader->mutex) lock (0x0C04F2C8): <unowned>
...
Anyone have any suggestions or similar experiences?
Thanks
Dirk
--
View this message in context: http://cxf.547215.n5.nabble.com/WebSphere-8-5-hanging-CXF-STS-implementation-tp5725997.html
Sent from the cxf-user mailing list archive at Nabble.com.