You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cxf.apache.org by Jared Shapiro <ja...@rockyou.com> on 2012/06/05 20:32:12 UTC

mysterious exception when hitting cxf service via load balancer with cxf client

We have a service set up running in tomcat 7.0.26 with cxf version 2.5.2.

currently it is set up on two hosts behind a hardware load balancer.
it is working flawlessly when hit by a php client using the standard
php soap library and when each server is hit directly with a java cxf
client.
when a java cxf client hits the service via the load balancer, we
intermittently see very long lags and the many times, the following
exception:

java.lang.RuntimeException: Couldn't parse stream.
      at org.apache.cxf.staxutils.StaxUtils.createXMLStreamReader(StaxUtils.java:1181)
      at org.apache.cxf.interceptor.StaxInInterceptor.handleMessage(StaxInInterceptor.java:104)
      at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:263)
      at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:123)
      at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:207)
      at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:213)
      at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:193)
      at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:126)
      at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:185)
      at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:108)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:641)
      at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:164)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
      at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:197)
      at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:171)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
      at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
      at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
      at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:987)
      at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:579)
      at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:307)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
      at java.lang.Thread.run(Thread.java:679)
Caused by: com.ctc.wstx.exc.WstxIOException: Read timed out
      at com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:536)
      at com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:585)
      at com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:610)
      at com.ctc.wstx.stax.WstxInputFactory.createXMLStreamReader(WstxInputFactory.java:316)
      at org.apache.cxf.staxutils.StaxUtils.createXMLStreamReader(StaxUtils.java:1179)
      ... 31 more
Caused by: java.net.SocketTimeoutException: Read timed out
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:146)
      at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:532)
      at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:501)
      at org.apache.coyote.http11.InternalInputBuffer$InputStreamInputBuffer.doRead(InternalInputBuffer.java:563)
      at org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:118)
      at org.apache.coyote.http11.AbstractInputBuffer.doRead(AbstractInputBuffer.java:326)
      at org.apache.coyote.Request.doRead(Request.java:422)
      at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:290)
      at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:431)
      at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:315)
      at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:200)
      at java.io.FilterInputStream.read(FilterInputStream.java:133)
      at com.ctc.wstx.io.BaseReader.readBytes(BaseReader.java:155)
      at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:368)
      at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:111)
      at com.ctc.wstx.io.ReaderBootstrapper.initialLoad(ReaderBootstrapper.java:250)
      at com.ctc.wstx.io.ReaderBootstrapper.bootstrapInput(ReaderBootstrapper.java:133)
      at com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:531)
      ... 35 more

we have checked the contents of the request by comparing a tcp dump
that the client sends vs what is received on the server on the other
side of the load balancer and they match.

again, the cxf client works perfectly when hitting each machine
directly. it is only when going through the hardware load balancer
that we see the issue, yet we've verified that the actual contents of
the request passes through the load balancer unchanged.

any advice on what may be going on here and how to resolve it would be
very helpful.

thanks

Re: mysterious exception when hitting cxf service via load balancer with cxf client

Posted by Daniel Kulp <dk...@apache.org>.
No idea.  Strange.

I see a filter in there:
net.bull.javamelody.MonitoringFilter

Would that filter be doing something with the stream?  Maybe write a very 
simple servlet that just grabs the InputStream from the request and tried to 
read it to see if that works?   Not really sure what else to suggest.

Dan





On Tuesday, June 05, 2012 11:32:12 AM Jared Shapiro wrote:
> We have a service set up running in tomcat 7.0.26 with cxf version 2.5.2.
> 
> currently it is set up on two hosts behind a hardware load balancer.
> it is working flawlessly when hit by a php client using the standard
> php soap library and when each server is hit directly with a java cxf
> client.
> when a java cxf client hits the service via the load balancer, we
> intermittently see very long lags and the many times, the following
> exception:
> 
> java.lang.RuntimeException: Couldn't parse stream.
>       at
> org.apache.cxf.staxutils.StaxUtils.createXMLStreamReader(StaxUtils.java:1
> 181) at
> org.apache.cxf.interceptor.StaxInInterceptor.handleMessage(StaxInIntercep
> tor.java:104) at
> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorCh
> ain.java:263) at
> org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiatio
> nObserver.java:123) at
> org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTP
> Destination.java:207) at
> org.apache.cxf.transport.servlet.ServletController.invokeDestination(Serv
> letController.java:213) at
> org.apache.cxf.transport.servlet.ServletController.invoke(ServletControll
> er.java:193) at
> org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringS
> ervlet.java:126) at
> org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(Abstra
> ctHTTPServlet.java:185) at
> org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPS
> ervlet.java:108) at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:641) at
> org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTP
> Servlet.java:164) at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicat
> ionFilterChain.java:305) at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilte
> rChain.java:210) at
> net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:197)
> at
> net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:171)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicat
> ionFilterChain.java:243) at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilte
> rChain.java:210) at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve
> .java:224) at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve
> .java:169) at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorB
> ase.java:472) at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:
> 168) at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:
> 98) at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.j
> ava:118) at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:40
> 7) at
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Pr
> ocessor.java:987) at
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Abst
> ractProtocol.java:579) at
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.ja
> va:307) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java
> :1110) at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav
> a:603) at java.lang.Thread.run(Thread.java:679)
> Caused by: com.ctc.wstx.exc.WstxIOException: Read timed out
>       at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:536)
> at com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:585)
> at com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:610)
> at
> com.ctc.wstx.stax.WstxInputFactory.createXMLStreamReader(WstxInputFactory
> .java:316) at
> org.apache.cxf.staxutils.StaxUtils.createXMLStreamReader(StaxUtils.java:1
> 179) ... 31 more
> Caused by: java.net.SocketTimeoutException: Read timed out
>       at java.net.SocketInputStream.socketRead0(Native Method)
>       at java.net.SocketInputStream.read(SocketInputStream.java:146)
>       at
> org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.jav
> a:532) at
> org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.jav
> a:501) at
> org.apache.coyote.http11.InternalInputBuffer$InputStreamInputBuffer.doRea
> d(InternalInputBuffer.java:563) at
> org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInput
> Filter.java:118) at
> org.apache.coyote.http11.AbstractInputBuffer.doRead(AbstractInputBuffer.j
> ava:326) at org.apache.coyote.Request.doRead(Request.java:422)
>       at
> org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:
> 290) at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:431)
> at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:315)
> at
> org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.ja
> va:200) at java.io.FilterInputStream.read(FilterInputStream.java:133) at
> com.ctc.wstx.io.BaseReader.readBytes(BaseReader.java:155) at
> com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:368) at
> com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:111)
>       at
> com.ctc.wstx.io.ReaderBootstrapper.initialLoad(ReaderBootstrapper.java:25
> 0) at
> com.ctc.wstx.io.ReaderBootstrapper.bootstrapInput(ReaderBootstrapper.java
> :133) at
> com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:531)
> ... 35 more
> 
> we have checked the contents of the request by comparing a tcp dump
> that the client sends vs what is received on the server on the other
> side of the load balancer and they match.
> 
> again, the cxf client works perfectly when hitting each machine
> directly. it is only when going through the hardware load balancer
> that we see the issue, yet we've verified that the actual contents of
> the request passes through the load balancer unchanged.
> 
> any advice on what may be going on here and how to resolve it would be
> very helpful.
> 
> thanks
-- 
Daniel Kulp
dkulp@apache.org - http://dankulp.com/blog
Talend Community Coder - http://coders.talend.com