You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cxf.apache.org by Veit Guna <Ve...@gmx.de> on 2015/09/02 12:33:17 UTC
NPE with message body writer during load
Hi.
I'm using CXF 3.0.4 in client proxy mode, using my server REST interfaces.
In addition I'm using the httpclient cxf module to workaround earlier problems with concurrency
under load (as suggested by Sergey in a post some months ago here on the mailing list).
For Multipart handling I'm using the jersey MultiPart JAX-RS MessageBodyWriter on the server
as well as on the client.
Under normal circumstances all goes well. But sometimes, especially on the startup-phase of the load test,
the CXF client proxy dies with the exception below during a Multipart POST. Any clue what might cause this?
0f5ba696ffe4 fns-service.log Caused by: org.apache.cxf.interceptor.Fault: No message body writer has been found for class org.glassfish.jersey.media.multipart.MultiPart, ContentType: multipart/mixed
0f5ba696ffe4 fns-service.log at org.apache.cxf.jaxrs.client.ClientProxyImpl$BodyWriter.doWriteBody(ClientProxyImpl.java:814)
0f5ba696ffe4 fns-service.log at org.apache.cxf.jaxrs.client.AbstractClient$AbstractBodyWriter.handleMessage(AbstractClient.java:1042)
0f5ba696ffe4 fns-service.log at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307)
0f5ba696ffe4 fns-service.log at org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:624)
0f5ba696ffe4 fns-service.log at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:674)
0f5ba696ffe4 fns-service.log at org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:224)
0f5ba696ffe4 fns-service.log at com.sun.proxy.$Proxy484.create(Unknown Source)
0f5ba696ffe4 fns-service.log at my.class.doSomething(MyClass.java:383)
0f5ba696ffe4 fns-service.log ... 54 more
0f5ba696ffe4 fns-service.log Caused by: javax.ws.rs.ProcessingException: No message body writer has been found for class org.glassfish.jersey.media.multipart.MultiPart, ContentType: multipart/mixed
0f5ba696ffe4 fns-service.log at org.apache.cxf.jaxrs.client.AbstractClient.reportMessageHandlerProblem(AbstractClient.java:741)
0f5ba696ffe4 fns-service.log at org.apache.cxf.jaxrs.client.AbstractClient.writeBody(AbstractClient.java:470)
0f5ba696ffe4 fns-service.log at org.apache.cxf.jaxrs.client.ClientProxyImpl$BodyWriter.doWriteBody(ClientProxyImpl.java:804)
0f5ba696ffe4 fns-service.log ... 62 more
0f5ba696ffe4 fns-service.log Caused by: java.lang.NullPointerException
0f5ba696ffe4 fns-service.log at org.apache.cxf.jaxrs.impl.tl.ThreadLocalProviders.getMessageBodyWriter(ThreadLocalProviders.java:46)
0f5ba696ffe4 fns-service.log at org.glassfish.jersey.media.multipart.internal.MultiPartWriter.writeTo(MultiPartWriter.java:222)
0f5ba696ffe4 fns-service.log at org.glassfish.jersey.media.multipart.internal.MultiPartWriter.writeTo(MultiPartWriter.java:79)
0f5ba696ffe4 fns-service.log at org.apache.cxf.jaxrs.utils.JAXRSUtils.writeMessageBody(JAXRSUtils.java:1379)
0f5ba696ffe4 fns-service.log at org.apache.cxf.jaxrs.client.AbstractClient.writeBody(AbstractClient.java:456)
0f5ba696ffe4 fns-service.log ... 63 more
Re: NPE with message body writer during load
Posted by Sergey Beryozkin <sb...@gmail.com>.
This particular issue has been addressed:
https://issues.apache.org/jira/browse/CXF-6593
However this is a short term solution and a more effective solution
would need to be done in a new trunk.
The way the thread local proxies are managed is somewhat complex in CXF
JAX-RS - partially because the bulk of it was written many years back
and also because the same provider or resource instance can be shared
between multiple endpoints. We can investigate how the code can be
re-written when a new Java 8 trunk opens
Cheers, Sergey
On 10/09/15 11:49, Sergey Beryozkin wrote:
> FYI, two more similar issues have been reported in the last couple of days.
>
> I start suspecting it might be a WeakHashMap optimization I did. In
> TomEE, the guys wanted to do a clean refresh of CXF (JAX-RS) module and
> the fact the bus was holding strong references to CXF context
> implementations such as UriInfoImpl, etc, was problematic (I'll need to
> ask Romain for few more details on that). So 'refresh' situations can
> possibly release the thread local proxies's content, just speculating
> but may be it is related
>
> So I'm on this issue now :-)
>
> Thanks, Sergey
>
>
> On 02/09/15 12:24, Sergey Beryozkin wrote:
>> Hi
>>
>> When CXF JAX-RS injects thread-local proxies into providers/resource
>> classes it keeps field and method proxy maps on the bus. I recall there
>> was a problem in one of the complex setups where there were many
>> endpoints with multiple Jackson instances (as opposed to a single
>> instance shared between the endpoints) where somehow the provider whose
>> thread local proxy was reset depending on the order of the provider
>> initialization.
>>
>> I vaguely recall the side-effect was caused by the fact there were
>> different bus instances involved too, this case is now tested in this
>> context:
>>
>> https://github.com/apache/cxf/blob/master/systests/jaxrs/src/test/resources/jaxrs_jackson_provider/WEB-INF/beans.xml
>>
>>
>>
>> (one of the endpoints has a custom bus)
>>
>> Perhaps when a client is set up in a complex, possibly multi-threaded
>> environment some side-effects with setting tl proxies are possible.
>> Example if it is a single/default bus that is shared between all the
>> clients then if we have multiple threads then they affect the th maps
>> kept on the bus.
>> Please experiment with trying to have each client instance have its own
>> Bus ? (You can create a unique bus with BusFactory and then set it on
>> JAXRSClientFactoryBean before creating a proxy) - if that helps than you
>> can limit such a setup to a test env only unless the actual production
>> setup is also multi-threaded
>>
>> HTH, Sergey
>>
>> On 02/09/15 11:33, Veit Guna wrote:
>>> Hi.
>>>
>>> I'm using CXF 3.0.4 in client proxy mode, using my server REST
>>> interfaces.
>>> In addition I'm using the httpclient cxf module to workaround earlier
>>> problems with concurrency
>>> under load (as suggested by Sergey in a post some months ago here on
>>> the mailing list).
>>>
>>> For Multipart handling I'm using the jersey MultiPart JAX-RS
>>> MessageBodyWriter on the server
>>> as well as on the client.
>>>
>>> Under normal circumstances all goes well. But sometimes, especially on
>>> the startup-phase of the load test,
>>> the CXF client proxy dies with the exception below during a Multipart
>>> POST. Any clue what might cause this?
>>>
>>> 0f5ba696ffe4 fns-service.log Caused by:
>>> org.apache.cxf.interceptor.Fault: No message body writer has been
>>> found for class org.glassfish.jersey.media.multipart.MultiPart,
>>> ContentType: multipart/mixed
>>> 0f5ba696ffe4 fns-service.log at
>>> org.apache.cxf.jaxrs.client.ClientProxyImpl$BodyWriter.doWriteBody(ClientProxyImpl.java:814)
>>>
>>>
>>> 0f5ba696ffe4 fns-service.log at
>>> org.apache.cxf.jaxrs.client.AbstractClient$AbstractBodyWriter.handleMessage(AbstractClient.java:1042)
>>>
>>>
>>> 0f5ba696ffe4 fns-service.log at
>>> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307)
>>>
>>>
>>> 0f5ba696ffe4 fns-service.log at
>>> org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:624)
>>>
>>>
>>> 0f5ba696ffe4 fns-service.log at
>>> org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:674)
>>>
>>>
>>> 0f5ba696ffe4 fns-service.log at
>>> org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:224)
>>>
>>>
>>> 0f5ba696ffe4 fns-service.log at
>>> com.sun.proxy.$Proxy484.create(Unknown Source)
>>> 0f5ba696ffe4 fns-service.log at
>>> my.class.doSomething(MyClass.java:383)
>>> 0f5ba696ffe4 fns-service.log ... 54 more
>>> 0f5ba696ffe4 fns-service.log Caused by:
>>> javax.ws.rs.ProcessingException: No message body writer has been found
>>> for class org.glassfish.jersey.media.multipart.MultiPart, ContentType:
>>> multipart/mixed
>>> 0f5ba696ffe4 fns-service.log at
>>> org.apache.cxf.jaxrs.client.AbstractClient.reportMessageHandlerProblem(AbstractClient.java:741)
>>>
>>>
>>> 0f5ba696ffe4 fns-service.log at
>>> org.apache.cxf.jaxrs.client.AbstractClient.writeBody(AbstractClient.java:470)
>>>
>>>
>>> 0f5ba696ffe4 fns-service.log at
>>> org.apache.cxf.jaxrs.client.ClientProxyImpl$BodyWriter.doWriteBody(ClientProxyImpl.java:804)
>>>
>>>
>>> 0f5ba696ffe4 fns-service.log ... 62 more
>>> 0f5ba696ffe4 fns-service.log Caused by: java.lang.NullPointerException
>>> 0f5ba696ffe4 fns-service.log at
>>> org.apache.cxf.jaxrs.impl.tl.ThreadLocalProviders.getMessageBodyWriter(ThreadLocalProviders.java:46)
>>>
>>>
>>> 0f5ba696ffe4 fns-service.log at
>>> org.glassfish.jersey.media.multipart.internal.MultiPartWriter.writeTo(MultiPartWriter.java:222)
>>>
>>>
>>> 0f5ba696ffe4 fns-service.log at
>>> org.glassfish.jersey.media.multipart.internal.MultiPartWriter.writeTo(MultiPartWriter.java:79)
>>>
>>>
>>> 0f5ba696ffe4 fns-service.log at
>>> org.apache.cxf.jaxrs.utils.JAXRSUtils.writeMessageBody(JAXRSUtils.java:1379)
>>>
>>>
>>> 0f5ba696ffe4 fns-service.log at
>>> org.apache.cxf.jaxrs.client.AbstractClient.writeBody(AbstractClient.java:456)
>>>
>>>
>>> 0f5ba696ffe4 fns-service.log ... 63 more
>>>
>>
>>
>
>
--
Sergey Beryozkin
Talend Community Coders
http://coders.talend.com/
Re: NPE with message body writer during load
Posted by Sergey Beryozkin <sb...@gmail.com>.
FYI, two more similar issues have been reported in the last couple of days.
I start suspecting it might be a WeakHashMap optimization I did. In
TomEE, the guys wanted to do a clean refresh of CXF (JAX-RS) module and
the fact the bus was holding strong references to CXF context
implementations such as UriInfoImpl, etc, was problematic (I'll need to
ask Romain for few more details on that). So 'refresh' situations can
possibly release the thread local proxies's content, just speculating
but may be it is related
So I'm on this issue now :-)
Thanks, Sergey
On 02/09/15 12:24, Sergey Beryozkin wrote:
> Hi
>
> When CXF JAX-RS injects thread-local proxies into providers/resource
> classes it keeps field and method proxy maps on the bus. I recall there
> was a problem in one of the complex setups where there were many
> endpoints with multiple Jackson instances (as opposed to a single
> instance shared between the endpoints) where somehow the provider whose
> thread local proxy was reset depending on the order of the provider
> initialization.
>
> I vaguely recall the side-effect was caused by the fact there were
> different bus instances involved too, this case is now tested in this
> context:
>
> https://github.com/apache/cxf/blob/master/systests/jaxrs/src/test/resources/jaxrs_jackson_provider/WEB-INF/beans.xml
>
>
> (one of the endpoints has a custom bus)
>
> Perhaps when a client is set up in a complex, possibly multi-threaded
> environment some side-effects with setting tl proxies are possible.
> Example if it is a single/default bus that is shared between all the
> clients then if we have multiple threads then they affect the th maps
> kept on the bus.
> Please experiment with trying to have each client instance have its own
> Bus ? (You can create a unique bus with BusFactory and then set it on
> JAXRSClientFactoryBean before creating a proxy) - if that helps than you
> can limit such a setup to a test env only unless the actual production
> setup is also multi-threaded
>
> HTH, Sergey
>
> On 02/09/15 11:33, Veit Guna wrote:
>> Hi.
>>
>> I'm using CXF 3.0.4 in client proxy mode, using my server REST
>> interfaces.
>> In addition I'm using the httpclient cxf module to workaround earlier
>> problems with concurrency
>> under load (as suggested by Sergey in a post some months ago here on
>> the mailing list).
>>
>> For Multipart handling I'm using the jersey MultiPart JAX-RS
>> MessageBodyWriter on the server
>> as well as on the client.
>>
>> Under normal circumstances all goes well. But sometimes, especially on
>> the startup-phase of the load test,
>> the CXF client proxy dies with the exception below during a Multipart
>> POST. Any clue what might cause this?
>>
>> 0f5ba696ffe4 fns-service.log Caused by:
>> org.apache.cxf.interceptor.Fault: No message body writer has been
>> found for class org.glassfish.jersey.media.multipart.MultiPart,
>> ContentType: multipart/mixed
>> 0f5ba696ffe4 fns-service.log at
>> org.apache.cxf.jaxrs.client.ClientProxyImpl$BodyWriter.doWriteBody(ClientProxyImpl.java:814)
>>
>> 0f5ba696ffe4 fns-service.log at
>> org.apache.cxf.jaxrs.client.AbstractClient$AbstractBodyWriter.handleMessage(AbstractClient.java:1042)
>>
>> 0f5ba696ffe4 fns-service.log at
>> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307)
>>
>> 0f5ba696ffe4 fns-service.log at
>> org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:624)
>>
>> 0f5ba696ffe4 fns-service.log at
>> org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:674)
>>
>> 0f5ba696ffe4 fns-service.log at
>> org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:224)
>>
>> 0f5ba696ffe4 fns-service.log at
>> com.sun.proxy.$Proxy484.create(Unknown Source)
>> 0f5ba696ffe4 fns-service.log at
>> my.class.doSomething(MyClass.java:383)
>> 0f5ba696ffe4 fns-service.log ... 54 more
>> 0f5ba696ffe4 fns-service.log Caused by:
>> javax.ws.rs.ProcessingException: No message body writer has been found
>> for class org.glassfish.jersey.media.multipart.MultiPart, ContentType:
>> multipart/mixed
>> 0f5ba696ffe4 fns-service.log at
>> org.apache.cxf.jaxrs.client.AbstractClient.reportMessageHandlerProblem(AbstractClient.java:741)
>>
>> 0f5ba696ffe4 fns-service.log at
>> org.apache.cxf.jaxrs.client.AbstractClient.writeBody(AbstractClient.java:470)
>>
>> 0f5ba696ffe4 fns-service.log at
>> org.apache.cxf.jaxrs.client.ClientProxyImpl$BodyWriter.doWriteBody(ClientProxyImpl.java:804)
>>
>> 0f5ba696ffe4 fns-service.log ... 62 more
>> 0f5ba696ffe4 fns-service.log Caused by: java.lang.NullPointerException
>> 0f5ba696ffe4 fns-service.log at
>> org.apache.cxf.jaxrs.impl.tl.ThreadLocalProviders.getMessageBodyWriter(ThreadLocalProviders.java:46)
>>
>> 0f5ba696ffe4 fns-service.log at
>> org.glassfish.jersey.media.multipart.internal.MultiPartWriter.writeTo(MultiPartWriter.java:222)
>>
>> 0f5ba696ffe4 fns-service.log at
>> org.glassfish.jersey.media.multipart.internal.MultiPartWriter.writeTo(MultiPartWriter.java:79)
>>
>> 0f5ba696ffe4 fns-service.log at
>> org.apache.cxf.jaxrs.utils.JAXRSUtils.writeMessageBody(JAXRSUtils.java:1379)
>>
>> 0f5ba696ffe4 fns-service.log at
>> org.apache.cxf.jaxrs.client.AbstractClient.writeBody(AbstractClient.java:456)
>>
>> 0f5ba696ffe4 fns-service.log ... 63 more
>>
>
>
--
Sergey Beryozkin
Talend Community Coders
http://coders.talend.com/
Re: NPE with message body writer during load
Posted by Sergey Beryozkin <sb...@gmail.com>.
Hi
When CXF JAX-RS injects thread-local proxies into providers/resource
classes it keeps field and method proxy maps on the bus. I recall there
was a problem in one of the complex setups where there were many
endpoints with multiple Jackson instances (as opposed to a single
instance shared between the endpoints) where somehow the provider whose
thread local proxy was reset depending on the order of the provider
initialization.
I vaguely recall the side-effect was caused by the fact there were
different bus instances involved too, this case is now tested in this
context:
https://github.com/apache/cxf/blob/master/systests/jaxrs/src/test/resources/jaxrs_jackson_provider/WEB-INF/beans.xml
(one of the endpoints has a custom bus)
Perhaps when a client is set up in a complex, possibly multi-threaded
environment some side-effects with setting tl proxies are possible.
Example if it is a single/default bus that is shared between all the
clients then if we have multiple threads then they affect the th maps
kept on the bus.
Please experiment with trying to have each client instance have its own
Bus ? (You can create a unique bus with BusFactory and then set it on
JAXRSClientFactoryBean before creating a proxy) - if that helps than you
can limit such a setup to a test env only unless the actual production
setup is also multi-threaded
HTH, Sergey
On 02/09/15 11:33, Veit Guna wrote:
> Hi.
>
> I'm using CXF 3.0.4 in client proxy mode, using my server REST interfaces.
> In addition I'm using the httpclient cxf module to workaround earlier problems with concurrency
> under load (as suggested by Sergey in a post some months ago here on the mailing list).
>
> For Multipart handling I'm using the jersey MultiPart JAX-RS MessageBodyWriter on the server
> as well as on the client.
>
> Under normal circumstances all goes well. But sometimes, especially on the startup-phase of the load test,
> the CXF client proxy dies with the exception below during a Multipart POST. Any clue what might cause this?
>
> 0f5ba696ffe4 fns-service.log Caused by: org.apache.cxf.interceptor.Fault: No message body writer has been found for class org.glassfish.jersey.media.multipart.MultiPart, ContentType: multipart/mixed
> 0f5ba696ffe4 fns-service.log at org.apache.cxf.jaxrs.client.ClientProxyImpl$BodyWriter.doWriteBody(ClientProxyImpl.java:814)
> 0f5ba696ffe4 fns-service.log at org.apache.cxf.jaxrs.client.AbstractClient$AbstractBodyWriter.handleMessage(AbstractClient.java:1042)
> 0f5ba696ffe4 fns-service.log at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307)
> 0f5ba696ffe4 fns-service.log at org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:624)
> 0f5ba696ffe4 fns-service.log at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:674)
> 0f5ba696ffe4 fns-service.log at org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:224)
> 0f5ba696ffe4 fns-service.log at com.sun.proxy.$Proxy484.create(Unknown Source)
> 0f5ba696ffe4 fns-service.log at my.class.doSomething(MyClass.java:383)
> 0f5ba696ffe4 fns-service.log ... 54 more
> 0f5ba696ffe4 fns-service.log Caused by: javax.ws.rs.ProcessingException: No message body writer has been found for class org.glassfish.jersey.media.multipart.MultiPart, ContentType: multipart/mixed
> 0f5ba696ffe4 fns-service.log at org.apache.cxf.jaxrs.client.AbstractClient.reportMessageHandlerProblem(AbstractClient.java:741)
> 0f5ba696ffe4 fns-service.log at org.apache.cxf.jaxrs.client.AbstractClient.writeBody(AbstractClient.java:470)
> 0f5ba696ffe4 fns-service.log at org.apache.cxf.jaxrs.client.ClientProxyImpl$BodyWriter.doWriteBody(ClientProxyImpl.java:804)
> 0f5ba696ffe4 fns-service.log ... 62 more
> 0f5ba696ffe4 fns-service.log Caused by: java.lang.NullPointerException
> 0f5ba696ffe4 fns-service.log at org.apache.cxf.jaxrs.impl.tl.ThreadLocalProviders.getMessageBodyWriter(ThreadLocalProviders.java:46)
> 0f5ba696ffe4 fns-service.log at org.glassfish.jersey.media.multipart.internal.MultiPartWriter.writeTo(MultiPartWriter.java:222)
> 0f5ba696ffe4 fns-service.log at org.glassfish.jersey.media.multipart.internal.MultiPartWriter.writeTo(MultiPartWriter.java:79)
> 0f5ba696ffe4 fns-service.log at org.apache.cxf.jaxrs.utils.JAXRSUtils.writeMessageBody(JAXRSUtils.java:1379)
> 0f5ba696ffe4 fns-service.log at org.apache.cxf.jaxrs.client.AbstractClient.writeBody(AbstractClient.java:456)
> 0f5ba696ffe4 fns-service.log ... 63 more
>
--
Sergey Beryozkin
Talend Community Coders
http://coders.talend.com/