You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@servicemix.apache.org by "Kurt Westerfeld (JIRA)" <ji...@apache.org> on 2011/02/09 04:40:57 UTC
[jira] Created: (SM-2054) MessageExchange Leak in CXF SE Proxy
Per-Request
MessageExchange Leak in CXF SE Proxy Per-Request
------------------------------------------------
Key: SM-2054
URL: https://issues.apache.org/jira/browse/SM-2054
Project: ServiceMix
Issue Type: Bug
Components: servicemix-cxf-se
Affects Versions: 4.0
Environment: Fuse ESB 4.3.03
Reporter: Kurt Westerfeld
Priority: Critical
We have discovered a memory leak with the CXFSE proxy bean, which is causing
load tests for our Servicemix application to fail over time.
The issue seems to be related to how CXF org.apache.cxf.endpoint.ClientImpl
manages something called the "response context". This class copies all
property values of the caller's message to the response context, including
the marshalled response. In the case of Servicemix CXFSE, this means that
the actual JBI MessageExchange is copied to the response context.
The leak occurs because the client maintains a thread-based map of thread
to Map<>, which is provided as the response context should the caller
choose to ask for it. In the case of the CXFSE proxy bean, most use
cases would never ask, since they are programmatically wired into the
application. Granted, each invocation of a method on the proxy can
clear the prior response context, but in our case with many threads
calling methods across a dozen proxies, soon each client is holding
onto a message exchange, including the sent and received payload over
the NMR.
In our application, we actually create a CxfSeProxyFactoryBean manually
to create proxies to some of our services within our testing framework.
For these proxies, I was able to remove some of this leak by adding an
interceptor to the CXF interceptor chain, very late in the list of
interceptors, to clear the response context:
public static class ContextCleanupInterceptor extends AbstractPhaseInterceptor<Message>
{
private Client client;
public ContextCleanupInterceptor( Client client )
{
super( Phase.POST_LOGICAL_ENDING );
this.client = client;
}
@Override
public void handleMessage( Message message ) throws Fault
{
client.getResponseContext().clear();
}
@Override
public void handleFault( Message message )
{
client.getResponseContext().clear();
}
}
And add this to the proxy. Unfortunately, there is no way to install this interceptor
decleratively that I know of, since the "client" in this case is constructed by the
bean and not exposed to spring. Many of our services have <cxfse:proxy> spring bean
wirings that we cannot intercept in this way.
Since this is such a major potential problem, I would suggest adding a similar interceptor
to the client "port" proxy within CxfSeProxyFactoryBean such that the response context
is cleaned up this way. At the very least, any property values of the response context
such as "org.apache.servicemix.*" should be removed. I have noticed that both the
request and response context at times also receive a property called "java.lang.ref.Method",
holding onto the method being called. This is really bad for the perm gen on redeploy
of an application, since this can cause prior classes to be leaked on the permanent generation
in the JVM.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Moved: (SMXCOMP-855) MessageExchange Leak in CXF SE Proxy
Per-Request
Posted by "Jean-Baptiste Onofré (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SMXCOMP-855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Baptiste Onofré moved SM-2054 to SMXCOMP-855:
--------------------------------------------------
Component/s: (was: servicemix-cxf-se)
servicemix-cxf-se
Affects Version/s: (was: 4.0)
2011.01
Key: SMXCOMP-855 (was: SM-2054)
Project: ServiceMix Components (was: ServiceMix)
> MessageExchange Leak in CXF SE Proxy Per-Request
> ------------------------------------------------
>
> Key: SMXCOMP-855
> URL: https://issues.apache.org/jira/browse/SMXCOMP-855
> Project: ServiceMix Components
> Issue Type: Bug
> Components: servicemix-cxf-se
> Affects Versions: 2011.01
> Environment: Fuse ESB 4.3.03
> Reporter: Kurt Westerfeld
> Assignee: Freeman Fang
> Priority: Critical
> Attachments: SM-2054.patch
>
>
> We have discovered a memory leak with the CXFSE proxy bean, which is causing
> load tests for our Servicemix application to fail over time.
> The issue seems to be related to how CXF org.apache.cxf.endpoint.ClientImpl
> manages something called the "response context". This class copies all
> property values of the caller's message to the response context, including
> the marshalled response. In the case of Servicemix CXFSE, this means that
> the actual JBI MessageExchange is copied to the response context.
> The leak occurs because the client maintains a thread-based map of thread
> to Map<>, which is provided as the response context should the caller
> choose to ask for it. In the case of the CXFSE proxy bean, most use
> cases would never ask, since they are programmatically wired into the
> application. Granted, each invocation of a method on the proxy can
> clear the prior response context, but in our case with many threads
> calling methods across a dozen proxies, soon each client is holding
> onto a message exchange, including the sent and received payload over
> the NMR.
> In our application, we actually create a CxfSeProxyFactoryBean manually
> to create proxies to some of our services within our testing framework.
> For these proxies, I was able to remove some of this leak by adding an
> interceptor to the CXF interceptor chain, very late in the list of
> interceptors, to clear the response context:
> public static class ContextCleanupInterceptor extends AbstractPhaseInterceptor<Message>
> {
> private Client client;
> public ContextCleanupInterceptor( Client client )
> {
> super( Phase.POST_LOGICAL_ENDING );
> this.client = client;
> }
> @Override
> public void handleMessage( Message message ) throws Fault
> {
> client.getResponseContext().clear();
> }
> @Override
> public void handleFault( Message message )
> {
> client.getResponseContext().clear();
> }
> }
> And add this to the proxy. Unfortunately, there is no way to install this interceptor
> decleratively that I know of, since the "client" in this case is constructed by the
> bean and not exposed to spring. Many of our services have <cxfse:proxy> spring bean
> wirings that we cannot intercept in this way.
> Since this is such a major potential problem, I would suggest adding a similar interceptor
> to the client "port" proxy within CxfSeProxyFactoryBean such that the response context
> is cleaned up this way. At the very least, any property values of the response context
> such as "org.apache.servicemix.*" should be removed. I have noticed that both the
> request and response context at times also receive a property called "java.lang.ref.Method",
> holding onto the method being called. This is really bad for the perm gen on redeploy
> of an application, since this can cause prior classes to be leaked on the permanent generation
> in the JVM.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (SM-2054) MessageExchange Leak in CXF SE Proxy
Per-Request
Posted by "Kurt Westerfeld (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SM-2054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kurt Westerfeld updated SM-2054:
--------------------------------
Attachment: SM-2054.patch
Adding patch for this issue.
This patch adds a new cxfse proxy property called "clearClientResponseContext", which is "true" by default. When set, an interceptor is added which clears the client response context after proxy invocation.
This patch was tested on our performance scenario and memory remained stable. Without this patch, response contexts would pile up on all threads for each proxy created, and eventually the JVM old generation would fill up.
Please look at this and include in later releases!
> MessageExchange Leak in CXF SE Proxy Per-Request
> ------------------------------------------------
>
> Key: SM-2054
> URL: https://issues.apache.org/jira/browse/SM-2054
> Project: ServiceMix
> Issue Type: Bug
> Components: servicemix-cxf-se
> Affects Versions: 4.0
> Environment: Fuse ESB 4.3.03
> Reporter: Kurt Westerfeld
> Priority: Critical
> Attachments: SM-2054.patch
>
>
> We have discovered a memory leak with the CXFSE proxy bean, which is causing
> load tests for our Servicemix application to fail over time.
> The issue seems to be related to how CXF org.apache.cxf.endpoint.ClientImpl
> manages something called the "response context". This class copies all
> property values of the caller's message to the response context, including
> the marshalled response. In the case of Servicemix CXFSE, this means that
> the actual JBI MessageExchange is copied to the response context.
> The leak occurs because the client maintains a thread-based map of thread
> to Map<>, which is provided as the response context should the caller
> choose to ask for it. In the case of the CXFSE proxy bean, most use
> cases would never ask, since they are programmatically wired into the
> application. Granted, each invocation of a method on the proxy can
> clear the prior response context, but in our case with many threads
> calling methods across a dozen proxies, soon each client is holding
> onto a message exchange, including the sent and received payload over
> the NMR.
> In our application, we actually create a CxfSeProxyFactoryBean manually
> to create proxies to some of our services within our testing framework.
> For these proxies, I was able to remove some of this leak by adding an
> interceptor to the CXF interceptor chain, very late in the list of
> interceptors, to clear the response context:
> public static class ContextCleanupInterceptor extends AbstractPhaseInterceptor<Message>
> {
> private Client client;
> public ContextCleanupInterceptor( Client client )
> {
> super( Phase.POST_LOGICAL_ENDING );
> this.client = client;
> }
> @Override
> public void handleMessage( Message message ) throws Fault
> {
> client.getResponseContext().clear();
> }
> @Override
> public void handleFault( Message message )
> {
> client.getResponseContext().clear();
> }
> }
> And add this to the proxy. Unfortunately, there is no way to install this interceptor
> decleratively that I know of, since the "client" in this case is constructed by the
> bean and not exposed to spring. Many of our services have <cxfse:proxy> spring bean
> wirings that we cannot intercept in this way.
> Since this is such a major potential problem, I would suggest adding a similar interceptor
> to the client "port" proxy within CxfSeProxyFactoryBean such that the response context
> is cleaned up this way. At the very least, any property values of the response context
> such as "org.apache.servicemix.*" should be removed. I have noticed that both the
> request and response context at times also receive a property called "java.lang.ref.Method",
> holding onto the method being called. This is really bad for the perm gen on redeploy
> of an application, since this can cause prior classes to be leaked on the permanent generation
> in the JVM.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (SMXCOMP-855) MessageExchange Leak in CXF SE Proxy
Per-Request
Posted by "Freeman Fang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SMXCOMP-855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Freeman Fang resolved SMXCOMP-855.
----------------------------------
Resolution: Fixed
Fix Version/s: 2011.02
apply patch on behalf of Kurt Westerfeld with thanks
http://svn.apache.org/viewvc?rev=1072880&view=rev
> MessageExchange Leak in CXF SE Proxy Per-Request
> ------------------------------------------------
>
> Key: SMXCOMP-855
> URL: https://issues.apache.org/jira/browse/SMXCOMP-855
> Project: ServiceMix Components
> Issue Type: Bug
> Components: servicemix-cxf-se
> Affects Versions: 2011.01
> Environment: Fuse ESB 4.3.03
> Reporter: Kurt Westerfeld
> Assignee: Freeman Fang
> Priority: Critical
> Fix For: 2011.02
>
> Attachments: SM-2054.patch
>
>
> We have discovered a memory leak with the CXFSE proxy bean, which is causing
> load tests for our Servicemix application to fail over time.
> The issue seems to be related to how CXF org.apache.cxf.endpoint.ClientImpl
> manages something called the "response context". This class copies all
> property values of the caller's message to the response context, including
> the marshalled response. In the case of Servicemix CXFSE, this means that
> the actual JBI MessageExchange is copied to the response context.
> The leak occurs because the client maintains a thread-based map of thread
> to Map<>, which is provided as the response context should the caller
> choose to ask for it. In the case of the CXFSE proxy bean, most use
> cases would never ask, since they are programmatically wired into the
> application. Granted, each invocation of a method on the proxy can
> clear the prior response context, but in our case with many threads
> calling methods across a dozen proxies, soon each client is holding
> onto a message exchange, including the sent and received payload over
> the NMR.
> In our application, we actually create a CxfSeProxyFactoryBean manually
> to create proxies to some of our services within our testing framework.
> For these proxies, I was able to remove some of this leak by adding an
> interceptor to the CXF interceptor chain, very late in the list of
> interceptors, to clear the response context:
> public static class ContextCleanupInterceptor extends AbstractPhaseInterceptor<Message>
> {
> private Client client;
> public ContextCleanupInterceptor( Client client )
> {
> super( Phase.POST_LOGICAL_ENDING );
> this.client = client;
> }
> @Override
> public void handleMessage( Message message ) throws Fault
> {
> client.getResponseContext().clear();
> }
> @Override
> public void handleFault( Message message )
> {
> client.getResponseContext().clear();
> }
> }
> And add this to the proxy. Unfortunately, there is no way to install this interceptor
> decleratively that I know of, since the "client" in this case is constructed by the
> bean and not exposed to spring. Many of our services have <cxfse:proxy> spring bean
> wirings that we cannot intercept in this way.
> Since this is such a major potential problem, I would suggest adding a similar interceptor
> to the client "port" proxy within CxfSeProxyFactoryBean such that the response context
> is cleaned up this way. At the very least, any property values of the response context
> such as "org.apache.servicemix.*" should be removed. I have noticed that both the
> request and response context at times also receive a property called "java.lang.ref.Method",
> holding onto the method being called. This is really bad for the perm gen on redeploy
> of an application, since this can cause prior classes to be leaked on the permanent generation
> in the JVM.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Assigned: (SM-2054) MessageExchange Leak in CXF SE Proxy
Per-Request
Posted by "Freeman Fang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SM-2054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Freeman Fang reassigned SM-2054:
--------------------------------
Assignee: Freeman Fang
> MessageExchange Leak in CXF SE Proxy Per-Request
> ------------------------------------------------
>
> Key: SM-2054
> URL: https://issues.apache.org/jira/browse/SM-2054
> Project: ServiceMix
> Issue Type: Bug
> Components: servicemix-cxf-se
> Affects Versions: 4.0
> Environment: Fuse ESB 4.3.03
> Reporter: Kurt Westerfeld
> Assignee: Freeman Fang
> Priority: Critical
> Attachments: SM-2054.patch
>
>
> We have discovered a memory leak with the CXFSE proxy bean, which is causing
> load tests for our Servicemix application to fail over time.
> The issue seems to be related to how CXF org.apache.cxf.endpoint.ClientImpl
> manages something called the "response context". This class copies all
> property values of the caller's message to the response context, including
> the marshalled response. In the case of Servicemix CXFSE, this means that
> the actual JBI MessageExchange is copied to the response context.
> The leak occurs because the client maintains a thread-based map of thread
> to Map<>, which is provided as the response context should the caller
> choose to ask for it. In the case of the CXFSE proxy bean, most use
> cases would never ask, since they are programmatically wired into the
> application. Granted, each invocation of a method on the proxy can
> clear the prior response context, but in our case with many threads
> calling methods across a dozen proxies, soon each client is holding
> onto a message exchange, including the sent and received payload over
> the NMR.
> In our application, we actually create a CxfSeProxyFactoryBean manually
> to create proxies to some of our services within our testing framework.
> For these proxies, I was able to remove some of this leak by adding an
> interceptor to the CXF interceptor chain, very late in the list of
> interceptors, to clear the response context:
> public static class ContextCleanupInterceptor extends AbstractPhaseInterceptor<Message>
> {
> private Client client;
> public ContextCleanupInterceptor( Client client )
> {
> super( Phase.POST_LOGICAL_ENDING );
> this.client = client;
> }
> @Override
> public void handleMessage( Message message ) throws Fault
> {
> client.getResponseContext().clear();
> }
> @Override
> public void handleFault( Message message )
> {
> client.getResponseContext().clear();
> }
> }
> And add this to the proxy. Unfortunately, there is no way to install this interceptor
> decleratively that I know of, since the "client" in this case is constructed by the
> bean and not exposed to spring. Many of our services have <cxfse:proxy> spring bean
> wirings that we cannot intercept in this way.
> Since this is such a major potential problem, I would suggest adding a similar interceptor
> to the client "port" proxy within CxfSeProxyFactoryBean such that the response context
> is cleaned up this way. At the very least, any property values of the response context
> such as "org.apache.servicemix.*" should be removed. I have noticed that both the
> request and response context at times also receive a property called "java.lang.ref.Method",
> holding onto the method being called. This is really bad for the perm gen on redeploy
> of an application, since this can cause prior classes to be leaked on the permanent generation
> in the JVM.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Work started: (SMXCOMP-855) MessageExchange Leak in CXF SE
Proxy Per-Request
Posted by "Freeman Fang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SMXCOMP-855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on SMXCOMP-855 started by Freeman Fang.
> MessageExchange Leak in CXF SE Proxy Per-Request
> ------------------------------------------------
>
> Key: SMXCOMP-855
> URL: https://issues.apache.org/jira/browse/SMXCOMP-855
> Project: ServiceMix Components
> Issue Type: Bug
> Components: servicemix-cxf-se
> Affects Versions: 2011.01
> Environment: Fuse ESB 4.3.03
> Reporter: Kurt Westerfeld
> Assignee: Freeman Fang
> Priority: Critical
> Attachments: SM-2054.patch
>
>
> We have discovered a memory leak with the CXFSE proxy bean, which is causing
> load tests for our Servicemix application to fail over time.
> The issue seems to be related to how CXF org.apache.cxf.endpoint.ClientImpl
> manages something called the "response context". This class copies all
> property values of the caller's message to the response context, including
> the marshalled response. In the case of Servicemix CXFSE, this means that
> the actual JBI MessageExchange is copied to the response context.
> The leak occurs because the client maintains a thread-based map of thread
> to Map<>, which is provided as the response context should the caller
> choose to ask for it. In the case of the CXFSE proxy bean, most use
> cases would never ask, since they are programmatically wired into the
> application. Granted, each invocation of a method on the proxy can
> clear the prior response context, but in our case with many threads
> calling methods across a dozen proxies, soon each client is holding
> onto a message exchange, including the sent and received payload over
> the NMR.
> In our application, we actually create a CxfSeProxyFactoryBean manually
> to create proxies to some of our services within our testing framework.
> For these proxies, I was able to remove some of this leak by adding an
> interceptor to the CXF interceptor chain, very late in the list of
> interceptors, to clear the response context:
> public static class ContextCleanupInterceptor extends AbstractPhaseInterceptor<Message>
> {
> private Client client;
> public ContextCleanupInterceptor( Client client )
> {
> super( Phase.POST_LOGICAL_ENDING );
> this.client = client;
> }
> @Override
> public void handleMessage( Message message ) throws Fault
> {
> client.getResponseContext().clear();
> }
> @Override
> public void handleFault( Message message )
> {
> client.getResponseContext().clear();
> }
> }
> And add this to the proxy. Unfortunately, there is no way to install this interceptor
> decleratively that I know of, since the "client" in this case is constructed by the
> bean and not exposed to spring. Many of our services have <cxfse:proxy> spring bean
> wirings that we cannot intercept in this way.
> Since this is such a major potential problem, I would suggest adding a similar interceptor
> to the client "port" proxy within CxfSeProxyFactoryBean such that the response context
> is cleaned up this way. At the very least, any property values of the response context
> such as "org.apache.servicemix.*" should be removed. I have noticed that both the
> request and response context at times also receive a property called "java.lang.ref.Method",
> holding onto the method being called. This is really bad for the perm gen on redeploy
> of an application, since this can cause prior classes to be leaked on the permanent generation
> in the JVM.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (SM-2054) MessageExchange Leak in CXF SE Proxy
Per-Request
Posted by "Kurt Westerfeld (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SM-2054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992325#comment-12992325 ]
Kurt Westerfeld commented on SM-2054:
-------------------------------------
Note: the interceptor here is conceptual-only. There are likely reasons (for example) the method and other data is placed into the response context. However, some of this data is obviously transient, such as the exchange and should be removed after the invocation.
> MessageExchange Leak in CXF SE Proxy Per-Request
> ------------------------------------------------
>
> Key: SM-2054
> URL: https://issues.apache.org/jira/browse/SM-2054
> Project: ServiceMix
> Issue Type: Bug
> Components: servicemix-cxf-se
> Affects Versions: 4.0
> Environment: Fuse ESB 4.3.03
> Reporter: Kurt Westerfeld
> Priority: Critical
>
> We have discovered a memory leak with the CXFSE proxy bean, which is causing
> load tests for our Servicemix application to fail over time.
> The issue seems to be related to how CXF org.apache.cxf.endpoint.ClientImpl
> manages something called the "response context". This class copies all
> property values of the caller's message to the response context, including
> the marshalled response. In the case of Servicemix CXFSE, this means that
> the actual JBI MessageExchange is copied to the response context.
> The leak occurs because the client maintains a thread-based map of thread
> to Map<>, which is provided as the response context should the caller
> choose to ask for it. In the case of the CXFSE proxy bean, most use
> cases would never ask, since they are programmatically wired into the
> application. Granted, each invocation of a method on the proxy can
> clear the prior response context, but in our case with many threads
> calling methods across a dozen proxies, soon each client is holding
> onto a message exchange, including the sent and received payload over
> the NMR.
> In our application, we actually create a CxfSeProxyFactoryBean manually
> to create proxies to some of our services within our testing framework.
> For these proxies, I was able to remove some of this leak by adding an
> interceptor to the CXF interceptor chain, very late in the list of
> interceptors, to clear the response context:
> public static class ContextCleanupInterceptor extends AbstractPhaseInterceptor<Message>
> {
> private Client client;
> public ContextCleanupInterceptor( Client client )
> {
> super( Phase.POST_LOGICAL_ENDING );
> this.client = client;
> }
> @Override
> public void handleMessage( Message message ) throws Fault
> {
> client.getResponseContext().clear();
> }
> @Override
> public void handleFault( Message message )
> {
> client.getResponseContext().clear();
> }
> }
> And add this to the proxy. Unfortunately, there is no way to install this interceptor
> decleratively that I know of, since the "client" in this case is constructed by the
> bean and not exposed to spring. Many of our services have <cxfse:proxy> spring bean
> wirings that we cannot intercept in this way.
> Since this is such a major potential problem, I would suggest adding a similar interceptor
> to the client "port" proxy within CxfSeProxyFactoryBean such that the response context
> is cleaned up this way. At the very least, any property values of the response context
> such as "org.apache.servicemix.*" should be removed. I have noticed that both the
> request and response context at times also receive a property called "java.lang.ref.Method",
> holding onto the method being called. This is really bad for the perm gen on redeploy
> of an application, since this can cause prior classes to be leaked on the permanent generation
> in the JVM.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira