You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cxf.apache.org by anoopPrasad <an...@huawei.com> on 2008/10/10 14:09:12 UTC
Re: Memory leak in CXF HTTP transport in Solaris ?
Dear Bharat,
I have observed the same behavior. Allow me to present the details u seek.
Server:
Created a Sample HTTP Service with CXF 2.0.3(publishing Greeter interface
provided in sample )
Service was published using :
Endpoint ep = Endpoint.publish("http://localhost:9000/Greeter", new
GreeterImpl());
and GreeterImpl has "String greetMe(String me)" which just returns a string.
Server is running on a Solaris 10 machine with java version "1.5.0_14"
Server does nothing else.
The libs in classpath are
-classpath
lib:/lib/cxf/asm-2.2.3.jar:lib/cxf/commons-logging-1.1.jar:lib/cxf/cxf-2.0.3-incubator.jar:lib/cxf/geronimo-activation_1.1_spec-1.
0-M1.jar:lib/cxf/geronimo-annotation_1.0_spec-1.1.jar:lib/cxf/geronimo-javamail_1.4_spec-1.0-M1.jar:lib/cxf/geronimo-jms_1.1_spec-
1.1.jar:lib/cxf/geronimo-servlet_2.5_spec-1.1-M1.jar:lib/cxf/geronimo-ws-metadata_2.0_spec-1.1.1.jar:lib/cxf/jaxb-api-2.0.jar:lib/
cxf/jaxb-impl-2.0.5.jar:lib/cxf/jaxb-xjc-2.0.jar:lib/cxf/jaxen-1.1.jar:lib/cxf/jaxws-api-2.0.jar:lib/cxf/jetty-6.1.12.rc2.jar:lib/
cxf/jetty-util-6.1.12.rc2.jar:lib/cxf/jra-1.0-alpha-4.jar:lib/cxf/js-1.6R5.jar:lib/cxf/neethi-2.0.2.jar:lib/cxf/saaj-api-1.3.jar:l
ib/cxf/saaj-impl-1.3.jar:lib/cxf/slf4j-api-1.3.1.jar:lib/cxf/slf4j-jdk14-1.3.1.jar:lib/cxf/spring-beans-2.0.6.jar:lib/cxf/spring-c
ontext-2.0.6.jar:lib/cxf/spring-core-2.0.6.jar:lib/cxf/stax-api-1.0.1.jar:lib/cxf/stax-utils-20060502.jar:lib/cxf/wsdl4j-1.6.1.jar
:lib/cxf/wss4j-1.5.1.jar:lib/cxf/wstx-asl-3.2.1.jar:lib/cxf/xalan-2.7.0.jar:lib/cxf/xbean-2.2.0.jar:lib/cxf/xml-apis-1.3.02.jar:li
b/cxf/xml-resolver-1.2.jar:lib/cxf/XmlSchema-1.3.2.jar:lib/activemq-core-5.0.0.jar:lib/geronimo-j2ee-management_1.0_spec-1.0.jar
Server is run with JVM parameters "-Xmx129m -XX:MaxPermSize=128m ". No other
tuning parameters added in server[However, a lot of other tuning were tried
(like, -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled
-XX:+CMSPermGenSweepingEnabled) and found that the result is not favorable]
Client:
Client written in CXF, sends requests continuously to the server.(infinite
loop)
Behavior:
The memory of the server continously increases from ~90M to ~350M which
never comes down.
But through jconsole we know that the Heap, or non-heap memory hasn't
increased much.
Below table contains mem usage details over a period of 1Hr (800,000
Request/Response)
prstat Heap Non-Heap CMS PermGen
Total RSS Used Comitd Max Used Comitd Max Used Comitd Max
176 103 3 36 120 16 30 163 14 28 131
184 124 2 35 120 20 24 163 16 20 131
184 129 2 13 120 22 25 163 16 20 131
200 145 2 19 120 22 25 163 16 20 131
204 149 2 45 120 22 25 163 16 20 131
253 207 5 18 120 19 20 163 14 16 131
Same test was carried out with CXF Version 2.1.2 and it behaves the same
way.
OS : Solaris 10
root@SUN50 # psrinfo -v
Status of virtual processor 0 as of: 03/31/2000 22:38:54
on-line since 03/25/2000 04:24:44.
The sparcv9 processor operates at 1593 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 1 as of: 03/31/2000 22:38:54
on-line since 03/25/2000 04:24:44.
The sparcv9 processor operates at 1593 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 2 as of: 03/31/2000 22:38:54
on-line since 03/25/2000 04:24:44.
The sparcv9 processor operates at 1593 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 3 as of: 03/31/2000 22:38:54
on-line since 03/25/2000 04:24:43.
The sparcv9 processor operates at 1593 MHz,
and has a sparcv9 floating point processor.
root@SUN50 # prtconf | grep Memory
Memory size: 8192 Megabytes
Please help us identify the reason for this mem increase.
regards
anoopPrasad
bharath-5 wrote:
>
> Hi Hubert,
>
> Is the leak replicable? Is is that the memory was just high or it was a
> leak? Did you see an OOM?
> Will be of great help if you can send the test case.
> Also let me know which CXF version you are using?
>
> Thanks,
> Bharath
> IBM - India Software Labs
>
> http://thoughts.bharathganesh.com
>
> On Tue, Sep 23, 2008 at 5:36 AM, Sky-Tiger
> <da...@gmail.com>wrote:
>
>>
>> Dear Dan,
>> I wrote a very simple application using CXF, just like Helloworld.
>> Firstly, I run the application using JMS as tranport layer,
>> erverything is ok.
>> But when i change the JMS with HTTP, I find the memory will
>> increase
>> slowly. And such increasing can not be checked by Jconsole.
>> Are there some issues in CXF or Jetty for memory leak?
>>
>> Regards
>>
>> Hubert.
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Memory-leak--tp19619011p19619011.html
>> Sent from the cxf-dev mailing list archive at Nabble.com.
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Memory-leak--tp19619011p19917037.html
Sent from the cxf-dev mailing list archive at Nabble.com.
Re: Memory leak in CXF HTTP transport in Solaris ?
Posted by Bharath Ganesh <bh...@gmail.com>.
I have verified the same test on Windows with the same JDK. My memory usage
looks stable.
Could anyone try on Solaris?
Anoop,
Have you tried the test on any non-solaris OS?
//The memory of the server continously increases from ~90M to ~350M which
never comes down.
How did you check this?
Thanks,
Bharath
On Fri, Oct 10, 2008 at 5:39 PM, anoopPrasad <an...@huawei.com> wrote:
>
> Dear Bharat,
>
> I have observed the same behavior. Allow me to present the details u seek.
>
> Server:
> Created a Sample HTTP Service with CXF 2.0.3(publishing Greeter interface
> provided in sample )
> Service was published using :
> Endpoint ep = Endpoint.publish("http://localhost:9000/Greeter", new
> GreeterImpl());
> and GreeterImpl has "String greetMe(String me)" which just returns a
> string.
> Server is running on a Solaris 10 machine with java version "1.5.0_14"
> Server does nothing else.
>
> The libs in classpath are
> -classpath
>
> lib:/lib/cxf/asm-2.2.3.jar:lib/cxf/commons-logging-1.1.jar:lib/cxf/cxf-2.0.3-incubator.jar:lib/cxf/geronimo-activation_1.1_spec-1.
>
> 0-M1.jar:lib/cxf/geronimo-annotation_1.0_spec-1.1.jar:lib/cxf/geronimo-javamail_1.4_spec-1.0-M1.jar:lib/cxf/geronimo-jms_1.1_spec-
>
> 1.1.jar:lib/cxf/geronimo-servlet_2.5_spec-1.1-M1.jar:lib/cxf/geronimo-ws-metadata_2.0_spec-1.1.1.jar:lib/cxf/jaxb-api-2.0.jar:lib/
>
> cxf/jaxb-impl-2.0.5.jar:lib/cxf/jaxb-xjc-2.0.jar:lib/cxf/jaxen-1.1.jar:lib/cxf/jaxws-api-2.0.jar:lib/cxf/jetty-6.1.12.rc2.jar:lib/
>
> cxf/jetty-util-6.1.12.rc2.jar:lib/cxf/jra-1.0-alpha-4.jar:lib/cxf/js-1.6R5.jar:lib/cxf/neethi-2.0.2.jar:lib/cxf/saaj-api-1.3.jar:l
>
> ib/cxf/saaj-impl-1.3.jar:lib/cxf/slf4j-api-1.3.1.jar:lib/cxf/slf4j-jdk14-1.3.1.jar:lib/cxf/spring-beans-2.0.6.jar:lib/cxf/spring-c
>
> ontext-2.0.6.jar:lib/cxf/spring-core-2.0.6.jar:lib/cxf/stax-api-1.0.1.jar:lib/cxf/stax-utils-20060502.jar:lib/cxf/wsdl4j-1.6.1.jar
>
> :lib/cxf/wss4j-1.5.1.jar:lib/cxf/wstx-asl-3.2.1.jar:lib/cxf/xalan-2.7.0.jar:lib/cxf/xbean-2.2.0.jar:lib/cxf/xml-apis-1.3.02.jar:li
>
> b/cxf/xml-resolver-1.2.jar:lib/cxf/XmlSchema-1.3.2.jar:lib/activemq-core-5.0.0.jar:lib/geronimo-j2ee-management_1.0_spec-1.0.jar
>
> Server is run with JVM parameters "-Xmx129m -XX:MaxPermSize=128m ". No
> other
> tuning parameters added in server[However, a lot of other tuning were tried
> (like, -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled
> -XX:+CMSPermGenSweepingEnabled) and found that the result is not favorable]
>
> Client:
> Client written in CXF, sends requests continuously to the server.(infinite
> loop)
>
> Behavior:
> The memory of the server continously increases from ~90M to ~350M which
> never comes down.
> But through jconsole we know that the Heap, or non-heap memory hasn't
> increased much.
> Below table contains mem usage details over a period of 1Hr (800,000
> Request/Response)
>
> prstat Heap Non-Heap
> CMS PermGen
> Total RSS Used Comitd Max Used Comitd Max
> Used Comitd Max
> 176 103 3 36 120 16 30 163
> 14 28 131
> 184 124 2 35 120 20 24 163
> 16 20 131
> 184 129 2 13 120 22 25 163
> 16 20 131
> 200 145 2 19 120 22 25 163
> 16 20 131
> 204 149 2 45 120 22 25 163
> 16 20 131
> 253 207 5 18 120 19 20 163
> 14 16 131
>
> Same test was carried out with CXF Version 2.1.2 and it behaves the same
> way.
>
>
>
> OS : Solaris 10
> root@SUN50 # psrinfo -v
> Status of virtual processor 0 as of: 03/31/2000 22:38:54
> on-line since 03/25/2000 04:24:44.
> The sparcv9 processor operates at 1593 MHz,
> and has a sparcv9 floating point processor.
> Status of virtual processor 1 as of: 03/31/2000 22:38:54
> on-line since 03/25/2000 04:24:44.
> The sparcv9 processor operates at 1593 MHz,
> and has a sparcv9 floating point processor.
> Status of virtual processor 2 as of: 03/31/2000 22:38:54
> on-line since 03/25/2000 04:24:44.
> The sparcv9 processor operates at 1593 MHz,
> and has a sparcv9 floating point processor.
> Status of virtual processor 3 as of: 03/31/2000 22:38:54
> on-line since 03/25/2000 04:24:43.
> The sparcv9 processor operates at 1593 MHz,
> and has a sparcv9 floating point processor.
>
> root@SUN50 # prtconf | grep Memory
> Memory size: 8192 Megabytes
>
>
> Please help us identify the reason for this mem increase.
>
> regards
> anoopPrasad
>
>
>
> bharath-5 wrote:
> >
> > Hi Hubert,
> >
> > Is the leak replicable? Is is that the memory was just high or it was a
> > leak? Did you see an OOM?
> > Will be of great help if you can send the test case.
> > Also let me know which CXF version you are using?
> >
> > Thanks,
> > Bharath
> > IBM - India Software Labs
> >
> > http://thoughts.bharathganesh.com
> >
> > On Tue, Sep 23, 2008 at 5:36 AM, Sky-Tiger
> > <da...@gmail.com>wrote:
> >
> >>
> >> Dear Dan,
> >> I wrote a very simple application using CXF, just like Helloworld.
> >> Firstly, I run the application using JMS as tranport layer,
> >> erverything is ok.
> >> But when i change the JMS with HTTP, I find the memory will
> >> increase
> >> slowly. And such increasing can not be checked by Jconsole.
> >> Are there some issues in CXF or Jetty for memory leak?
> >>
> >> Regards
> >>
> >> Hubert.
> >>
> >> --
> >> View this message in context:
> >> http://www.nabble.com/Memory-leak--tp19619011p19619011.html
> >> Sent from the cxf-dev mailing list archive at Nabble.com.
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Memory-leak--tp19619011p19917037.html
> Sent from the cxf-dev mailing list archive at Nabble.com.
>
>
Re: Memory leak in CXF HTTP transport in Solaris ?
Posted by Bharath Ganesh <bh...@gmail.com>.
Yes running without NIO might narrow down..
I saw a related bug[1], but closed saying Not replicable.
1. http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6208845
On Wed, Oct 15, 2008 at 12:05 AM, Daniel Kulp <dk...@apache.org> wrote:
> On Tuesday 14 October 2008 2:19:44 pm anoopPrasad wrote:
> > Dear Dan/Bharat,
> >
> > I have checked Heap,non-Heap, and perm Gen Memory usage of the process
> > through JCOnsole and OptimizeIT. And like Dan observed, its stable, more
> or
> > less.
> > (Like in the report in the first post)
> >
> > But still the OS (Solaris) reports a Resident Memory increase as well as
> > the swap memory increase.
> >
> > From the "pmap" report I have observed that after a number of requests a
> > lot of the anonymus [anon] memory blocks are allocates, which is never
> > reclaimed.
>
> I have NO idea how to debug something down there. If it's outside the
> java
> heaps, it's down in native code. :-(
>
>
>
> > In the beginning we were also doubting PermGen space and Jetty server.
> But
> > it turns out that PermGen allocation is fine , no considerable
> > increase(~3M) at all.
>
> Right. That's what I was seeing as well.
>
>
> > About Jetty we are still doubtful.Since JMS transport is fine, it has to
> be
> > either Jetty problem or
> > the way servlets are handled in CXF Code.
>
> Well, you could TRY building a war, deploying in tomcat, and trying that.
> 95% of the http handling code is shared between the servlet version and the
> jetty version. That said, if there was an issue there, I would expect it
> to
> show up as objects on the java heap.
>
> One thing you COULD try is modify the
> org.apache.cxf.transport.http_jetty.JettyHTTPServerEngine class and change
> the getHTTPConnectorFactory() call stuff to return the non-NIO version of
> the
> connector. It could be direct nio buffers leaking or something.
>
>
> > Could you please have a look at the following post and tell me whether it
> > applies to CXF.
> > http://blogs.sun.com/fkieviet/entry/classloader_leaks_the_dreaded_java
>
> I don't think so. In the standalone case, nothing is being
> deployed/undeployed/etc.... so classes shouldn't ever need to be unloaded.
>
> Dan
>
>
>
> > >
> >
> > If it's in the process memory space, that's probably a bug in either
> Jetty
> > (nio stuff it does) or in the JDK itself. Nothing we can do about either
> > of those.
> > If we locate the problem, we can try to fix it.
> >
> >
> > Thank you very much.
> >
> > regards
> > anoopPrasad
>
>
>
> --
> Daniel Kulp
> dkulp@apache.org
> http://dankulp.com/blog
>
Re: Memory leak in CXF HTTP transport in Solaris ?
Posted by Daniel Kulp <dk...@apache.org>.
On Tuesday 14 October 2008 2:19:44 pm anoopPrasad wrote:
> Dear Dan/Bharat,
>
> I have checked Heap,non-Heap, and perm Gen Memory usage of the process
> through JCOnsole and OptimizeIT. And like Dan observed, its stable, more or
> less.
> (Like in the report in the first post)
>
> But still the OS (Solaris) reports a Resident Memory increase as well as
> the swap memory increase.
>
> From the "pmap" report I have observed that after a number of requests a
> lot of the anonymus [anon] memory blocks are allocates, which is never
> reclaimed.
I have NO idea how to debug something down there. If it's outside the java
heaps, it's down in native code. :-(
> In the beginning we were also doubting PermGen space and Jetty server. But
> it turns out that PermGen allocation is fine , no considerable
> increase(~3M) at all.
Right. That's what I was seeing as well.
> About Jetty we are still doubtful.Since JMS transport is fine, it has to be
> either Jetty problem or
> the way servlets are handled in CXF Code.
Well, you could TRY building a war, deploying in tomcat, and trying that.
95% of the http handling code is shared between the servlet version and the
jetty version. That said, if there was an issue there, I would expect it to
show up as objects on the java heap.
One thing you COULD try is modify the
org.apache.cxf.transport.http_jetty.JettyHTTPServerEngine class and change
the getHTTPConnectorFactory() call stuff to return the non-NIO version of the
connector. It could be direct nio buffers leaking or something.
> Could you please have a look at the following post and tell me whether it
> applies to CXF.
> http://blogs.sun.com/fkieviet/entry/classloader_leaks_the_dreaded_java
I don't think so. In the standalone case, nothing is being
deployed/undeployed/etc.... so classes shouldn't ever need to be unloaded.
Dan
> >
>
> If it's in the process memory space, that's probably a bug in either Jetty
> (nio stuff it does) or in the JDK itself. Nothing we can do about either
> of those.
> If we locate the problem, we can try to fix it.
>
>
> Thank you very much.
>
> regards
> anoopPrasad
--
Daniel Kulp
dkulp@apache.org
http://dankulp.com/blog
Re: Memory leak in CXF HTTP transport in Solaris ?
Posted by anoopPrasad <an...@huawei.com>.
Adding the Link again from the last post:
Could you please have a look at the following post and tell me whether it
applies to CXF.
http://blogs.sun.com/fkieviet/entry/classloader_leaks_the_dreaded_java
~
anoopPrasad
anoopPrasad wrote:
>
> Dear Dan/Bharat,
>
> I have checked Heap,non-Heap, and perm Gen Memory usage of the process
> through JCOnsole and OptimizeIT. And like Dan observed, its stable, more
> or less.
> (Like in the report in the first post)
>
> But still the OS (Solaris) reports a Resident Memory increase as well as
> the swap memory increase.
>
> From the "pmap" report I have observed that after a number of requests a
> lot of the anonymus [anon] memory blocks are allocates, which is never
> reclaimed.
>
> In the beginning we were also doubting PermGen space and Jetty server. But
> it turns out that PermGen allocation is fine , no considerable
> increase(~3M) at all.
>
> About Jetty we are still doubtful.Since JMS transport is fine, it has to
> be either Jetty problem or
> the way servlets are handled in CXF Code.
>
> Could you please have a look at the following post and tell me whether it
> applies to CXF.
> http://blogs.sun.com/fkieviet/entry/classloader_leaks_the_dreaded_java
>
>
> If it's in the process memory space, that's probably a bug in either Jetty
> (nio stuff it does) or in the JDK itself. Nothing we can do about either
> of
> those.
> If we locate the problem, we can try to fix it.
>
>
> Thank you very much.
>
> regards
> anoopPrasad
>
>
--
View this message in context: http://www.nabble.com/Memory-leak--tp19619011p19979358.html
Sent from the cxf-dev mailing list archive at Nabble.com.
Re: Memory leak in CXF HTTP transport in Solaris ?
Posted by anoopPrasad <an...@huawei.com>.
Dear Dan/Bharat,
I have checked Heap,non-Heap, and perm Gen Memory usage of the process
through JCOnsole and OptimizeIT. And like Dan observed, its stable, more or
less.
(Like in the report in the first post)
But still the OS (Solaris) reports a Resident Memory increase as well as the
swap memory increase.
>From the "pmap" report I have observed that after a number of requests a
lot of the anonymus [anon] memory blocks are allocates, which is never
reclaimed.
In the beginning we were also doubting PermGen space and Jetty server. But
it turns out that PermGen allocation is fine , no considerable increase(~3M)
at all.
About Jetty we are still doubtful.Since JMS transport is fine, it has to be
either Jetty problem or
the way servlets are handled in CXF Code.
Could you please have a look at the following post and tell me whether it
applies to CXF.
http://blogs.sun.com/fkieviet/entry/classloader_leaks_the_dreaded_java
If it's in the process memory space, that's probably a bug in either Jetty
(nio stuff it does) or in the JDK itself. Nothing we can do about either of
those.
If we locate the problem, we can try to fix it.
Thank you very much.
regards
anoopPrasad
--
View this message in context: http://www.nabble.com/Memory-leak--tp19619011p19979302.html
Sent from the cxf-dev mailing list archive at Nabble.com.