You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-dev@axis.apache.org by Aleksander Slominski <as...@cs.indiana.edu> on 2004/06/14 19:46:32 UTC
severe memory leakage [Re: out of memory exception for current AXIS
version from CVS
hi,
with tomcat service running on JVM maxed memory to 1GB
setenv JAVA_OPTS -Xmx1024m
./catalina.sh start
after running full benchmark :
/opt/j2sdk/bin/java -Dmachine.name=LINUX_JDK142 -Dserver.name=AXIS_1_2
soap_bench.BenchClient http://server:8080/axis/services/Benchmark1
200000 aa 10,100,1000,5000,10000,25000,50000,75000,100000
that took few hours to finish and here is what i got next day few hours
after test was finished:
106 processes: 105 sleeping, 1 running, 0 zombie, 0 stopped
CPU states: 0.0% user, 1.6% system, 0.0% nice, 98.4% idle
Mem: 1033500K total, 1019936K used, 13564K free, 14904K buffers
Swap: 4112560K total, 1580800K used, 2531760K free, 37460K cached
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
2453 aslom 17 0 1040 1040 804 R 6.2 0.1 0:04 top
1762 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:06 java
1763 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:00 java
1764 aslom 9 0 983M 347M 75840 S 0.0 34.4 7:06 java
1765 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:07 java
1766 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:15 java
1767 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:00 java
1768 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:00 java
1769 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:03 java
1770 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:32 java
1772 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:00 java
1775 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:00 java
1776 aslom 9 0 983M 347M 75840 S 0.0 34.4 2:14 java
1777 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:00 java
so that strongly suggest that there is severe memory leak in AXIS Java
if high (913MB!) memoery usage stays for hours not garbage collected ...
alek
Aleksander Slominski wrote:
> hi,
>
> there was some discussion abot memory problems and it seems i have a
> way to experiment and maybe find how to fix it (i did not see Peter
> Molettiere actually posting soemthing to JIRA on how to reproduce his
> problems? AFAICT only other open issues are
> http://nagoya.apache.org/jira/browse/AXIS-1270
> http://nagoya.apache.org/jira/browse/AXIS-1202
> http://nagoya.apache.org/jira/browse/AXIS-1095)
>
> here is how to reproduce this OME (got it on linux but probably is the
> same on any OS) with the latest AXIS from CVS:
>
> compile current axis and generate war file (ant war - wil take some
> time ...)
> get tomcat 4.1.30, deploy current axis.war, modify to disable
> multi-ref in WEB-INF\server-config.wsdd
> set <parameter name="sendXsiTypes" value="false"/> and <parameter
> name="sendMultiRefs" value="false"/>
>
> then download source code and compile from
> http://www.extreme.indiana.edu/~aslom/bnp/wsperf/axis_java/
> <>
> after service is deployed start tomcat (running JDK 1.4.2on linux
> without any chnages to -Xmx i.e. /opt/j2sdk/bin/java
> -Djava.endorsed.dirs=/home/aslom/Forge/tomcat/common/endorsed
> -classpath
> /opt/j2sdk/lib/tools.jar:/home/aslom/Forge/tomcat/bin/bootstrap.jar
> -Dcatalina.base=/home/aslom/Forge/tomcat
> -Dcatalina.home=/home/aslom/Forge/tomcat
> -Djava.io.tmpdir=/home/aslom/Forge/tomcat/temp
> org.apache.catalina.startup.Bootstrap start)
>
> and run client with it:
> java -Dmachine.name=LINUX_JDK142 -Dserver.name=AXIS_1_2
> soap_bench.BenchClient http://server:8080/axis/services/Benchmark1
> 200000 ev 10,100,1000,5000,10000,25000,50000,75000,100000
>
> the OME happens after running test two times, here are details:
>
> nimitz:~/Forge/driver> java -Dmachine.name=LINUX_JDK142
> -Dserver.name=AXIS_1_2 soap_bench.BenchClient
> http://server:8080/axis/services/Benchmark1 200000 ev
> 10,100,1000,5000,10000,25000,50000,75000,100000
> size=10
> invoking 20000 times for test v arraysSize=10
> Exception in thread "main"
> xsul.invoker.DynamicInfosetInvokerException: HTTP server error 500
> Internal Server E
> rror body="
> <html><head><title>Apache Tomcat/4.1.30 - Error
> report</title><STYLE><!--H1{font-family : sans-serif,Arial,Taho
> ma;color : white;background-color : #0086b2;} H3{font-family :
> sans-serif,Arial,Tahoma;color : white;background
> -color : #0086b2;} BODY{font-family : sans-serif,Arial,Tahoma;color :
> black;background-color : white;} B{color
> : white;background-color : #0086b2;} HR{color : #0086b2;} --></STYLE>
> </head><body><h1>HTTP Status 500 - </h1><
> HR size=\"1\" noshade><p><b>type</b> Exception
> report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>
> The server encountered an internal error () that prevented it from
> fulfilling this request.</u></p><p><b>except
> ion</b> <pre>javax.servlet.ServletException: Servlet execution threw
> an exception
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
>
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
>
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:256)
>
> at
> org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.j
>
> ava:643)
> at
> org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
>
> at
> org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>
> at
> org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.j
>
> ava:643)
> at
> org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
>
> at
> org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
> at
> org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2422)
>
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
>
> at
> org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.j
>
> ava:643)
> at
> org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:171)
>
> at
> org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.j
>
> ava:641)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:163)
>
> at
> org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.j
>
> ava:641)
> at
> org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
>
> at
> org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
>
> at
> org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.j
>
> ava:643)
> at
> org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
>
> at
> org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
> at
> org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:199)
> at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:828)
>
> at
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.jav
>
> a:700)
> at
> org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:584)
>
> at
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683)
>
> at java.lang.Thread.run(Thread.java:534)
> </pre></p><p><b>root cause</b> <pre>java.lang.OutOfMemoryError
> </pre></p><HR size=\"1\" noshade><h3>Apache
> Tomcat/4.1.30</h3></body></html>"
> at
> xsul.invoker.http.HttpDynamicInfosetInvoker.invokeXml(HttpDynamicInfosetInvoker.java:300)
>
> at
> xsul.invoker.soap_over_http.SoapHttpDynamicInfosetInvoker.invokeMessage(SoapHttpDynamicInfosetInvoke
>
> r.java:122)
> at
> xsul.soaprpc_client.SoapRpcInvocationHandler.invokeRemoteEndpoint(SoapRpcInvocationHandler.java:170)
>
> at
> xsul.soaprpc_client.SoapRpcInvocationHandler.invoke(SoapRpcInvocationHandler.java:101)
>
> at $Proxy0.echoVoid(Unknown Source)
> at soap_bench.BenchClient.runOneTest(BenchClient.java:338)
> at
> soap_bench.BenchClient.runTestsForDirection(BenchClient.java:193)
> at soap_bench.BenchClient.runTestsForSize(BenchClient.java:172)
> at soap_bench.BenchClient.main(BenchClient.java:143)
>
> HTH,
>
> alek
>
--
The best way to predict the future is to invent it - Alan Kay
Re: severe memory leakage [Re: out of memory exception for current
AXIS version from CVS
Posted by Aleksander Slominski <as...@cs.indiana.edu>.
Davanum Srinivas wrote:
>Am banging on the JAX-RPC TCK stuff...I'd REALLY appreciate any help
>in tracking down the source of this bug (a patch would be even
>better). Can u please start by logging the bug in JIRA?
>
>
http://nagoya.apache.org/jira/browse/AXIS-1391
alek
>On Mon, 14 Jun 2004 13:12:37 -0500, Aleksander Slominski
><as...@cs.indiana.edu> wrote:
>
>
>>as a followup: if you observe test progress on server machine using top
>>you will notice that about every 10-20 seconds in-memory size of tomcat
>>increases by 1MB ... probably the same would be happening on windows or
>>any other OS because of WORA ...
>>
>>
>>alek
>>
>>Aleksander Slominski wrote:
>>
>>
>>
>>>hi,
>>>
>>>with tomcat service running on JVM maxed memory to 1GB
>>>
>>>setenv JAVA_OPTS -Xmx1024m
>>>./catalina.sh start
>>>
>>>after running full benchmark :
>>>
>>>/opt/j2sdk/bin/java -Dmachine.name=LINUX_JDK142 -Dserver.name=AXIS_1_2
>>>soap_bench.BenchClient http://server:8080/axis/services/Benchmark1
>>>200000 aa 10,100,1000,5000,10000,25000,50000,75000,100000
>>>
>>>that took few hours to finish and here is what i got next day few
>>>hours after test was finished:
>>>
>>>106 processes: 105 sleeping, 1 running, 0 zombie, 0 stopped
>>>CPU states: 0.0% user, 1.6% system, 0.0% nice, 98.4% idle
>>>Mem: 1033500K total, 1019936K used, 13564K free, 14904K buffers
>>>Swap: 4112560K total, 1580800K used, 2531760K free, 37460K cached
>>>
>>> PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
>>>2453 aslom 17 0 1040 1040 804 R 6.2 0.1 0:04 top
>>>1762 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:06 java
>>>1763 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:00 java
>>>1764 aslom 9 0 983M 347M 75840 S 0.0 34.4 7:06 java
>>>1765 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:07 java
>>>1766 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:15 java
>>>1767 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:00 java
>>>1768 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:00 java
>>>1769 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:03 java
>>>1770 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:32 java
>>>1772 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:00 java
>>>1775 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:00 java
>>>1776 aslom 9 0 983M 347M 75840 S 0.0 34.4 2:14 java
>>>1777 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:00 java
>>>
>>>
>>>so that strongly suggest that there is severe memory leak in AXIS Java
>>>if high (913MB!) memoery usage stays for hours not garbage collected ...
>>>
>>>alek
>>>
>>>Aleksander Slominski wrote:
>>>
>>>
>>>
>>>>hi,
>>>>
>>>>there was some discussion abot memory problems and it seems i have a
>>>>way to experiment and maybe find how to fix it (i did not see Peter
>>>>Molettiere actually posting soemthing to JIRA on how to reproduce his
>>>>problems? AFAICT only other open issues are
>>>>http://nagoya.apache.org/jira/browse/AXIS-1270
>>>>http://nagoya.apache.org/jira/browse/AXIS-1202
>>>>http://nagoya.apache.org/jira/browse/AXIS-1095)
>>>>
>>>>here is how to reproduce this OME (got it on linux but probably is
>>>>the same on any OS) with the latest AXIS from CVS:
>>>>
>>>>compile current axis and generate war file (ant war - wil take some
>>>>time ...)
>>>>get tomcat 4.1.30, deploy current axis.war, modify to disable
>>>>multi-ref in WEB-INF\server-config.wsdd
>>>>set <parameter name="sendXsiTypes" value="false"/> and <parameter
>>>>name="sendMultiRefs" value="false"/>
>>>>
>>>>then download source code and compile from
>>>>http://www.extreme.indiana.edu/~aslom/bnp/wsperf/axis_java/
>>>><>
>>>>after service is deployed start tomcat (running JDK 1.4.2on linux
>>>>without any chnages to -Xmx i.e. /opt/j2sdk/bin/java
>>>>-Djava.endorsed.dirs=/home/aslom/Forge/tomcat/common/endorsed
>>>>-classpath
>>>>/opt/j2sdk/lib/tools.jar:/home/aslom/Forge/tomcat/bin/bootstrap.jar
>>>>-Dcatalina.base=/home/aslom/Forge/tomcat
>>>>-Dcatalina.home=/home/aslom/Forge/tomcat
>>>>-Djava.io.tmpdir=/home/aslom/Forge/tomcat/temp
>>>>org.apache.catalina.startup.Bootstrap start)
>>>>
>>>>and run client with it:
>>>>java -Dmachine.name=LINUX_JDK142 -Dserver.name=AXIS_1_2
>>>>soap_bench.BenchClient http://server:8080/axis/services/Benchmark1
>>>>200000 ev 10,100,1000,5000,10000,25000,50000,75000,100000
>>>>
>>>>the OME happens after running test two times, here are details:
>>>>
>>>>nimitz:~/Forge/driver> java -Dmachine.name=LINUX_JDK142
>>>>-Dserver.name=AXIS_1_2 soap_bench.BenchClient
>>>>http://server:8080/axis/services/Benchmark1 200000 ev
>>>>10,100,1000,5000,10000,25000,50000,75000,100000
>>>>size=10
>>>>invoking 20000 times for test v arraysSize=10
>>>>Exception in thread "main"
>>>>xsul.invoker.DynamicInfosetInvokerException: HTTP server error 500
>>>>Internal Server E
>>>>rror body="
>>>><html><head><title>Apache Tomcat/4.1.30 - Error
>>>>report</title><STYLE><!--H1{font-family : sans-serif,Arial,Taho
>>>>ma;color : white;background-color : #0086b2;} H3{font-family :
>>>>sans-serif,Arial,Tahoma;color : white;background
>>>>-color : #0086b2;} BODY{font-family : sans-serif,Arial,Tahoma;color :
>>>>black;background-color : white;} B{color
>>>>: white;background-color : #0086b2;} HR{color : #0086b2;} --></STYLE>
>>>></head><body><h1>HTTP Status 500 - </h1><
>>>>HR size=\"1\" noshade><p><b>type</b> Exception
>>>>report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>
>>>>The server encountered an internal error () that prevented it from
>>>>fulfilling this request.</u></p><p><b>except
>>>>ion</b> <pre>javax.servlet.ServletException: Servlet execution threw
>>>>an exception
>>>> at
>>>>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
>>>>
>>>> at
>>>>org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
>>>>
>>>> at
>>>>org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:256)
>>>>
>>>> at
>>>>org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.j
>>>>
>>>>ava:643)
>>>> at
>>>>org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
>>>>
>>>> at
>>>>org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
>>>> at
>>>>org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>>>>
>>>> at
>>>>org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.j
>>>>
>>>>ava:643)
>>>> at
>>>>org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
>>>>
>>>> at
>>>>org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
>>>> at
>>>>org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2422)
>>>>
>>>> at
>>>>org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
>>>>
>>>> at
>>>>org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.j
>>>>
>>>>ava:643)
>>>> at
>>>>org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:171)
>>>>
>>>> at
>>>>org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.j
>>>>
>>>>ava:641)
>>>> at
>>>>org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:163)
>>>>
>>>> at
>>>>org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.j
>>>>
>>>>ava:641)
>>>> at
>>>>org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
>>>>
>>>> at
>>>>org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
>>>> at
>>>>org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
>>>>
>>>> at
>>>>org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.j
>>>>
>>>>ava:643)
>>>> at
>>>>org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
>>>>
>>>> at
>>>>org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
>>>> at
>>>>org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:199)
>>>> at
>>>>org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:828)
>>>>
>>>> at
>>>>org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.jav
>>>>
>>>>a:700)
>>>> at
>>>>org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:584)
>>>>
>>>> at
>>>>org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683)
>>>>
>>>> at java.lang.Thread.run(Thread.java:534)
>>>></pre></p><p><b>root cause</b> <pre>java.lang.OutOfMemoryError
>>>></pre></p><HR size=\"1\" noshade><h3>Apache
>>>>Tomcat/4.1.30</h3></body></html>"
>>>> at
>>>>xsul.invoker.http.HttpDynamicInfosetInvoker.invokeXml(HttpDynamicInfosetInvoker.java:300)
>>>>
>>>> at
>>>>xsul.invoker.soap_over_http.SoapHttpDynamicInfosetInvoker.invokeMessage(SoapHttpDynamicInfosetInvoke
>>>>
>>>>r.java:122)
>>>> at
>>>>xsul.soaprpc_client.SoapRpcInvocationHandler.invokeRemoteEndpoint(SoapRpcInvocationHandler.java:170)
>>>>
>>>> at
>>>>xsul.soaprpc_client.SoapRpcInvocationHandler.invoke(SoapRpcInvocationHandler.java:101)
>>>>
>>>> at $Proxy0.echoVoid(Unknown Source)
>>>> at soap_bench.BenchClient.runOneTest(BenchClient.java:338)
>>>> at
>>>>soap_bench.BenchClient.runTestsForDirection(BenchClient.java:193)
>>>> at soap_bench.BenchClient.runTestsForSize(BenchClient.java:172)
>>>> at soap_bench.BenchClient.main(BenchClient.java:143)
>>>>
>>>>HTH,
>>>>
>>>>alek
>>>>
>>>>
>>>>
>>>
>>>
>>--
>>The best way to predict the future is to invent it - Alan Kay
>>
>>
>>
>>
>
>
>
>
>
>
--
The best way to predict the future is to invent it - Alan Kay
Re: Re: severe memory leakage [Re: out of memory exception for current AXIS version from CVS
Posted by Davanum Srinivas <da...@gmail.com>.
Am banging on the JAX-RPC TCK stuff...I'd REALLY appreciate any help
in tracking down the source of this bug (a patch would be even
better). Can u please start by logging the bug in JIRA?
thanks,
dims
On Mon, 14 Jun 2004 13:12:37 -0500, Aleksander Slominski
<as...@cs.indiana.edu> wrote:
>
> as a followup: if you observe test progress on server machine using top
> you will notice that about every 10-20 seconds in-memory size of tomcat
> increases by 1MB ... probably the same would be happening on windows or
> any other OS because of WORA ...
>
>
> alek
>
> Aleksander Slominski wrote:
>
> > hi,
> >
> > with tomcat service running on JVM maxed memory to 1GB
> >
> > setenv JAVA_OPTS -Xmx1024m
> > ./catalina.sh start
> >
> > after running full benchmark :
> >
> > /opt/j2sdk/bin/java -Dmachine.name=LINUX_JDK142 -Dserver.name=AXIS_1_2
> > soap_bench.BenchClient http://server:8080/axis/services/Benchmark1
> > 200000 aa 10,100,1000,5000,10000,25000,50000,75000,100000
> >
> > that took few hours to finish and here is what i got next day few
> > hours after test was finished:
> >
> > 106 processes: 105 sleeping, 1 running, 0 zombie, 0 stopped
> > CPU states: 0.0% user, 1.6% system, 0.0% nice, 98.4% idle
> > Mem: 1033500K total, 1019936K used, 13564K free, 14904K buffers
> > Swap: 4112560K total, 1580800K used, 2531760K free, 37460K cached
> >
> > PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
> > 2453 aslom 17 0 1040 1040 804 R 6.2 0.1 0:04 top
> > 1762 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:06 java
> > 1763 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:00 java
> > 1764 aslom 9 0 983M 347M 75840 S 0.0 34.4 7:06 java
> > 1765 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:07 java
> > 1766 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:15 java
> > 1767 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:00 java
> > 1768 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:00 java
> > 1769 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:03 java
> > 1770 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:32 java
> > 1772 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:00 java
> > 1775 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:00 java
> > 1776 aslom 9 0 983M 347M 75840 S 0.0 34.4 2:14 java
> > 1777 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:00 java
> >
> >
> > so that strongly suggest that there is severe memory leak in AXIS Java
> > if high (913MB!) memoery usage stays for hours not garbage collected ...
> >
> > alek
> >
> > Aleksander Slominski wrote:
> >
> >> hi,
> >>
> >> there was some discussion abot memory problems and it seems i have a
> >> way to experiment and maybe find how to fix it (i did not see Peter
> >> Molettiere actually posting soemthing to JIRA on how to reproduce his
> >> problems? AFAICT only other open issues are
> >> http://nagoya.apache.org/jira/browse/AXIS-1270
> >> http://nagoya.apache.org/jira/browse/AXIS-1202
> >> http://nagoya.apache.org/jira/browse/AXIS-1095)
> >>
> >> here is how to reproduce this OME (got it on linux but probably is
> >> the same on any OS) with the latest AXIS from CVS:
> >>
> >> compile current axis and generate war file (ant war - wil take some
> >> time ...)
> >> get tomcat 4.1.30, deploy current axis.war, modify to disable
> >> multi-ref in WEB-INF\server-config.wsdd
> >> set <parameter name="sendXsiTypes" value="false"/> and <parameter
> >> name="sendMultiRefs" value="false"/>
> >>
> >> then download source code and compile from
> >> http://www.extreme.indiana.edu/~aslom/bnp/wsperf/axis_java/
> >> <>
> >> after service is deployed start tomcat (running JDK 1.4.2on linux
> >> without any chnages to -Xmx i.e. /opt/j2sdk/bin/java
> >> -Djava.endorsed.dirs=/home/aslom/Forge/tomcat/common/endorsed
> >> -classpath
> >> /opt/j2sdk/lib/tools.jar:/home/aslom/Forge/tomcat/bin/bootstrap.jar
> >> -Dcatalina.base=/home/aslom/Forge/tomcat
> >> -Dcatalina.home=/home/aslom/Forge/tomcat
> >> -Djava.io.tmpdir=/home/aslom/Forge/tomcat/temp
> >> org.apache.catalina.startup.Bootstrap start)
> >>
> >> and run client with it:
> >> java -Dmachine.name=LINUX_JDK142 -Dserver.name=AXIS_1_2
> >> soap_bench.BenchClient http://server:8080/axis/services/Benchmark1
> >> 200000 ev 10,100,1000,5000,10000,25000,50000,75000,100000
> >>
> >> the OME happens after running test two times, here are details:
> >>
> >> nimitz:~/Forge/driver> java -Dmachine.name=LINUX_JDK142
> >> -Dserver.name=AXIS_1_2 soap_bench.BenchClient
> >> http://server:8080/axis/services/Benchmark1 200000 ev
> >> 10,100,1000,5000,10000,25000,50000,75000,100000
> >> size=10
> >> invoking 20000 times for test v arraysSize=10
> >> Exception in thread "main"
> >> xsul.invoker.DynamicInfosetInvokerException: HTTP server error 500
> >> Internal Server E
> >> rror body="
> >> <html><head><title>Apache Tomcat/4.1.30 - Error
> >> report</title><STYLE><!--H1{font-family : sans-serif,Arial,Taho
> >> ma;color : white;background-color : #0086b2;} H3{font-family :
> >> sans-serif,Arial,Tahoma;color : white;background
> >> -color : #0086b2;} BODY{font-family : sans-serif,Arial,Tahoma;color :
> >> black;background-color : white;} B{color
> >> : white;background-color : #0086b2;} HR{color : #0086b2;} --></STYLE>
> >> </head><body><h1>HTTP Status 500 - </h1><
> >> HR size=\"1\" noshade><p><b>type</b> Exception
> >> report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>
> >> The server encountered an internal error () that prevented it from
> >> fulfilling this request.</u></p><p><b>except
> >> ion</b> <pre>javax.servlet.ServletException: Servlet execution threw
> >> an exception
> >> at
> >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
> >>
> >> at
> >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
> >>
> >> at
> >> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:256)
> >>
> >> at
> >> org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.j
> >>
> >> ava:643)
> >> at
> >> org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
> >>
> >> at
> >> org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
> >> at
> >> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> >>
> >> at
> >> org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.j
> >>
> >> ava:643)
> >> at
> >> org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
> >>
> >> at
> >> org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
> >> at
> >> org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2422)
> >>
> >> at
> >> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
> >>
> >> at
> >> org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.j
> >>
> >> ava:643)
> >> at
> >> org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:171)
> >>
> >> at
> >> org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.j
> >>
> >> ava:641)
> >> at
> >> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:163)
> >>
> >> at
> >> org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.j
> >>
> >> ava:641)
> >> at
> >> org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
> >>
> >> at
> >> org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
> >> at
> >> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
> >>
> >> at
> >> org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.j
> >>
> >> ava:643)
> >> at
> >> org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
> >>
> >> at
> >> org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
> >> at
> >> org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:199)
> >> at
> >> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:828)
> >>
> >> at
> >> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.jav
> >>
> >> a:700)
> >> at
> >> org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:584)
> >>
> >> at
> >> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683)
> >>
> >> at java.lang.Thread.run(Thread.java:534)
> >> </pre></p><p><b>root cause</b> <pre>java.lang.OutOfMemoryError
> >> </pre></p><HR size=\"1\" noshade><h3>Apache
> >> Tomcat/4.1.30</h3></body></html>"
> >> at
> >> xsul.invoker.http.HttpDynamicInfosetInvoker.invokeXml(HttpDynamicInfosetInvoker.java:300)
> >>
> >> at
> >> xsul.invoker.soap_over_http.SoapHttpDynamicInfosetInvoker.invokeMessage(SoapHttpDynamicInfosetInvoke
> >>
> >> r.java:122)
> >> at
> >> xsul.soaprpc_client.SoapRpcInvocationHandler.invokeRemoteEndpoint(SoapRpcInvocationHandler.java:170)
> >>
> >> at
> >> xsul.soaprpc_client.SoapRpcInvocationHandler.invoke(SoapRpcInvocationHandler.java:101)
> >>
> >> at $Proxy0.echoVoid(Unknown Source)
> >> at soap_bench.BenchClient.runOneTest(BenchClient.java:338)
> >> at
> >> soap_bench.BenchClient.runTestsForDirection(BenchClient.java:193)
> >> at soap_bench.BenchClient.runTestsForSize(BenchClient.java:172)
> >> at soap_bench.BenchClient.main(BenchClient.java:143)
> >>
> >> HTH,
> >>
> >> alek
> >>
> >
> >
>
> --
> The best way to predict the future is to invent it - Alan Kay
>
>
--
Davanum Srinivas - http://webservices.apache.org/~dims/
Re: severe memory leakage [Re: out of memory exception for current
AXIS version from CVS
Posted by Aleksander Slominski <as...@cs.indiana.edu>.
as a followup: if you observe test progress on server machine using top
you will notice that about every 10-20 seconds in-memory size of tomcat
increases by 1MB ... probably the same would be happening on windows or
any other OS because of WORA ...
alek
Aleksander Slominski wrote:
> hi,
>
> with tomcat service running on JVM maxed memory to 1GB
>
> setenv JAVA_OPTS -Xmx1024m
> ./catalina.sh start
>
> after running full benchmark :
>
> /opt/j2sdk/bin/java -Dmachine.name=LINUX_JDK142 -Dserver.name=AXIS_1_2
> soap_bench.BenchClient http://server:8080/axis/services/Benchmark1
> 200000 aa 10,100,1000,5000,10000,25000,50000,75000,100000
>
> that took few hours to finish and here is what i got next day few
> hours after test was finished:
>
> 106 processes: 105 sleeping, 1 running, 0 zombie, 0 stopped
> CPU states: 0.0% user, 1.6% system, 0.0% nice, 98.4% idle
> Mem: 1033500K total, 1019936K used, 13564K free, 14904K buffers
> Swap: 4112560K total, 1580800K used, 2531760K free, 37460K cached
>
> PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
> 2453 aslom 17 0 1040 1040 804 R 6.2 0.1 0:04 top
> 1762 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:06 java
> 1763 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:00 java
> 1764 aslom 9 0 983M 347M 75840 S 0.0 34.4 7:06 java
> 1765 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:07 java
> 1766 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:15 java
> 1767 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:00 java
> 1768 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:00 java
> 1769 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:03 java
> 1770 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:32 java
> 1772 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:00 java
> 1775 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:00 java
> 1776 aslom 9 0 983M 347M 75840 S 0.0 34.4 2:14 java
> 1777 aslom 9 0 983M 347M 75840 S 0.0 34.4 0:00 java
>
>
> so that strongly suggest that there is severe memory leak in AXIS Java
> if high (913MB!) memoery usage stays for hours not garbage collected ...
>
> alek
>
> Aleksander Slominski wrote:
>
>> hi,
>>
>> there was some discussion abot memory problems and it seems i have a
>> way to experiment and maybe find how to fix it (i did not see Peter
>> Molettiere actually posting soemthing to JIRA on how to reproduce his
>> problems? AFAICT only other open issues are
>> http://nagoya.apache.org/jira/browse/AXIS-1270
>> http://nagoya.apache.org/jira/browse/AXIS-1202
>> http://nagoya.apache.org/jira/browse/AXIS-1095)
>>
>> here is how to reproduce this OME (got it on linux but probably is
>> the same on any OS) with the latest AXIS from CVS:
>>
>> compile current axis and generate war file (ant war - wil take some
>> time ...)
>> get tomcat 4.1.30, deploy current axis.war, modify to disable
>> multi-ref in WEB-INF\server-config.wsdd
>> set <parameter name="sendXsiTypes" value="false"/> and <parameter
>> name="sendMultiRefs" value="false"/>
>>
>> then download source code and compile from
>> http://www.extreme.indiana.edu/~aslom/bnp/wsperf/axis_java/
>> <>
>> after service is deployed start tomcat (running JDK 1.4.2on linux
>> without any chnages to -Xmx i.e. /opt/j2sdk/bin/java
>> -Djava.endorsed.dirs=/home/aslom/Forge/tomcat/common/endorsed
>> -classpath
>> /opt/j2sdk/lib/tools.jar:/home/aslom/Forge/tomcat/bin/bootstrap.jar
>> -Dcatalina.base=/home/aslom/Forge/tomcat
>> -Dcatalina.home=/home/aslom/Forge/tomcat
>> -Djava.io.tmpdir=/home/aslom/Forge/tomcat/temp
>> org.apache.catalina.startup.Bootstrap start)
>>
>> and run client with it:
>> java -Dmachine.name=LINUX_JDK142 -Dserver.name=AXIS_1_2
>> soap_bench.BenchClient http://server:8080/axis/services/Benchmark1
>> 200000 ev 10,100,1000,5000,10000,25000,50000,75000,100000
>>
>> the OME happens after running test two times, here are details:
>>
>> nimitz:~/Forge/driver> java -Dmachine.name=LINUX_JDK142
>> -Dserver.name=AXIS_1_2 soap_bench.BenchClient
>> http://server:8080/axis/services/Benchmark1 200000 ev
>> 10,100,1000,5000,10000,25000,50000,75000,100000
>> size=10
>> invoking 20000 times for test v arraysSize=10
>> Exception in thread "main"
>> xsul.invoker.DynamicInfosetInvokerException: HTTP server error 500
>> Internal Server E
>> rror body="
>> <html><head><title>Apache Tomcat/4.1.30 - Error
>> report</title><STYLE><!--H1{font-family : sans-serif,Arial,Taho
>> ma;color : white;background-color : #0086b2;} H3{font-family :
>> sans-serif,Arial,Tahoma;color : white;background
>> -color : #0086b2;} BODY{font-family : sans-serif,Arial,Tahoma;color :
>> black;background-color : white;} B{color
>> : white;background-color : #0086b2;} HR{color : #0086b2;} --></STYLE>
>> </head><body><h1>HTTP Status 500 - </h1><
>> HR size=\"1\" noshade><p><b>type</b> Exception
>> report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>
>> The server encountered an internal error () that prevented it from
>> fulfilling this request.</u></p><p><b>except
>> ion</b> <pre>javax.servlet.ServletException: Servlet execution threw
>> an exception
>> at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
>>
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
>>
>> at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:256)
>>
>> at
>> org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.j
>>
>> ava:643)
>> at
>> org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
>>
>> at
>> org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
>> at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>>
>> at
>> org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.j
>>
>> ava:643)
>> at
>> org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
>>
>> at
>> org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
>> at
>> org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2422)
>>
>> at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
>>
>> at
>> org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.j
>>
>> ava:643)
>> at
>> org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:171)
>>
>> at
>> org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.j
>>
>> ava:641)
>> at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:163)
>>
>> at
>> org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.j
>>
>> ava:641)
>> at
>> org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
>>
>> at
>> org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
>> at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
>>
>> at
>> org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.j
>>
>> ava:643)
>> at
>> org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
>>
>> at
>> org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
>> at
>> org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:199)
>> at
>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:828)
>>
>> at
>> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.jav
>>
>> a:700)
>> at
>> org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:584)
>>
>> at
>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683)
>>
>> at java.lang.Thread.run(Thread.java:534)
>> </pre></p><p><b>root cause</b> <pre>java.lang.OutOfMemoryError
>> </pre></p><HR size=\"1\" noshade><h3>Apache
>> Tomcat/4.1.30</h3></body></html>"
>> at
>> xsul.invoker.http.HttpDynamicInfosetInvoker.invokeXml(HttpDynamicInfosetInvoker.java:300)
>>
>> at
>> xsul.invoker.soap_over_http.SoapHttpDynamicInfosetInvoker.invokeMessage(SoapHttpDynamicInfosetInvoke
>>
>> r.java:122)
>> at
>> xsul.soaprpc_client.SoapRpcInvocationHandler.invokeRemoteEndpoint(SoapRpcInvocationHandler.java:170)
>>
>> at
>> xsul.soaprpc_client.SoapRpcInvocationHandler.invoke(SoapRpcInvocationHandler.java:101)
>>
>> at $Proxy0.echoVoid(Unknown Source)
>> at soap_bench.BenchClient.runOneTest(BenchClient.java:338)
>> at
>> soap_bench.BenchClient.runTestsForDirection(BenchClient.java:193)
>> at soap_bench.BenchClient.runTestsForSize(BenchClient.java:172)
>> at soap_bench.BenchClient.main(BenchClient.java:143)
>>
>> HTH,
>>
>> alek
>>
>
>
--
The best way to predict the future is to invent it - Alan Kay
Re: severe memory leakage [Re: out of memory exception for current
AXIS version from CVS
Posted by Aleksander Slominski <as...@cs.indiana.edu>.
Sanjiva Weerawarana wrote:
>"Aleksander Slominski" <as...@cs.indiana.edu> writes:
>
>
>>so that strongly suggest that there is severe memory leak in AXIS Java
>>if high (913MB!) memoery usage stays for hours not garbage collected ...
>>
>>
>
>Its been a while since I did perf stuff on a unix box (have to go
>back to my Purdue days :-)), but IIRC Unix doesn't actually shrink
>processes. That is, while you can grow a process, you can't actually
>shrink .. so even if GC is picking up the memory that won't show
>up via "ps".
>
>
when i used defaul -Xms i have got OutofMemoryError so i thin it is
indication of the problem and no other java process grows to 900MB ...
>Did you record any info from the JVM and see what it thinks?
>
>
i will try to do this when i finsih with performance tests.
thanks,
alek
--
The best way to predict the future is to invent it - Alan Kay
Re: severe memory leakage [Re: out of memory exception for current AXIS version from CVS
Posted by Sanjiva Weerawarana <sa...@opensource.lk>.
"Aleksander Slominski" <as...@cs.indiana.edu> writes:
>
> so that strongly suggest that there is severe memory leak in AXIS Java
> if high (913MB!) memoery usage stays for hours not garbage collected ...
Its been a while since I did perf stuff on a unix box (have to go
back to my Purdue days :-)), but IIRC Unix doesn't actually shrink
processes. That is, while you can grow a process, you can't actually
shrink .. so even if GC is picking up the memory that won't show
up via "ps".
Did you record any info from the JVM and see what it thinks?
Sanjiva.