You are viewing a plain text version of this content. The canonical link for it is here.
Posted to c-dev@axis.apache.org by Aleksander Slominski <as...@cs.indiana.edu> on 2004/06/12 13:34:17 UTC
more on AXIS-Java/C++ server side performance
Davanum Srinivas wrote:
>Alek,
>
>Where is the sample code for your XSOAP server-side code? does it
>create a java object from the soap request?
>
hi dims,
first let me emphasize that i do not look on this benchmark as only to
indicate performance but also a tool
now that i said that i think i can demonstrate usefulness of such tests:
to my big surprise after running tests i can see that AXIS-C++ is two
times slower than AXIS-Java (in this sense the test work to help detect
possible huge performance bottlenecks may ultimately lead to much better
implementation).
anyway this is really strange - i would expect C++ to be faster :)
below is everything you need to reproduce test results yourself and
maybe somebody will have an idea what is going on with AXIS-C++ ...
the test driver (client side) is still not fully finished but here is
working version:
http://www.extreme.indiana.edu/~aslom/bnp/wsperf/driver/
after downloading do source classpath.sh (or classpath.bat on windows)
to set CLASSPATH
run self test:
java -Dserver.name=SELF_TEST soap_bench.BenchDriver http://localhost:3434
and here is full description of command line:
java -Dserver.name=SERVICE_NAME soap_bench.Bench URL total
{rsea}{bdisva} [arraySize][,arraySize][,...]
parameters in order:
1. URL of service
2. total number of elements to send (default 10K)
3. specification of what to send : two letters
[rse] means receive, send, or echo (a == all)
[bdisva] means base64, double, int, string, void (only applies to
echo), a == all methods;
4. list of arraySizes (optional for void) - default to 10
and you can set test metadata such as -Dmachine.name=...
-Dserver.name=... (it is included in tabulated results good for
spreadsheet input and to draw graphs)
those are optional system properties:
-Dsmoke_test - will show you what messages were received and check that
all works fine
-DAXISCPP - will add SOAPAction header and xsi:type for BASE64 (so
AXIS-C++ is happy)
-Dlog=trace.xsul.http.client:ALL -will show all messages and HTTP
headers (good for debugging)
-Dlog=:ALL - if you are doing desperate debugging
use smoke test to get code to work with driver - it will just send 3
messages and will print what it received - combine it with -Dlog=... to
get more fine grained information on what is happening and if/when
driver is failing.
the way i would do to get test driver to work with a SOAP service is to
it in multiple esteps and to get first to work different echo methods
a) echoVoid
java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200 ev
10,100
echoInts
java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200 ei
10,100
echoDoubles
java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200 ed
10,100
echoStrings
java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200 es
10,100
echoBytes (as Base64)
java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200 eb
10,100
b) all echo methods
java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200 ea
10,100
c) all methods
java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200 aa
10,100
and then run real tests
----------------------
For AXIS-Java
get code from http://www.extreme.indiana.edu/~aslom/bnp/wsperf/axis_java/
and deploy benchmark service in tomcat - to do it just modify
build.properties and do ant all
run test
java -Dmachine.name=YOUR_MACHINE_NAME -Dserver.name=AXIS_1_2_CVS
soap_bench.BenchClient http://server:8080/axis/services/Benchmark1
200000 aa 10,1000,10000
----------------------
For AXIS-C++
get code from http://www.extreme.indiana.edu/~aslom/bnp/wsperf/axis_cpp/
copy soap_bench/ to samples/server (and client respectively)
modify top level Makefile.am to include soap_bench (see example)
rebuilt .so library
add to list of deployed services (see deploy.wsdd in soap_bench)
restart apache server
run test (make sure -DAXISCPP parameter is included!!!!!)
java -DAXISCPP -Dmachine.name=YOUR_MACHINE_NAME
-Dserver.name=AXISCPP_1_0_CVS
soap_bench.BenchClient http://server:8080/axis/Benchmark1 200000 ea
10,1000,10000
NOTE: 'aa' will not work as all send tests do not work (and i do not
know why). instead try ea (echo tests) and sa (send tests) - if you want
to see what is not working use sa and enable logging (-Dlog=:ALL)
--------------------------
For XSOAP4
start service: java soap_bench.BenchServer 8077
run test driver:
java -Dmachine.name=YOUR_MACHINE_NAME -Dserver.name=XSOAP4
soap_bench.BenchClient http://server:8077 200000 aa 10,1000,10000
----------------------
comments are welcome especially concerning performance differences
between Java and C++ and how to get send tests to work with AXIS C++...
thanks,
alek
>On Thu, 03 Jun 2004 12:58:41 -0500, Aleksander Slominski
><as...@cs.indiana.edu> wrote:
>
>
>>Davanum Srinivas wrote:
>>
>>
>>
>>>You need to turn streaming on the client-side as well. see
>>>samples/perf in latest cvs.
>>>
>>> binding._setProperty(org.apache.axis.client.Call.STREAMING_PROPERTY,
>>>Boolean.TRUE);
>>>
>>>
>>>
>>>
>>i am interested in testing server side performance (as server side is
>>typically the bottleneck) so i use the same client to access all
>>services to measure only factors that are effecting server side
>>performance (CPU, memory footprint, scalability).
>>
>>hopefully soon i will have AXIS-C++ numbers - i am really close now to
>>get them after few weeks of struggling with C++ (Java configuration and
>>build tools are much easier even when dealing with JAR hell ...)
>>
>>so it should be interesting to see how AXIS-Java and AXIS-C++ compare.
>>
>>however i am also *very* interested in very long running web services so
>>Peter post about _huge_ memory leaks that are easily reproducible looked
>>very interesting to me :) is it a property of data graph he send or is
>>it just a bug ...
>>
>>thanks,
>>
>>alek
>>
>>
>>
>>>-- dims
>>>
>>>On Thu, 03 Jun 2004 12:27:18 -0500, Aleksander Slominski
>>><as...@cs.indiana.edu> wrote:
>>>
>>>
>>>
>>>
>>>>Peter,
>>>>
>>>>please post more details how to reproduce the memory leak and i will add
>>>>to what i have for WS/SOAP performance tests at:
>>>>http://www.extreme.indiana.edu/~aslom/bnp/wsperf/
>>>>
>>>>thanks,
>>>>
>>>>alek
>>>>
>>>>Peter Molettiere wrote:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>We're currently looking into a massive memory leak in axis, that no
>>>>>one here seems willing to admit. We're close to having a minimal test
>>>>>case which demonstrates the leak we're seeing. I don't think it's the
>>>>>same one you're seeing however, as our leak happens 1) when the
>>>>>server is sitting doing nothing (but only after axis has been
>>>>>loaded), and 2) every time we serialize a large message. The
>>>>>suggestions on the list about switching to using attachments doesn't
>>>>>fix the leak sadly. And setting a larger heap size only delays the
>>>>>inevitable Out Of Memory Exception.
>>>>>
>>>>>
>>>>>
>>>>>>From our testing, it looks like it takes on the order of 700MB to
>>>>
>>>>
>>>>>serialize a 30MB java object graph into a 3MB SOAP stream.
>>>>>
>>>>>I've posted about this before, but no one has responded. Hopefully
>>>>>when we post the test case someone will actually take a look at it.
>>>>>There are other threads with test cases which currently have "we're
>>>>>looking at this now" as the only response, but as they are all pretty
>>>>>old threads, my guess is that this problem either isn't believed or
>>>>>is being swept under the carpet. Maybe we're the only people using
>>>>>axis for large messages.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>--
>>>>>
>>>>>On Jun 3, 2004, at 9:38 AM, jira@apache.org wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>The following comment has been added to this issue:
>>>>>>
>>>>>> Author: Jennifer Jackson
>>>>>> Created: Thu, 3 Jun 2004 9:37 AM
>>>>>> Body:
>>>>>>Has anyone looked into this? I haven't used Axis because of this
>>>>>>problem and doing a quick search it seems others have come across
>>>>>>memory leaks also. I narrowed it down to the Call object not
>>>>>>cleaning out its member vars (the whole Deserialization process).
>>>>>>If you do that the only thing stuck in memory is the call object
>>>>>>since its part of the ThreadLocal object.
>>>>>>---------------------------------------------------------------------
>>>>>>View this comment:
>>>>>> http://issues.apache.org/jira/browse/AXIS-935?
>>>>>>page=comments#action_35890
>>>>>>
>>>>>>---------------------------------------------------------------------
>>>>>>View the issue:
>>>>>> http://issues.apache.org/jira/browse/AXIS-935
>>>>>>
>>>>>>Here is an overview of the issue:
>>>>>>---------------------------------------------------------------------
>>>>>> Key: AXIS-935
>>>>>> Summary: Using ThreadLocal Call objects in a servlet memory leak
>>>>>>both on client and server side
>>>>>> Type: Bug
>>>>>>
>>>>>> Status: Open
>>>>>>
>>>>>> Project: Axis
>>>>>>Components:
>>>>>> Basic Architecture
>>>>>> Versions:
>>>>>> 1.1rc2
>>>>>>
>>>>>> Assignee: Axis Developers Mailing List
>>>>>> Reporter: Jennifer Jackson
>>>>>>
>>>>>> Created: Mon, 30 Jun 2003 12:12 PM
>>>>>> Updated: Thu, 3 Jun 2004 9:37 AM
>>>>>>Environment: Operating System: All
>>>>>>Platform: All
>>>>>>
>>>>>>Description:
>>>>>>I discovered using Axis in tomcat causes a memory leak.
>>>>>>
>>>>>>It seems using threadlocal in the classes causes a memory leak for
>>>>>>each Call
>>>>>>that is created. Threadlocal sticks around until the calling thread
>>>>>>dies, and
>>>>>>since tomcat keeps threads alive to answer new calls, the
>>>>>>threadlocal data
>>>>>>never goes away. The Call object gets stuck in the threadlocal
>>>>>>data, which
>>>>>>contains the entire soap response and all objects. This is a BIG
>>>>>>memory leak.
>>>>>>I initially cleared it by forcing all Call objects to clean their
>>>>>>member vars
>>>>>>after I was done with a call, then I realized the source was from
>>>>>>the thread
>>>>>>local class. Here is some info I found about threadlocal:
>>>>>>
>>>>>>This is stated in the Sun Javadocs for ThreadLocal:
>>>>>>
>>>>>> Each thread holds an implicit reference to its copy of a
>>>>>> ThreadLocal as long as the thread is alive and the
>>>>>> ThreadLocal object is accessible; after a thread goes
>>>>>> away, all of its copies of ThreadLocal variables are
>>>>>> subject to garbage collection (unless other references
>>>>>> to these copies exist).
>>>>>>
>>>>>>So, this means that ANY APPLICATION that uses PreparedStatements in
>>>>>>a thread
>>>>>>that 1) either does a lot of PreparedStatements or 2) never dies
>>>>>>(i.e., a
>>>>>>main thread) will ALWAYS eventually have an out of memory error.
>>>>>>Simply put,
>>>>>>this is a MEMORY LEAK. I imagine that the leak is very small, the
>>>>>>ThreadLocal object only contains one member variable, maybe 64 bytes
>>>>>>or less
>>>>>>(depending on the VM implementation). So, our 60,000
>>>>>>PreparedStatements of 2
>>>>>>ThreadLocals each times 64 bytes (my wild guess) is 7.5MB.
>>>>>>
>>>>>>Ideas? I've never used threadlocal myself so this is new to me.
>>>>>>
>>>>>>
>>>>>>---------------------------------------------------------------------
>>>>>>JIRA INFORMATION:
>>>>>>This message is automatically generated by JIRA.
>>>>>>
>>>>>>If you think it was sent incorrectly contact one of the administrators:
>>>>>> http://issues.apache.org/jira/secure/Administrators.jspa
>>>>>>
>>>>>>If you want more information on JIRA, or have a bug to report see:
>>>>>> http://www.atlassian.com/software/jira
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>--
>>>>The best way to predict the future is to invent it - Alan Kay
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>--
>>The best way to predict the future is to invent it - Alan Kay
>>
>>
>>
>>
--
The best way to predict the future is to invent it - Alan Kay
Re: more on AXIS-Java/C++ server side performance
Posted by Aleksander Slominski <as...@cs.indiana.edu>.
Susantha Kumara wrote:
>Hi Alek
>
>You are doing a great work...!
>
>Which parser did you use ?. Xerces or Expat.
>
expat.
>Last time we did
>performance tests we found that Axis C++ is faster with Expat. 3 times
>faster with Expat than Xerces.
>
>
ha tis ehat i expected as expat is supposed ot be the fastest XML parser
on C.
i will post more results and a graph soon.
thanks,
alek
>>-----Original Message-----
>>From: Aleksander Slominski [mailto:aslom@cs.indiana.edu]
>>Sent: Saturday, June 12, 2004 5:34 PM
>>To: Apache AXIS C Developers List; 'axis-dev@ws.apache.org'
>>Subject: more on AXIS-Java/C++ server side performance
>>
>>Davanum Srinivas wrote:
>>
>>
>>
>>>Alek,
>>>
>>>Where is the sample code for your XSOAP server-side code? does it
>>>create a java object from the soap request?
>>>
>>>
>>>
>>hi dims,
>>
>>first let me emphasize that i do not look on this benchmark as only to
>>indicate performance but also a tool
>>
>>now that i said that i think i can demonstrate usefulness of such
>>
>>
>tests:
>
>
>>to my big surprise after running tests i can see that AXIS-C++ is two
>>times slower than AXIS-Java (in this sense the test work to help
>>
>>
>detect
>
>
>>possible huge performance bottlenecks may ultimately lead to much
>>
>>
>better
>
>
>>implementation).
>>
>>anyway this is really strange - i would expect C++ to be faster :)
>>
>>below is everything you need to reproduce test results yourself and
>>maybe somebody will have an idea what is going on with AXIS-C++ ...
>>
>>the test driver (client side) is still not fully finished but here is
>>working version:
>>http://www.extreme.indiana.edu/~aslom/bnp/wsperf/driver/
>>
>>after downloading do source classpath.sh (or classpath.bat on windows)
>>to set CLASSPATH
>>
>>run self test:
>>java -Dserver.name=SELF_TEST soap_bench.BenchDriver
>>
>>
>http://localhost:3434
>
>
>>and here is full description of command line:
>>
>>java -Dserver.name=SERVICE_NAME soap_bench.Bench URL total
>>{rsea}{bdisva} [arraySize][,arraySize][,...]
>>
>>parameters in order:
>>1. URL of service
>>2. total number of elements to send (default 10K)
>>3. specification of what to send : two letters
>> [rse] means receive, send, or echo (a == all)
>> [bdisva] means base64, double, int, string, void (only applies to
>>echo), a == all methods;
>>4. list of arraySizes (optional for void) - default to 10
>>and you can set test metadata such as -Dmachine.name=...
>>-Dserver.name=... (it is included in tabulated results good for
>>spreadsheet input and to draw graphs)
>>
>>those are optional system properties:
>>-Dsmoke_test - will show you what messages were received and check
>>
>>
>that
>
>
>>all works fine
>>-DAXISCPP - will add SOAPAction header and xsi:type for BASE64 (so
>>AXIS-C++ is happy)
>>-Dlog=trace.xsul.http.client:ALL -will show all messages and HTTP
>>headers (good for debugging)
>>-Dlog=:ALL - if you are doing desperate debugging
>>
>>use smoke test to get code to work with driver - it will just send 3
>>messages and will print what it received - combine it with -Dlog=...
>>
>>
>to
>
>
>>get more fine grained information on what is happening and if/when
>>driver is failing.
>>
>>the way i would do to get test driver to work with a SOAP service is
>>
>>
>to
>
>
>>it in multiple esteps and to get first to work different echo methods
>>
>>a) echoVoid
>>
>>java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200
>>
>>
>ev
>
>
>>10,100
>>
>>echoInts
>>
>>java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200
>>
>>
>ei
>
>
>>10,100
>>
>>echoDoubles
>>
>>java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200
>>
>>
>ed
>
>
>>10,100
>>
>>echoStrings
>>
>>java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200
>>
>>
>es
>
>
>>10,100
>>
>>echoBytes (as Base64)
>>
>>java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200
>>
>>
>eb
>
>
>>10,100
>>
>>b) all echo methods
>>
>>java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200
>>
>>
>ea
>
>
>>10,100
>>
>>c) all methods
>>
>>java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200
>>
>>
>aa
>
>
>>10,100
>>
>>and then run real tests
>>
>>----------------------
>>For AXIS-Java
>>
>>get code from
>>
>>
>http://www.extreme.indiana.edu/~aslom/bnp/wsperf/axis_java/
>
>
>>and deploy benchmark service in tomcat - to do it just modify
>>build.properties and do ant all
>>
>>run test
>>java -Dmachine.name=YOUR_MACHINE_NAME -Dserver.name=AXIS_1_2_CVS
>>soap_bench.BenchClient http://server:8080/axis/services/Benchmark1
>>200000 aa 10,1000,10000
>>
>>----------------------
>>For AXIS-C++
>>
>>get code from
>>
>>
>http://www.extreme.indiana.edu/~aslom/bnp/wsperf/axis_cpp/
>
>
>>copy soap_bench/ to samples/server (and client respectively)
>>modify top level Makefile.am to include soap_bench (see example)
>>rebuilt .so library
>>add to list of deployed services (see deploy.wsdd in soap_bench)
>>restart apache server
>>
>>run test (make sure -DAXISCPP parameter is included!!!!!)
>>java -DAXISCPP -Dmachine.name=YOUR_MACHINE_NAME
>>-Dserver.name=AXISCPP_1_0_CVS
>>soap_bench.BenchClient http://server:8080/axis/Benchmark1 200000 ea
>>10,1000,10000
>>
>>NOTE: 'aa' will not work as all send tests do not work (and i do not
>>know why). instead try ea (echo tests) and sa (send tests) - if you
>>
>>
>want
>
>
>>to see what is not working use sa and enable logging (-Dlog=:ALL)
>>
>>--------------------------
>>For XSOAP4
>>
>>start service: java soap_bench.BenchServer 8077
>>
>>run test driver:
>>java -Dmachine.name=YOUR_MACHINE_NAME -Dserver.name=XSOAP4
>>soap_bench.BenchClient http://server:8077 200000 aa 10,1000,10000
>>
>>----------------------
>>
>>comments are welcome especially concerning performance differences
>>between Java and C++ and how to get send tests to work with AXIS
>>
>>
>C++...
>
>
>>thanks,
>>
>>alek
>>
>>
>>
>>>On Thu, 03 Jun 2004 12:58:41 -0500, Aleksander Slominski
>>><as...@cs.indiana.edu> wrote:
>>>
>>>
>>>
>>>
>>>>Davanum Srinivas wrote:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>You need to turn streaming on the client-side as well. see
>>>>>samples/perf in latest cvs.
>>>>>
>>>>>
>>>>>
>>>>>
>>binding._setProperty(org.apache.axis.client.Call.STREAMING_PROPERTY,
>>
>>
>>>>>Boolean.TRUE);
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>i am interested in testing server side performance (as server side
>>>>
>>>>
>is
>
>
>>>>typically the bottleneck) so i use the same client to access all
>>>>services to measure only factors that are effecting server side
>>>>performance (CPU, memory footprint, scalability).
>>>>
>>>>hopefully soon i will have AXIS-C++ numbers - i am really close now
>>>>
>>>>
>to
>
>
>>>>get them after few weeks of struggling with C++ (Java configuration
>>>>
>>>>
>and
>
>
>>>>build tools are much easier even when dealing with JAR hell ...)
>>>>
>>>>so it should be interesting to see how AXIS-Java and AXIS-C++
>>>>
>>>>
>compare.
>
>
>>>>however i am also *very* interested in very long running web
>>>>
>>>>
>services so
>
>
>>>>Peter post about _huge_ memory leaks that are easily reproducible
>>>>
>>>>
>looked
>
>
>>>>very interesting to me :) is it a property of data graph he send or
>>>>
>>>>
>is
>
>
>>>>it just a bug ...
>>>>
>>>>thanks,
>>>>
>>>>alek
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>-- dims
>>>>>
>>>>>On Thu, 03 Jun 2004 12:27:18 -0500, Aleksander Slominski
>>>>><as...@cs.indiana.edu> wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>Peter,
>>>>>>
>>>>>>please post more details how to reproduce the memory leak and i
>>>>>>
>>>>>>
>will
>
>
>>add
>>
>>
>>>>>>to what i have for WS/SOAP performance tests at:
>>>>>>http://www.extreme.indiana.edu/~aslom/bnp/wsperf/
>>>>>>
>>>>>>thanks,
>>>>>>
>>>>>>alek
>>>>>>
>>>>>>Peter Molettiere wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>>We're currently looking into a massive memory leak in axis, that
>>>>>>>
>>>>>>>
>no
>
>
>>>>>>>one here seems willing to admit. We're close to having a minimal
>>>>>>>
>>>>>>>
>>test
>>
>>
>>>>>>>case which demonstrates the leak we're seeing. I don't think
>>>>>>>
>>>>>>>
>it's
>
>
>>the
>>
>>
>>>>>>>same one you're seeing however, as our leak happens 1) when the
>>>>>>>server is sitting doing nothing (but only after axis has been
>>>>>>>loaded), and 2) every time we serialize a large message. The
>>>>>>>suggestions on the list about switching to using attachments
>>>>>>>
>>>>>>>
>doesn't
>
>
>>>>>>>fix the leak sadly. And setting a larger heap size only delays
>>>>>>>
>>>>>>>
>the
>
>
>>>>>>>inevitable Out Of Memory Exception.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>From our testing, it looks like it takes on the order of 700MB to
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>>serialize a 30MB java object graph into a 3MB SOAP stream.
>>>>>>>
>>>>>>>I've posted about this before, but no one has responded.
>>>>>>>
>>>>>>>
>Hopefully
>
>
>>>>>>>when we post the test case someone will actually take a look at
>>>>>>>
>>>>>>>
>it.
>
>
>>>>>>>There are other threads with test cases which currently have
>>>>>>>
>>>>>>>
>"we're
>
>
>>>>>>>looking at this now" as the only response, but as they are all
>>>>>>>
>>>>>>>
>>pretty
>>
>>
>>>>>>>old threads, my guess is that this problem either isn't believed
>>>>>>>
>>>>>>>
>or
>
>
>>>>>>>is being swept under the carpet. Maybe we're the only people
>>>>>>>
>>>>>>>
>using
>
>
>>>>>>>axis for large messages.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>--
>>>>>>>
>>>>>>>On Jun 3, 2004, at 9:38 AM, jira@apache.org wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>The following comment has been added to this issue:
>>>>>>>>
>>>>>>>> Author: Jennifer Jackson
>>>>>>>> Created: Thu, 3 Jun 2004 9:37 AM
>>>>>>>> Body:
>>>>>>>>Has anyone looked into this? I haven't used Axis because of
>>>>>>>>
>>>>>>>>
>this
>
>
>>>>>>>>problem and doing a quick search it seems others have come
>>>>>>>>
>>>>>>>>
>across
>
>
>>>>>>>>memory leaks also. I narrowed it down to the Call object not
>>>>>>>>cleaning out its member vars (the whole Deserialization
>>>>>>>>
>>>>>>>>
>process).
>
>
>>>>>>>>If you do that the only thing stuck in memory is the call
>>>>>>>>
>>>>>>>>
>object
>
>
>>>>>>>>since its part of the ThreadLocal object.
>>>>>>>>
>>>>>>>>
>>>>>>>------------------------------------------------------------------
>>>>>>>
>>>>>>>
>--
>
>
>>-
>>
>>
>>>>>>>>View this comment:
>>>>>>>> http://issues.apache.org/jira/browse/AXIS-935?
>>>>>>>>page=comments#action_35890
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>------------------------------------------------------------------
>>>>>>>
>>>>>>>
>--
>
>
>>-
>>
>>
>>>>>>>>View the issue:
>>>>>>>>http://issues.apache.org/jira/browse/AXIS-935
>>>>>>>>
>>>>>>>>Here is an overview of the issue:
>>>>>>>>
>>>>>>>>
>>>>>>>------------------------------------------------------------------
>>>>>>>
>>>>>>>
>--
>
>
>>-
>>
>>
>>>>>>>> Key: AXIS-935
>>>>>>>> Summary: Using ThreadLocal Call objects in a servlet memory
>>>>>>>>
>>>>>>>>
>leak
>
>
>>>>>>>>both on client and server side
>>>>>>>> Type: Bug
>>>>>>>>
>>>>>>>> Status: Open
>>>>>>>>
>>>>>>>> Project: Axis
>>>>>>>>Components:
>>>>>>>> Basic Architecture
>>>>>>>> Versions:
>>>>>>>> 1.1rc2
>>>>>>>>
>>>>>>>> Assignee: Axis Developers Mailing List
>>>>>>>> Reporter: Jennifer Jackson
>>>>>>>>
>>>>>>>> Created: Mon, 30 Jun 2003 12:12 PM
>>>>>>>> Updated: Thu, 3 Jun 2004 9:37 AM
>>>>>>>>Environment: Operating System: All
>>>>>>>>Platform: All
>>>>>>>>
>>>>>>>>Description:
>>>>>>>>I discovered using Axis in tomcat causes a memory leak.
>>>>>>>>
>>>>>>>>It seems using threadlocal in the classes causes a memory leak
>>>>>>>>
>>>>>>>>
>for
>
>
>>>>>>>>each Call
>>>>>>>>that is created. Threadlocal sticks around until the calling
>>>>>>>>
>>>>>>>>
>thread
>
>
>>>>>>>>dies, and
>>>>>>>>since tomcat keeps threads alive to answer new calls, the
>>>>>>>>threadlocal data
>>>>>>>>never goes away. The Call object gets stuck in the threadlocal
>>>>>>>>data, which
>>>>>>>>contains the entire soap response and all objects. This is a
>>>>>>>>
>>>>>>>>
>BIG
>
>
>>>>>>>>memory leak.
>>>>>>>>I initially cleared it by forcing all Call objects to clean
>>>>>>>>
>>>>>>>>
>their
>
>
>>>>>>>>member vars
>>>>>>>>after I was done with a call, then I realized the source was
>>>>>>>>
>>>>>>>>
>from
>
>
>>>>>>>>the thread
>>>>>>>>local class. Here is some info I found about threadlocal:
>>>>>>>>
>>>>>>>>This is stated in the Sun Javadocs for ThreadLocal:
>>>>>>>>
>>>>>>>> Each thread holds an implicit reference to its copy of a
>>>>>>>> ThreadLocal as long as the thread is alive and the
>>>>>>>> ThreadLocal object is accessible; after a thread goes
>>>>>>>> away, all of its copies of ThreadLocal variables are
>>>>>>>> subject to garbage collection (unless other references
>>>>>>>> to these copies exist).
>>>>>>>>
>>>>>>>>So, this means that ANY APPLICATION that uses PreparedStatements
>>>>>>>>
>>>>>>>>
>in
>
>
>>>>>>>>a thread
>>>>>>>>that 1) either does a lot of PreparedStatements or 2) never dies
>>>>>>>>(i.e., a
>>>>>>>>main thread) will ALWAYS eventually have an out of memory error.
>>>>>>>>Simply put,
>>>>>>>>this is a MEMORY LEAK. I imagine that the leak is very small,
>>>>>>>>
>>>>>>>>
>the
>
>
>>>>>>>>ThreadLocal object only contains one member variable, maybe 64
>>>>>>>>
>>>>>>>>
>bytes
>
>
>>>>>>>>or less
>>>>>>>>(depending on the VM implementation). So, our 60,000
>>>>>>>>PreparedStatements of 2
>>>>>>>>ThreadLocals each times 64 bytes (my wild guess) is 7.5MB.
>>>>>>>>
>>>>>>>>Ideas? I've never used threadlocal myself so this is new to me.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>------------------------------------------------------------------
>>>>>>>
>>>>>>>
>--
>
>
>>-
>>
>>
>>>>>>>>JIRA INFORMATION:
>>>>>>>>This message is automatically generated by JIRA.
>>>>>>>>
>>>>>>>>If you think it was sent incorrectly contact one of the
>>>>>>>>
>>>>>>>>
>>administrators:
>>
>>
>>>>>>>> http://issues.apache.org/jira/secure/Administrators.jspa
>>>>>>>>
>>>>>>>>If you want more information on JIRA, or have a bug to report
>>>>>>>>
>>>>>>>>
>see:
>
>
>>>>>>>> http://www.atlassian.com/software/jira
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>--
>>>>>>The best way to predict the future is to invent it - Alan Kay
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>--
>>>>The best way to predict the future is to invent it - Alan Kay
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>--
>>The best way to predict the future is to invent it - Alan Kay
>>
>>
>
>
>
>
--
The best way to predict the future is to invent it - Alan Kay
RE: more on AXIS-Java/C++ server side performance
Posted by Susantha Kumara <su...@opensource.lk>.
Hi Alek
You are doing a great work...!
Which parser did you use ?. Xerces or Expat. Last time we did
performance tests we found that Axis C++ is faster with Expat. 3 times
faster with Expat than Xerces.
Thanks,
---
Susantha Kumara
Virtusa (pvt) Ltd.
Office : +94112714385
Mobile : +94777420453
> -----Original Message-----
> From: Aleksander Slominski [mailto:aslom@cs.indiana.edu]
> Sent: Saturday, June 12, 2004 5:34 PM
> To: Apache AXIS C Developers List; 'axis-dev@ws.apache.org'
> Subject: more on AXIS-Java/C++ server side performance
>
> Davanum Srinivas wrote:
>
> >Alek,
> >
> >Where is the sample code for your XSOAP server-side code? does it
> >create a java object from the soap request?
> >
> hi dims,
>
> first let me emphasize that i do not look on this benchmark as only to
> indicate performance but also a tool
>
> now that i said that i think i can demonstrate usefulness of such
tests:
> to my big surprise after running tests i can see that AXIS-C++ is two
> times slower than AXIS-Java (in this sense the test work to help
detect
> possible huge performance bottlenecks may ultimately lead to much
better
> implementation).
>
> anyway this is really strange - i would expect C++ to be faster :)
>
> below is everything you need to reproduce test results yourself and
> maybe somebody will have an idea what is going on with AXIS-C++ ...
>
> the test driver (client side) is still not fully finished but here is
> working version:
> http://www.extreme.indiana.edu/~aslom/bnp/wsperf/driver/
>
> after downloading do source classpath.sh (or classpath.bat on windows)
> to set CLASSPATH
>
> run self test:
> java -Dserver.name=SELF_TEST soap_bench.BenchDriver
http://localhost:3434
>
> and here is full description of command line:
>
> java -Dserver.name=SERVICE_NAME soap_bench.Bench URL total
> {rsea}{bdisva} [arraySize][,arraySize][,...]
>
> parameters in order:
> 1. URL of service
> 2. total number of elements to send (default 10K)
> 3. specification of what to send : two letters
> [rse] means receive, send, or echo (a == all)
> [bdisva] means base64, double, int, string, void (only applies to
> echo), a == all methods;
> 4. list of arraySizes (optional for void) - default to 10
> and you can set test metadata such as -Dmachine.name=...
> -Dserver.name=... (it is included in tabulated results good for
> spreadsheet input and to draw graphs)
>
> those are optional system properties:
> -Dsmoke_test - will show you what messages were received and check
that
> all works fine
> -DAXISCPP - will add SOAPAction header and xsi:type for BASE64 (so
> AXIS-C++ is happy)
> -Dlog=trace.xsul.http.client:ALL -will show all messages and HTTP
> headers (good for debugging)
> -Dlog=:ALL - if you are doing desperate debugging
>
> use smoke test to get code to work with driver - it will just send 3
> messages and will print what it received - combine it with -Dlog=...
to
> get more fine grained information on what is happening and if/when
> driver is failing.
>
> the way i would do to get test driver to work with a SOAP service is
to
> it in multiple esteps and to get first to work different echo methods
>
> a) echoVoid
>
> java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200
ev
> 10,100
>
> echoInts
>
> java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200
ei
> 10,100
>
> echoDoubles
>
> java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200
ed
> 10,100
>
> echoStrings
>
> java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200
es
> 10,100
>
> echoBytes (as Base64)
>
> java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200
eb
> 10,100
>
> b) all echo methods
>
> java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200
ea
> 10,100
>
> c) all methods
>
> java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200
aa
> 10,100
>
> and then run real tests
>
> ----------------------
> For AXIS-Java
>
> get code from
http://www.extreme.indiana.edu/~aslom/bnp/wsperf/axis_java/
> and deploy benchmark service in tomcat - to do it just modify
> build.properties and do ant all
>
> run test
> java -Dmachine.name=YOUR_MACHINE_NAME -Dserver.name=AXIS_1_2_CVS
> soap_bench.BenchClient http://server:8080/axis/services/Benchmark1
> 200000 aa 10,1000,10000
>
> ----------------------
> For AXIS-C++
>
> get code from
http://www.extreme.indiana.edu/~aslom/bnp/wsperf/axis_cpp/
> copy soap_bench/ to samples/server (and client respectively)
> modify top level Makefile.am to include soap_bench (see example)
> rebuilt .so library
> add to list of deployed services (see deploy.wsdd in soap_bench)
> restart apache server
>
> run test (make sure -DAXISCPP parameter is included!!!!!)
> java -DAXISCPP -Dmachine.name=YOUR_MACHINE_NAME
> -Dserver.name=AXISCPP_1_0_CVS
> soap_bench.BenchClient http://server:8080/axis/Benchmark1 200000 ea
> 10,1000,10000
>
> NOTE: 'aa' will not work as all send tests do not work (and i do not
> know why). instead try ea (echo tests) and sa (send tests) - if you
want
> to see what is not working use sa and enable logging (-Dlog=:ALL)
>
> --------------------------
> For XSOAP4
>
> start service: java soap_bench.BenchServer 8077
>
> run test driver:
> java -Dmachine.name=YOUR_MACHINE_NAME -Dserver.name=XSOAP4
> soap_bench.BenchClient http://server:8077 200000 aa 10,1000,10000
>
> ----------------------
>
> comments are welcome especially concerning performance differences
> between Java and C++ and how to get send tests to work with AXIS
C++...
>
> thanks,
>
> alek
>
> >On Thu, 03 Jun 2004 12:58:41 -0500, Aleksander Slominski
> ><as...@cs.indiana.edu> wrote:
> >
> >
> >>Davanum Srinivas wrote:
> >>
> >>
> >>
> >>>You need to turn streaming on the client-side as well. see
> >>>samples/perf in latest cvs.
> >>>
> >>>
> binding._setProperty(org.apache.axis.client.Call.STREAMING_PROPERTY,
> >>>Boolean.TRUE);
> >>>
> >>>
> >>>
> >>>
> >>i am interested in testing server side performance (as server side
is
> >>typically the bottleneck) so i use the same client to access all
> >>services to measure only factors that are effecting server side
> >>performance (CPU, memory footprint, scalability).
> >>
> >>hopefully soon i will have AXIS-C++ numbers - i am really close now
to
> >>get them after few weeks of struggling with C++ (Java configuration
and
> >>build tools are much easier even when dealing with JAR hell ...)
> >>
> >>so it should be interesting to see how AXIS-Java and AXIS-C++
compare.
> >>
> >>however i am also *very* interested in very long running web
services so
> >>Peter post about _huge_ memory leaks that are easily reproducible
looked
> >>very interesting to me :) is it a property of data graph he send or
is
> >>it just a bug ...
> >>
> >>thanks,
> >>
> >>alek
> >>
> >>
> >>
> >>>-- dims
> >>>
> >>>On Thu, 03 Jun 2004 12:27:18 -0500, Aleksander Slominski
> >>><as...@cs.indiana.edu> wrote:
> >>>
> >>>
> >>>
> >>>
> >>>>Peter,
> >>>>
> >>>>please post more details how to reproduce the memory leak and i
will
> add
> >>>>to what i have for WS/SOAP performance tests at:
> >>>>http://www.extreme.indiana.edu/~aslom/bnp/wsperf/
> >>>>
> >>>>thanks,
> >>>>
> >>>>alek
> >>>>
> >>>>Peter Molettiere wrote:
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>We're currently looking into a massive memory leak in axis, that
no
> >>>>>one here seems willing to admit. We're close to having a minimal
> test
> >>>>>case which demonstrates the leak we're seeing. I don't think
it's
> the
> >>>>>same one you're seeing however, as our leak happens 1) when the
> >>>>>server is sitting doing nothing (but only after axis has been
> >>>>>loaded), and 2) every time we serialize a large message. The
> >>>>>suggestions on the list about switching to using attachments
doesn't
> >>>>>fix the leak sadly. And setting a larger heap size only delays
the
> >>>>>inevitable Out Of Memory Exception.
> >>>>>
> >>>>>
> >>>>>
> >>>>>From our testing, it looks like it takes on the order of 700MB to
> >>>>
> >>>>
> >>>>>serialize a 30MB java object graph into a 3MB SOAP stream.
> >>>>>
> >>>>>I've posted about this before, but no one has responded.
Hopefully
> >>>>>when we post the test case someone will actually take a look at
it.
> >>>>>There are other threads with test cases which currently have
"we're
> >>>>>looking at this now" as the only response, but as they are all
> pretty
> >>>>>old threads, my guess is that this problem either isn't believed
or
> >>>>>is being swept under the carpet. Maybe we're the only people
using
> >>>>>axis for large messages.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>--
> >>>>>
> >>>>>On Jun 3, 2004, at 9:38 AM, jira@apache.org wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>The following comment has been added to this issue:
> >>>>>>
> >>>>>> Author: Jennifer Jackson
> >>>>>> Created: Thu, 3 Jun 2004 9:37 AM
> >>>>>> Body:
> >>>>>>Has anyone looked into this? I haven't used Axis because of
this
> >>>>>>problem and doing a quick search it seems others have come
across
> >>>>>>memory leaks also. I narrowed it down to the Call object not
> >>>>>>cleaning out its member vars (the whole Deserialization
process).
> >>>>>>If you do that the only thing stuck in memory is the call
object
> >>>>>>since its part of the ThreadLocal object.
>
>>>>>>------------------------------------------------------------------
--
> -
> >>>>>>View this comment:
> >>>>>> http://issues.apache.org/jira/browse/AXIS-935?
> >>>>>>page=comments#action_35890
> >>>>>>
>
>>>>>>------------------------------------------------------------------
--
> -
> >>>>>>View the issue:
> >>>>>> http://issues.apache.org/jira/browse/AXIS-935
> >>>>>>
> >>>>>>Here is an overview of the issue:
>
>>>>>>------------------------------------------------------------------
--
> -
> >>>>>> Key: AXIS-935
> >>>>>> Summary: Using ThreadLocal Call objects in a servlet memory
leak
> >>>>>>both on client and server side
> >>>>>> Type: Bug
> >>>>>>
> >>>>>> Status: Open
> >>>>>>
> >>>>>> Project: Axis
> >>>>>>Components:
> >>>>>> Basic Architecture
> >>>>>> Versions:
> >>>>>> 1.1rc2
> >>>>>>
> >>>>>> Assignee: Axis Developers Mailing List
> >>>>>> Reporter: Jennifer Jackson
> >>>>>>
> >>>>>> Created: Mon, 30 Jun 2003 12:12 PM
> >>>>>> Updated: Thu, 3 Jun 2004 9:37 AM
> >>>>>>Environment: Operating System: All
> >>>>>>Platform: All
> >>>>>>
> >>>>>>Description:
> >>>>>>I discovered using Axis in tomcat causes a memory leak.
> >>>>>>
> >>>>>>It seems using threadlocal in the classes causes a memory leak
for
> >>>>>>each Call
> >>>>>>that is created. Threadlocal sticks around until the calling
thread
> >>>>>>dies, and
> >>>>>>since tomcat keeps threads alive to answer new calls, the
> >>>>>>threadlocal data
> >>>>>>never goes away. The Call object gets stuck in the threadlocal
> >>>>>>data, which
> >>>>>>contains the entire soap response and all objects. This is a
BIG
> >>>>>>memory leak.
> >>>>>>I initially cleared it by forcing all Call objects to clean
their
> >>>>>>member vars
> >>>>>>after I was done with a call, then I realized the source was
from
> >>>>>>the thread
> >>>>>>local class. Here is some info I found about threadlocal:
> >>>>>>
> >>>>>>This is stated in the Sun Javadocs for ThreadLocal:
> >>>>>>
> >>>>>> Each thread holds an implicit reference to its copy of a
> >>>>>> ThreadLocal as long as the thread is alive and the
> >>>>>> ThreadLocal object is accessible; after a thread goes
> >>>>>> away, all of its copies of ThreadLocal variables are
> >>>>>> subject to garbage collection (unless other references
> >>>>>> to these copies exist).
> >>>>>>
> >>>>>>So, this means that ANY APPLICATION that uses PreparedStatements
in
> >>>>>>a thread
> >>>>>>that 1) either does a lot of PreparedStatements or 2) never dies
> >>>>>>(i.e., a
> >>>>>>main thread) will ALWAYS eventually have an out of memory error.
> >>>>>>Simply put,
> >>>>>>this is a MEMORY LEAK. I imagine that the leak is very small,
the
> >>>>>>ThreadLocal object only contains one member variable, maybe 64
bytes
> >>>>>>or less
> >>>>>>(depending on the VM implementation). So, our 60,000
> >>>>>>PreparedStatements of 2
> >>>>>>ThreadLocals each times 64 bytes (my wild guess) is 7.5MB.
> >>>>>>
> >>>>>>Ideas? I've never used threadlocal myself so this is new to me.
> >>>>>>
> >>>>>>
>
>>>>>>------------------------------------------------------------------
--
> -
> >>>>>>JIRA INFORMATION:
> >>>>>>This message is automatically generated by JIRA.
> >>>>>>
> >>>>>>If you think it was sent incorrectly contact one of the
> administrators:
> >>>>>> http://issues.apache.org/jira/secure/Administrators.jspa
> >>>>>>
> >>>>>>If you want more information on JIRA, or have a bug to report
see:
> >>>>>> http://www.atlassian.com/software/jira
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>--
> >>>>The best way to predict the future is to invent it - Alan Kay
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>--
> >>The best way to predict the future is to invent it - Alan Kay
> >>
> >>
> >>
> >>
>
>
> --
> The best way to predict the future is to invent it - Alan Kay