You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-dev@axis.apache.org by ji...@apache.org on 2004/06/03 18:38:53 UTC
[jira] Commented: (AXIS-935) Using ThreadLocal Call objects in a servlet memory leak both on client and server side
The following comment has been added to this issue:
Author: Jennifer Jackson
Created: Thu, 3 Jun 2004 9:37 AM
Body:
Has anyone looked into this? I haven't used Axis because of this problem and doing a quick search it seems others have come across memory leaks also. I narrowed it down to the Call object not cleaning out its member vars (the whole Deserialization process). If you do that the only thing stuck in memory is the call object since its part of the ThreadLocal object.
---------------------------------------------------------------------
View this comment:
http://issues.apache.org/jira/browse/AXIS-935?page=comments#action_35890
---------------------------------------------------------------------
View the issue:
http://issues.apache.org/jira/browse/AXIS-935
Here is an overview of the issue:
---------------------------------------------------------------------
Key: AXIS-935
Summary: Using ThreadLocal Call objects in a servlet memory leak both on client and server side
Type: Bug
Status: Open
Project: Axis
Components:
Basic Architecture
Versions:
1.1rc2
Assignee: Axis Developers Mailing List
Reporter: Jennifer Jackson
Created: Mon, 30 Jun 2003 12:12 PM
Updated: Thu, 3 Jun 2004 9:37 AM
Environment: Operating System: All
Platform: All
Description:
I discovered using Axis in tomcat causes a memory leak.
It seems using threadlocal in the classes causes a memory leak for each Call
that is created. Threadlocal sticks around until the calling thread dies, and
since tomcat keeps threads alive to answer new calls, the threadlocal data
never goes away. The Call object gets stuck in the threadlocal data, which
contains the entire soap response and all objects. This is a BIG memory leak.
I initially cleared it by forcing all Call objects to clean their member vars
after I was done with a call, then I realized the source was from the thread
local class. Here is some info I found about threadlocal:
This is stated in the Sun Javadocs for ThreadLocal:
Each thread holds an implicit reference to its copy of a
ThreadLocal as long as the thread is alive and the
ThreadLocal object is accessible; after a thread goes
away, all of its copies of ThreadLocal variables are
subject to garbage collection (unless other references
to these copies exist).
So, this means that ANY APPLICATION that uses PreparedStatements in a thread
that 1) either does a lot of PreparedStatements or 2) never dies (i.e., a
main thread) will ALWAYS eventually have an out of memory error. Simply put,
this is a MEMORY LEAK. I imagine that the leak is very small, the
ThreadLocal object only contains one member variable, maybe 64 bytes or less
(depending on the VM implementation). So, our 60,000 PreparedStatements of 2
ThreadLocals each times 64 bytes (my wild guess) is 7.5MB.
Ideas? I've never used threadlocal myself so this is new to me.
---------------------------------------------------------------------
JIRA INFORMATION:
This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
Re: [jira] Commented: (AXIS-935) Using ThreadLocal Call objects in
a servlet memory leak both on client and server side
Posted by "David D. Lucas" <dd...@lse.com>.
We are using Axis with large messages in a long runner server with
thousands of requests in testing and our responses are usually 100KB to
10MB of binary data processed using Base64 (non-attachment).
Please post a test case and I will take a look at it with a profiler.
Include a JUnit as part of the test case.
One thing you must look at is the web server, JVM version, version of
axis and the xerces version that is actually being picked up. Use the
happyaxis.jsp and send us the output so we know what is in your
classpath and where.
I don't want to sound repeatitive, but have you verified that you are
closing all your result sets, statements, and connections ? You might
be losing memory if you are never closing them.
I use ThreadLocal a lot and have never had issues with leaks. Are you
using Native IO by any chance ?
Best wishes,
Dave
Peter Molettiere wrote:
>
>
> We're currently looking into a massive memory leak in axis, that no one
> here seems willing to admit. We're close to having a minimal test case
> which demonstrates the leak we're seeing. I don't think it's the same
> one you're seeing however, as our leak happens 1) when the server is
> sitting doing nothing (but only after axis has been loaded), and 2)
> every time we serialize a large message. The suggestions on the list
> about switching to using attachments doesn't fix the leak sadly. And
> setting a larger heap size only delays the inevitable Out Of Memory
> Exception.
>
> From our testing, it looks like it takes on the order of 700MB to
> serialize a 30MB java object graph into a 3MB SOAP stream.
>
> I've posted about this before, but no one has responded. Hopefully when
> we post the test case someone will actually take a look at it. There
> are other threads with test cases which currently have "we're looking
> at this now" as the only response, but as they are all pretty old
> threads, my guess is that this problem either isn't believed or is
> being swept under the carpet. Maybe we're the only people using axis
> for large messages.
>
> Hopefully, we'll have more info in the next couple days.
>
> --Peter
>
> On Jun 3, 2004, at 9:38 AM, jira@apache.org wrote:
>
>> The following comment has been added to this issue:
>>
>> Author: Jennifer Jackson
>> Created: Thu, 3 Jun 2004 9:37 AM
>> Body:
>> Has anyone looked into this? I haven't used Axis because of this
>> problem and doing a quick search it seems others have come across
>> memory leaks also. I narrowed it down to the Call object not
>> cleaning out its member vars (the whole Deserialization process). If
>> you do that the only thing stuck in memory is the call object since
>> its part of the ThreadLocal object.
>> ---------------------------------------------------------------------
>> View this comment:
>> http://issues.apache.org/jira/browse/AXIS-935?
>> page=comments#action_35890
>>
>> ---------------------------------------------------------------------
>> View the issue:
>> http://issues.apache.org/jira/browse/AXIS-935
>>
>> Here is an overview of the issue:
>> ---------------------------------------------------------------------
>> Key: AXIS-935
>> Summary: Using ThreadLocal Call objects in a servlet memory leak
>> both on client and server side
>> Type: Bug
>>
>> Status: Open
>>
>> Project: Axis
>> Components:
>> Basic Architecture
>> Versions:
>> 1.1rc2
>>
>> Assignee: Axis Developers Mailing List
>> Reporter: Jennifer Jackson
>>
>> Created: Mon, 30 Jun 2003 12:12 PM
>> Updated: Thu, 3 Jun 2004 9:37 AM
>> Environment: Operating System: All
>> Platform: All
>>
>> Description:
>> I discovered using Axis in tomcat causes a memory leak.
>>
>> It seems using threadlocal in the classes causes a memory leak for
>> each Call
>> that is created. Threadlocal sticks around until the calling thread
>> dies, and
>> since tomcat keeps threads alive to answer new calls, the threadlocal
>> data
>> never goes away. The Call object gets stuck in the threadlocal data,
>> which
>> contains the entire soap response and all objects. This is a BIG
>> memory leak.
>> I initially cleared it by forcing all Call objects to clean their
>> member vars
>> after I was done with a call, then I realized the source was from the
>> thread
>> local class. Here is some info I found about threadlocal:
>>
>> This is stated in the Sun Javadocs for ThreadLocal:
>>
>> Each thread holds an implicit reference to its copy of a
>> ThreadLocal as long as the thread is alive and the
>> ThreadLocal object is accessible; after a thread goes
>> away, all of its copies of ThreadLocal variables are
>> subject to garbage collection (unless other references
>> to these copies exist).
>>
>> So, this means that ANY APPLICATION that uses PreparedStatements in a
>> thread
>> that 1) either does a lot of PreparedStatements or 2) never dies
>> (i.e., a
>> main thread) will ALWAYS eventually have an out of memory error.
>> Simply put,
>> this is a MEMORY LEAK. I imagine that the leak is very small, the
>> ThreadLocal object only contains one member variable, maybe 64 bytes
>> or less
>> (depending on the VM implementation). So, our 60,000
>> PreparedStatements of 2
>> ThreadLocals each times 64 bytes (my wild guess) is 7.5MB.
>>
>> Ideas? I've never used threadlocal myself so this is new to me.
>>
>>
>> ---------------------------------------------------------------------
>> JIRA INFORMATION:
>> This message is automatically generated by JIRA.
>>
>> If you think it was sent incorrectly contact one of the administrators:
>> http://issues.apache.org/jira/secure/Administrators.jspa
>>
>> If you want more information on JIRA, or have a bug to report see:
>> http://www.atlassian.com/software/jira
>
>
--
+------------------------------------------------------------+
| David Lucas mailto: ddlucas @ lse.com |
| Lucas Software Engineering, Inc. (740) 964-6248 Voice |
| Unix,Java,C++,CORBA,XML,EJB (614) 668-4020 Mobile |
| Middleware,Frameworks (888) 866-4728 Fax/Msg |
+------------------------------------------------------------+
| GPS Location: 40.0150 deg Lat, -82.6378 deg Long |
| IMHC: "Jesus Christ is the way, the truth, and the life." |
| IMHC: "I know where I am; I know where I'm going." <>< |
+------------------------------------------------------------+
Notes: PGP Key Block=http://www.lse.com/~ddlucas/pgpblock.txt
IMHO="in my humble opinion" IMHC="in my humble conviction"
All trademarks above are those of their respective owners.
Re: more on AXIS-Java/C++ server side performance
Posted by Aleksander Slominski <as...@cs.indiana.edu>.
Susantha Kumara wrote:
>Hi Alek
>
>You are doing a great work...!
>
>Which parser did you use ?. Xerces or Expat.
>
expat.
>Last time we did
>performance tests we found that Axis C++ is faster with Expat. 3 times
>faster with Expat than Xerces.
>
>
ha tis ehat i expected as expat is supposed ot be the fastest XML parser
on C.
i will post more results and a graph soon.
thanks,
alek
>>-----Original Message-----
>>From: Aleksander Slominski [mailto:aslom@cs.indiana.edu]
>>Sent: Saturday, June 12, 2004 5:34 PM
>>To: Apache AXIS C Developers List; 'axis-dev@ws.apache.org'
>>Subject: more on AXIS-Java/C++ server side performance
>>
>>Davanum Srinivas wrote:
>>
>>
>>
>>>Alek,
>>>
>>>Where is the sample code for your XSOAP server-side code? does it
>>>create a java object from the soap request?
>>>
>>>
>>>
>>hi dims,
>>
>>first let me emphasize that i do not look on this benchmark as only to
>>indicate performance but also a tool
>>
>>now that i said that i think i can demonstrate usefulness of such
>>
>>
>tests:
>
>
>>to my big surprise after running tests i can see that AXIS-C++ is two
>>times slower than AXIS-Java (in this sense the test work to help
>>
>>
>detect
>
>
>>possible huge performance bottlenecks may ultimately lead to much
>>
>>
>better
>
>
>>implementation).
>>
>>anyway this is really strange - i would expect C++ to be faster :)
>>
>>below is everything you need to reproduce test results yourself and
>>maybe somebody will have an idea what is going on with AXIS-C++ ...
>>
>>the test driver (client side) is still not fully finished but here is
>>working version:
>>http://www.extreme.indiana.edu/~aslom/bnp/wsperf/driver/
>>
>>after downloading do source classpath.sh (or classpath.bat on windows)
>>to set CLASSPATH
>>
>>run self test:
>>java -Dserver.name=SELF_TEST soap_bench.BenchDriver
>>
>>
>http://localhost:3434
>
>
>>and here is full description of command line:
>>
>>java -Dserver.name=SERVICE_NAME soap_bench.Bench URL total
>>{rsea}{bdisva} [arraySize][,arraySize][,...]
>>
>>parameters in order:
>>1. URL of service
>>2. total number of elements to send (default 10K)
>>3. specification of what to send : two letters
>> [rse] means receive, send, or echo (a == all)
>> [bdisva] means base64, double, int, string, void (only applies to
>>echo), a == all methods;
>>4. list of arraySizes (optional for void) - default to 10
>>and you can set test metadata such as -Dmachine.name=...
>>-Dserver.name=... (it is included in tabulated results good for
>>spreadsheet input and to draw graphs)
>>
>>those are optional system properties:
>>-Dsmoke_test - will show you what messages were received and check
>>
>>
>that
>
>
>>all works fine
>>-DAXISCPP - will add SOAPAction header and xsi:type for BASE64 (so
>>AXIS-C++ is happy)
>>-Dlog=trace.xsul.http.client:ALL -will show all messages and HTTP
>>headers (good for debugging)
>>-Dlog=:ALL - if you are doing desperate debugging
>>
>>use smoke test to get code to work with driver - it will just send 3
>>messages and will print what it received - combine it with -Dlog=...
>>
>>
>to
>
>
>>get more fine grained information on what is happening and if/when
>>driver is failing.
>>
>>the way i would do to get test driver to work with a SOAP service is
>>
>>
>to
>
>
>>it in multiple esteps and to get first to work different echo methods
>>
>>a) echoVoid
>>
>>java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200
>>
>>
>ev
>
>
>>10,100
>>
>>echoInts
>>
>>java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200
>>
>>
>ei
>
>
>>10,100
>>
>>echoDoubles
>>
>>java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200
>>
>>
>ed
>
>
>>10,100
>>
>>echoStrings
>>
>>java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200
>>
>>
>es
>
>
>>10,100
>>
>>echoBytes (as Base64)
>>
>>java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200
>>
>>
>eb
>
>
>>10,100
>>
>>b) all echo methods
>>
>>java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200
>>
>>
>ea
>
>
>>10,100
>>
>>c) all methods
>>
>>java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200
>>
>>
>aa
>
>
>>10,100
>>
>>and then run real tests
>>
>>----------------------
>>For AXIS-Java
>>
>>get code from
>>
>>
>http://www.extreme.indiana.edu/~aslom/bnp/wsperf/axis_java/
>
>
>>and deploy benchmark service in tomcat - to do it just modify
>>build.properties and do ant all
>>
>>run test
>>java -Dmachine.name=YOUR_MACHINE_NAME -Dserver.name=AXIS_1_2_CVS
>>soap_bench.BenchClient http://server:8080/axis/services/Benchmark1
>>200000 aa 10,1000,10000
>>
>>----------------------
>>For AXIS-C++
>>
>>get code from
>>
>>
>http://www.extreme.indiana.edu/~aslom/bnp/wsperf/axis_cpp/
>
>
>>copy soap_bench/ to samples/server (and client respectively)
>>modify top level Makefile.am to include soap_bench (see example)
>>rebuilt .so library
>>add to list of deployed services (see deploy.wsdd in soap_bench)
>>restart apache server
>>
>>run test (make sure -DAXISCPP parameter is included!!!!!)
>>java -DAXISCPP -Dmachine.name=YOUR_MACHINE_NAME
>>-Dserver.name=AXISCPP_1_0_CVS
>>soap_bench.BenchClient http://server:8080/axis/Benchmark1 200000 ea
>>10,1000,10000
>>
>>NOTE: 'aa' will not work as all send tests do not work (and i do not
>>know why). instead try ea (echo tests) and sa (send tests) - if you
>>
>>
>want
>
>
>>to see what is not working use sa and enable logging (-Dlog=:ALL)
>>
>>--------------------------
>>For XSOAP4
>>
>>start service: java soap_bench.BenchServer 8077
>>
>>run test driver:
>>java -Dmachine.name=YOUR_MACHINE_NAME -Dserver.name=XSOAP4
>>soap_bench.BenchClient http://server:8077 200000 aa 10,1000,10000
>>
>>----------------------
>>
>>comments are welcome especially concerning performance differences
>>between Java and C++ and how to get send tests to work with AXIS
>>
>>
>C++...
>
>
>>thanks,
>>
>>alek
>>
>>
>>
>>>On Thu, 03 Jun 2004 12:58:41 -0500, Aleksander Slominski
>>><as...@cs.indiana.edu> wrote:
>>>
>>>
>>>
>>>
>>>>Davanum Srinivas wrote:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>You need to turn streaming on the client-side as well. see
>>>>>samples/perf in latest cvs.
>>>>>
>>>>>
>>>>>
>>>>>
>>binding._setProperty(org.apache.axis.client.Call.STREAMING_PROPERTY,
>>
>>
>>>>>Boolean.TRUE);
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>i am interested in testing server side performance (as server side
>>>>
>>>>
>is
>
>
>>>>typically the bottleneck) so i use the same client to access all
>>>>services to measure only factors that are effecting server side
>>>>performance (CPU, memory footprint, scalability).
>>>>
>>>>hopefully soon i will have AXIS-C++ numbers - i am really close now
>>>>
>>>>
>to
>
>
>>>>get them after few weeks of struggling with C++ (Java configuration
>>>>
>>>>
>and
>
>
>>>>build tools are much easier even when dealing with JAR hell ...)
>>>>
>>>>so it should be interesting to see how AXIS-Java and AXIS-C++
>>>>
>>>>
>compare.
>
>
>>>>however i am also *very* interested in very long running web
>>>>
>>>>
>services so
>
>
>>>>Peter post about _huge_ memory leaks that are easily reproducible
>>>>
>>>>
>looked
>
>
>>>>very interesting to me :) is it a property of data graph he send or
>>>>
>>>>
>is
>
>
>>>>it just a bug ...
>>>>
>>>>thanks,
>>>>
>>>>alek
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>-- dims
>>>>>
>>>>>On Thu, 03 Jun 2004 12:27:18 -0500, Aleksander Slominski
>>>>><as...@cs.indiana.edu> wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>Peter,
>>>>>>
>>>>>>please post more details how to reproduce the memory leak and i
>>>>>>
>>>>>>
>will
>
>
>>add
>>
>>
>>>>>>to what i have for WS/SOAP performance tests at:
>>>>>>http://www.extreme.indiana.edu/~aslom/bnp/wsperf/
>>>>>>
>>>>>>thanks,
>>>>>>
>>>>>>alek
>>>>>>
>>>>>>Peter Molettiere wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>>We're currently looking into a massive memory leak in axis, that
>>>>>>>
>>>>>>>
>no
>
>
>>>>>>>one here seems willing to admit. We're close to having a minimal
>>>>>>>
>>>>>>>
>>test
>>
>>
>>>>>>>case which demonstrates the leak we're seeing. I don't think
>>>>>>>
>>>>>>>
>it's
>
>
>>the
>>
>>
>>>>>>>same one you're seeing however, as our leak happens 1) when the
>>>>>>>server is sitting doing nothing (but only after axis has been
>>>>>>>loaded), and 2) every time we serialize a large message. The
>>>>>>>suggestions on the list about switching to using attachments
>>>>>>>
>>>>>>>
>doesn't
>
>
>>>>>>>fix the leak sadly. And setting a larger heap size only delays
>>>>>>>
>>>>>>>
>the
>
>
>>>>>>>inevitable Out Of Memory Exception.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>From our testing, it looks like it takes on the order of 700MB to
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>>serialize a 30MB java object graph into a 3MB SOAP stream.
>>>>>>>
>>>>>>>I've posted about this before, but no one has responded.
>>>>>>>
>>>>>>>
>Hopefully
>
>
>>>>>>>when we post the test case someone will actually take a look at
>>>>>>>
>>>>>>>
>it.
>
>
>>>>>>>There are other threads with test cases which currently have
>>>>>>>
>>>>>>>
>"we're
>
>
>>>>>>>looking at this now" as the only response, but as they are all
>>>>>>>
>>>>>>>
>>pretty
>>
>>
>>>>>>>old threads, my guess is that this problem either isn't believed
>>>>>>>
>>>>>>>
>or
>
>
>>>>>>>is being swept under the carpet. Maybe we're the only people
>>>>>>>
>>>>>>>
>using
>
>
>>>>>>>axis for large messages.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>--
>>>>>>>
>>>>>>>On Jun 3, 2004, at 9:38 AM, jira@apache.org wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>The following comment has been added to this issue:
>>>>>>>>
>>>>>>>> Author: Jennifer Jackson
>>>>>>>> Created: Thu, 3 Jun 2004 9:37 AM
>>>>>>>> Body:
>>>>>>>>Has anyone looked into this? I haven't used Axis because of
>>>>>>>>
>>>>>>>>
>this
>
>
>>>>>>>>problem and doing a quick search it seems others have come
>>>>>>>>
>>>>>>>>
>across
>
>
>>>>>>>>memory leaks also. I narrowed it down to the Call object not
>>>>>>>>cleaning out its member vars (the whole Deserialization
>>>>>>>>
>>>>>>>>
>process).
>
>
>>>>>>>>If you do that the only thing stuck in memory is the call
>>>>>>>>
>>>>>>>>
>object
>
>
>>>>>>>>since its part of the ThreadLocal object.
>>>>>>>>
>>>>>>>>
>>>>>>>------------------------------------------------------------------
>>>>>>>
>>>>>>>
>--
>
>
>>-
>>
>>
>>>>>>>>View this comment:
>>>>>>>> http://issues.apache.org/jira/browse/AXIS-935?
>>>>>>>>page=comments#action_35890
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>------------------------------------------------------------------
>>>>>>>
>>>>>>>
>--
>
>
>>-
>>
>>
>>>>>>>>View the issue:
>>>>>>>>http://issues.apache.org/jira/browse/AXIS-935
>>>>>>>>
>>>>>>>>Here is an overview of the issue:
>>>>>>>>
>>>>>>>>
>>>>>>>------------------------------------------------------------------
>>>>>>>
>>>>>>>
>--
>
>
>>-
>>
>>
>>>>>>>> Key: AXIS-935
>>>>>>>> Summary: Using ThreadLocal Call objects in a servlet memory
>>>>>>>>
>>>>>>>>
>leak
>
>
>>>>>>>>both on client and server side
>>>>>>>> Type: Bug
>>>>>>>>
>>>>>>>> Status: Open
>>>>>>>>
>>>>>>>> Project: Axis
>>>>>>>>Components:
>>>>>>>> Basic Architecture
>>>>>>>> Versions:
>>>>>>>> 1.1rc2
>>>>>>>>
>>>>>>>> Assignee: Axis Developers Mailing List
>>>>>>>> Reporter: Jennifer Jackson
>>>>>>>>
>>>>>>>> Created: Mon, 30 Jun 2003 12:12 PM
>>>>>>>> Updated: Thu, 3 Jun 2004 9:37 AM
>>>>>>>>Environment: Operating System: All
>>>>>>>>Platform: All
>>>>>>>>
>>>>>>>>Description:
>>>>>>>>I discovered using Axis in tomcat causes a memory leak.
>>>>>>>>
>>>>>>>>It seems using threadlocal in the classes causes a memory leak
>>>>>>>>
>>>>>>>>
>for
>
>
>>>>>>>>each Call
>>>>>>>>that is created. Threadlocal sticks around until the calling
>>>>>>>>
>>>>>>>>
>thread
>
>
>>>>>>>>dies, and
>>>>>>>>since tomcat keeps threads alive to answer new calls, the
>>>>>>>>threadlocal data
>>>>>>>>never goes away. The Call object gets stuck in the threadlocal
>>>>>>>>data, which
>>>>>>>>contains the entire soap response and all objects. This is a
>>>>>>>>
>>>>>>>>
>BIG
>
>
>>>>>>>>memory leak.
>>>>>>>>I initially cleared it by forcing all Call objects to clean
>>>>>>>>
>>>>>>>>
>their
>
>
>>>>>>>>member vars
>>>>>>>>after I was done with a call, then I realized the source was
>>>>>>>>
>>>>>>>>
>from
>
>
>>>>>>>>the thread
>>>>>>>>local class. Here is some info I found about threadlocal:
>>>>>>>>
>>>>>>>>This is stated in the Sun Javadocs for ThreadLocal:
>>>>>>>>
>>>>>>>> Each thread holds an implicit reference to its copy of a
>>>>>>>> ThreadLocal as long as the thread is alive and the
>>>>>>>> ThreadLocal object is accessible; after a thread goes
>>>>>>>> away, all of its copies of ThreadLocal variables are
>>>>>>>> subject to garbage collection (unless other references
>>>>>>>> to these copies exist).
>>>>>>>>
>>>>>>>>So, this means that ANY APPLICATION that uses PreparedStatements
>>>>>>>>
>>>>>>>>
>in
>
>
>>>>>>>>a thread
>>>>>>>>that 1) either does a lot of PreparedStatements or 2) never dies
>>>>>>>>(i.e., a
>>>>>>>>main thread) will ALWAYS eventually have an out of memory error.
>>>>>>>>Simply put,
>>>>>>>>this is a MEMORY LEAK. I imagine that the leak is very small,
>>>>>>>>
>>>>>>>>
>the
>
>
>>>>>>>>ThreadLocal object only contains one member variable, maybe 64
>>>>>>>>
>>>>>>>>
>bytes
>
>
>>>>>>>>or less
>>>>>>>>(depending on the VM implementation). So, our 60,000
>>>>>>>>PreparedStatements of 2
>>>>>>>>ThreadLocals each times 64 bytes (my wild guess) is 7.5MB.
>>>>>>>>
>>>>>>>>Ideas? I've never used threadlocal myself so this is new to me.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>------------------------------------------------------------------
>>>>>>>
>>>>>>>
>--
>
>
>>-
>>
>>
>>>>>>>>JIRA INFORMATION:
>>>>>>>>This message is automatically generated by JIRA.
>>>>>>>>
>>>>>>>>If you think it was sent incorrectly contact one of the
>>>>>>>>
>>>>>>>>
>>administrators:
>>
>>
>>>>>>>> http://issues.apache.org/jira/secure/Administrators.jspa
>>>>>>>>
>>>>>>>>If you want more information on JIRA, or have a bug to report
>>>>>>>>
>>>>>>>>
>see:
>
>
>>>>>>>> http://www.atlassian.com/software/jira
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>--
>>>>>>The best way to predict the future is to invent it - Alan Kay
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>--
>>>>The best way to predict the future is to invent it - Alan Kay
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>--
>>The best way to predict the future is to invent it - Alan Kay
>>
>>
>
>
>
>
--
The best way to predict the future is to invent it - Alan Kay
RE: more on AXIS-Java/C++ server side performance
Posted by Susantha Kumara <su...@opensource.lk>.
Hi Alek
You are doing a great work...!
Which parser did you use ?. Xerces or Expat. Last time we did
performance tests we found that Axis C++ is faster with Expat. 3 times
faster with Expat than Xerces.
Thanks,
---
Susantha Kumara
Virtusa (pvt) Ltd.
Office : +94112714385
Mobile : +94777420453
> -----Original Message-----
> From: Aleksander Slominski [mailto:aslom@cs.indiana.edu]
> Sent: Saturday, June 12, 2004 5:34 PM
> To: Apache AXIS C Developers List; 'axis-dev@ws.apache.org'
> Subject: more on AXIS-Java/C++ server side performance
>
> Davanum Srinivas wrote:
>
> >Alek,
> >
> >Where is the sample code for your XSOAP server-side code? does it
> >create a java object from the soap request?
> >
> hi dims,
>
> first let me emphasize that i do not look on this benchmark as only to
> indicate performance but also a tool
>
> now that i said that i think i can demonstrate usefulness of such
tests:
> to my big surprise after running tests i can see that AXIS-C++ is two
> times slower than AXIS-Java (in this sense the test work to help
detect
> possible huge performance bottlenecks may ultimately lead to much
better
> implementation).
>
> anyway this is really strange - i would expect C++ to be faster :)
>
> below is everything you need to reproduce test results yourself and
> maybe somebody will have an idea what is going on with AXIS-C++ ...
>
> the test driver (client side) is still not fully finished but here is
> working version:
> http://www.extreme.indiana.edu/~aslom/bnp/wsperf/driver/
>
> after downloading do source classpath.sh (or classpath.bat on windows)
> to set CLASSPATH
>
> run self test:
> java -Dserver.name=SELF_TEST soap_bench.BenchDriver
http://localhost:3434
>
> and here is full description of command line:
>
> java -Dserver.name=SERVICE_NAME soap_bench.Bench URL total
> {rsea}{bdisva} [arraySize][,arraySize][,...]
>
> parameters in order:
> 1. URL of service
> 2. total number of elements to send (default 10K)
> 3. specification of what to send : two letters
> [rse] means receive, send, or echo (a == all)
> [bdisva] means base64, double, int, string, void (only applies to
> echo), a == all methods;
> 4. list of arraySizes (optional for void) - default to 10
> and you can set test metadata such as -Dmachine.name=...
> -Dserver.name=... (it is included in tabulated results good for
> spreadsheet input and to draw graphs)
>
> those are optional system properties:
> -Dsmoke_test - will show you what messages were received and check
that
> all works fine
> -DAXISCPP - will add SOAPAction header and xsi:type for BASE64 (so
> AXIS-C++ is happy)
> -Dlog=trace.xsul.http.client:ALL -will show all messages and HTTP
> headers (good for debugging)
> -Dlog=:ALL - if you are doing desperate debugging
>
> use smoke test to get code to work with driver - it will just send 3
> messages and will print what it received - combine it with -Dlog=...
to
> get more fine grained information on what is happening and if/when
> driver is failing.
>
> the way i would do to get test driver to work with a SOAP service is
to
> it in multiple esteps and to get first to work different echo methods
>
> a) echoVoid
>
> java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200
ev
> 10,100
>
> echoInts
>
> java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200
ei
> 10,100
>
> echoDoubles
>
> java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200
ed
> 10,100
>
> echoStrings
>
> java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200
es
> 10,100
>
> echoBytes (as Base64)
>
> java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200
eb
> 10,100
>
> b) all echo methods
>
> java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200
ea
> 10,100
>
> c) all methods
>
> java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200
aa
> 10,100
>
> and then run real tests
>
> ----------------------
> For AXIS-Java
>
> get code from
http://www.extreme.indiana.edu/~aslom/bnp/wsperf/axis_java/
> and deploy benchmark service in tomcat - to do it just modify
> build.properties and do ant all
>
> run test
> java -Dmachine.name=YOUR_MACHINE_NAME -Dserver.name=AXIS_1_2_CVS
> soap_bench.BenchClient http://server:8080/axis/services/Benchmark1
> 200000 aa 10,1000,10000
>
> ----------------------
> For AXIS-C++
>
> get code from
http://www.extreme.indiana.edu/~aslom/bnp/wsperf/axis_cpp/
> copy soap_bench/ to samples/server (and client respectively)
> modify top level Makefile.am to include soap_bench (see example)
> rebuilt .so library
> add to list of deployed services (see deploy.wsdd in soap_bench)
> restart apache server
>
> run test (make sure -DAXISCPP parameter is included!!!!!)
> java -DAXISCPP -Dmachine.name=YOUR_MACHINE_NAME
> -Dserver.name=AXISCPP_1_0_CVS
> soap_bench.BenchClient http://server:8080/axis/Benchmark1 200000 ea
> 10,1000,10000
>
> NOTE: 'aa' will not work as all send tests do not work (and i do not
> know why). instead try ea (echo tests) and sa (send tests) - if you
want
> to see what is not working use sa and enable logging (-Dlog=:ALL)
>
> --------------------------
> For XSOAP4
>
> start service: java soap_bench.BenchServer 8077
>
> run test driver:
> java -Dmachine.name=YOUR_MACHINE_NAME -Dserver.name=XSOAP4
> soap_bench.BenchClient http://server:8077 200000 aa 10,1000,10000
>
> ----------------------
>
> comments are welcome especially concerning performance differences
> between Java and C++ and how to get send tests to work with AXIS
C++...
>
> thanks,
>
> alek
>
> >On Thu, 03 Jun 2004 12:58:41 -0500, Aleksander Slominski
> ><as...@cs.indiana.edu> wrote:
> >
> >
> >>Davanum Srinivas wrote:
> >>
> >>
> >>
> >>>You need to turn streaming on the client-side as well. see
> >>>samples/perf in latest cvs.
> >>>
> >>>
> binding._setProperty(org.apache.axis.client.Call.STREAMING_PROPERTY,
> >>>Boolean.TRUE);
> >>>
> >>>
> >>>
> >>>
> >>i am interested in testing server side performance (as server side
is
> >>typically the bottleneck) so i use the same client to access all
> >>services to measure only factors that are effecting server side
> >>performance (CPU, memory footprint, scalability).
> >>
> >>hopefully soon i will have AXIS-C++ numbers - i am really close now
to
> >>get them after few weeks of struggling with C++ (Java configuration
and
> >>build tools are much easier even when dealing with JAR hell ...)
> >>
> >>so it should be interesting to see how AXIS-Java and AXIS-C++
compare.
> >>
> >>however i am also *very* interested in very long running web
services so
> >>Peter post about _huge_ memory leaks that are easily reproducible
looked
> >>very interesting to me :) is it a property of data graph he send or
is
> >>it just a bug ...
> >>
> >>thanks,
> >>
> >>alek
> >>
> >>
> >>
> >>>-- dims
> >>>
> >>>On Thu, 03 Jun 2004 12:27:18 -0500, Aleksander Slominski
> >>><as...@cs.indiana.edu> wrote:
> >>>
> >>>
> >>>
> >>>
> >>>>Peter,
> >>>>
> >>>>please post more details how to reproduce the memory leak and i
will
> add
> >>>>to what i have for WS/SOAP performance tests at:
> >>>>http://www.extreme.indiana.edu/~aslom/bnp/wsperf/
> >>>>
> >>>>thanks,
> >>>>
> >>>>alek
> >>>>
> >>>>Peter Molettiere wrote:
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>We're currently looking into a massive memory leak in axis, that
no
> >>>>>one here seems willing to admit. We're close to having a minimal
> test
> >>>>>case which demonstrates the leak we're seeing. I don't think
it's
> the
> >>>>>same one you're seeing however, as our leak happens 1) when the
> >>>>>server is sitting doing nothing (but only after axis has been
> >>>>>loaded), and 2) every time we serialize a large message. The
> >>>>>suggestions on the list about switching to using attachments
doesn't
> >>>>>fix the leak sadly. And setting a larger heap size only delays
the
> >>>>>inevitable Out Of Memory Exception.
> >>>>>
> >>>>>
> >>>>>
> >>>>>From our testing, it looks like it takes on the order of 700MB to
> >>>>
> >>>>
> >>>>>serialize a 30MB java object graph into a 3MB SOAP stream.
> >>>>>
> >>>>>I've posted about this before, but no one has responded.
Hopefully
> >>>>>when we post the test case someone will actually take a look at
it.
> >>>>>There are other threads with test cases which currently have
"we're
> >>>>>looking at this now" as the only response, but as they are all
> pretty
> >>>>>old threads, my guess is that this problem either isn't believed
or
> >>>>>is being swept under the carpet. Maybe we're the only people
using
> >>>>>axis for large messages.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>--
> >>>>>
> >>>>>On Jun 3, 2004, at 9:38 AM, jira@apache.org wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>The following comment has been added to this issue:
> >>>>>>
> >>>>>> Author: Jennifer Jackson
> >>>>>> Created: Thu, 3 Jun 2004 9:37 AM
> >>>>>> Body:
> >>>>>>Has anyone looked into this? I haven't used Axis because of
this
> >>>>>>problem and doing a quick search it seems others have come
across
> >>>>>>memory leaks also. I narrowed it down to the Call object not
> >>>>>>cleaning out its member vars (the whole Deserialization
process).
> >>>>>>If you do that the only thing stuck in memory is the call
object
> >>>>>>since its part of the ThreadLocal object.
>
>>>>>>------------------------------------------------------------------
--
> -
> >>>>>>View this comment:
> >>>>>> http://issues.apache.org/jira/browse/AXIS-935?
> >>>>>>page=comments#action_35890
> >>>>>>
>
>>>>>>------------------------------------------------------------------
--
> -
> >>>>>>View the issue:
> >>>>>> http://issues.apache.org/jira/browse/AXIS-935
> >>>>>>
> >>>>>>Here is an overview of the issue:
>
>>>>>>------------------------------------------------------------------
--
> -
> >>>>>> Key: AXIS-935
> >>>>>> Summary: Using ThreadLocal Call objects in a servlet memory
leak
> >>>>>>both on client and server side
> >>>>>> Type: Bug
> >>>>>>
> >>>>>> Status: Open
> >>>>>>
> >>>>>> Project: Axis
> >>>>>>Components:
> >>>>>> Basic Architecture
> >>>>>> Versions:
> >>>>>> 1.1rc2
> >>>>>>
> >>>>>> Assignee: Axis Developers Mailing List
> >>>>>> Reporter: Jennifer Jackson
> >>>>>>
> >>>>>> Created: Mon, 30 Jun 2003 12:12 PM
> >>>>>> Updated: Thu, 3 Jun 2004 9:37 AM
> >>>>>>Environment: Operating System: All
> >>>>>>Platform: All
> >>>>>>
> >>>>>>Description:
> >>>>>>I discovered using Axis in tomcat causes a memory leak.
> >>>>>>
> >>>>>>It seems using threadlocal in the classes causes a memory leak
for
> >>>>>>each Call
> >>>>>>that is created. Threadlocal sticks around until the calling
thread
> >>>>>>dies, and
> >>>>>>since tomcat keeps threads alive to answer new calls, the
> >>>>>>threadlocal data
> >>>>>>never goes away. The Call object gets stuck in the threadlocal
> >>>>>>data, which
> >>>>>>contains the entire soap response and all objects. This is a
BIG
> >>>>>>memory leak.
> >>>>>>I initially cleared it by forcing all Call objects to clean
their
> >>>>>>member vars
> >>>>>>after I was done with a call, then I realized the source was
from
> >>>>>>the thread
> >>>>>>local class. Here is some info I found about threadlocal:
> >>>>>>
> >>>>>>This is stated in the Sun Javadocs for ThreadLocal:
> >>>>>>
> >>>>>> Each thread holds an implicit reference to its copy of a
> >>>>>> ThreadLocal as long as the thread is alive and the
> >>>>>> ThreadLocal object is accessible; after a thread goes
> >>>>>> away, all of its copies of ThreadLocal variables are
> >>>>>> subject to garbage collection (unless other references
> >>>>>> to these copies exist).
> >>>>>>
> >>>>>>So, this means that ANY APPLICATION that uses PreparedStatements
in
> >>>>>>a thread
> >>>>>>that 1) either does a lot of PreparedStatements or 2) never dies
> >>>>>>(i.e., a
> >>>>>>main thread) will ALWAYS eventually have an out of memory error.
> >>>>>>Simply put,
> >>>>>>this is a MEMORY LEAK. I imagine that the leak is very small,
the
> >>>>>>ThreadLocal object only contains one member variable, maybe 64
bytes
> >>>>>>or less
> >>>>>>(depending on the VM implementation). So, our 60,000
> >>>>>>PreparedStatements of 2
> >>>>>>ThreadLocals each times 64 bytes (my wild guess) is 7.5MB.
> >>>>>>
> >>>>>>Ideas? I've never used threadlocal myself so this is new to me.
> >>>>>>
> >>>>>>
>
>>>>>>------------------------------------------------------------------
--
> -
> >>>>>>JIRA INFORMATION:
> >>>>>>This message is automatically generated by JIRA.
> >>>>>>
> >>>>>>If you think it was sent incorrectly contact one of the
> administrators:
> >>>>>> http://issues.apache.org/jira/secure/Administrators.jspa
> >>>>>>
> >>>>>>If you want more information on JIRA, or have a bug to report
see:
> >>>>>> http://www.atlassian.com/software/jira
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>--
> >>>>The best way to predict the future is to invent it - Alan Kay
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>--
> >>The best way to predict the future is to invent it - Alan Kay
> >>
> >>
> >>
> >>
>
>
> --
> The best way to predict the future is to invent it - Alan Kay
more on AXIS-Java/C++ server side performance
Posted by Aleksander Slominski <as...@cs.indiana.edu>.
Davanum Srinivas wrote:
>Alek,
>
>Where is the sample code for your XSOAP server-side code? does it
>create a java object from the soap request?
>
hi dims,
first let me emphasize that i do not look on this benchmark as only to
indicate performance but also a tool
now that i said that i think i can demonstrate usefulness of such tests:
to my big surprise after running tests i can see that AXIS-C++ is two
times slower than AXIS-Java (in this sense the test work to help detect
possible huge performance bottlenecks may ultimately lead to much better
implementation).
anyway this is really strange - i would expect C++ to be faster :)
below is everything you need to reproduce test results yourself and
maybe somebody will have an idea what is going on with AXIS-C++ ...
the test driver (client side) is still not fully finished but here is
working version:
http://www.extreme.indiana.edu/~aslom/bnp/wsperf/driver/
after downloading do source classpath.sh (or classpath.bat on windows)
to set CLASSPATH
run self test:
java -Dserver.name=SELF_TEST soap_bench.BenchDriver http://localhost:3434
and here is full description of command line:
java -Dserver.name=SERVICE_NAME soap_bench.Bench URL total
{rsea}{bdisva} [arraySize][,arraySize][,...]
parameters in order:
1. URL of service
2. total number of elements to send (default 10K)
3. specification of what to send : two letters
[rse] means receive, send, or echo (a == all)
[bdisva] means base64, double, int, string, void (only applies to
echo), a == all methods;
4. list of arraySizes (optional for void) - default to 10
and you can set test metadata such as -Dmachine.name=...
-Dserver.name=... (it is included in tabulated results good for
spreadsheet input and to draw graphs)
those are optional system properties:
-Dsmoke_test - will show you what messages were received and check that
all works fine
-DAXISCPP - will add SOAPAction header and xsi:type for BASE64 (so
AXIS-C++ is happy)
-Dlog=trace.xsul.http.client:ALL -will show all messages and HTTP
headers (good for debugging)
-Dlog=:ALL - if you are doing desperate debugging
use smoke test to get code to work with driver - it will just send 3
messages and will print what it received - combine it with -Dlog=... to
get more fine grained information on what is happening and if/when
driver is failing.
the way i would do to get test driver to work with a SOAP service is to
it in multiple esteps and to get first to work different echo methods
a) echoVoid
java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200 ev
10,100
echoInts
java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200 ei
10,100
echoDoubles
java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200 ed
10,100
echoStrings
java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200 es
10,100
echoBytes (as Base64)
java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200 eb
10,100
b) all echo methods
java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200 ea
10,100
c) all methods
java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200 aa
10,100
and then run real tests
----------------------
For AXIS-Java
get code from http://www.extreme.indiana.edu/~aslom/bnp/wsperf/axis_java/
and deploy benchmark service in tomcat - to do it just modify
build.properties and do ant all
run test
java -Dmachine.name=YOUR_MACHINE_NAME -Dserver.name=AXIS_1_2_CVS
soap_bench.BenchClient http://server:8080/axis/services/Benchmark1
200000 aa 10,1000,10000
----------------------
For AXIS-C++
get code from http://www.extreme.indiana.edu/~aslom/bnp/wsperf/axis_cpp/
copy soap_bench/ to samples/server (and client respectively)
modify top level Makefile.am to include soap_bench (see example)
rebuilt .so library
add to list of deployed services (see deploy.wsdd in soap_bench)
restart apache server
run test (make sure -DAXISCPP parameter is included!!!!!)
java -DAXISCPP -Dmachine.name=YOUR_MACHINE_NAME
-Dserver.name=AXISCPP_1_0_CVS
soap_bench.BenchClient http://server:8080/axis/Benchmark1 200000 ea
10,1000,10000
NOTE: 'aa' will not work as all send tests do not work (and i do not
know why). instead try ea (echo tests) and sa (send tests) - if you want
to see what is not working use sa and enable logging (-Dlog=:ALL)
--------------------------
For XSOAP4
start service: java soap_bench.BenchServer 8077
run test driver:
java -Dmachine.name=YOUR_MACHINE_NAME -Dserver.name=XSOAP4
soap_bench.BenchClient http://server:8077 200000 aa 10,1000,10000
----------------------
comments are welcome especially concerning performance differences
between Java and C++ and how to get send tests to work with AXIS C++...
thanks,
alek
>On Thu, 03 Jun 2004 12:58:41 -0500, Aleksander Slominski
><as...@cs.indiana.edu> wrote:
>
>
>>Davanum Srinivas wrote:
>>
>>
>>
>>>You need to turn streaming on the client-side as well. see
>>>samples/perf in latest cvs.
>>>
>>> binding._setProperty(org.apache.axis.client.Call.STREAMING_PROPERTY,
>>>Boolean.TRUE);
>>>
>>>
>>>
>>>
>>i am interested in testing server side performance (as server side is
>>typically the bottleneck) so i use the same client to access all
>>services to measure only factors that are effecting server side
>>performance (CPU, memory footprint, scalability).
>>
>>hopefully soon i will have AXIS-C++ numbers - i am really close now to
>>get them after few weeks of struggling with C++ (Java configuration and
>>build tools are much easier even when dealing with JAR hell ...)
>>
>>so it should be interesting to see how AXIS-Java and AXIS-C++ compare.
>>
>>however i am also *very* interested in very long running web services so
>>Peter post about _huge_ memory leaks that are easily reproducible looked
>>very interesting to me :) is it a property of data graph he send or is
>>it just a bug ...
>>
>>thanks,
>>
>>alek
>>
>>
>>
>>>-- dims
>>>
>>>On Thu, 03 Jun 2004 12:27:18 -0500, Aleksander Slominski
>>><as...@cs.indiana.edu> wrote:
>>>
>>>
>>>
>>>
>>>>Peter,
>>>>
>>>>please post more details how to reproduce the memory leak and i will add
>>>>to what i have for WS/SOAP performance tests at:
>>>>http://www.extreme.indiana.edu/~aslom/bnp/wsperf/
>>>>
>>>>thanks,
>>>>
>>>>alek
>>>>
>>>>Peter Molettiere wrote:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>We're currently looking into a massive memory leak in axis, that no
>>>>>one here seems willing to admit. We're close to having a minimal test
>>>>>case which demonstrates the leak we're seeing. I don't think it's the
>>>>>same one you're seeing however, as our leak happens 1) when the
>>>>>server is sitting doing nothing (but only after axis has been
>>>>>loaded), and 2) every time we serialize a large message. The
>>>>>suggestions on the list about switching to using attachments doesn't
>>>>>fix the leak sadly. And setting a larger heap size only delays the
>>>>>inevitable Out Of Memory Exception.
>>>>>
>>>>>
>>>>>
>>>>>>From our testing, it looks like it takes on the order of 700MB to
>>>>
>>>>
>>>>>serialize a 30MB java object graph into a 3MB SOAP stream.
>>>>>
>>>>>I've posted about this before, but no one has responded. Hopefully
>>>>>when we post the test case someone will actually take a look at it.
>>>>>There are other threads with test cases which currently have "we're
>>>>>looking at this now" as the only response, but as they are all pretty
>>>>>old threads, my guess is that this problem either isn't believed or
>>>>>is being swept under the carpet. Maybe we're the only people using
>>>>>axis for large messages.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>--
>>>>>
>>>>>On Jun 3, 2004, at 9:38 AM, jira@apache.org wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>The following comment has been added to this issue:
>>>>>>
>>>>>> Author: Jennifer Jackson
>>>>>> Created: Thu, 3 Jun 2004 9:37 AM
>>>>>> Body:
>>>>>>Has anyone looked into this? I haven't used Axis because of this
>>>>>>problem and doing a quick search it seems others have come across
>>>>>>memory leaks also. I narrowed it down to the Call object not
>>>>>>cleaning out its member vars (the whole Deserialization process).
>>>>>>If you do that the only thing stuck in memory is the call object
>>>>>>since its part of the ThreadLocal object.
>>>>>>---------------------------------------------------------------------
>>>>>>View this comment:
>>>>>> http://issues.apache.org/jira/browse/AXIS-935?
>>>>>>page=comments#action_35890
>>>>>>
>>>>>>---------------------------------------------------------------------
>>>>>>View the issue:
>>>>>> http://issues.apache.org/jira/browse/AXIS-935
>>>>>>
>>>>>>Here is an overview of the issue:
>>>>>>---------------------------------------------------------------------
>>>>>> Key: AXIS-935
>>>>>> Summary: Using ThreadLocal Call objects in a servlet memory leak
>>>>>>both on client and server side
>>>>>> Type: Bug
>>>>>>
>>>>>> Status: Open
>>>>>>
>>>>>> Project: Axis
>>>>>>Components:
>>>>>> Basic Architecture
>>>>>> Versions:
>>>>>> 1.1rc2
>>>>>>
>>>>>> Assignee: Axis Developers Mailing List
>>>>>> Reporter: Jennifer Jackson
>>>>>>
>>>>>> Created: Mon, 30 Jun 2003 12:12 PM
>>>>>> Updated: Thu, 3 Jun 2004 9:37 AM
>>>>>>Environment: Operating System: All
>>>>>>Platform: All
>>>>>>
>>>>>>Description:
>>>>>>I discovered using Axis in tomcat causes a memory leak.
>>>>>>
>>>>>>It seems using threadlocal in the classes causes a memory leak for
>>>>>>each Call
>>>>>>that is created. Threadlocal sticks around until the calling thread
>>>>>>dies, and
>>>>>>since tomcat keeps threads alive to answer new calls, the
>>>>>>threadlocal data
>>>>>>never goes away. The Call object gets stuck in the threadlocal
>>>>>>data, which
>>>>>>contains the entire soap response and all objects. This is a BIG
>>>>>>memory leak.
>>>>>>I initially cleared it by forcing all Call objects to clean their
>>>>>>member vars
>>>>>>after I was done with a call, then I realized the source was from
>>>>>>the thread
>>>>>>local class. Here is some info I found about threadlocal:
>>>>>>
>>>>>>This is stated in the Sun Javadocs for ThreadLocal:
>>>>>>
>>>>>> Each thread holds an implicit reference to its copy of a
>>>>>> ThreadLocal as long as the thread is alive and the
>>>>>> ThreadLocal object is accessible; after a thread goes
>>>>>> away, all of its copies of ThreadLocal variables are
>>>>>> subject to garbage collection (unless other references
>>>>>> to these copies exist).
>>>>>>
>>>>>>So, this means that ANY APPLICATION that uses PreparedStatements in
>>>>>>a thread
>>>>>>that 1) either does a lot of PreparedStatements or 2) never dies
>>>>>>(i.e., a
>>>>>>main thread) will ALWAYS eventually have an out of memory error.
>>>>>>Simply put,
>>>>>>this is a MEMORY LEAK. I imagine that the leak is very small, the
>>>>>>ThreadLocal object only contains one member variable, maybe 64 bytes
>>>>>>or less
>>>>>>(depending on the VM implementation). So, our 60,000
>>>>>>PreparedStatements of 2
>>>>>>ThreadLocals each times 64 bytes (my wild guess) is 7.5MB.
>>>>>>
>>>>>>Ideas? I've never used threadlocal myself so this is new to me.
>>>>>>
>>>>>>
>>>>>>---------------------------------------------------------------------
>>>>>>JIRA INFORMATION:
>>>>>>This message is automatically generated by JIRA.
>>>>>>
>>>>>>If you think it was sent incorrectly contact one of the administrators:
>>>>>> http://issues.apache.org/jira/secure/Administrators.jspa
>>>>>>
>>>>>>If you want more information on JIRA, or have a bug to report see:
>>>>>> http://www.atlassian.com/software/jira
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>--
>>>>The best way to predict the future is to invent it - Alan Kay
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>--
>>The best way to predict the future is to invent it - Alan Kay
>>
>>
>>
>>
--
The best way to predict the future is to invent it - Alan Kay
more on AXIS-Java/C++ server side performance
Posted by Aleksander Slominski <as...@cs.indiana.edu>.
Davanum Srinivas wrote:
>Alek,
>
>Where is the sample code for your XSOAP server-side code? does it
>create a java object from the soap request?
>
hi dims,
first let me emphasize that i do not look on this benchmark as only to
indicate performance but also a tool
now that i said that i think i can demonstrate usefulness of such tests:
to my big surprise after running tests i can see that AXIS-C++ is two
times slower than AXIS-Java (in this sense the test work to help detect
possible huge performance bottlenecks may ultimately lead to much better
implementation).
anyway this is really strange - i would expect C++ to be faster :)
below is everything you need to reproduce test results yourself and
maybe somebody will have an idea what is going on with AXIS-C++ ...
the test driver (client side) is still not fully finished but here is
working version:
http://www.extreme.indiana.edu/~aslom/bnp/wsperf/driver/
after downloading do source classpath.sh (or classpath.bat on windows)
to set CLASSPATH
run self test:
java -Dserver.name=SELF_TEST soap_bench.BenchDriver http://localhost:3434
and here is full description of command line:
java -Dserver.name=SERVICE_NAME soap_bench.Bench URL total
{rsea}{bdisva} [arraySize][,arraySize][,...]
parameters in order:
1. URL of service
2. total number of elements to send (default 10K)
3. specification of what to send : two letters
[rse] means receive, send, or echo (a == all)
[bdisva] means base64, double, int, string, void (only applies to
echo), a == all methods;
4. list of arraySizes (optional for void) - default to 10
and you can set test metadata such as -Dmachine.name=...
-Dserver.name=... (it is included in tabulated results good for
spreadsheet input and to draw graphs)
those are optional system properties:
-Dsmoke_test - will show you what messages were received and check that
all works fine
-DAXISCPP - will add SOAPAction header and xsi:type for BASE64 (so
AXIS-C++ is happy)
-Dlog=trace.xsul.http.client:ALL -will show all messages and HTTP
headers (good for debugging)
-Dlog=:ALL - if you are doing desperate debugging
use smoke test to get code to work with driver - it will just send 3
messages and will print what it received - combine it with -Dlog=... to
get more fine grained information on what is happening and if/when
driver is failing.
the way i would do to get test driver to work with a SOAP service is to
it in multiple esteps and to get first to work different echo methods
a) echoVoid
java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200 ev
10,100
echoInts
java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200 ei
10,100
echoDoubles
java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200 ed
10,100
echoStrings
java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200 es
10,100
echoBytes (as Base64)
java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200 eb
10,100
b) all echo methods
java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200 ea
10,100
c) all methods
java -Dsmoke_test -Dserver.name=FOO soap_bench.BenchClient <URL> 200 aa
10,100
and then run real tests
----------------------
For AXIS-Java
get code from http://www.extreme.indiana.edu/~aslom/bnp/wsperf/axis_java/
and deploy benchmark service in tomcat - to do it just modify
build.properties and do ant all
run test
java -Dmachine.name=YOUR_MACHINE_NAME -Dserver.name=AXIS_1_2_CVS
soap_bench.BenchClient http://server:8080/axis/services/Benchmark1
200000 aa 10,1000,10000
----------------------
For AXIS-C++
get code from http://www.extreme.indiana.edu/~aslom/bnp/wsperf/axis_cpp/
copy soap_bench/ to samples/server (and client respectively)
modify top level Makefile.am to include soap_bench (see example)
rebuilt .so library
add to list of deployed services (see deploy.wsdd in soap_bench)
restart apache server
run test (make sure -DAXISCPP parameter is included!!!!!)
java -DAXISCPP -Dmachine.name=YOUR_MACHINE_NAME
-Dserver.name=AXISCPP_1_0_CVS
soap_bench.BenchClient http://server:8080/axis/Benchmark1 200000 ea
10,1000,10000
NOTE: 'aa' will not work as all send tests do not work (and i do not
know why). instead try ea (echo tests) and sa (send tests) - if you want
to see what is not working use sa and enable logging (-Dlog=:ALL)
--------------------------
For XSOAP4
start service: java soap_bench.BenchServer 8077
run test driver:
java -Dmachine.name=YOUR_MACHINE_NAME -Dserver.name=XSOAP4
soap_bench.BenchClient http://server:8077 200000 aa 10,1000,10000
----------------------
comments are welcome especially concerning performance differences
between Java and C++ and how to get send tests to work with AXIS C++...
thanks,
alek
>On Thu, 03 Jun 2004 12:58:41 -0500, Aleksander Slominski
><as...@cs.indiana.edu> wrote:
>
>
>>Davanum Srinivas wrote:
>>
>>
>>
>>>You need to turn streaming on the client-side as well. see
>>>samples/perf in latest cvs.
>>>
>>> binding._setProperty(org.apache.axis.client.Call.STREAMING_PROPERTY,
>>>Boolean.TRUE);
>>>
>>>
>>>
>>>
>>i am interested in testing server side performance (as server side is
>>typically the bottleneck) so i use the same client to access all
>>services to measure only factors that are effecting server side
>>performance (CPU, memory footprint, scalability).
>>
>>hopefully soon i will have AXIS-C++ numbers - i am really close now to
>>get them after few weeks of struggling with C++ (Java configuration and
>>build tools are much easier even when dealing with JAR hell ...)
>>
>>so it should be interesting to see how AXIS-Java and AXIS-C++ compare.
>>
>>however i am also *very* interested in very long running web services so
>>Peter post about _huge_ memory leaks that are easily reproducible looked
>>very interesting to me :) is it a property of data graph he send or is
>>it just a bug ...
>>
>>thanks,
>>
>>alek
>>
>>
>>
>>>-- dims
>>>
>>>On Thu, 03 Jun 2004 12:27:18 -0500, Aleksander Slominski
>>><as...@cs.indiana.edu> wrote:
>>>
>>>
>>>
>>>
>>>>Peter,
>>>>
>>>>please post more details how to reproduce the memory leak and i will add
>>>>to what i have for WS/SOAP performance tests at:
>>>>http://www.extreme.indiana.edu/~aslom/bnp/wsperf/
>>>>
>>>>thanks,
>>>>
>>>>alek
>>>>
>>>>Peter Molettiere wrote:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>We're currently looking into a massive memory leak in axis, that no
>>>>>one here seems willing to admit. We're close to having a minimal test
>>>>>case which demonstrates the leak we're seeing. I don't think it's the
>>>>>same one you're seeing however, as our leak happens 1) when the
>>>>>server is sitting doing nothing (but only after axis has been
>>>>>loaded), and 2) every time we serialize a large message. The
>>>>>suggestions on the list about switching to using attachments doesn't
>>>>>fix the leak sadly. And setting a larger heap size only delays the
>>>>>inevitable Out Of Memory Exception.
>>>>>
>>>>>
>>>>>
>>>>>>From our testing, it looks like it takes on the order of 700MB to
>>>>
>>>>
>>>>>serialize a 30MB java object graph into a 3MB SOAP stream.
>>>>>
>>>>>I've posted about this before, but no one has responded. Hopefully
>>>>>when we post the test case someone will actually take a look at it.
>>>>>There are other threads with test cases which currently have "we're
>>>>>looking at this now" as the only response, but as they are all pretty
>>>>>old threads, my guess is that this problem either isn't believed or
>>>>>is being swept under the carpet. Maybe we're the only people using
>>>>>axis for large messages.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>--
>>>>>
>>>>>On Jun 3, 2004, at 9:38 AM, jira@apache.org wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>The following comment has been added to this issue:
>>>>>>
>>>>>> Author: Jennifer Jackson
>>>>>> Created: Thu, 3 Jun 2004 9:37 AM
>>>>>> Body:
>>>>>>Has anyone looked into this? I haven't used Axis because of this
>>>>>>problem and doing a quick search it seems others have come across
>>>>>>memory leaks also. I narrowed it down to the Call object not
>>>>>>cleaning out its member vars (the whole Deserialization process).
>>>>>>If you do that the only thing stuck in memory is the call object
>>>>>>since its part of the ThreadLocal object.
>>>>>>---------------------------------------------------------------------
>>>>>>View this comment:
>>>>>> http://issues.apache.org/jira/browse/AXIS-935?
>>>>>>page=comments#action_35890
>>>>>>
>>>>>>---------------------------------------------------------------------
>>>>>>View the issue:
>>>>>> http://issues.apache.org/jira/browse/AXIS-935
>>>>>>
>>>>>>Here is an overview of the issue:
>>>>>>---------------------------------------------------------------------
>>>>>> Key: AXIS-935
>>>>>> Summary: Using ThreadLocal Call objects in a servlet memory leak
>>>>>>both on client and server side
>>>>>> Type: Bug
>>>>>>
>>>>>> Status: Open
>>>>>>
>>>>>> Project: Axis
>>>>>>Components:
>>>>>> Basic Architecture
>>>>>> Versions:
>>>>>> 1.1rc2
>>>>>>
>>>>>> Assignee: Axis Developers Mailing List
>>>>>> Reporter: Jennifer Jackson
>>>>>>
>>>>>> Created: Mon, 30 Jun 2003 12:12 PM
>>>>>> Updated: Thu, 3 Jun 2004 9:37 AM
>>>>>>Environment: Operating System: All
>>>>>>Platform: All
>>>>>>
>>>>>>Description:
>>>>>>I discovered using Axis in tomcat causes a memory leak.
>>>>>>
>>>>>>It seems using threadlocal in the classes causes a memory leak for
>>>>>>each Call
>>>>>>that is created. Threadlocal sticks around until the calling thread
>>>>>>dies, and
>>>>>>since tomcat keeps threads alive to answer new calls, the
>>>>>>threadlocal data
>>>>>>never goes away. The Call object gets stuck in the threadlocal
>>>>>>data, which
>>>>>>contains the entire soap response and all objects. This is a BIG
>>>>>>memory leak.
>>>>>>I initially cleared it by forcing all Call objects to clean their
>>>>>>member vars
>>>>>>after I was done with a call, then I realized the source was from
>>>>>>the thread
>>>>>>local class. Here is some info I found about threadlocal:
>>>>>>
>>>>>>This is stated in the Sun Javadocs for ThreadLocal:
>>>>>>
>>>>>> Each thread holds an implicit reference to its copy of a
>>>>>> ThreadLocal as long as the thread is alive and the
>>>>>> ThreadLocal object is accessible; after a thread goes
>>>>>> away, all of its copies of ThreadLocal variables are
>>>>>> subject to garbage collection (unless other references
>>>>>> to these copies exist).
>>>>>>
>>>>>>So, this means that ANY APPLICATION that uses PreparedStatements in
>>>>>>a thread
>>>>>>that 1) either does a lot of PreparedStatements or 2) never dies
>>>>>>(i.e., a
>>>>>>main thread) will ALWAYS eventually have an out of memory error.
>>>>>>Simply put,
>>>>>>this is a MEMORY LEAK. I imagine that the leak is very small, the
>>>>>>ThreadLocal object only contains one member variable, maybe 64 bytes
>>>>>>or less
>>>>>>(depending on the VM implementation). So, our 60,000
>>>>>>PreparedStatements of 2
>>>>>>ThreadLocals each times 64 bytes (my wild guess) is 7.5MB.
>>>>>>
>>>>>>Ideas? I've never used threadlocal myself so this is new to me.
>>>>>>
>>>>>>
>>>>>>---------------------------------------------------------------------
>>>>>>JIRA INFORMATION:
>>>>>>This message is automatically generated by JIRA.
>>>>>>
>>>>>>If you think it was sent incorrectly contact one of the administrators:
>>>>>> http://issues.apache.org/jira/secure/Administrators.jspa
>>>>>>
>>>>>>If you want more information on JIRA, or have a bug to report see:
>>>>>> http://www.atlassian.com/software/jira
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>--
>>>>The best way to predict the future is to invent it - Alan Kay
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>--
>>The best way to predict the future is to invent it - Alan Kay
>>
>>
>>
>>
--
The best way to predict the future is to invent it - Alan Kay
Re: AXIS-Java/C++ server side performance / memory footprint
Posted by Davanum Srinivas <da...@gmail.com>.
Alek,
Where is the sample code for your XSOAP server-side code? does it
create a java object from the soap request?
thanks,
dims
On Thu, 03 Jun 2004 12:58:41 -0500, Aleksander Slominski
<as...@cs.indiana.edu> wrote:
>
> Davanum Srinivas wrote:
>
> >You need to turn streaming on the client-side as well. see
> >samples/perf in latest cvs.
> >
> > binding._setProperty(org.apache.axis.client.Call.STREAMING_PROPERTY,
> >Boolean.TRUE);
> >
> >
> i am interested in testing server side performance (as server side is
> typically the bottleneck) so i use the same client to access all
> services to measure only factors that are effecting server side
> performance (CPU, memory footprint, scalability).
>
> hopefully soon i will have AXIS-C++ numbers - i am really close now to
> get them after few weeks of struggling with C++ (Java configuration and
> build tools are much easier even when dealing with JAR hell ...)
>
> so it should be interesting to see how AXIS-Java and AXIS-C++ compare.
>
> however i am also *very* interested in very long running web services so
> Peter post about _huge_ memory leaks that are easily reproducible looked
> very interesting to me :) is it a property of data graph he send or is
> it just a bug ...
>
> thanks,
>
> alek
>
> >-- dims
> >
> >On Thu, 03 Jun 2004 12:27:18 -0500, Aleksander Slominski
> ><as...@cs.indiana.edu> wrote:
> >
> >
> >>Peter,
> >>
> >>please post more details how to reproduce the memory leak and i will add
> >>to what i have for WS/SOAP performance tests at:
> >>http://www.extreme.indiana.edu/~aslom/bnp/wsperf/
> >>
> >>thanks,
> >>
> >>alek
> >>
> >>Peter Molettiere wrote:
> >>
> >>
> >>
> >>>We're currently looking into a massive memory leak in axis, that no
> >>>one here seems willing to admit. We're close to having a minimal test
> >>>case which demonstrates the leak we're seeing. I don't think it's the
> >>>same one you're seeing however, as our leak happens 1) when the
> >>>server is sitting doing nothing (but only after axis has been
> >>>loaded), and 2) every time we serialize a large message. The
> >>>suggestions on the list about switching to using attachments doesn't
> >>>fix the leak sadly. And setting a larger heap size only delays the
> >>>inevitable Out Of Memory Exception.
> >>>
> >>>From our testing, it looks like it takes on the order of 700MB to
> >>>serialize a 30MB java object graph into a 3MB SOAP stream.
> >>>
> >>>I've posted about this before, but no one has responded. Hopefully
> >>>when we post the test case someone will actually take a look at it.
> >>>There are other threads with test cases which currently have "we're
> >>>looking at this now" as the only response, but as they are all pretty
> >>>old threads, my guess is that this problem either isn't believed or
> >>>is being swept under the carpet. Maybe we're the only people using
> >>>axis for large messages.
> >>>
> >>>
> >>>
> >>>
> >>>--
> >>>
> >>>On Jun 3, 2004, at 9:38 AM, jira@apache.org wrote:
> >>>
> >>>
> >>>
> >>>>The following comment has been added to this issue:
> >>>>
> >>>> Author: Jennifer Jackson
> >>>> Created: Thu, 3 Jun 2004 9:37 AM
> >>>> Body:
> >>>>Has anyone looked into this? I haven't used Axis because of this
> >>>>problem and doing a quick search it seems others have come across
> >>>>memory leaks also. I narrowed it down to the Call object not
> >>>>cleaning out its member vars (the whole Deserialization process).
> >>>>If you do that the only thing stuck in memory is the call object
> >>>>since its part of the ThreadLocal object.
> >>>>---------------------------------------------------------------------
> >>>>View this comment:
> >>>> http://issues.apache.org/jira/browse/AXIS-935?
> >>>>page=comments#action_35890
> >>>>
> >>>>---------------------------------------------------------------------
> >>>>View the issue:
> >>>> http://issues.apache.org/jira/browse/AXIS-935
> >>>>
> >>>>Here is an overview of the issue:
> >>>>---------------------------------------------------------------------
> >>>> Key: AXIS-935
> >>>> Summary: Using ThreadLocal Call objects in a servlet memory leak
> >>>>both on client and server side
> >>>> Type: Bug
> >>>>
> >>>> Status: Open
> >>>>
> >>>> Project: Axis
> >>>> Components:
> >>>> Basic Architecture
> >>>> Versions:
> >>>> 1.1rc2
> >>>>
> >>>> Assignee: Axis Developers Mailing List
> >>>> Reporter: Jennifer Jackson
> >>>>
> >>>> Created: Mon, 30 Jun 2003 12:12 PM
> >>>> Updated: Thu, 3 Jun 2004 9:37 AM
> >>>>Environment: Operating System: All
> >>>>Platform: All
> >>>>
> >>>>Description:
> >>>>I discovered using Axis in tomcat causes a memory leak.
> >>>>
> >>>>It seems using threadlocal in the classes causes a memory leak for
> >>>>each Call
> >>>>that is created. Threadlocal sticks around until the calling thread
> >>>>dies, and
> >>>>since tomcat keeps threads alive to answer new calls, the
> >>>>threadlocal data
> >>>>never goes away. The Call object gets stuck in the threadlocal
> >>>>data, which
> >>>>contains the entire soap response and all objects. This is a BIG
> >>>>memory leak.
> >>>>I initially cleared it by forcing all Call objects to clean their
> >>>>member vars
> >>>>after I was done with a call, then I realized the source was from
> >>>>the thread
> >>>>local class. Here is some info I found about threadlocal:
> >>>>
> >>>>This is stated in the Sun Javadocs for ThreadLocal:
> >>>>
> >>>> Each thread holds an implicit reference to its copy of a
> >>>> ThreadLocal as long as the thread is alive and the
> >>>> ThreadLocal object is accessible; after a thread goes
> >>>> away, all of its copies of ThreadLocal variables are
> >>>> subject to garbage collection (unless other references
> >>>> to these copies exist).
> >>>>
> >>>>So, this means that ANY APPLICATION that uses PreparedStatements in
> >>>>a thread
> >>>>that 1) either does a lot of PreparedStatements or 2) never dies
> >>>>(i.e., a
> >>>>main thread) will ALWAYS eventually have an out of memory error.
> >>>>Simply put,
> >>>>this is a MEMORY LEAK. I imagine that the leak is very small, the
> >>>>ThreadLocal object only contains one member variable, maybe 64 bytes
> >>>>or less
> >>>>(depending on the VM implementation). So, our 60,000
> >>>>PreparedStatements of 2
> >>>>ThreadLocals each times 64 bytes (my wild guess) is 7.5MB.
> >>>>
> >>>>Ideas? I've never used threadlocal myself so this is new to me.
> >>>>
> >>>>
> >>>>---------------------------------------------------------------------
> >>>>JIRA INFORMATION:
> >>>>This message is automatically generated by JIRA.
> >>>>
> >>>>If you think it was sent incorrectly contact one of the administrators:
> >>>> http://issues.apache.org/jira/secure/Administrators.jspa
> >>>>
> >>>>If you want more information on JIRA, or have a bug to report see:
> >>>> http://www.atlassian.com/software/jira
> >>>>
> >>>>
> >>>
> >>>
> >>--
> >>The best way to predict the future is to invent it - Alan Kay
> >>
> >>
> >>
> >>
>
>
> --
> The best way to predict the future is to invent it - Alan Kay
>
>
AXIS-Java/C++ server side performance / memory footprint
Posted by Aleksander Slominski <as...@cs.indiana.edu>.
Davanum Srinivas wrote:
>You need to turn streaming on the client-side as well. see
>samples/perf in latest cvs.
>
> binding._setProperty(org.apache.axis.client.Call.STREAMING_PROPERTY,
>Boolean.TRUE);
>
>
i am interested in testing server side performance (as server side is
typically the bottleneck) so i use the same client to access all
services to measure only factors that are effecting server side
performance (CPU, memory footprint, scalability).
hopefully soon i will have AXIS-C++ numbers - i am really close now to
get them after few weeks of struggling with C++ (Java configuration and
build tools are much easier even when dealing with JAR hell ...)
so it should be interesting to see how AXIS-Java and AXIS-C++ compare.
however i am also *very* interested in very long running web services so
Peter post about _huge_ memory leaks that are easily reproducible looked
very interesting to me :) is it a property of data graph he send or is
it just a bug ...
thanks,
alek
>-- dims
>
>On Thu, 03 Jun 2004 12:27:18 -0500, Aleksander Slominski
><as...@cs.indiana.edu> wrote:
>
>
>>Peter,
>>
>>please post more details how to reproduce the memory leak and i will add
>>to what i have for WS/SOAP performance tests at:
>>http://www.extreme.indiana.edu/~aslom/bnp/wsperf/
>>
>>thanks,
>>
>>alek
>>
>>Peter Molettiere wrote:
>>
>>
>>
>>>We're currently looking into a massive memory leak in axis, that no
>>>one here seems willing to admit. We're close to having a minimal test
>>>case which demonstrates the leak we're seeing. I don't think it's the
>>>same one you're seeing however, as our leak happens 1) when the
>>>server is sitting doing nothing (but only after axis has been
>>>loaded), and 2) every time we serialize a large message. The
>>>suggestions on the list about switching to using attachments doesn't
>>>fix the leak sadly. And setting a larger heap size only delays the
>>>inevitable Out Of Memory Exception.
>>>
>>>>From our testing, it looks like it takes on the order of 700MB to
>>>serialize a 30MB java object graph into a 3MB SOAP stream.
>>>
>>>I've posted about this before, but no one has responded. Hopefully
>>>when we post the test case someone will actually take a look at it.
>>>There are other threads with test cases which currently have "we're
>>>looking at this now" as the only response, but as they are all pretty
>>>old threads, my guess is that this problem either isn't believed or
>>>is being swept under the carpet. Maybe we're the only people using
>>>axis for large messages.
>>>
>>>
>>>
>>>
>>>--
>>>
>>>On Jun 3, 2004, at 9:38 AM, jira@apache.org wrote:
>>>
>>>
>>>
>>>>The following comment has been added to this issue:
>>>>
>>>> Author: Jennifer Jackson
>>>> Created: Thu, 3 Jun 2004 9:37 AM
>>>> Body:
>>>>Has anyone looked into this? I haven't used Axis because of this
>>>>problem and doing a quick search it seems others have come across
>>>>memory leaks also. I narrowed it down to the Call object not
>>>>cleaning out its member vars (the whole Deserialization process).
>>>>If you do that the only thing stuck in memory is the call object
>>>>since its part of the ThreadLocal object.
>>>>---------------------------------------------------------------------
>>>>View this comment:
>>>> http://issues.apache.org/jira/browse/AXIS-935?
>>>>page=comments#action_35890
>>>>
>>>>---------------------------------------------------------------------
>>>>View the issue:
>>>> http://issues.apache.org/jira/browse/AXIS-935
>>>>
>>>>Here is an overview of the issue:
>>>>---------------------------------------------------------------------
>>>> Key: AXIS-935
>>>> Summary: Using ThreadLocal Call objects in a servlet memory leak
>>>>both on client and server side
>>>> Type: Bug
>>>>
>>>> Status: Open
>>>>
>>>> Project: Axis
>>>> Components:
>>>> Basic Architecture
>>>> Versions:
>>>> 1.1rc2
>>>>
>>>> Assignee: Axis Developers Mailing List
>>>> Reporter: Jennifer Jackson
>>>>
>>>> Created: Mon, 30 Jun 2003 12:12 PM
>>>> Updated: Thu, 3 Jun 2004 9:37 AM
>>>>Environment: Operating System: All
>>>>Platform: All
>>>>
>>>>Description:
>>>>I discovered using Axis in tomcat causes a memory leak.
>>>>
>>>>It seems using threadlocal in the classes causes a memory leak for
>>>>each Call
>>>>that is created. Threadlocal sticks around until the calling thread
>>>>dies, and
>>>>since tomcat keeps threads alive to answer new calls, the
>>>>threadlocal data
>>>>never goes away. The Call object gets stuck in the threadlocal
>>>>data, which
>>>>contains the entire soap response and all objects. This is a BIG
>>>>memory leak.
>>>>I initially cleared it by forcing all Call objects to clean their
>>>>member vars
>>>>after I was done with a call, then I realized the source was from
>>>>the thread
>>>>local class. Here is some info I found about threadlocal:
>>>>
>>>>This is stated in the Sun Javadocs for ThreadLocal:
>>>>
>>>> Each thread holds an implicit reference to its copy of a
>>>> ThreadLocal as long as the thread is alive and the
>>>> ThreadLocal object is accessible; after a thread goes
>>>> away, all of its copies of ThreadLocal variables are
>>>> subject to garbage collection (unless other references
>>>> to these copies exist).
>>>>
>>>>So, this means that ANY APPLICATION that uses PreparedStatements in
>>>>a thread
>>>>that 1) either does a lot of PreparedStatements or 2) never dies
>>>>(i.e., a
>>>>main thread) will ALWAYS eventually have an out of memory error.
>>>>Simply put,
>>>>this is a MEMORY LEAK. I imagine that the leak is very small, the
>>>>ThreadLocal object only contains one member variable, maybe 64 bytes
>>>>or less
>>>>(depending on the VM implementation). So, our 60,000
>>>>PreparedStatements of 2
>>>>ThreadLocals each times 64 bytes (my wild guess) is 7.5MB.
>>>>
>>>>Ideas? I've never used threadlocal myself so this is new to me.
>>>>
>>>>
>>>>---------------------------------------------------------------------
>>>>JIRA INFORMATION:
>>>>This message is automatically generated by JIRA.
>>>>
>>>>If you think it was sent incorrectly contact one of the administrators:
>>>> http://issues.apache.org/jira/secure/Administrators.jspa
>>>>
>>>>If you want more information on JIRA, or have a bug to report see:
>>>> http://www.atlassian.com/software/jira
>>>>
>>>>
>>>
>>>
>>--
>>The best way to predict the future is to invent it - Alan Kay
>>
>>
>>
>>
--
The best way to predict the future is to invent it - Alan Kay
Re: Re: [jira] Commented: (AXIS-935) Using ThreadLocal Call objects in a servlet memory leak both on client and server side
Posted by Davanum Srinivas <da...@gmail.com>.
Alek,
You need to turn streaming on the client-side as well. see
samples/perf in latest cvs.
binding._setProperty(org.apache.axis.client.Call.STREAMING_PROPERTY,
Boolean.TRUE);
-- dims
On Thu, 03 Jun 2004 12:27:18 -0500, Aleksander Slominski
<as...@cs.indiana.edu> wrote:
>
> Peter,
>
> please post more details how to reproduce the memory leak and i will add
> to what i have for WS/SOAP performance tests at:
> http://www.extreme.indiana.edu/~aslom/bnp/wsperf/
>
> thanks,
>
> alek
>
> Peter Molettiere wrote:
>
> >
> >
> > We're currently looking into a massive memory leak in axis, that no
> > one here seems willing to admit. We're close to having a minimal test
> > case which demonstrates the leak we're seeing. I don't think it's the
> > same one you're seeing however, as our leak happens 1) when the
> > server is sitting doing nothing (but only after axis has been
> > loaded), and 2) every time we serialize a large message. The
> > suggestions on the list about switching to using attachments doesn't
> > fix the leak sadly. And setting a larger heap size only delays the
> > inevitable Out Of Memory Exception.
> >
> > From our testing, it looks like it takes on the order of 700MB to
> > serialize a 30MB java object graph into a 3MB SOAP stream.
> >
> > I've posted about this before, but no one has responded. Hopefully
> > when we post the test case someone will actually take a look at it.
> > There are other threads with test cases which currently have "we're
> > looking at this now" as the only response, but as they are all pretty
> > old threads, my guess is that this problem either isn't believed or
> > is being swept under the carpet. Maybe we're the only people using
> > axis for large messages.
> >
> >
>
> >
> > --
> >
> > On Jun 3, 2004, at 9:38 AM, jira@apache.org wrote:
> >
> >> The following comment has been added to this issue:
> >>
> >> Author: Jennifer Jackson
> >> Created: Thu, 3 Jun 2004 9:37 AM
> >> Body:
> >> Has anyone looked into this? I haven't used Axis because of this
> >> problem and doing a quick search it seems others have come across
> >> memory leaks also. I narrowed it down to the Call object not
> >> cleaning out its member vars (the whole Deserialization process).
> >> If you do that the only thing stuck in memory is the call object
> >> since its part of the ThreadLocal object.
> >> ---------------------------------------------------------------------
> >> View this comment:
> >> http://issues.apache.org/jira/browse/AXIS-935?
> >> page=comments#action_35890
> >>
> >> ---------------------------------------------------------------------
> >> View the issue:
> >> http://issues.apache.org/jira/browse/AXIS-935
> >>
> >> Here is an overview of the issue:
> >> ---------------------------------------------------------------------
> >> Key: AXIS-935
> >> Summary: Using ThreadLocal Call objects in a servlet memory leak
> >> both on client and server side
> >> Type: Bug
> >>
> >> Status: Open
> >>
> >> Project: Axis
> >> Components:
> >> Basic Architecture
> >> Versions:
> >> 1.1rc2
> >>
> >> Assignee: Axis Developers Mailing List
> >> Reporter: Jennifer Jackson
> >>
> >> Created: Mon, 30 Jun 2003 12:12 PM
> >> Updated: Thu, 3 Jun 2004 9:37 AM
> >> Environment: Operating System: All
> >> Platform: All
> >>
> >> Description:
> >> I discovered using Axis in tomcat causes a memory leak.
> >>
> >> It seems using threadlocal in the classes causes a memory leak for
> >> each Call
> >> that is created. Threadlocal sticks around until the calling thread
> >> dies, and
> >> since tomcat keeps threads alive to answer new calls, the
> >> threadlocal data
> >> never goes away. The Call object gets stuck in the threadlocal
> >> data, which
> >> contains the entire soap response and all objects. This is a BIG
> >> memory leak.
> >> I initially cleared it by forcing all Call objects to clean their
> >> member vars
> >> after I was done with a call, then I realized the source was from
> >> the thread
> >> local class. Here is some info I found about threadlocal:
> >>
> >> This is stated in the Sun Javadocs for ThreadLocal:
> >>
> >> Each thread holds an implicit reference to its copy of a
> >> ThreadLocal as long as the thread is alive and the
> >> ThreadLocal object is accessible; after a thread goes
> >> away, all of its copies of ThreadLocal variables are
> >> subject to garbage collection (unless other references
> >> to these copies exist).
> >>
> >> So, this means that ANY APPLICATION that uses PreparedStatements in
> >> a thread
> >> that 1) either does a lot of PreparedStatements or 2) never dies
> >> (i.e., a
> >> main thread) will ALWAYS eventually have an out of memory error.
> >> Simply put,
> >> this is a MEMORY LEAK. I imagine that the leak is very small, the
> >> ThreadLocal object only contains one member variable, maybe 64 bytes
> >> or less
> >> (depending on the VM implementation). So, our 60,000
> >> PreparedStatements of 2
> >> ThreadLocals each times 64 bytes (my wild guess) is 7.5MB.
> >>
> >> Ideas? I've never used threadlocal myself so this is new to me.
> >>
> >>
> >> ---------------------------------------------------------------------
> >> JIRA INFORMATION:
> >> This message is automatically generated by JIRA.
> >>
> >> If you think it was sent incorrectly contact one of the administrators:
> >> http://issues.apache.org/jira/secure/Administrators.jspa
> >>
> >> If you want more information on JIRA, or have a bug to report see:
> >> http://www.atlassian.com/software/jira
> >
> >
>
>
> --
> The best way to predict the future is to invent it - Alan Kay
>
>
Re: [jira] Commented: (AXIS-935) Using ThreadLocal Call objects in a servlet memory leak both on client and server side
Posted by Peter Molettiere <pi...@axonstudios.net>.
As soon as we have the test case properly packaged, we'll post it to
jira and here.
--Peter
On Jun 3, 2004, at 10:27 AM, Aleksander Slominski wrote:
> Peter,
>
> please post more details how to reproduce the memory leak and i will
> add to what i have for WS/SOAP performance tests at:
> http://www.extreme.indiana.edu/~aslom/bnp/wsperf/
>
> thanks,
>
> alek
>
> Peter Molettiere wrote:
>
>>
>>
>> We're currently looking into a massive memory leak in axis, that no
>> one here seems willing to admit. We're close to having a minimal
>> test case which demonstrates the leak we're seeing. I don't think
>> it's the same one you're seeing however, as our leak happens 1) when
>> the server is sitting doing nothing (but only after axis has been
>> loaded), and 2) every time we serialize a large message. The
>> suggestions on the list about switching to using attachments doesn't
>> fix the leak sadly. And setting a larger heap size only delays the
>> inevitable Out Of Memory Exception.
>>
>> From our testing, it looks like it takes on the order of 700MB to
>> serialize a 30MB java object graph into a 3MB SOAP stream.
>>
>> I've posted about this before, but no one has responded. Hopefully
>> when we post the test case someone will actually take a look at it.
>> There are other threads with test cases which currently have "we're
>> looking at this now" as the only response, but as they are all
>> pretty old threads, my guess is that this problem either isn't
>> believed or is being swept under the carpet. Maybe we're the only
>> people using axis for large messages.
>>
>>
>
>>
>> --
>>
>> On Jun 3, 2004, at 9:38 AM, jira@apache.org wrote:
>>
>>> The following comment has been added to this issue:
>>>
>>> Author: Jennifer Jackson
>>> Created: Thu, 3 Jun 2004 9:37 AM
>>> Body:
>>> Has anyone looked into this? I haven't used Axis because of this
>>> problem and doing a quick search it seems others have come across
>>> memory leaks also. I narrowed it down to the Call object not
>>> cleaning out its member vars (the whole Deserialization process).
>>> If you do that the only thing stuck in memory is the call object
>>> since its part of the ThreadLocal object.
>>> ---------------------------------------------------------------------
>>> View this comment:
>>> http://issues.apache.org/jira/browse/AXIS-935?
>>> page=comments#action_35890
>>>
>>> ---------------------------------------------------------------------
>>> View the issue:
>>> http://issues.apache.org/jira/browse/AXIS-935
>>>
>>> Here is an overview of the issue:
>>> ---------------------------------------------------------------------
>>> Key: AXIS-935
>>> Summary: Using ThreadLocal Call objects in a servlet memory leak
>>> both on client and server side
>>> Type: Bug
>>>
>>> Status: Open
>>>
>>> Project: Axis
>>> Components:
>>> Basic Architecture
>>> Versions:
>>> 1.1rc2
>>>
>>> Assignee: Axis Developers Mailing List
>>> Reporter: Jennifer Jackson
>>>
>>> Created: Mon, 30 Jun 2003 12:12 PM
>>> Updated: Thu, 3 Jun 2004 9:37 AM
>>> Environment: Operating System: All
>>> Platform: All
>>>
>>> Description:
>>> I discovered using Axis in tomcat causes a memory leak.
>>>
>>> It seems using threadlocal in the classes causes a memory leak for
>>> each Call
>>> that is created. Threadlocal sticks around until the calling thread
>>> dies, and
>>> since tomcat keeps threads alive to answer new calls, the
>>> threadlocal data
>>> never goes away. The Call object gets stuck in the threadlocal
>>> data, which
>>> contains the entire soap response and all objects. This is a BIG
>>> memory leak.
>>> I initially cleared it by forcing all Call objects to clean their
>>> member vars
>>> after I was done with a call, then I realized the source was from
>>> the thread
>>> local class. Here is some info I found about threadlocal:
>>>
>>> This is stated in the Sun Javadocs for ThreadLocal:
>>>
>>> Each thread holds an implicit reference to its copy of a
>>> ThreadLocal as long as the thread is alive and the
>>> ThreadLocal object is accessible; after a thread goes
>>> away, all of its copies of ThreadLocal variables are
>>> subject to garbage collection (unless other references
>>> to these copies exist).
>>>
>>> So, this means that ANY APPLICATION that uses PreparedStatements in
>>> a thread
>>> that 1) either does a lot of PreparedStatements or 2) never dies
>>> (i.e., a
>>> main thread) will ALWAYS eventually have an out of memory error.
>>> Simply put,
>>> this is a MEMORY LEAK. I imagine that the leak is very small, the
>>> ThreadLocal object only contains one member variable, maybe 64 bytes
>>> or less
>>> (depending on the VM implementation). So, our 60,000
>>> PreparedStatements of 2
>>> ThreadLocals each times 64 bytes (my wild guess) is 7.5MB.
>>>
>>> Ideas? I've never used threadlocal myself so this is new to me.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> JIRA INFORMATION:
>>> This message is automatically generated by JIRA.
>>>
>>> If you think it was sent incorrectly contact one of the
>>> administrators:
>>> http://issues.apache.org/jira/secure/Administrators.jspa
>>>
>>> If you want more information on JIRA, or have a bug to report see:
>>> http://www.atlassian.com/software/jira
>>
>>
>
>
> --
> The best way to predict the future is to invent it - Alan Kay
Re: Re: [jira] Commented: (AXIS-935) Using ThreadLocal Call objects in a servlet memory leak both on client and server side
Posted by Davanum Srinivas <da...@gmail.com>.
Alek,
Are u using Tomcat or SimpleAxisServer? Are u using Commons HTTPClient
or not? Is it possible to narrow down the problem scenario? (XSOAP
Client against Axis Server vs Axis Client against XSOAP Server etc).
Any help in drilling down into perf problems will be much appreciated.
-- dims
On Thu, 03 Jun 2004 12:27:18 -0500, Aleksander Slominski
<as...@cs.indiana.edu> wrote:
>
> Peter,
>
> please post more details how to reproduce the memory leak and i will add
> to what i have for WS/SOAP performance tests at:
> http://www.extreme.indiana.edu/~aslom/bnp/wsperf/
>
> thanks,
>
> alek
>
> Peter Molettiere wrote:
>
> >
> >
> > We're currently looking into a massive memory leak in axis, that no
> > one here seems willing to admit. We're close to having a minimal test
> > case which demonstrates the leak we're seeing. I don't think it's the
> > same one you're seeing however, as our leak happens 1) when the
> > server is sitting doing nothing (but only after axis has been
> > loaded), and 2) every time we serialize a large message. The
> > suggestions on the list about switching to using attachments doesn't
> > fix the leak sadly. And setting a larger heap size only delays the
> > inevitable Out Of Memory Exception.
> >
> > From our testing, it looks like it takes on the order of 700MB to
> > serialize a 30MB java object graph into a 3MB SOAP stream.
> >
> > I've posted about this before, but no one has responded. Hopefully
> > when we post the test case someone will actually take a look at it.
> > There are other threads with test cases which currently have "we're
> > looking at this now" as the only response, but as they are all pretty
> > old threads, my guess is that this problem either isn't believed or
> > is being swept under the carpet. Maybe we're the only people using
> > axis for large messages.
> >
> >
>
> >
> > --
> >
> > On Jun 3, 2004, at 9:38 AM, jira@apache.org wrote:
> >
> >> The following comment has been added to this issue:
> >>
> >> Author: Jennifer Jackson
> >> Created: Thu, 3 Jun 2004 9:37 AM
> >> Body:
> >> Has anyone looked into this? I haven't used Axis because of this
> >> problem and doing a quick search it seems others have come across
> >> memory leaks also. I narrowed it down to the Call object not
> >> cleaning out its member vars (the whole Deserialization process).
> >> If you do that the only thing stuck in memory is the call object
> >> since its part of the ThreadLocal object.
> >> ---------------------------------------------------------------------
> >> View this comment:
> >> http://issues.apache.org/jira/browse/AXIS-935?
> >> page=comments#action_35890
> >>
> >> ---------------------------------------------------------------------
> >> View the issue:
> >> http://issues.apache.org/jira/browse/AXIS-935
> >>
> >> Here is an overview of the issue:
> >> ---------------------------------------------------------------------
> >> Key: AXIS-935
> >> Summary: Using ThreadLocal Call objects in a servlet memory leak
> >> both on client and server side
> >> Type: Bug
> >>
> >> Status: Open
> >>
> >> Project: Axis
> >> Components:
> >> Basic Architecture
> >> Versions:
> >> 1.1rc2
> >>
> >> Assignee: Axis Developers Mailing List
> >> Reporter: Jennifer Jackson
> >>
> >> Created: Mon, 30 Jun 2003 12:12 PM
> >> Updated: Thu, 3 Jun 2004 9:37 AM
> >> Environment: Operating System: All
> >> Platform: All
> >>
> >> Description:
> >> I discovered using Axis in tomcat causes a memory leak.
> >>
> >> It seems using threadlocal in the classes causes a memory leak for
> >> each Call
> >> that is created. Threadlocal sticks around until the calling thread
> >> dies, and
> >> since tomcat keeps threads alive to answer new calls, the
> >> threadlocal data
> >> never goes away. The Call object gets stuck in the threadlocal
> >> data, which
> >> contains the entire soap response and all objects. This is a BIG
> >> memory leak.
> >> I initially cleared it by forcing all Call objects to clean their
> >> member vars
> >> after I was done with a call, then I realized the source was from
> >> the thread
> >> local class. Here is some info I found about threadlocal:
> >>
> >> This is stated in the Sun Javadocs for ThreadLocal:
> >>
> >> Each thread holds an implicit reference to its copy of a
> >> ThreadLocal as long as the thread is alive and the
> >> ThreadLocal object is accessible; after a thread goes
> >> away, all of its copies of ThreadLocal variables are
> >> subject to garbage collection (unless other references
> >> to these copies exist).
> >>
> >> So, this means that ANY APPLICATION that uses PreparedStatements in
> >> a thread
> >> that 1) either does a lot of PreparedStatements or 2) never dies
> >> (i.e., a
> >> main thread) will ALWAYS eventually have an out of memory error.
> >> Simply put,
> >> this is a MEMORY LEAK. I imagine that the leak is very small, the
> >> ThreadLocal object only contains one member variable, maybe 64 bytes
> >> or less
> >> (depending on the VM implementation). So, our 60,000
> >> PreparedStatements of 2
> >> ThreadLocals each times 64 bytes (my wild guess) is 7.5MB.
> >>
> >> Ideas? I've never used threadlocal myself so this is new to me.
> >>
> >>
> >> ---------------------------------------------------------------------
> >> JIRA INFORMATION:
> >> This message is automatically generated by JIRA.
> >>
> >> If you think it was sent incorrectly contact one of the administrators:
> >> http://issues.apache.org/jira/secure/Administrators.jspa
> >>
> >> If you want more information on JIRA, or have a bug to report see:
> >> http://www.atlassian.com/software/jira
> >
> >
>
>
> --
> The best way to predict the future is to invent it - Alan Kay
>
>
Re: [jira] Commented: (AXIS-935) Using ThreadLocal Call objects in
a servlet memory leak both on client and server side
Posted by Aleksander Slominski <as...@cs.indiana.edu>.
Peter,
please post more details how to reproduce the memory leak and i will add
to what i have for WS/SOAP performance tests at:
http://www.extreme.indiana.edu/~aslom/bnp/wsperf/
thanks,
alek
Peter Molettiere wrote:
>
>
> We're currently looking into a massive memory leak in axis, that no
> one here seems willing to admit. We're close to having a minimal test
> case which demonstrates the leak we're seeing. I don't think it's the
> same one you're seeing however, as our leak happens 1) when the
> server is sitting doing nothing (but only after axis has been
> loaded), and 2) every time we serialize a large message. The
> suggestions on the list about switching to using attachments doesn't
> fix the leak sadly. And setting a larger heap size only delays the
> inevitable Out Of Memory Exception.
>
> From our testing, it looks like it takes on the order of 700MB to
> serialize a 30MB java object graph into a 3MB SOAP stream.
>
> I've posted about this before, but no one has responded. Hopefully
> when we post the test case someone will actually take a look at it.
> There are other threads with test cases which currently have "we're
> looking at this now" as the only response, but as they are all pretty
> old threads, my guess is that this problem either isn't believed or
> is being swept under the carpet. Maybe we're the only people using
> axis for large messages.
>
>
>
> --
>
> On Jun 3, 2004, at 9:38 AM, jira@apache.org wrote:
>
>> The following comment has been added to this issue:
>>
>> Author: Jennifer Jackson
>> Created: Thu, 3 Jun 2004 9:37 AM
>> Body:
>> Has anyone looked into this? I haven't used Axis because of this
>> problem and doing a quick search it seems others have come across
>> memory leaks also. I narrowed it down to the Call object not
>> cleaning out its member vars (the whole Deserialization process).
>> If you do that the only thing stuck in memory is the call object
>> since its part of the ThreadLocal object.
>> ---------------------------------------------------------------------
>> View this comment:
>> http://issues.apache.org/jira/browse/AXIS-935?
>> page=comments#action_35890
>>
>> ---------------------------------------------------------------------
>> View the issue:
>> http://issues.apache.org/jira/browse/AXIS-935
>>
>> Here is an overview of the issue:
>> ---------------------------------------------------------------------
>> Key: AXIS-935
>> Summary: Using ThreadLocal Call objects in a servlet memory leak
>> both on client and server side
>> Type: Bug
>>
>> Status: Open
>>
>> Project: Axis
>> Components:
>> Basic Architecture
>> Versions:
>> 1.1rc2
>>
>> Assignee: Axis Developers Mailing List
>> Reporter: Jennifer Jackson
>>
>> Created: Mon, 30 Jun 2003 12:12 PM
>> Updated: Thu, 3 Jun 2004 9:37 AM
>> Environment: Operating System: All
>> Platform: All
>>
>> Description:
>> I discovered using Axis in tomcat causes a memory leak.
>>
>> It seems using threadlocal in the classes causes a memory leak for
>> each Call
>> that is created. Threadlocal sticks around until the calling thread
>> dies, and
>> since tomcat keeps threads alive to answer new calls, the
>> threadlocal data
>> never goes away. The Call object gets stuck in the threadlocal
>> data, which
>> contains the entire soap response and all objects. This is a BIG
>> memory leak.
>> I initially cleared it by forcing all Call objects to clean their
>> member vars
>> after I was done with a call, then I realized the source was from
>> the thread
>> local class. Here is some info I found about threadlocal:
>>
>> This is stated in the Sun Javadocs for ThreadLocal:
>>
>> Each thread holds an implicit reference to its copy of a
>> ThreadLocal as long as the thread is alive and the
>> ThreadLocal object is accessible; after a thread goes
>> away, all of its copies of ThreadLocal variables are
>> subject to garbage collection (unless other references
>> to these copies exist).
>>
>> So, this means that ANY APPLICATION that uses PreparedStatements in
>> a thread
>> that 1) either does a lot of PreparedStatements or 2) never dies
>> (i.e., a
>> main thread) will ALWAYS eventually have an out of memory error.
>> Simply put,
>> this is a MEMORY LEAK. I imagine that the leak is very small, the
>> ThreadLocal object only contains one member variable, maybe 64 bytes
>> or less
>> (depending on the VM implementation). So, our 60,000
>> PreparedStatements of 2
>> ThreadLocals each times 64 bytes (my wild guess) is 7.5MB.
>>
>> Ideas? I've never used threadlocal myself so this is new to me.
>>
>>
>> ---------------------------------------------------------------------
>> JIRA INFORMATION:
>> This message is automatically generated by JIRA.
>>
>> If you think it was sent incorrectly contact one of the administrators:
>> http://issues.apache.org/jira/secure/Administrators.jspa
>>
>> If you want more information on JIRA, or have a bug to report see:
>> http://www.atlassian.com/software/jira
>
>
--
The best way to predict the future is to invent it - Alan Kay
Re: [jira] Commented: (AXIS-935) Using ThreadLocal Call objects in a servlet memory leak both on client and server side
Posted by Davanum Srinivas <da...@gmail.com>.
Peter,
If i can recreate this on my machine, i promise to fix it ASAP :) Just
create a jira bug (http://issues.apache.org/jira) with the sample code
and instructions.
-- dims
On Thu, 3 Jun 2004 10:18:45 -0700, Peter Molettiere
<pi...@axonstudios.net> wrote:
>
>
>
> We're currently looking into a massive memory leak in axis, that no one
> here seems willing to admit. We're close to having a minimal test case
> which demonstrates the leak we're seeing. I don't think it's the same
> one you're seeing however, as our leak happens 1) when the server is
> sitting doing nothing (but only after axis has been loaded), and 2)
> every time we serialize a large message. The suggestions on the list
> about switching to using attachments doesn't fix the leak sadly. And
> setting a larger heap size only delays the inevitable Out Of Memory
> Exception.
>
> From our testing, it looks like it takes on the order of 700MB to
> serialize a 30MB java object graph into a 3MB SOAP stream.
>
> I've posted about this before, but no one has responded. Hopefully when
> we post the test case someone will actually take a look at it. There
> are other threads with test cases which currently have "we're looking
> at this now" as the only response, but as they are all pretty old
> threads, my guess is that this problem either isn't believed or is
> being swept under the carpet. Maybe we're the only people using axis
> for large messages.
>
> Hopefully, we'll have more info in the next couple days.
>
> --Peter
>
> On Jun 3, 2004, at 9:38 AM, jira@apache.org wrote:
>
> > The following comment has been added to this issue:
> >
> > Author: Jennifer Jackson
> > Created: Thu, 3 Jun 2004 9:37 AM
> > Body:
> > Has anyone looked into this? I haven't used Axis because of this
> > problem and doing a quick search it seems others have come across
> > memory leaks also. I narrowed it down to the Call object not cleaning
> > out its member vars (the whole Deserialization process). If you do
> > that the only thing stuck in memory is the call object since its part
> > of the ThreadLocal object.
> > ---------------------------------------------------------------------
> > View this comment:
> >
> > http://issues.apache.org/jira/browse/AXIS-935?
> > page=comments#action_35890
> >
> > ---------------------------------------------------------------------
> > View the issue:
> > http://issues.apache.org/jira/browse/AXIS-935
> >
> > Here is an overview of the issue:
> > ---------------------------------------------------------------------
> > Key: AXIS-935
> > Summary: Using ThreadLocal Call objects in a servlet memory leak
> > both on client and server side
> > Type: Bug
> >
> > Status: Open
> >
> > Project: Axis
> > Components:
> > Basic Architecture
> > Versions:
> > 1.1rc2
> >
> > Assignee: Axis Developers Mailing List
> > Reporter: Jennifer Jackson
> >
> > Created: Mon, 30 Jun 2003 12:12 PM
> > Updated: Thu, 3 Jun 2004 9:37 AM
> > Environment: Operating System: All
> > Platform: All
> >
> > Description:
> > I discovered using Axis in tomcat causes a memory leak.
> >
> > It seems using threadlocal in the classes causes a memory leak for
> > each Call
> > that is created. Threadlocal sticks around until the calling thread
> > dies, and
> > since tomcat keeps threads alive to answer new calls, the threadlocal
> > data
> > never goes away. The Call object gets stuck in the threadlocal data,
> > which
> > contains the entire soap response and all objects. This is a BIG
> > memory leak.
> > I initially cleared it by forcing all Call objects to clean their
> > member vars
> > after I was done with a call, then I realized the source was from the
> > thread
> > local class. Here is some info I found about threadlocal:
> >
> > This is stated in the Sun Javadocs for ThreadLocal:
> >
> > Each thread holds an implicit reference to its copy of a
> > ThreadLocal as long as the thread is alive and the
> > ThreadLocal object is accessible; after a thread goes
> > away, all of its copies of ThreadLocal variables are
> > subject to garbage collection (unless other references
> > to these copies exist).
> >
> > So, this means that ANY APPLICATION that uses PreparedStatements in a
> > thread
> > that 1) either does a lot of PreparedStatements or 2) never dies
> > (i.e., a
> > main thread) will ALWAYS eventually have an out of memory error.
> > Simply put,
> > this is a MEMORY LEAK. I imagine that the leak is very small, the
> > ThreadLocal object only contains one member variable, maybe 64 bytes
> > or less
> > (depending on the VM implementation). So, our 60,000
> > PreparedStatements of 2
> > ThreadLocals each times 64 bytes (my wild guess) is 7.5MB.
> >
> > Ideas? I've never used threadlocal myself so this is new to me.
> >
> >
> > ---------------------------------------------------------------------
> > JIRA INFORMATION:
> > This message is automatically generated by JIRA.
> >
> > If you think it was sent incorrectly contact one of the administrators:
> > http://issues.apache.org/jira/secure/Administrators.jspa
> >
> > If you want more information on JIRA, or have a bug to report see:
> > http://www.atlassian.com/software/jira
>
>
Re: [jira] Commented: (AXIS-935) Using ThreadLocal Call objects in a servlet memory leak both on client and server side
Posted by Peter Molettiere <pi...@axonstudios.net>.
We're currently looking into a massive memory leak in axis, that no one
here seems willing to admit. We're close to having a minimal test case
which demonstrates the leak we're seeing. I don't think it's the same
one you're seeing however, as our leak happens 1) when the server is
sitting doing nothing (but only after axis has been loaded), and 2)
every time we serialize a large message. The suggestions on the list
about switching to using attachments doesn't fix the leak sadly. And
setting a larger heap size only delays the inevitable Out Of Memory
Exception.
From our testing, it looks like it takes on the order of 700MB to
serialize a 30MB java object graph into a 3MB SOAP stream.
I've posted about this before, but no one has responded. Hopefully when
we post the test case someone will actually take a look at it. There
are other threads with test cases which currently have "we're looking
at this now" as the only response, but as they are all pretty old
threads, my guess is that this problem either isn't believed or is
being swept under the carpet. Maybe we're the only people using axis
for large messages.
Hopefully, we'll have more info in the next couple days.
--Peter
On Jun 3, 2004, at 9:38 AM, jira@apache.org wrote:
> The following comment has been added to this issue:
>
> Author: Jennifer Jackson
> Created: Thu, 3 Jun 2004 9:37 AM
> Body:
> Has anyone looked into this? I haven't used Axis because of this
> problem and doing a quick search it seems others have come across
> memory leaks also. I narrowed it down to the Call object not cleaning
> out its member vars (the whole Deserialization process). If you do
> that the only thing stuck in memory is the call object since its part
> of the ThreadLocal object.
> ---------------------------------------------------------------------
> View this comment:
>
> http://issues.apache.org/jira/browse/AXIS-935?
> page=comments#action_35890
>
> ---------------------------------------------------------------------
> View the issue:
> http://issues.apache.org/jira/browse/AXIS-935
>
> Here is an overview of the issue:
> ---------------------------------------------------------------------
> Key: AXIS-935
> Summary: Using ThreadLocal Call objects in a servlet memory leak
> both on client and server side
> Type: Bug
>
> Status: Open
>
> Project: Axis
> Components:
> Basic Architecture
> Versions:
> 1.1rc2
>
> Assignee: Axis Developers Mailing List
> Reporter: Jennifer Jackson
>
> Created: Mon, 30 Jun 2003 12:12 PM
> Updated: Thu, 3 Jun 2004 9:37 AM
> Environment: Operating System: All
> Platform: All
>
> Description:
> I discovered using Axis in tomcat causes a memory leak.
>
> It seems using threadlocal in the classes causes a memory leak for
> each Call
> that is created. Threadlocal sticks around until the calling thread
> dies, and
> since tomcat keeps threads alive to answer new calls, the threadlocal
> data
> never goes away. The Call object gets stuck in the threadlocal data,
> which
> contains the entire soap response and all objects. This is a BIG
> memory leak.
> I initially cleared it by forcing all Call objects to clean their
> member vars
> after I was done with a call, then I realized the source was from the
> thread
> local class. Here is some info I found about threadlocal:
>
> This is stated in the Sun Javadocs for ThreadLocal:
>
> Each thread holds an implicit reference to its copy of a
> ThreadLocal as long as the thread is alive and the
> ThreadLocal object is accessible; after a thread goes
> away, all of its copies of ThreadLocal variables are
> subject to garbage collection (unless other references
> to these copies exist).
>
> So, this means that ANY APPLICATION that uses PreparedStatements in a
> thread
> that 1) either does a lot of PreparedStatements or 2) never dies
> (i.e., a
> main thread) will ALWAYS eventually have an out of memory error.
> Simply put,
> this is a MEMORY LEAK. I imagine that the leak is very small, the
> ThreadLocal object only contains one member variable, maybe 64 bytes
> or less
> (depending on the VM implementation). So, our 60,000
> PreparedStatements of 2
> ThreadLocals each times 64 bytes (my wild guess) is 7.5MB.
>
> Ideas? I've never used threadlocal myself so this is new to me.
>
>
> ---------------------------------------------------------------------
> JIRA INFORMATION:
> This message is automatically generated by JIRA.
>
> If you think it was sent incorrectly contact one of the administrators:
> http://issues.apache.org/jira/secure/Administrators.jspa
>
> If you want more information on JIRA, or have a bug to report see:
> http://www.atlassian.com/software/jira