You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cxf.apache.org by Joe Sunday <su...@csh.rit.edu> on 2007/09/04 20:05:43 UTC
Memory leak in client?
Using 2.0.1, creating a client from a large wsdl (~1 MB) in my
classpath using generated stubs increases memory usage by about 20
MB, which never gets garbage collected. Later clients created from
the same WSDL url cause similar increases until I get an out of
memory condition.
static {
WSDL_URL = MyService.class.getClassLoader().getResource
("myService.wsdl");
SERVICE_QNAME = new QName("urn:myService", "MyService");
}
MyService service = new MyService(WSDL_URL, SERVICE_QNAME);
MyPort client = service.getMyPort();
BindingProvider provider = (BindingProvider)client;
provider.getRequestContext().put
(BindingProvider.ENDPOINT_ADDRESS_PROPERTY, endpoint.getURL());
Client c = ClientProxy.getClient(client);
HTTPConduit conduit = (HTTPConduit)c.getConduit();
conduit.getClient().setAllowChunking(false);
// ... SSL setup too ...
Working through a heap dump now, but does this look familiar to anyone?
--Joe
Re: Memory leak in client?
Posted by Joe Sunday <su...@csh.rit.edu>.
Thank you, that seems to do the trick, although I believe the first
line is actually:
BusFactory.newInstance().createBus();
--Joe
On Sep 10, 2007, at 10:12 PM, Willem Jiang wrote:
> Hi Joe,
>
> Yes, you could set a bus to all your port, but you need write some
> no JAX-WS compatible codes.
>
> Here is a code snippet for setting the default bus for all the thread
>
> Bus bus = BusFactory.createBus();
> // in you working thread start up
> BusFactory.setDefaultBus(bus);
> BusFactory.setThreadDefaultBus(bus);
> // then just use the JAX-WS API to get the port.
>
> Please see my other comments in the mail.
>
> Joe Sunday wrote:
>> Reading through that I'm still not clear.. Is the bus associated
>> to the running JVM and shared for all ports, or is it per-port?
>>
>> To rephrase back to my actual problem, which is painfully high
>> memory usage when trying to talk to a complex endpoint where the
>> WSDL is just under a meg, particularly when used against multiple
>> endpoints from multiple threads.
>>
>> 1) Is it safe to use a single port from multiple threads
>> concurrently?
> I don't think so.
>> 2) Is it safe to use multiple ports to different endpoints within
>> a thread concurrently?
> It should be safe.
>> 3) Is bus shutdown the proper way to free the 20 MB or so per
>> thread/port when I'm done with a port (but not done using CXF in
>> my application)
> It is no need for us the shutdown the bus per port invocation ,
> because it will take a long time to load a bus.
>> 4) Is there any way CXF can re-use all or some of the service
>> model objects it builds from a WSDL the next time I bind it
>> against a different endpoint? It seems like 98% of that 20 MB
>> model should be the same objects every time.
> The bus holds the reference of WSDL manager which holds the WSDL
> definitions which will eat up lots of memory for big wsdl file. If
> we share a same bus with all the ports, I think the memory
> consumption will be a low level.
Re: Memory leak in client?
Posted by Willem Jiang <ni...@iona.com>.
Hi Joe,
Yes, you could set a bus to all your port, but you need write some no
JAX-WS compatible codes.
Here is a code snippet for setting the default bus for all the thread
Bus bus = BusFactory.createBus();
// in you working thread start up
BusFactory.setDefaultBus(bus);
BusFactory.setThreadDefaultBus(bus);
// then just use the JAX-WS API to get the port.
Please see my other comments in the mail.
Joe Sunday wrote:
> Reading through that I'm still not clear.. Is the bus associated to
> the running JVM and shared for all ports, or is it per-port?
>
> To rephrase back to my actual problem, which is painfully high memory
> usage when trying to talk to a complex endpoint where the WSDL is just
> under a meg, particularly when used against multiple endpoints from
> multiple threads.
>
> 1) Is it safe to use a single port from multiple threads concurrently?
I don't think so.
> 2) Is it safe to use multiple ports to different endpoints within a
> thread concurrently?
It should be safe.
> 3) Is bus shutdown the proper way to free the 20 MB or so per
> thread/port when I'm done with a port (but not done using CXF in my
> application)
It is no need for us the shutdown the bus per port invocation , because
it will take a long time to load a bus.
> 4) Is there any way CXF can re-use all or some of the service model
> objects it builds from a WSDL the next time I bind it against a
> different endpoint? It seems like 98% of that 20 MB model should be
> the same objects every time.
The bus holds the reference of WSDL manager which holds the WSDL
definitions which will eat up lots of memory for big wsdl file. If we
share a same bus with all the ports, I think the memory consumption will
be a low level.
>
> --Joe
>
> On Sep 7, 2007, at 2:59 AM, Willem Jiang wrote:
>
>> Hi
>> If you use the Jaxws API getPort to create the client proxy. CXF will
>> try to get a default bus first ,
>> if the default bus is not exit, it will create a bus and set default
>> bus and default thread local bus for you.
>>
>> So the thread local object in your case should be the bus :)
>> Here are some code snippets which can help you to know better about
>> the default Bus and bus.shutdown()
>> 1. Bus creation
>> public CXFBusImpl(Map<Class, Object> extensions) {
>> if (extensions == null) {
>> extensions = new ConcurrentHashMap<Class, Object>();
>> } else {
>> extensions = new ConcurrentHashMap<Class,
>> Object>(extensions);
>> }
>> this.extensions = extensions;
>> state = BusState.INITIAL;
>> CXFBusFactory.possiblySetDefaultBus(this);
>> }
>>
>> 2. CXFBusFactory.possiblySetDefaultBus(this);
>> public static synchronized boolean possiblySetDefaultBus(Bus bus) {
>> // there is a thread location variable which store the default
>> thread local bus
>> if (localBus.get() == null) {
>> localBus.set(bus);
>> }
>> if (defaultBus == null) {
>> defaultBus = bus; return true;
>> }
>> return false;
>> }
>>
>> 3. CXFBusImpl.shutdown(boolean wait)
>> public void shutdown(boolean wait) {
>> BusLifeCycleManager lifeCycleManager =
>> this.getExtension(BusLifeCycleManager.class);
>> if (null != lifeCycleManager) {
>> lifeCycleManager.preShutdown();
>> }
>> synchronized (this) {
>> state = BusState.SHUTDOWN;
>> notifyAll();
>> }
>> if (null != lifeCycleManager) {
>> lifeCycleManager.postShutdown();
>> }
>> if (BusFactory.getDefaultBus(false) == this) {
>> BusFactory.setDefaultBus(null);
>> }
>> }
>> You can find more information about bus by reading the source code.
>>
>> CXFBusImpl
>> https://svn.apache.org/repos/asf/incubator/cxf/trunk/rt/core/src/main/java/org/apache/cxf/bus/CXFBusImpl.java
>>
>> CXFBusFactory
>> https://svn.apache.org/repos/asf/incubator/cxf/trunk/rt/core/src/main/java/org/apache/cxf/bus/CXFBusFactory.java
>>
>>
>> Willem.
>>
>> Joe Sunday wrote:
>>> Is that a done with the client or done with the application call?
>>>
>>> If there's something I need to call when I'm done with a particular
>>> port, I can deal with that. I don't see it documented anywhere though.
>>>
>>> --Joe
>>>
>>> On Sep 6, 2007, at 3:21 PM, Daniel Kulp wrote:
>>>
>>>>
>>>> Joe,
>>>>
>>>> One thing to keep in mind: the Bus would still be around. You'd
>>>> need to
>>>> get the default Bus and call shutdown() on it if you're completely
>>>> done.
>>>>
>>>> Dan
>>>>
>>>> --
>>>> J. Daniel Kulp
>>>> Principal Engineer
>>>> IONA
>>>> P: 781-902-8727 C: 508-380-7194
>>>> daniel.kulp@iona.com
>>>> http://www.dankulp.com/blog
>>>
>
Willem.
Re: Memory leak in client?
Posted by Joe Sunday <su...@csh.rit.edu>.
Reading through that I'm still not clear.. Is the bus associated to
the running JVM and shared for all ports, or is it per-port?
To rephrase back to my actual problem, which is painfully high memory
usage when trying to talk to a complex endpoint where the WSDL is
just under a meg, particularly when used against multiple endpoints
from multiple threads.
1) Is it safe to use a single port from multiple threads concurrently?
2) Is it safe to use multiple ports to different endpoints within a
thread concurrently?
3) Is bus shutdown the proper way to free the 20 MB or so per thread/
port when I'm done with a port (but not done using CXF in my
application)
4) Is there any way CXF can re-use all or some of the service model
objects it builds from a WSDL the next time I bind it against a
different endpoint? It seems like 98% of that 20 MB model should be
the same objects every time.
--Joe
On Sep 7, 2007, at 2:59 AM, Willem Jiang wrote:
> Hi
> If you use the Jaxws API getPort to create the client proxy. CXF
> will try to get a default bus first ,
> if the default bus is not exit, it will create a bus and set
> default bus and default thread local bus for you.
>
> So the thread local object in your case should be the bus :)
> Here are some code snippets which can help you to know better about
> the default Bus and bus.shutdown()
> 1. Bus creation
> public CXFBusImpl(Map<Class, Object> extensions) {
> if (extensions == null) {
> extensions = new ConcurrentHashMap<Class, Object>();
> } else {
> extensions = new ConcurrentHashMap<Class, Object>
> (extensions);
> }
> this.extensions = extensions;
> state = BusState.INITIAL;
> CXFBusFactory.possiblySetDefaultBus(this);
> }
>
> 2. CXFBusFactory.possiblySetDefaultBus(this);
> public static synchronized boolean possiblySetDefaultBus(Bus bus) {
> // there is a thread location variable which store the
> default thread local bus
> if (localBus.get() == null) {
> localBus.set(bus);
> }
> if (defaultBus == null) {
> defaultBus = bus; return true;
> }
> return false;
> }
>
> 3. CXFBusImpl.shutdown(boolean wait)
> public void shutdown(boolean wait) {
> BusLifeCycleManager lifeCycleManager = this.getExtension
> (BusLifeCycleManager.class);
> if (null != lifeCycleManager) {
> lifeCycleManager.preShutdown();
> }
> synchronized (this) {
> state = BusState.SHUTDOWN;
> notifyAll();
> }
> if (null != lifeCycleManager) {
> lifeCycleManager.postShutdown();
> }
> if (BusFactory.getDefaultBus(false) == this) {
> BusFactory.setDefaultBus(null);
> }
> }
> You can find more information about bus by reading the source code.
>
> CXFBusImpl https://svn.apache.org/repos/asf/incubator/cxf/trunk/rt/
> core/src/main/java/org/apache/cxf/bus/CXFBusImpl.java
> CXFBusFactory https://svn.apache.org/repos/asf/incubator/cxf/trunk/
> rt/core/src/main/java/org/apache/cxf/bus/CXFBusFactory.java
>
> Willem.
>
> Joe Sunday wrote:
>> Is that a done with the client or done with the application call?
>>
>> If there's something I need to call when I'm done with a
>> particular port, I can deal with that. I don't see it documented
>> anywhere though.
>>
>> --Joe
>>
>> On Sep 6, 2007, at 3:21 PM, Daniel Kulp wrote:
>>
>>>
>>> Joe,
>>>
>>> One thing to keep in mind: the Bus would still be around. You'd
>>> need to
>>> get the default Bus and call shutdown() on it if you're
>>> completely done.
>>>
>>> Dan
>>>
>>> --
>>> J. Daniel Kulp
>>> Principal Engineer
>>> IONA
>>> P: 781-902-8727 C: 508-380-7194
>>> daniel.kulp@iona.com
>>> http://www.dankulp.com/blog
>>
Re: Memory leak in client?
Posted by Willem Jiang <ni...@iona.com>.
Hi
If you use the Jaxws API getPort to create the client proxy. CXF will
try to get a default bus first ,
if the default bus is not exit, it will create a bus and set default bus
and default thread local bus for you.
So the thread local object in your case should be the bus :)
Here are some code snippets which can help you to know better about the
default Bus and bus.shutdown()
1. Bus creation
public CXFBusImpl(Map<Class, Object> extensions) {
if (extensions == null) {
extensions = new ConcurrentHashMap<Class, Object>();
} else {
extensions = new ConcurrentHashMap<Class, Object>(extensions);
}
this.extensions = extensions;
state = BusState.INITIAL;
CXFBusFactory.possiblySetDefaultBus(this);
}
2. CXFBusFactory.possiblySetDefaultBus(this);
public static synchronized boolean possiblySetDefaultBus(Bus bus) {
// there is a thread location variable which store the default
thread local bus
if (localBus.get() == null) {
localBus.set(bus);
}
if (defaultBus == null) {
defaultBus = bus;
return true;
}
return false;
}
3. CXFBusImpl.shutdown(boolean wait)
public void shutdown(boolean wait) {
BusLifeCycleManager lifeCycleManager =
this.getExtension(BusLifeCycleManager.class);
if (null != lifeCycleManager) {
lifeCycleManager.preShutdown();
}
synchronized (this) {
state = BusState.SHUTDOWN;
notifyAll();
}
if (null != lifeCycleManager) {
lifeCycleManager.postShutdown();
}
if (BusFactory.getDefaultBus(false) == this) {
BusFactory.setDefaultBus(null);
}
}
You can find more information about bus by reading the source code.
CXFBusImpl
https://svn.apache.org/repos/asf/incubator/cxf/trunk/rt/core/src/main/java/org/apache/cxf/bus/CXFBusImpl.java
CXFBusFactory
https://svn.apache.org/repos/asf/incubator/cxf/trunk/rt/core/src/main/java/org/apache/cxf/bus/CXFBusFactory.java
Willem.
Joe Sunday wrote:
> Is that a done with the client or done with the application call?
>
> If there's something I need to call when I'm done with a particular
> port, I can deal with that. I don't see it documented anywhere though.
>
> --Joe
>
> On Sep 6, 2007, at 3:21 PM, Daniel Kulp wrote:
>
>>
>> Joe,
>>
>> One thing to keep in mind: the Bus would still be around. You'd
>> need to
>> get the default Bus and call shutdown() on it if you're completely done.
>>
>> Dan
>>
>> --
>> J. Daniel Kulp
>> Principal Engineer
>> IONA
>> P: 781-902-8727 C: 508-380-7194
>> daniel.kulp@iona.com
>> http://www.dankulp.com/blog
>
Re: Memory leak in client?
Posted by Joe Sunday <su...@csh.rit.edu>.
Is that a done with the client or done with the application call?
If there's something I need to call when I'm done with a particular
port, I can deal with that. I don't see it documented anywhere though.
--Joe
On Sep 6, 2007, at 3:21 PM, Daniel Kulp wrote:
>
> Joe,
>
> One thing to keep in mind: the Bus would still be around. You'd
> need to
> get the default Bus and call shutdown() on it if you're completely
> done.
>
> Dan
>
> --
> J. Daniel Kulp
> Principal Engineer
> IONA
> P: 781-902-8727 C: 508-380-7194
> daniel.kulp@iona.com
> http://www.dankulp.com/blog
Re: Memory leak in client?
Posted by Daniel Kulp <dk...@apache.org>.
Joe,
One thing to keep in mind: the Bus would still be around. You'd need to
get the default Bus and call shutdown() on it if you're completely done.
Dan
On Wednesday 05 September 2007, Joe Sunday wrote:
> On Sep 5, 2007, at 2:07 AM, Liu, Jervis wrote:
> >> -----Original Message-----
> >> From: Joe Sunday [mailto:sunday@csh.rit.edu]
> >> Sent: 2007?9?5? 12:11
> >> To: cxf-user@incubator.apache.org
> >> Subject: Re: Memory leak in client?
> >>
> >>
> >> I changed it so I only create the service object once and then each
> >> time I need a new client use getMyPort and set it up.. The docs
> >> aren't exactly clear, but it looks like I get distinct proxies back
> >> on each call, so I'm assuming that's safe? If I then call
> >> service.getMyPort() across multiple threads?
> >
> > As you already noticed, getMyPort(which calls
> > org.apache.cxf.jaxws.ServiceImpl.createPort(QName,
> > EndpointReferenceType, Class<T>) indeed creates a new proxy object
> > every time, thus a new ServiceFactoryBean, a new service model etc
> > for every new proxy. If what you are trying to do is invoking a
> > same web service several times, you should try to use the same
> > proxy instead.
>
> That's not always possible. The client is running inside a CXF
> service, and the endpoint and session to the remote end change over
> time. I shouldn't have to cache 100 ports for the life of the
> program. I re-use the same port where I can, but when I'm done with
> it it shouldn't keep holding memory.
>
> >> It still leaks about 6 megs each time I call getMyPort() and make a
> >> few remote requests. After any references I have to the new port
> >> object are gone and I've forced a few GCs,it looks like a bunch of
> >> data being left in a ThreadLocal. I'm still trying to unwind what
> >> happens behind the scenes when I setup a client fully, but I see a
> >> couple ThreadLocals in JaxWsClientProxy that might not be getting
> >> cleared?
> >
> > If you are sure all references to the proxy object have gone, and
> > there are still some objects not cleaned by GC, then it might be a
> > memory leak on the client side. Would you be able to reproduce same
> > symptoms using a very simple WSDL, for example, the hello_world
> > sample in CXF distribution?
>
> Here's my test that approximates what I'm really doing... I've got a
> stack of worker threads that each one will occasionally create a port
> and a few requests, and then let it go. The threads stick around to
> get re-used. If I run this test, the following happens:
>
> 10 threads / 50 invokes() per thread:
> 1) Beginning execution 1,512k
> 2) Workers finished and idle 5,500k
> 3) Telling workers to die 5,500k
> 4) Idle state 1,736k
>
> 10 threads / 100 invokes() per thread:
> 1) Beginning execution 1,512k
> 2) Workers finished and idle 5,761k
> 3) Telling workers to die 5,761k
> 4) Idle state 1,732k
>
> 20 threads / 50 invokes() per thread:
> 1) Beginning execution 1,512k
> 2) Workers finished and idle 12,052k
> 3) Telling workers to die 10,814k
> 4) Idle state 2,268k
>
> 20 threads / 100 invokes() per thread:
> 1) Beginning execution 1,512k
> 2) Workers finished and idle 11,245k
> 3) Telling workers to die 9,983k
> 4) Idle state 2,403k
>
> If I make invoke() a no-op as a baseline:
> 1) Beginning execution 1,522k
> 2) Workers finished and idle 1,520k
> 3) Telling workers to die 1,504k
> 4) Idle state 1,504k
>
> I would expect for CXF, (3) would be around the same value as (4),
> since there should not be live references to the port objects hanging
> around at that point. It looks like something's getting put into a
> ThreadLocal that isn't getting collected until the thread dies even
> after I've lost the reference to the port object.
>
> This also makes me wonder, if I am caching the port objects for a
> brief time, is it safe to pass control of a port from one thread to
> another? My actual worker operations may migrate threads between
> outgoing requests, so can I pass the port safely to the new thread?
>
> package soap;
>
> import java.io.File;
> import java.net.MalformedURLException;
> import java.net.URL;
> import javax.xml.namespace.QName;
> import java.util.concurrent.CountDownLatch;
>
> public final class Greeter_SoapPort_Client {
>
> private static final QName SERVICE_NAME = new QName("http://
> apache.org/hello_world_soap_http", "SOAPService");
> private static final URL WSDL = Greeter.class.getClassLoader
> ().getResource("hello_world.wsdl");
>
> private Greeter_SoapPort_Client() {
> }
>
> public static void main(String args[]) throws Exception {
> int tCount = 10;
> CountDownLatch latch = new CountDownLatch(tCount);
>
> SOAPService ss = new SOAPService(WSDL, SERVICE_NAME);
> System.out.println("Waiting to attach jconsole");
> try { Thread.sleep(7 * 1000); }
> catch (InterruptedException e) { }
>
> System.out.println("1) Beginning execution " + mem());
>
> Thread[] workers = new Thread[tCount];
> for (int i = 0; i < tCount; i++) {
> workers[i] = new Thread(
> new Worker(ss, latch));
> workers[i].setName("Worker " + i);
> workers[i].start();
> }
>
> try { latch.await(); }
> catch (InterruptedException e) { }
>
> System.out.println("2) Workers finished and idle " + mem());
>
> try { Thread.sleep(10 * 1000); }
> catch (InterruptedException e) { }
>
> System.out.println("3) Telling workers to die " + mem());
> for (int i = 0; i < tCount; i++) {
> synchronized (workers[i]) {
> workers[i].notify();
> workers[i] = null;
> }
> }
>
> try { Thread.sleep(10 * 1000); }
> catch (InterruptedException e) { }
>
> System.out.println("4) Idle state " + mem());
> }
>
> public static String mem() {
> System.gc();
> System.gc();
> long b = Runtime.getRuntime().totalMemory() -
> Runtime.getRuntime().freeMemory();
> return String.format("%,dk", b / 1024);
>
> }
>
> private static class Worker implements Runnable {
> SOAPService ss;
> CountDownLatch latch;
>
> Worker(SOAPService ss, CountDownLatch latch) {
> this.ss = ss;
> this.latch = latch;
> }
>
> public void run() {
> for (int i = 0; i < 100; i++) {
> invoke();
> }
> latch.countDown();
> synchronized (Thread.currentThread()) {
> try { Thread.currentThread().wait(); }
> catch (InterruptedException e) { }
> }
> }
>
> // Force this into a separate stack frame
> public void invoke() {
> Greeter port = ss.getSoapPort();
> String ret = port.sayHi();
> }
> }
> }
>
>
> --Joe
--
J. Daniel Kulp
Principal Engineer
IONA
P: 781-902-8727 C: 508-380-7194
daniel.kulp@iona.com
http://www.dankulp.com/blog
Re: Memory leak in client?
Posted by Joe Sunday <su...@csh.rit.edu>.
On Sep 5, 2007, at 2:07 AM, Liu, Jervis wrote:
>
>
>> -----Original Message-----
>> From: Joe Sunday [mailto:sunday@csh.rit.edu]
>> Sent: 2007?9?5? 12:11
>> To: cxf-user@incubator.apache.org
>> Subject: Re: Memory leak in client?
>>
>>
>> I changed it so I only create the service object once and then each
>> time I need a new client use getMyPort and set it up.. The docs
>> aren't exactly clear, but it looks like I get distinct proxies back
>> on each call, so I'm assuming that's safe? If I then call
>> service.getMyPort() across multiple threads?
>>
>
> As you already noticed, getMyPort(which calls
> org.apache.cxf.jaxws.ServiceImpl.createPort(QName,
> EndpointReferenceType, Class<T>) indeed creates a new proxy object
> every time, thus a new ServiceFactoryBean, a new service model etc
> for every new proxy. If what you are trying to do is invoking a
> same web service several times, you should try to use the same
> proxy instead.
That's not always possible. The client is running inside a CXF
service, and the endpoint and session to the remote end change over
time. I shouldn't have to cache 100 ports for the life of the
program. I re-use the same port where I can, but when I'm done with
it it shouldn't keep holding memory.
>
>> It still leaks about 6 megs each time I call getMyPort() and make a
>> few remote requests. After any references I have to the new port
>> object are gone and I've forced a few GCs,it looks like a bunch of
>> data being left in a ThreadLocal. I'm still trying to unwind what
>> happens behind the scenes when I setup a client fully, but I see a
>> couple ThreadLocals in JaxWsClientProxy that might not be getting
>> cleared?
>>
>
> If you are sure all references to the proxy object have gone, and
> there are still some objects not cleaned by GC, then it might be a
> memory leak on the client side. Would you be able to reproduce same
> symptoms using a very simple WSDL, for example, the hello_world
> sample in CXF distribution?
Here's my test that approximates what I'm really doing... I've got a
stack of worker threads that each one will occasionally create a port
and a few requests, and then let it go. The threads stick around to
get re-used. If I run this test, the following happens:
10 threads / 50 invokes() per thread:
1) Beginning execution 1,512k
2) Workers finished and idle 5,500k
3) Telling workers to die 5,500k
4) Idle state 1,736k
10 threads / 100 invokes() per thread:
1) Beginning execution 1,512k
2) Workers finished and idle 5,761k
3) Telling workers to die 5,761k
4) Idle state 1,732k
20 threads / 50 invokes() per thread:
1) Beginning execution 1,512k
2) Workers finished and idle 12,052k
3) Telling workers to die 10,814k
4) Idle state 2,268k
20 threads / 100 invokes() per thread:
1) Beginning execution 1,512k
2) Workers finished and idle 11,245k
3) Telling workers to die 9,983k
4) Idle state 2,403k
If I make invoke() a no-op as a baseline:
1) Beginning execution 1,522k
2) Workers finished and idle 1,520k
3) Telling workers to die 1,504k
4) Idle state 1,504k
I would expect for CXF, (3) would be around the same value as (4),
since there should not be live references to the port objects hanging
around at that point. It looks like something's getting put into a
ThreadLocal that isn't getting collected until the thread dies even
after I've lost the reference to the port object.
This also makes me wonder, if I am caching the port objects for a
brief time, is it safe to pass control of a port from one thread to
another? My actual worker operations may migrate threads between
outgoing requests, so can I pass the port safely to the new thread?
package soap;
import java.io.File;
import java.net.MalformedURLException;
import java.net.URL;
import javax.xml.namespace.QName;
import java.util.concurrent.CountDownLatch;
public final class Greeter_SoapPort_Client {
private static final QName SERVICE_NAME = new QName("http://
apache.org/hello_world_soap_http", "SOAPService");
private static final URL WSDL = Greeter.class.getClassLoader
().getResource("hello_world.wsdl");
private Greeter_SoapPort_Client() {
}
public static void main(String args[]) throws Exception {
int tCount = 10;
CountDownLatch latch = new CountDownLatch(tCount);
SOAPService ss = new SOAPService(WSDL, SERVICE_NAME);
System.out.println("Waiting to attach jconsole");
try { Thread.sleep(7 * 1000); }
catch (InterruptedException e) { }
System.out.println("1) Beginning execution " + mem());
Thread[] workers = new Thread[tCount];
for (int i = 0; i < tCount; i++) {
workers[i] = new Thread(
new Worker(ss, latch));
workers[i].setName("Worker " + i);
workers[i].start();
}
try { latch.await(); }
catch (InterruptedException e) { }
System.out.println("2) Workers finished and idle " + mem());
try { Thread.sleep(10 * 1000); }
catch (InterruptedException e) { }
System.out.println("3) Telling workers to die " + mem());
for (int i = 0; i < tCount; i++) {
synchronized (workers[i]) {
workers[i].notify();
workers[i] = null;
}
}
try { Thread.sleep(10 * 1000); }
catch (InterruptedException e) { }
System.out.println("4) Idle state " + mem());
}
public static String mem() {
System.gc();
System.gc();
long b = Runtime.getRuntime().totalMemory() -
Runtime.getRuntime().freeMemory();
return String.format("%,dk", b / 1024);
}
private static class Worker implements Runnable {
SOAPService ss;
CountDownLatch latch;
Worker(SOAPService ss, CountDownLatch latch) {
this.ss = ss;
this.latch = latch;
}
public void run() {
for (int i = 0; i < 100; i++) {
invoke();
}
latch.countDown();
synchronized (Thread.currentThread()) {
try { Thread.currentThread().wait(); }
catch (InterruptedException e) { }
}
}
// Force this into a separate stack frame
public void invoke() {
Greeter port = ss.getSoapPort();
String ret = port.sayHi();
}
}
}
--Joe
RE: Memory leak in client?
Posted by "Liu, Jervis" <jl...@iona.com>.
> -----Original Message-----
> From: Joe Sunday [mailto:sunday@csh.rit.edu]
> Sent: 2007?9?5? 12:11
> To: cxf-user@incubator.apache.org
> Subject: Re: Memory leak in client?
>
>
> I changed it so I only create the service object once and then each
> time I need a new client use getMyPort and set it up.. The docs
> aren't exactly clear, but it looks like I get distinct proxies back
> on each call, so I'm assuming that's safe? If I then call
> service.getMyPort() across multiple threads?
>
As you already noticed, getMyPort(which calls org.apache.cxf.jaxws.ServiceImpl.createPort(QName, EndpointReferenceType, Class<T>) indeed creates a new proxy object every time, thus a new ServiceFactoryBean, a new service model etc for every new proxy. If what you are trying to do is invoking a same web service several times, you should try to use the same proxy instead.
> It still leaks about 6 megs each time I call getMyPort() and make a
> few remote requests. After any references I have to the new port
> object are gone and I've forced a few GCs,it looks like a bunch of
> data being left in a ThreadLocal. I'm still trying to unwind what
> happens behind the scenes when I setup a client fully, but I see a
> couple ThreadLocals in JaxWsClientProxy that might not be getting
> cleared?
>
If you are sure all references to the proxy object have gone, and there are still some objects not cleaned by GC, then it might be a memory leak on the client side. Would you be able to reproduce same symptoms using a very simple WSDL, for example, the hello_world sample in CXF distribution?
For example, following code snippet is modified based on hello_world sample to show a memory leak :
................
SOAPService ss = new SOAPService(wsdlURL, SERVICE_NAME);
try (int =0; i <1000; i++ ) {
Greeter port = ss.getSoapPort();
String resp;
System.out.println("Invoking sayHi...");
resp = port.sayHi();
System.out.println("Server responded with: " + resp);
System.out.println();
}
If you run this, and you see the memory usage increased steadily after 1000 loops, then its definitely a memory leak.
Jervis
> --Joe
>
> On Sep 4, 2007, at 10:10 PM, Liu, Jervis wrote:
>
> > Hi Joe, based on the information you supplied, I wont necessarily
> > say there is a memory leak. Internally CXF runtime maintains a
> > service model which is built either from WSDL or from POJO,
> both on
> > the client side and server side. And this service model will keep
> > alive until the server or client gets shut down. It can
> happen that
> > a service model takes about 20MB for a WSDL of 1 MB. Having this
> > said, we did fix several memory leaks before, for example, if you
> > start a CXF client and invoke an operation several times then you
> > find the memory usage increases steadily, then it is likely that
> > you run into a memory leak problem.
> >
> > Cheers,
> > Jervis
> >
> >
> >> -----Original Message-----
> >> From: Joe Sunday [mailto:sunday@csh.rit.edu]
> >> Sent: 2007?9?5? 2:06
> >> To: cxf-user@incubator.apache.org
> >> Subject: Memory leak in client?
> >>
> >>
> >> Using 2.0.1, creating a client from a large wsdl (~1 MB) in my
> >> classpath using generated stubs increases memory usage by about 20
> >> MB, which never gets garbage collected. Later clients created from
> >> the same WSDL url cause similar increases until I get an out of
> >> memory condition.
> >>
> >> static {
> >> WSDL_URL = MyService.class.getClassLoader().getResource
> >> ("myService.wsdl");
> >> SERVICE_QNAME = new QName("urn:myService", "MyService");
> >> }
> >>
> >> MyService service = new MyService(WSDL_URL, SERVICE_QNAME);
> >> MyPort client = service.getMyPort();
> >> BindingProvider provider = (BindingProvider)client;
> >> provider.getRequestContext().put
> >> (BindingProvider.ENDPOINT_ADDRESS_PROPERTY, endpoint.getURL());
> >> Client c = ClientProxy.getClient(client);
> >> HTTPConduit conduit = (HTTPConduit)c.getConduit();
> >> conduit.getClient().setAllowChunking(false);
> >> // ... SSL setup too ...
> >>
> >> Working through a heap dump now, but does this look familiar
> >> to anyone?
> >>
> >> --Joe
> >>
> >>
> >
> > ----------------------------
> > IONA Technologies PLC (registered in Ireland)
> > Registered Number: 171387
> > Registered Address: The IONA Building, Shelbourne Road, Dublin 4,
> > Ireland
>
----------------------------
IONA Technologies PLC (registered in Ireland)
Registered Number: 171387
Registered Address: The IONA Building, Shelbourne Road, Dublin 4, Ireland
Re: Memory leak in client?
Posted by Joe Sunday <su...@csh.rit.edu>.
I changed it so I only create the service object once and then each
time I need a new client use getMyPort and set it up.. The docs
aren't exactly clear, but it looks like I get distinct proxies back
on each call, so I'm assuming that's safe? If I then call
service.getMyPort() across multiple threads?
It still leaks about 6 megs each time I call getMyPort() and make a
few remote requests. After any references I have to the new port
object are gone and I've forced a few GCs,it looks like a bunch of
data being left in a ThreadLocal. I'm still trying to unwind what
happens behind the scenes when I setup a client fully, but I see a
couple ThreadLocals in JaxWsClientProxy that might not be getting
cleared?
--Joe
On Sep 4, 2007, at 10:10 PM, Liu, Jervis wrote:
> Hi Joe, based on the information you supplied, I wont necessarily
> say there is a memory leak. Internally CXF runtime maintains a
> service model which is built either from WSDL or from POJO, both on
> the client side and server side. And this service model will keep
> alive until the server or client gets shut down. It can happen that
> a service model takes about 20MB for a WSDL of 1 MB. Having this
> said, we did fix several memory leaks before, for example, if you
> start a CXF client and invoke an operation several times then you
> find the memory usage increases steadily, then it is likely that
> you run into a memory leak problem.
>
> Cheers,
> Jervis
>
>
>> -----Original Message-----
>> From: Joe Sunday [mailto:sunday@csh.rit.edu]
>> Sent: 2007?9?5? 2:06
>> To: cxf-user@incubator.apache.org
>> Subject: Memory leak in client?
>>
>>
>> Using 2.0.1, creating a client from a large wsdl (~1 MB) in my
>> classpath using generated stubs increases memory usage by about 20
>> MB, which never gets garbage collected. Later clients created from
>> the same WSDL url cause similar increases until I get an out of
>> memory condition.
>>
>> static {
>> WSDL_URL = MyService.class.getClassLoader().getResource
>> ("myService.wsdl");
>> SERVICE_QNAME = new QName("urn:myService", "MyService");
>> }
>>
>> MyService service = new MyService(WSDL_URL, SERVICE_QNAME);
>> MyPort client = service.getMyPort();
>> BindingProvider provider = (BindingProvider)client;
>> provider.getRequestContext().put
>> (BindingProvider.ENDPOINT_ADDRESS_PROPERTY, endpoint.getURL());
>> Client c = ClientProxy.getClient(client);
>> HTTPConduit conduit = (HTTPConduit)c.getConduit();
>> conduit.getClient().setAllowChunking(false);
>> // ... SSL setup too ...
>>
>> Working through a heap dump now, but does this look familiar
>> to anyone?
>>
>> --Joe
>>
>>
>
> ----------------------------
> IONA Technologies PLC (registered in Ireland)
> Registered Number: 171387
> Registered Address: The IONA Building, Shelbourne Road, Dublin 4,
> Ireland
RE: Memory leak in client?
Posted by "Liu, Jervis" <jl...@iona.com>.
Hi Joe, based on the information you supplied, I wont necessarily say there is a memory leak. Internally CXF runtime maintains a service model which is built either from WSDL or from POJO, both on the client side and server side. And this service model will keep alive until the server or client gets shut down. It can happen that a service model takes about 20MB for a WSDL of 1 MB. Having this said, we did fix several memory leaks before, for example, if you start a CXF client and invoke an operation several times then you find the memory usage increases steadily, then it is likely that you run into a memory leak problem.
Cheers,
Jervis
> -----Original Message-----
> From: Joe Sunday [mailto:sunday@csh.rit.edu]
> Sent: 2007?9?5? 2:06
> To: cxf-user@incubator.apache.org
> Subject: Memory leak in client?
>
>
> Using 2.0.1, creating a client from a large wsdl (~1 MB) in my
> classpath using generated stubs increases memory usage by about 20
> MB, which never gets garbage collected. Later clients created from
> the same WSDL url cause similar increases until I get an out of
> memory condition.
>
> static {
> WSDL_URL = MyService.class.getClassLoader().getResource
> ("myService.wsdl");
> SERVICE_QNAME = new QName("urn:myService", "MyService");
> }
>
> MyService service = new MyService(WSDL_URL, SERVICE_QNAME);
> MyPort client = service.getMyPort();
> BindingProvider provider = (BindingProvider)client;
> provider.getRequestContext().put
> (BindingProvider.ENDPOINT_ADDRESS_PROPERTY, endpoint.getURL());
> Client c = ClientProxy.getClient(client);
> HTTPConduit conduit = (HTTPConduit)c.getConduit();
> conduit.getClient().setAllowChunking(false);
> // ... SSL setup too ...
>
> Working through a heap dump now, but does this look familiar
> to anyone?
>
> --Joe
>
>
----------------------------
IONA Technologies PLC (registered in Ireland)
Registered Number: 171387
Registered Address: The IONA Building, Shelbourne Road, Dublin 4, Ireland