You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@curator.apache.org by Joe Littlejohn <jo...@gmail.com> on 2014/04/29 12:55:55 UTC
Memory leak when using service providers
Hi,
I've observed a memory leak in our production system using Curator service
discovery. I can replicate the problem with 2.4.1 and 2.4.2, though I
haven't tested any older versions.
This test shows the problem:
https://www.refheap.com/82891
(Hopefully this test is enough to demonstrate the problem. I haven't used
the TestingServer so you'll need a zk instance with a registered service.)
If you run it and watch the process with jvisualvm you'll see that the heap
grows and grows as the test is running. Taking a heap dump will reveal
thousands of ServiceInstance and ServiceCacheImpl instances that are
retained even though the provider is closed after each usage. The
references appear to be traced back to the PathChildrenCache. I realise
it's possible to retain ServiceProvider instances, but this appears to be a
leak that shouldn't occur if the provider is correctly closed each time.
Can anyone comment on whether this is indeed a problem? Am I missing
something?
Cheers
Re: Re: Memory leak when using service providers
Posted by shuxingzhang1988 <sh...@163.com>.
hi joe,
Thank you for pointing out my mistakes. I agree with you .
2014-04-30
shuxingzhang1988
发件人：Jordan Zimmerman <jo...@jordanzimmerman.com>
发送时间：2014-04-30 00:24
主题：Re: Memory leak when using service providers
收件人："Joe Littlejohn"<jo...@curator.apache.org>
抄送：
If this is a real issue can someone please open a Jira for it?
-Jordan
From: Joe Littlejohn joelittlejohn@gmail.com
Reply: user@curator.apache.org user@curator.apache.org
Date: April 29, 2014 at 10:54:41 AM
To: user@curator.apache.org user@curator.apache.org
Subject: Re: Memory leak when using service providers
References to the ServiceDiscovery instance will not stop GC, only references in the other direction would stop GC. The problem here is some cached values that are not cleaned up when a provider is closed, and the caches maintain references that stop the ServiceProvider instances from being cleaned up correctly.
When a provider is closed, the service discovery is notified (see org.apache.curator.x.discovery.details.ServiceProviderImpl#close). This allows the ServiceDiscovery to remove that provider from the list of providers that it will close when the ServiceDiscovery itself is closed.
On 29 April 2014 16:14, shuxingzhang1988 <sh...@163.com> wrote:
Hi
I think it maybe not a problem. In the test code,the so many serviceprovders have the reference to the first servicediscovery which is not released a,so they can not gc .
At 2014-04-29 18:55:55,"Joe Littlejohn" <jo...@gmail.com> wrote:
Hi,
I've observed a memory leak in our production system using Curator service discovery. I can replicate the problem with 2.4.1 and 2.4.2, though I haven't tested any older versions.
This test shows the problem:
https://www.refheap.com/82891
(Hopefully this test is enough to demonstrate the problem. I haven't used the TestingServer so you'll need a zk instance with a registered service.)
If you run it and watch the process with jvisualvm you'll see that the heap grows and grows as the test is running. Taking a heap dump will reveal thousands of ServiceInstance and ServiceCacheImpl instances that are retained even though the provider is closed after each usage. The references appear to be traced back to the PathChildrenCache. I realise it's possible to retain ServiceProvider instances, but this appears to be a leak that shouldn't occur if the provider is correctly closed each time.
Can anyone comment on whether this is indeed a problem? Am I missing something?
Cheers
Re: Memory leak when using service providers
Posted by Joe Littlejohn <jo...@gmail.com>.
Thanks Jordan. I thought you might want to have a look over the problem
before raising a ticket but I've added the details here:
https://issues.apache.org/jira/browse/CURATOR-105
Cheers
On 29 April 2014 17:24, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
> If this is a real issue can someone please open a Jira for it?
>
> -Jordan
>
>
> From: Joe Littlejohn joelittlejohn@gmail.com
> Reply: user@curator.apache.org user@curator.apache.org
> Date: April 29, 2014 at 10:54:41 AM
> To: user@curator.apache.org user@curator.apache.org
> Subject: Re: Memory leak when using service providers
>
> References to the ServiceDiscovery instance will not stop GC, only
> references in the other direction would stop GC. The problem here is some
> cached values that are not cleaned up when a provider is closed, and the
> caches maintain references that stop the ServiceProvider instances from
> being cleaned up correctly.
>
> When a provider is closed, the service discovery is notified (see
> org.apache.curator.x.discovery.details.ServiceProviderImpl#close). This
> allows the ServiceDiscovery to remove that provider from the list of
> providers that it will close when the ServiceDiscovery itself is closed.
>
>
> On 29 April 2014 16:14, shuxingzhang1988 <sh...@163.com> wrote:
>
>> Hi
>>
>> I think it maybe not a problem. In the test code,the so many
>> serviceprovders have the reference to the first servicediscovery which is
>> not released a,so they can not gc .
>>
>>
>>
>>
>>
>> At 2014-04-29 18:55:55,"Joe Littlejohn" <jo...@gmail.com> wrote:
>>
>> Hi,
>>
>> I've observed a memory leak in our production system using Curator
>> service discovery. I can replicate the problem with 2.4.1 and 2.4.2, though
>> I haven't tested any older versions.
>>
>> This test shows the problem:
>>
>> https://www.refheap.com/82891
>>
>> (Hopefully this test is enough to demonstrate the problem. I haven't used
>> the TestingServer so you'll need a zk instance with a registered service.)
>>
>> If you run it and watch the process with jvisualvm you'll see that the
>> heap grows and grows as the test is running. Taking a heap dump will reveal
>> thousands of ServiceInstance and ServiceCacheImpl instances that are
>> retained even though the provider is closed after each usage. The
>> references appear to be traced back to the PathChildrenCache. I realise
>> it's possible to retain ServiceProvider instances, but this appears to be a
>> leak that shouldn't occur if the provider is correctly closed each time.
>>
>> Can anyone comment on whether this is indeed a problem? Am I missing
>> something?
>>
>> Cheers
>>
>>
>>
>>
>
Re: Memory leak when using service providers
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
If this is a real issue can someone please open a Jira for it?
-Jordan
From: Joe Littlejohn joelittlejohn@gmail.com
Reply: user@curator.apache.org user@curator.apache.org
Date: April 29, 2014 at 10:54:41 AM
To: user@curator.apache.org user@curator.apache.org
Subject: Re: Memory leak when using service providers
References to the ServiceDiscovery instance will not stop GC, only references in the other direction would stop GC. The problem here is some cached values that are not cleaned up when a provider is closed, and the caches maintain references that stop the ServiceProvider instances from being cleaned up correctly.
When a provider is closed, the service discovery is notified (see org.apache.curator.x.discovery.details.ServiceProviderImpl#close). This allows the ServiceDiscovery to remove that provider from the list of providers that it will close when the ServiceDiscovery itself is closed.
On 29 April 2014 16:14, shuxingzhang1988 <sh...@163.com> wrote:
Hi
I think it maybe not a problem. In the test code,the so many serviceprovders have the reference to the first servicediscovery which is not released a,so they can not gc .
At 2014-04-29 18:55:55,"Joe Littlejohn" <jo...@gmail.com> wrote:
Hi,
I've observed a memory leak in our production system using Curator service discovery. I can replicate the problem with 2.4.1 and 2.4.2, though I haven't tested any older versions.
This test shows the problem:
https://www.refheap.com/82891
(Hopefully this test is enough to demonstrate the problem. I haven't used the TestingServer so you'll need a zk instance with a registered service.)
If you run it and watch the process with jvisualvm you'll see that the heap grows and grows as the test is running. Taking a heap dump will reveal thousands of ServiceInstance and ServiceCacheImpl instances that are retained even though the provider is closed after each usage. The references appear to be traced back to the PathChildrenCache. I realise it's possible to retain ServiceProvider instances, but this appears to be a leak that shouldn't occur if the provider is correctly closed each time.
Can anyone comment on whether this is indeed a problem? Am I missing something?
Cheers
Re: Memory leak when using service providers
Posted by Joe Littlejohn <jo...@gmail.com>.
References to the ServiceDiscovery instance will not stop GC, only
references in the other direction would stop GC. The problem here is some
cached values that are not cleaned up when a provider is closed, and the
caches maintain references that stop the ServiceProvider instances from
being cleaned up correctly.
When a provider is closed, the service discovery is notified (see
org.apache.curator.x.discovery.details.ServiceProviderImpl#close). This
allows the ServiceDiscovery to remove that provider from the list of
providers that it will close when the ServiceDiscovery itself is closed.
On 29 April 2014 16:14, shuxingzhang1988 <sh...@163.com> wrote:
> Hi
>
> I think it maybe not a problem. In the test code,the so many
> serviceprovders have the reference to the first servicediscovery which is
> not released a,so they can not gc .
>
>
>
>
>
> At 2014-04-29 18:55:55,"Joe Littlejohn" <jo...@gmail.com> wrote:
>
> Hi,
>
> I've observed a memory leak in our production system using Curator service
> discovery. I can replicate the problem with 2.4.1 and 2.4.2, though I
> haven't tested any older versions.
>
> This test shows the problem:
>
> https://www.refheap.com/82891
>
> (Hopefully this test is enough to demonstrate the problem. I haven't used
> the TestingServer so you'll need a zk instance with a registered service.)
>
> If you run it and watch the process with jvisualvm you'll see that the
> heap grows and grows as the test is running. Taking a heap dump will reveal
> thousands of ServiceInstance and ServiceCacheImpl instances that are
> retained even though the provider is closed after each usage. The
> references appear to be traced back to the PathChildrenCache. I realise
> it's possible to retain ServiceProvider instances, but this appears to be a
> leak that shouldn't occur if the provider is correctly closed each time.
>
> Can anyone comment on whether this is indeed a problem? Am I missing
> something?
>
> Cheers
>
>
>
>
Re:Memory leak when using service providers
Posted by shuxingzhang1988 <sh...@163.com>.
Hi
I think it maybe not a problem. In the test code,the so many serviceprovders have the reference to the first servicediscovery which is not released a,so they can not gc .
At 2014-04-29 18:55:55,"Joe Littlejohn" <jo...@gmail.com> wrote:
Hi,
I've observed a memory leak in our production system using Curator service discovery. I can replicate the problem with 2.4.1 and 2.4.2, though I haven't tested any older versions.
This test shows the problem:
https://www.refheap.com/82891
(Hopefully this test is enough to demonstrate the problem. I haven't used the TestingServer so you'll need a zk instance with a registered service.)
If you run it and watch the process with jvisualvm you'll see that the heap grows and grows as the test is running. Taking a heap dump will reveal thousands of ServiceInstance and ServiceCacheImpl instances that are retained even though the provider is closed after each usage. The references appear to be traced back to the PathChildrenCache. I realise it's possible to retain ServiceProvider instances, but this appears to be a leak that shouldn't occur if the provider is correctly closed each time.
Can anyone comment on whether this is indeed a problem? Am I missing something?
Cheers
Re: Memory leak when using service providers
Posted by Joe Littlejohn <jo...@gmail.com>.
Here's an example heap dump taken after running the test for a while:
https://dl.dropboxusercontent.com/u/10909453/heapdump-curator-leak.hprof.tar.gz
I recommend Eclipse MAT or VisualVM to analyse this.
On 29 April 2014 11:55, Joe Littlejohn <jo...@gmail.com> wrote:
> Hi,
>
> I've observed a memory leak in our production system using Curator service
> discovery. I can replicate the problem with 2.4.1 and 2.4.2, though I
> haven't tested any older versions.
>
> This test shows the problem:
>
> https://www.refheap.com/82891
>
> (Hopefully this test is enough to demonstrate the problem. I haven't used
> the TestingServer so you'll need a zk instance with a registered service.)
>
> If you run it and watch the process with jvisualvm you'll see that the
> heap grows and grows as the test is running. Taking a heap dump will reveal
> thousands of ServiceInstance and ServiceCacheImpl instances that are
> retained even though the provider is closed after each usage. The
> references appear to be traced back to the PathChildrenCache. I realise
> it's possible to retain ServiceProvider instances, but this appears to be a
> leak that shouldn't occur if the provider is correctly closed each time.
>
> Can anyone comment on whether this is indeed a problem? Am I missing
> something?
>
> Cheers
>