You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Abdul Patel <ab...@gmail.com> on 2018/06/04 13:52:13 UTC
3.11.2 memory leak
Hi All,
I recently upgraded my non prod cluster from 3.10 to 3.11.2.
It was working fine for a 1.5 weeks then suddenly nodetool info startee
reporting 80% and more memory consumption.
Intially it was 16gb configured, then i bumped to 20gb and rebooted all 4
nodes of cluster-single DC.
Now after 8 days i again see 80% + usage and its 16gb and above ..which we
never saw before .
Seems like memory leak bug?
Does anyone has any idea ? Our 3.11.2 release rollout has been halted
because of this.
If not 3.11.2 whats the next best stable release we have now?
Re: 3.11.2 memory leak
Posted by kurt greaves <ku...@instaclustr.com>.
Likely in the next few weeks.
On Mon., 23 Jul. 2018, 01:17 Abdul Patel, <ab...@gmail.com> wrote:
> Any idea when 3.11.3 is coming in?
>
> On Tuesday, June 19, 2018, kurt greaves <ku...@instaclustr.com> wrote:
>
>> At this point I'd wait for 3.11.3. If you can't, you can get away with
>> backporting a few repair fixes or just doing sub range repairs on 3.11.2
>>
>> On Wed., 20 Jun. 2018, 01:10 Abdul Patel, <ab...@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> Do we kmow whats the stable version for now if u wish to upgrade ?
>>>
>>> On Tuesday, June 5, 2018, Steinmaurer, Thomas <
>>> thomas.steinmaurer@dynatrace.com> wrote:
>>>
>>>> Jeff,
>>>>
>>>>
>>>>
>>>> FWIW, when talking about
>>>> https://issues.apache.org/jira/browse/CASSANDRA-13929, there is a
>>>> patch available since March without getting further attention.
>>>>
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Thomas
>>>>
>>>>
>>>>
>>>> *From:* Jeff Jirsa [mailto:jjirsa@gmail.com]
>>>> *Sent:* Dienstag, 05. Juni 2018 00:51
>>>> *To:* cassandra <us...@cassandra.apache.org>
>>>> *Subject:* Re: 3.11.2 memory leak
>>>>
>>>>
>>>>
>>>> There have been a few people who have reported it, but nobody (yet) has
>>>> offered a patch to fix it. It would be good to have a reliable way to
>>>> repro, and/or an analysis of a heap dump demonstrating the problem (what's
>>>> actually retained at the time you're OOM'ing).
>>>>
>>>>
>>>>
>>>> On Mon, Jun 4, 2018 at 6:52 AM, Abdul Patel <ab...@gmail.com>
>>>> wrote:
>>>>
>>>> Hi All,
>>>>
>>>>
>>>>
>>>> I recently upgraded my non prod cluster from 3.10 to 3.11.2.
>>>>
>>>> It was working fine for a 1.5 weeks then suddenly nodetool info startee
>>>> reporting 80% and more memory consumption.
>>>>
>>>> Intially it was 16gb configured, then i bumped to 20gb and rebooted all
>>>> 4 nodes of cluster-single DC.
>>>>
>>>> Now after 8 days i again see 80% + usage and its 16gb and above ..which
>>>> we never saw before .
>>>>
>>>> Seems like memory leak bug?
>>>>
>>>> Does anyone has any idea ? Our 3.11.2 release rollout has been halted
>>>> because of this.
>>>>
>>>> If not 3.11.2 whats the next best stable release we have now?
>>>>
>>>>
>>>> The contents of this e-mail are intended for the named addressee only.
>>>> It contains information that may be confidential. Unless you are the named
>>>> addressee or an authorized designee, you may not copy or use it, or
>>>> disclose it to anyone else. If you received it in error please notify us
>>>> immediately and then destroy it. Dynatrace Austria GmbH (registration
>>>> number FN 91482h) is a company registered in Linz whose registered office
>>>> is at 4040 Linz, Austria, Freistädterstraße 313
>>>>
>>>
Re: 3.11.2 memory leak
Posted by Abdul Patel <ab...@gmail.com>.
Any idea when 3.11.3 is coming in?
On Tuesday, June 19, 2018, kurt greaves <ku...@instaclustr.com> wrote:
> At this point I'd wait for 3.11.3. If you can't, you can get away with
> backporting a few repair fixes or just doing sub range repairs on 3.11.2
>
> On Wed., 20 Jun. 2018, 01:10 Abdul Patel, <ab...@gmail.com> wrote:
>
>> Hi All,
>>
>> Do we kmow whats the stable version for now if u wish to upgrade ?
>>
>> On Tuesday, June 5, 2018, Steinmaurer, Thomas <
>> thomas.steinmaurer@dynatrace.com> wrote:
>>
>>> Jeff,
>>>
>>>
>>>
>>> FWIW, when talking about https://issues.apache.org/
>>> jira/browse/CASSANDRA-13929, there is a patch available since March
>>> without getting further attention.
>>>
>>>
>>>
>>> Regards,
>>>
>>> Thomas
>>>
>>>
>>>
>>> *From:* Jeff Jirsa [mailto:jjirsa@gmail.com]
>>> *Sent:* Dienstag, 05. Juni 2018 00:51
>>> *To:* cassandra <us...@cassandra.apache.org>
>>> *Subject:* Re: 3.11.2 memory leak
>>>
>>>
>>>
>>> There have been a few people who have reported it, but nobody (yet) has
>>> offered a patch to fix it. It would be good to have a reliable way to
>>> repro, and/or an analysis of a heap dump demonstrating the problem (what's
>>> actually retained at the time you're OOM'ing).
>>>
>>>
>>>
>>> On Mon, Jun 4, 2018 at 6:52 AM, Abdul Patel <ab...@gmail.com> wrote:
>>>
>>> Hi All,
>>>
>>>
>>>
>>> I recently upgraded my non prod cluster from 3.10 to 3.11.2.
>>>
>>> It was working fine for a 1.5 weeks then suddenly nodetool info startee
>>> reporting 80% and more memory consumption.
>>>
>>> Intially it was 16gb configured, then i bumped to 20gb and rebooted all
>>> 4 nodes of cluster-single DC.
>>>
>>> Now after 8 days i again see 80% + usage and its 16gb and above ..which
>>> we never saw before .
>>>
>>> Seems like memory leak bug?
>>>
>>> Does anyone has any idea ? Our 3.11.2 release rollout has been halted
>>> because of this.
>>>
>>> If not 3.11.2 whats the next best stable release we have now?
>>>
>>>
>>> The contents of this e-mail are intended for the named addressee only.
>>> It contains information that may be confidential. Unless you are the named
>>> addressee or an authorized designee, you may not copy or use it, or
>>> disclose it to anyone else. If you received it in error please notify us
>>> immediately and then destroy it. Dynatrace Austria GmbH (registration
>>> number FN 91482h) is a company registered in Linz whose registered office
>>> is at 4040 Linz, Austria, Freistädterstraße 313
>>>
>>
Re: 3.11.2 memory leak
Posted by kurt greaves <ku...@instaclustr.com>.
At this point I'd wait for 3.11.3. If you can't, you can get away with
backporting a few repair fixes or just doing sub range repairs on 3.11.2
On Wed., 20 Jun. 2018, 01:10 Abdul Patel, <ab...@gmail.com> wrote:
> Hi All,
>
> Do we kmow whats the stable version for now if u wish to upgrade ?
>
> On Tuesday, June 5, 2018, Steinmaurer, Thomas <
> thomas.steinmaurer@dynatrace.com> wrote:
>
>> Jeff,
>>
>>
>>
>> FWIW, when talking about
>> https://issues.apache.org/jira/browse/CASSANDRA-13929, there is a patch
>> available since March without getting further attention.
>>
>>
>>
>> Regards,
>>
>> Thomas
>>
>>
>>
>> *From:* Jeff Jirsa [mailto:jjirsa@gmail.com]
>> *Sent:* Dienstag, 05. Juni 2018 00:51
>> *To:* cassandra <us...@cassandra.apache.org>
>> *Subject:* Re: 3.11.2 memory leak
>>
>>
>>
>> There have been a few people who have reported it, but nobody (yet) has
>> offered a patch to fix it. It would be good to have a reliable way to
>> repro, and/or an analysis of a heap dump demonstrating the problem (what's
>> actually retained at the time you're OOM'ing).
>>
>>
>>
>> On Mon, Jun 4, 2018 at 6:52 AM, Abdul Patel <ab...@gmail.com> wrote:
>>
>> Hi All,
>>
>>
>>
>> I recently upgraded my non prod cluster from 3.10 to 3.11.2.
>>
>> It was working fine for a 1.5 weeks then suddenly nodetool info startee
>> reporting 80% and more memory consumption.
>>
>> Intially it was 16gb configured, then i bumped to 20gb and rebooted all 4
>> nodes of cluster-single DC.
>>
>> Now after 8 days i again see 80% + usage and its 16gb and above ..which
>> we never saw before .
>>
>> Seems like memory leak bug?
>>
>> Does anyone has any idea ? Our 3.11.2 release rollout has been halted
>> because of this.
>>
>> If not 3.11.2 whats the next best stable release we have now?
>>
>>
>> The contents of this e-mail are intended for the named addressee only. It
>> contains information that may be confidential. Unless you are the named
>> addressee or an authorized designee, you may not copy or use it, or
>> disclose it to anyone else. If you received it in error please notify us
>> immediately and then destroy it. Dynatrace Austria GmbH (registration
>> number FN 91482h) is a company registered in Linz whose registered office
>> is at 4040 Linz, Austria, Freistädterstraße 313
>>
>
Re: 3.11.2 memory leak
Posted by Abdul Patel <ab...@gmail.com>.
Hi All,
Do we kmow whats the stable version for now if u wish to upgrade ?
On Tuesday, June 5, 2018, Steinmaurer, Thomas <
thomas.steinmaurer@dynatrace.com> wrote:
> Jeff,
>
>
>
> FWIW, when talking about https://issues.apache.org/
> jira/browse/CASSANDRA-13929, there is a patch available since March
> without getting further attention.
>
>
>
> Regards,
>
> Thomas
>
>
>
> *From:* Jeff Jirsa [mailto:jjirsa@gmail.com]
> *Sent:* Dienstag, 05. Juni 2018 00:51
> *To:* cassandra <us...@cassandra.apache.org>
> *Subject:* Re: 3.11.2 memory leak
>
>
>
> There have been a few people who have reported it, but nobody (yet) has
> offered a patch to fix it. It would be good to have a reliable way to
> repro, and/or an analysis of a heap dump demonstrating the problem (what's
> actually retained at the time you're OOM'ing).
>
>
>
> On Mon, Jun 4, 2018 at 6:52 AM, Abdul Patel <ab...@gmail.com> wrote:
>
> Hi All,
>
>
>
> I recently upgraded my non prod cluster from 3.10 to 3.11.2.
>
> It was working fine for a 1.5 weeks then suddenly nodetool info startee
> reporting 80% and more memory consumption.
>
> Intially it was 16gb configured, then i bumped to 20gb and rebooted all 4
> nodes of cluster-single DC.
>
> Now after 8 days i again see 80% + usage and its 16gb and above ..which we
> never saw before .
>
> Seems like memory leak bug?
>
> Does anyone has any idea ? Our 3.11.2 release rollout has been halted
> because of this.
>
> If not 3.11.2 whats the next best stable release we have now?
>
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
>
RE: 3.11.2 memory leak
Posted by "Steinmaurer, Thomas" <th...@dynatrace.com>.
Jeff,
FWIW, when talking about https://issues.apache.org/jira/browse/CASSANDRA-13929, there is a patch available since March without getting further attention.
Regards,
Thomas
From: Jeff Jirsa [mailto:jjirsa@gmail.com]
Sent: Dienstag, 05. Juni 2018 00:51
To: cassandra <us...@cassandra.apache.org>
Subject: Re: 3.11.2 memory leak
There have been a few people who have reported it, but nobody (yet) has offered a patch to fix it. It would be good to have a reliable way to repro, and/or an analysis of a heap dump demonstrating the problem (what's actually retained at the time you're OOM'ing).
On Mon, Jun 4, 2018 at 6:52 AM, Abdul Patel <ab...@gmail.com>> wrote:
Hi All,
I recently upgraded my non prod cluster from 3.10 to 3.11.2.
It was working fine for a 1.5 weeks then suddenly nodetool info startee reporting 80% and more memory consumption.
Intially it was 16gb configured, then i bumped to 20gb and rebooted all 4 nodes of cluster-single DC.
Now after 8 days i again see 80% + usage and its 16gb and above ..which we never saw before .
Seems like memory leak bug?
Does anyone has any idea ? Our 3.11.2 release rollout has been halted because of this.
If not 3.11.2 whats the next best stable release we have now?
The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria, Freistädterstraße 313
Re: 3.11.2 memory leak
Posted by Jeff Jirsa <jj...@gmail.com>.
There have been a few people who have reported it, but nobody (yet) has
offered a patch to fix it. It would be good to have a reliable way to
repro, and/or an analysis of a heap dump demonstrating the problem (what's
actually retained at the time you're OOM'ing).
On Mon, Jun 4, 2018 at 6:52 AM, Abdul Patel <ab...@gmail.com> wrote:
> Hi All,
>
> I recently upgraded my non prod cluster from 3.10 to 3.11.2.
> It was working fine for a 1.5 weeks then suddenly nodetool info startee
> reporting 80% and more memory consumption.
> Intially it was 16gb configured, then i bumped to 20gb and rebooted all 4
> nodes of cluster-single DC.
> Now after 8 days i again see 80% + usage and its 16gb and above ..which we
> never saw before .
> Seems like memory leak bug?
> Does anyone has any idea ? Our 3.11.2 release rollout has been halted
> because of this.
> If not 3.11.2 whats the next best stable release we have now?
>
Re: 3.11.2 memory leak
Posted by Elliott Sims <el...@backblaze.com>.
Are you seeing significant issues in terms of performance? Increased
garbage collection, long pauses, or even OutOfMemory? Which garbage
collector are you using and with what settings/thresholds? Since the JVM's
garbage-collected, a bigger heap can mean a problem or it can just mean
"hasn't gotten big enough for the collector to bother doing any work"
If it's genuinely having memory/heap pressure problems, it's probably worth
getting a heap dump and poking through it to see what's using the space.
For a heap that big, you'll probably need to run the Eclipse MAT CLI tools
against it then open the result in the GUI.
On Mon, Jun 4, 2018 at 6:52 AM, Abdul Patel <ab...@gmail.com> wrote:
> Hi All,
>
> I recently upgraded my non prod cluster from 3.10 to 3.11.2.
> It was working fine for a 1.5 weeks then suddenly nodetool info startee
> reporting 80% and more memory consumption.
> Intially it was 16gb configured, then i bumped to 20gb and rebooted all 4
> nodes of cluster-single DC.
> Now after 8 days i again see 80% + usage and its 16gb and above ..which we
> never saw before .
> Seems like memory leak bug?
> Does anyone has any idea ? Our 3.11.2 release rollout has been halted
> because of this.
> If not 3.11.2 whats the next best stable release we have now?
>