You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Phil Burress <ph...@gmail.com> on 2014/12/24 18:41:32 UTC
Nodes Dying in 2.1.2
Just upgraded our cluster from 2.1.1 to 2.1.2 and our nodes keep dying. The
kernel is killing the process due to out of memory:
kernel: Out of memory: Kill process 6267 (java) score 998 or sacrifice
child
Appears to only occur during compactions. We've tried playing with the heap
settings but nothing has worked thus far. We did not have this issue until
we upgraded. Anyone else run into this or have suggestions?
Thanks!
Phil
Re: Nodes Dying in 2.1.2
Posted by Phil Burress <ph...@gmail.com>.
Looks like someone else is experiencing almost exactly what we are seeing:
https://issues.apache.org/jira/browse/CASSANDRA-8552
On Mon, Dec 29, 2014 at 5:14 PM, Robert Coli <rc...@eventbrite.com> wrote:
> Might be https://issues.apache.org/jira/browse/CASSANDRA-8061 or one of
> the linked/duplicate tickets.
>
> =Rob
>
> On Mon, Dec 29, 2014 at 1:40 PM, Robert Coli <rc...@eventbrite.com> wrote:
>
>> On Wed, Dec 24, 2014 at 9:41 AM, Phil Burress <ph...@gmail.com>
>> wrote:
>>
>>> Just upgraded our cluster from 2.1.1 to 2.1.2 and our nodes keep dying.
>>> The kernel is killing the process due to out of memory:
>>>
>>
>> https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/
>>
>>
>> Appears to only occur during compactions. We've tried playing with the
>>> heap settings but nothing has worked thus far. We did not have this issue
>>> until we upgraded. Anyone else run into this or have suggestions?
>>>
>>
>> I would :
>>
>> 1) see if downgrade is possible (while unsupported, it probably is
>> possible) and downgrade if so
>> 2) search JIRA 2.1 era for related issues
>> 3) examine changes from 2.1.1 to 2.1.2 which relate to compaction
>> 4) file a JIRA describing yr experience if no prior one exists
>>
>> =Rob
>>
>
>
Re: Nodes Dying in 2.1.2
Posted by Robert Coli <rc...@eventbrite.com>.
Might be https://issues.apache.org/jira/browse/CASSANDRA-8061 or one of the
linked/duplicate tickets.
=Rob
On Mon, Dec 29, 2014 at 1:40 PM, Robert Coli <rc...@eventbrite.com> wrote:
> On Wed, Dec 24, 2014 at 9:41 AM, Phil Burress <ph...@gmail.com>
> wrote:
>
>> Just upgraded our cluster from 2.1.1 to 2.1.2 and our nodes keep dying.
>> The kernel is killing the process due to out of memory:
>>
>
> https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/
>
>
> Appears to only occur during compactions. We've tried playing with the
>> heap settings but nothing has worked thus far. We did not have this issue
>> until we upgraded. Anyone else run into this or have suggestions?
>>
>
> I would :
>
> 1) see if downgrade is possible (while unsupported, it probably is
> possible) and downgrade if so
> 2) search JIRA 2.1 era for related issues
> 3) examine changes from 2.1.1 to 2.1.2 which relate to compaction
> 4) file a JIRA describing yr experience if no prior one exists
>
> =Rob
>
Re: Nodes Dying in 2.1.2
Posted by Robert Coli <rc...@eventbrite.com>.
On Wed, Dec 24, 2014 at 9:41 AM, Phil Burress <ph...@gmail.com>
wrote:
> Just upgraded our cluster from 2.1.1 to 2.1.2 and our nodes keep dying.
> The kernel is killing the process due to out of memory:
>
https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/
Appears to only occur during compactions. We've tried playing with the heap
> settings but nothing has worked thus far. We did not have this issue until
> we upgraded. Anyone else run into this or have suggestions?
>
I would :
1) see if downgrade is possible (while unsupported, it probably is
possible) and downgrade if so
2) search JIRA 2.1 era for related issues
3) examine changes from 2.1.1 to 2.1.2 which relate to compaction
4) file a JIRA describing yr experience if no prior one exists
=Rob