You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Reynald Bourtembourg <re...@esrf.fr> on 2016/08/03 16:23:45 UTC
Re: [Marketing Mail] Re: Memory leak and lockup on our 2.2.7
Cassandra cluster.
Hi,
Maybe Ben was referring to this issue which has been mentioned recently
on this mailing list:
https://issues.apache.org/jira/browse/CASSANDRA-11887
Cheers,
Reynald
On 03/08/2016 18:09, Romain Hardouin wrote:
> >Curious why the 2.2 to 3.x upgrade path is risky at best.
> I guess that upgrade from 2.2 is less tested by DataStax QA because
> DSE4 used C* 2.1, not 2.2.
> I would say the safest upgrade is 2.1 to 3.0.x
>
> Best,
>
> Romain
>
Re: [Marketing Mail] Re: Memory leak and lockup on our 2.2.7
Cassandra cluster.
Posted by Jonathan Haddad <jo...@jonhaddad.com>.
In the future you may find SASI indexes useful for indexing Cassandra data.
Shameless blog post plug:
http://rustyrazorblade.com/2016/02/cassandra-secondary-index-preview-1/
Deep technical dive: http://www.doanduyhai.com/blog/?p=2058
On Thu, Aug 4, 2016 at 11:45 AM Kevin Burton <bu...@spinn3r.com> wrote:
> BTW. we think we tracked this down to using large partitions to implement
> inverted indexes. C* just doesn't do a reasonable job at all with large
> partitions so we're going to migrate this use case to using Elasticsearch
>
> On Wed, Aug 3, 2016 at 1:54 PM, Ben Slater <be...@instaclustr.com>
> wrote:
>
>> Yep, that was what I was referring to.
>>
>>
>> On Thu, 4 Aug 2016 2:24 am Reynald Bourtembourg <
>> reynald.bourtembourg@esrf.fr> wrote:
>>
>>> Hi,
>>>
>>> Maybe Ben was referring to this issue which has been mentioned recently
>>> on this mailing list:
>>> https://issues.apache.org/jira/browse/CASSANDRA-11887
>>>
>>> Cheers,
>>> Reynald
>>>
>>>
>>> On 03/08/2016 18:09, Romain Hardouin wrote:
>>>
>>> > Curious why the 2.2 to 3.x upgrade path is risky at best.
>>> I guess that upgrade from 2.2 is less tested by DataStax QA because DSE4
>>> used C* 2.1, not 2.2.
>>> I would say the safest upgrade is 2.1 to 3.0.x
>>>
>>> Best,
>>>
>>> Romain
>>>
>>>
>>> --
>> ————————
>> Ben Slater
>> Chief Product Officer
>> Instaclustr: Cassandra + Spark - Managed | Consulting | Support
>> +61 437 929 798
>>
>
>
>
> --
>
> We’re hiring if you know of any awesome Java Devops or Linux Operations
> Engineers!
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
>
>
Re: [Marketing Mail] Re: Memory leak and lockup on our 2.2.7
Cassandra cluster.
Posted by Kevin Burton <bu...@spinn3r.com>.
BTW. we think we tracked this down to using large partitions to implement
inverted indexes. C* just doesn't do a reasonable job at all with large
partitions so we're going to migrate this use case to using Elasticsearch
On Wed, Aug 3, 2016 at 1:54 PM, Ben Slater <be...@instaclustr.com>
wrote:
> Yep, that was what I was referring to.
>
>
> On Thu, 4 Aug 2016 2:24 am Reynald Bourtembourg <
> reynald.bourtembourg@esrf.fr> wrote:
>
>> Hi,
>>
>> Maybe Ben was referring to this issue which has been mentioned recently
>> on this mailing list:
>> https://issues.apache.org/jira/browse/CASSANDRA-11887
>>
>> Cheers,
>> Reynald
>>
>>
>> On 03/08/2016 18:09, Romain Hardouin wrote:
>>
>> > Curious why the 2.2 to 3.x upgrade path is risky at best.
>> I guess that upgrade from 2.2 is less tested by DataStax QA because DSE4
>> used C* 2.1, not 2.2.
>> I would say the safest upgrade is 2.1 to 3.0.x
>>
>> Best,
>>
>> Romain
>>
>>
>> --
> ————————
> Ben Slater
> Chief Product Officer
> Instaclustr: Cassandra + Spark - Managed | Consulting | Support
> +61 437 929 798
>
--
We’re hiring if you know of any awesome Java Devops or Linux Operations
Engineers!
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
Re: [Marketing Mail] Re: Memory leak and lockup on our 2.2.7
Cassandra cluster.
Posted by Ben Slater <be...@instaclustr.com>.
Yep, that was what I was referring to.
On Thu, 4 Aug 2016 2:24 am Reynald Bourtembourg <
reynald.bourtembourg@esrf.fr> wrote:
> Hi,
>
> Maybe Ben was referring to this issue which has been mentioned recently on
> this mailing list:
> https://issues.apache.org/jira/browse/CASSANDRA-11887
>
> Cheers,
> Reynald
>
>
> On 03/08/2016 18:09, Romain Hardouin wrote:
>
> > Curious why the 2.2 to 3.x upgrade path is risky at best.
> I guess that upgrade from 2.2 is less tested by DataStax QA because DSE4
> used C* 2.1, not 2.2.
> I would say the safest upgrade is 2.1 to 3.0.x
>
> Best,
>
> Romain
>
>
> --
————————
Ben Slater
Chief Product Officer
Instaclustr: Cassandra + Spark - Managed | Consulting | Support
+61 437 929 798
Re: [Marketing Mail] Re: Memory leak and lockup on our 2.2.7
Cassandra cluster.
Posted by Henrik Schröder <sk...@gmail.com>.
Have you tried using the G1 garbage collector instead of CMS?
We had the same issues that things were normally fine, but as soon as
something extraordinary happened, a node could go into GC hell and never
recover, and that could then spread to other nodes as they took up the
slack, trapping them in GC hell, and so on.
We did two things that helped us a lot; we switched to the G1GC, and we
switched to off-heap memtables. The second is pretty much a no-brainer, and
might even be default in 2.2.x, but do it if not. Switching to G1 needs to
be monitored closely, it has very different characteristics from CMS, but
it helped us in our case.
Both things are very easy to try out, it's just a config change and a node
restart, and if you have good monitoring you should be able to see how they
compare in the regular case and the extraordinary case.
/Henrik
On Wed, Aug 3, 2016 at 11:09 AM, Jonathan Haddad <jo...@jonhaddad.com> wrote:
> Kevin,
>
> "Our scheme uses large buckets of content where we write to a
> bucket/partition for 5 minutes, then move to a new one."
>
> Are you writing to a single partition and only that partition for 5
> minutes? If so, you should really rethink your data model. This method
> does not scale as you add nodes, it can only scale vertically.
>
> On Wed, Aug 3, 2016 at 9:24 AM Reynald Bourtembourg <
> reynald.bourtembourg@esrf.fr> wrote:
>
>> Hi,
>>
>> Maybe Ben was referring to this issue which has been mentioned recently
>> on this mailing list:
>> https://issues.apache.org/jira/browse/CASSANDRA-11887
>>
>> Cheers,
>> Reynald
>>
>>
>> On 03/08/2016 18:09, Romain Hardouin wrote:
>>
>> > Curious why the 2.2 to 3.x upgrade path is risky at best.
>> I guess that upgrade from 2.2 is less tested by DataStax QA because DSE4
>> used C* 2.1, not 2.2.
>> I would say the safest upgrade is 2.1 to 3.0.x
>>
>> Best,
>>
>> Romain
>>
>>
>>
Re: [Marketing Mail] Re: Memory leak and lockup on our 2.2.7
Cassandra cluster.
Posted by Kevin Burton <bu...@spinn3r.com>.
We usually use 100 per every 5 minutes.. but you're right. We might
actually move this use case over to using Elasticsearch in the next couple
of weeks.
On Wed, Aug 3, 2016 at 11:09 AM, Jonathan Haddad <jo...@jonhaddad.com> wrote:
> Kevin,
>
> "Our scheme uses large buckets of content where we write to a
> bucket/partition for 5 minutes, then move to a new one."
>
> Are you writing to a single partition and only that partition for 5
> minutes? If so, you should really rethink your data model. This method
> does not scale as you add nodes, it can only scale vertically.
>
> On Wed, Aug 3, 2016 at 9:24 AM Reynald Bourtembourg <
> reynald.bourtembourg@esrf.fr> wrote:
>
>> Hi,
>>
>> Maybe Ben was referring to this issue which has been mentioned recently
>> on this mailing list:
>> https://issues.apache.org/jira/browse/CASSANDRA-11887
>>
>> Cheers,
>> Reynald
>>
>>
>> On 03/08/2016 18:09, Romain Hardouin wrote:
>>
>> > Curious why the 2.2 to 3.x upgrade path is risky at best.
>> I guess that upgrade from 2.2 is less tested by DataStax QA because DSE4
>> used C* 2.1, not 2.2.
>> I would say the safest upgrade is 2.1 to 3.0.x
>>
>> Best,
>>
>> Romain
>>
>>
>>
--
We’re hiring if you know of any awesome Java Devops or Linux Operations
Engineers!
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
Re: [Marketing Mail] Re: Memory leak and lockup on our 2.2.7
Cassandra cluster.
Posted by Jonathan Haddad <jo...@jonhaddad.com>.
Kevin,
"Our scheme uses large buckets of content where we write to a
bucket/partition for 5 minutes, then move to a new one."
Are you writing to a single partition and only that partition for 5
minutes? If so, you should really rethink your data model. This method
does not scale as you add nodes, it can only scale vertically.
On Wed, Aug 3, 2016 at 9:24 AM Reynald Bourtembourg <
reynald.bourtembourg@esrf.fr> wrote:
> Hi,
>
> Maybe Ben was referring to this issue which has been mentioned recently on
> this mailing list:
> https://issues.apache.org/jira/browse/CASSANDRA-11887
>
> Cheers,
> Reynald
>
>
> On 03/08/2016 18:09, Romain Hardouin wrote:
>
> > Curious why the 2.2 to 3.x upgrade path is risky at best.
> I guess that upgrade from 2.2 is less tested by DataStax QA because DSE4
> used C* 2.1, not 2.2.
> I would say the safest upgrade is 2.1 to 3.0.x
>
> Best,
>
> Romain
>
>
>