Re: [Marketing Mail] Re: Memory leak and lockup on our 2.2.7 Cassandra cluster.

[Jonathan Haddad <jo...@jonhaddad.com>]

Kevin,

"Our scheme uses large buckets of content where we write to a
bucket/partition for 5 minutes, then move to a new one."

Are you writing to a single partition and only that partition for 5
minutes?  If so, you should really rethink your data model.  This method
does not scale as you add nodes, it can only scale vertically.

On Wed, Aug 3, 2016 at 9:24 AM Reynald Bourtembourg <
reynald.bourtembourg@esrf.fr> wrote:

> Hi,
>
> Maybe Ben was referring to this issue which has been mentioned recently on
> this mailing list:
> https://issues.apache.org/jira/browse/CASSANDRA-11887
>
> Cheers,
> Reynald
>
>
> On 03/08/2016 18:09, Romain Hardouin wrote:
>
> > Curious why the 2.2 to 3.x upgrade path is risky at best.
> I guess that upgrade from 2.2 is less tested by DataStax QA because DSE4
> used C* 2.1, not 2.2.
> I would say the safest upgrade is 2.1 to 3.0.x
>
> Best,
>
> Romain
>
>
>

Re: [Marketing Mail] Re: Memory leak and lockup on our 2.2.7 Cassandra cluster.

[Henrik Schröder <sk...@gmail.com>]

Have you tried using the G1 garbage collector instead of CMS?

We had the same issues that things were normally fine, but as soon as
something extraordinary happened, a node could go into GC hell and never
recover, and that could then spread to other nodes as they took up the
slack, trapping them in GC hell, and so on.

We did two things that helped us a lot; we switched to the G1GC, and we
switched to off-heap memtables. The second is pretty much a no-brainer, and
might even be default in 2.2.x, but do it if not. Switching to G1 needs to
be monitored closely, it has very different characteristics from CMS, but
it helped us in our case.

Both things are very easy to try out, it's just a config change and a node
restart, and if you have good monitoring you should be able to see how they
compare in the regular case and the extraordinary case.


/Henrik

On Wed, Aug 3, 2016 at 11:09 AM, Jonathan Haddad <jo...@jonhaddad.com> wrote:

> Kevin,
>
> "Our scheme uses large buckets of content where we write to a
> bucket/partition for 5 minutes, then move to a new one."
>
> Are you writing to a single partition and only that partition for 5
> minutes?  If so, you should really rethink your data model.  This method
> does not scale as you add nodes, it can only scale vertically.
>
> On Wed, Aug 3, 2016 at 9:24 AM Reynald Bourtembourg <
> reynald.bourtembourg@esrf.fr> wrote:
>
>> Hi,
>>
>> Maybe Ben was referring to this issue which has been mentioned recently
>> on this mailing list:
>> https://issues.apache.org/jira/browse/CASSANDRA-11887
>>
>> Cheers,
>> Reynald
>>
>>
>> On 03/08/2016 18:09, Romain Hardouin wrote:
>>
>> > Curious why the 2.2 to 3.x upgrade path is risky at best.
>> I guess that upgrade from 2.2 is less tested by DataStax QA because DSE4
>> used C* 2.1, not 2.2.
>> I would say the safest upgrade is 2.1 to 3.0.x
>>
>> Best,
>>
>> Romain
>>
>>
>>

Re: [Marketing Mail] Re: Memory leak and lockup on our 2.2.7 Cassandra cluster.

[Kevin Burton <bu...@spinn3r.com>]

We usually use 100 per every 5 minutes.. but you're right.  We might
actually move this use case over to using Elasticsearch in the next couple
of weeks.

On Wed, Aug 3, 2016 at 11:09 AM, Jonathan Haddad <jo...@jonhaddad.com> wrote:

> Kevin,
>
> "Our scheme uses large buckets of content where we write to a
> bucket/partition for 5 minutes, then move to a new one."
>
> Are you writing to a single partition and only that partition for 5
> minutes?  If so, you should really rethink your data model.  This method
> does not scale as you add nodes, it can only scale vertically.
>
> On Wed, Aug 3, 2016 at 9:24 AM Reynald Bourtembourg <
> reynald.bourtembourg@esrf.fr> wrote:
>
>> Hi,
>>
>> Maybe Ben was referring to this issue which has been mentioned recently
>> on this mailing list:
>> https://issues.apache.org/jira/browse/CASSANDRA-11887
>>
>> Cheers,
>> Reynald
>>
>>
>> On 03/08/2016 18:09, Romain Hardouin wrote:
>>
>> > Curious why the 2.2 to 3.x upgrade path is risky at best.
>> I guess that upgrade from 2.2 is less tested by DataStax QA because DSE4
>> used C* 2.1, not 2.2.
>> I would say the safest upgrade is 2.1 to 3.0.x
>>
>> Best,
>>
>> Romain
>>
>>
>>


-- 

We’re hiring if you know of any awesome Java Devops or Linux Operations
Engineers!

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>