Help with Cassandra Row Caches

[Oleg Tsvinev <ol...@gmail.com>]

Hi everybody,

We set row cache too high, 10000 or so and now all our 6 nodes fail
with OOM. I believe that high row cache causes OOMs.

Now, we trying to change row cache sizes using cassandra-cli update
column family command but Cassandra nodes keep dying.

Any help is appreciated, it's a production accident that lasts for too
long already.

Thank you,
  Oleg

Re: Help with Cassandra Row Caches

[Oleg Tsvinev <ol...@gmail.com>]

The problem was fixed by _decreasing_ heap size and applying changes
on the least loaded node.
Tyler, where saved caches are stored? /var/lib/cassandra/saved_caches?
Would it help relieving heap load?

On Fri, Nov 11, 2011 at 6:29 PM, Tyler Hobbs <ty...@datastax.com> wrote:
> You could also delete any saved caches, which might give you some more
> breathing room.
>
> On Fri, Nov 11, 2011 at 7:53 PM, Mohit Anchlia <mo...@gmail.com>
> wrote:
>>
>> Can you temporarily increase the size of Heap and try?
>>
>> On Fri, Nov 11, 2011 at 5:21 PM, Oleg Tsvinev <ol...@gmail.com>
>> wrote:
>> > Hi everybody,
>> >
>> > We set row cache too high, 10000 or so and now all our 6 nodes fail
>> > with OOM. I believe that high row cache causes OOMs.
>> >
>> > Now, we trying to change row cache sizes using cassandra-cli update
>> > column family command but Cassandra nodes keep dying.
>> >
>> > Any help is appreciated, it's a production accident that lasts for too
>> > long already.
>> >
>> > Thank you,
>> >  Oleg
>> >
>
>
>
> --
> Tyler Hobbs
> DataStax
>
>

Re: Help with Cassandra Row Caches

[Tyler Hobbs <ty...@datastax.com>]

You could also delete any saved caches, which might give you some more
breathing room.

On Fri, Nov 11, 2011 at 7:53 PM, Mohit Anchlia <mo...@gmail.com>wrote:

> Can you temporarily increase the size of Heap and try?
>
> On Fri, Nov 11, 2011 at 5:21 PM, Oleg Tsvinev <ol...@gmail.com>
> wrote:
> > Hi everybody,
> >
> > We set row cache too high, 10000 or so and now all our 6 nodes fail
> > with OOM. I believe that high row cache causes OOMs.
> >
> > Now, we trying to change row cache sizes using cassandra-cli update
> > column family command but Cassandra nodes keep dying.
> >
> > Any help is appreciated, it's a production accident that lasts for too
> > long already.
> >
> > Thank you,
> >  Oleg
> >
>



-- 
Tyler Hobbs
DataStax <http://datastax.com/>

Re: Help with Cassandra Row Caches

[Mohit Anchlia <mo...@gmail.com>]

Can you temporarily increase the size of Heap and try?

On Fri, Nov 11, 2011 at 5:21 PM, Oleg Tsvinev <ol...@gmail.com> wrote:
> Hi everybody,
>
> We set row cache too high, 10000 or so and now all our 6 nodes fail
> with OOM. I believe that high row cache causes OOMs.
>
> Now, we trying to change row cache sizes using cassandra-cli update
> column family command but Cassandra nodes keep dying.
>
> Any help is appreciated, it's a production accident that lasts for too
> long already.
>
> Thank you,
>  Oleg
>

Re: Help with Cassandra Row Caches

[Peter Schuller <pe...@infidyne.com>]

> We set row cache too high, 10000 or so and now all our 6 nodes fail
> with OOM. I believe that high row cache causes OOMs.

If you get into a fallen-and-cant-get-up scenario like this, I suggest
a rolling restart of nodes, and on each node:

(1) shut it down
(2) removed saved caches
(3) start it up
(4) (quickly) use nodetool to decrease the cache sizes *on that particular node*

Once that's done everywhere and everything is stable, a permanent and
cluster-wide change of cache sizes can be done by a schema change as
usual.

-- 
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)