Cassandra high heap utilization under heavy reads and writes.

[srmore <co...@gmail.com>]

Hello,
We moved to cassandra 1.2.9 from 1.0.11 to take advantage of the off-heap
bloom filters and other improvements.

We see a lot of messages dropped under high load conditions. We noticed
that when we do heavy read AND write simultaneously (we read first and
check whether the key exists if not we write it) Cassandra heap increases
dramatically and then gossip marks the node down (as a result of high load
on the node).


Under heavy 'reads only' we don't see this behavior.  Has anyone seen this
behavior ? any suggestions.

Thanks !

Re: Cassandra high heap utilization under heavy reads and writes.

[srmore <co...@gmail.com>]

So, I did a lot of dial turning and heap tuning (came across this nice
writeup about JVM tuning
http://blog.mikiobraun.de/2010/08/cassandra-gc-tuning.html) still no luck
with 1.2.9. I gave up and upgraded to 1.2.12 and since then things are much
much better. I don't run into the heap issue that I used to with 1.2.9.

1.2.12 also looks more stable and seems to work  for us.

-sandeep


On Mon, Nov 25, 2013 at 11:10 AM, Christopher J. Bottaro <
cjbottaro@academicworks.com> wrote:

> Yes, we saw this same behavior.
>
> A couple of months ago, we moved a large portion of our data out of
> Postgres and into Cassandra.  The initial migration was done in a
> "distributed" manner:  we had 600 (or 800, can't remember) processes
> reading from Postgres and writing to Cassandra in tight loops.  This caused
> the exact behavior you described.  We also did a read before a write.
>
> After we got through the initial data migration, our normal workload is
> *much* less writes (and reads for that matter) such that our cluster can
> easily handle it, so we didn't investigate further.
>
> -- C
>
>
> On Sat, Nov 23, 2013 at 10:55 PM, srmore <co...@gmail.com> wrote:
>
>> Hello,
>> We moved to cassandra 1.2.9 from 1.0.11 to take advantage of the off-heap
>> bloom filters and other improvements.
>>
>> We see a lot of messages dropped under high load conditions. We noticed
>> that when we do heavy read AND write simultaneously (we read first and
>> check whether the key exists if not we write it) Cassandra heap increases
>> dramatically and then gossip marks the node down (as a result of high load
>> on the node).
>>
>>
>> Under heavy 'reads only' we don't see this behavior.  Has anyone seen
>> this behavior ? any suggestions.
>>
>> Thanks !
>>
>>
>>
>

Re: Cassandra high heap utilization under heavy reads and writes.

["Christopher J. Bottaro" <cj...@academicworks.com>]

Yes, we saw this same behavior.

A couple of months ago, we moved a large portion of our data out of
Postgres and into Cassandra.  The initial migration was done in a
"distributed" manner:  we had 600 (or 800, can't remember) processes
reading from Postgres and writing to Cassandra in tight loops.  This caused
the exact behavior you described.  We also did a read before a write.

After we got through the initial data migration, our normal workload is
*much* less writes (and reads for that matter) such that our cluster can
easily handle it, so we didn't investigate further.

-- C


On Sat, Nov 23, 2013 at 10:55 PM, srmore <co...@gmail.com> wrote:

> Hello,
> We moved to cassandra 1.2.9 from 1.0.11 to take advantage of the off-heap
> bloom filters and other improvements.
>
> We see a lot of messages dropped under high load conditions. We noticed
> that when we do heavy read AND write simultaneously (we read first and
> check whether the key exists if not we write it) Cassandra heap increases
> dramatically and then gossip marks the node down (as a result of high load
> on the node).
>
>
> Under heavy 'reads only' we don't see this behavior.  Has anyone seen this
> behavior ? any suggestions.
>
> Thanks !
>
>
>