You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Stefan Reek <st...@unitedgames.com> on 2011/10/06 17:09:15 UTC
Batch writes getting slow
Hi guys,
We're currently testing an application against a very high load, which
runs against Cassandra 0.6.13 (I know, we just never got the time to
upgrade).
The nature of our app is that it will write to two different
SuperColumnFamilies in bursts, and to some other columnfamilies less
frequently.
About every 10 mins a very large number of subcolumns will be written
to those superColumnFamilies (> 1 million). The writes are mainly
overwrites, there are no deletes.
when I do a cfstats on the SuperColumnFamilies I see something like this:
SSTable count: 4
Space used (live): 1611778835
Space used (total): 1611778835
Memtable Columns Count: 82844
Memtable Data Size: 4439246
Memtable Switch Count: 5262
Read Count: 4633810
Read Latency: 3.727 ms.
Write Count: 29739347
Write Latency: 0.028 ms.
Pending Tasks: 0
Which tells me the Write Latency is not extremely high, but I can see
in my logs that some batch writes, which contain about 1000 entries,
take over 2 seconds to execute.
I can see that during the times the writing gets slow there are ~3000
pending tasks, but they disappear quickly.
Is there anything I can do to tune Cassandra so the writes won't take
that long? Maybe increasing the MemtableOperationsInMillions so the
memtables get flushed less often?
Or is the load just too high and should I solve this by adding more
nodes? We are currently testing with 3, replication factor 2.
I can also see that Cassandra gradually takes more and more memory,
eventually filling up the 16GB that is assigned to it, although it
doesn't go out of memory.
Is this normal behaviour? I expected to see more of a sawtooth...
All help is appreciated.
Thanks,
Stefan
Re: Batch writes getting slow
Posted by Jonathan Ellis <jb...@gmail.com>.
Look at the OOM section in
http://www.datastax.com/docs/0.6/troubleshooting/index
On Fri, Oct 7, 2011 at 3:47 AM, Stefan Reek <st...@unitedgames.com> wrote:
> Is it actually filling up enough to trigger an old-gen CMS gc?
>
>
> Yes, it fills up to the 16G and then it starts doing the CMS gc's which
> dramatically decreases the performance.
> I'm still not sure why it does this, as a nodetool info states the load as
> less than 4G.
> Any ideas?
>
>
> On 10/06/2011 06:15 PM, Jonathan Ellis wrote:
>>
>> On Thu, Oct 6, 2011 at 10:53 AM, Stefan Reek<st...@unitedgames.com>
>> wrote:
>>
>>>
>>> We do have the commitlogs on separate devices, are there any other basics
>>> that I could have forgotten, or
>>> any parameters that are important for write performance?
>>>
>>
>> 1.0 write performance is something like 30% better... I don't think
>> there's anything else you'll find for "free."
>>
>>
>>>
>>> As I understand it
>>> the flush thresholds mainly
>>> influence read performance instead of write performance.
>>>
>>
>> It can affect write performance if you're flushing really small
>> sstables, but I doubt that's the problem here.
>>
>>
>>>
>>> Would it make any difference to write the data with more threads from the
>>> client, as that's something we can easily tune.
>>>
>>
>> Not in this case because Cassandra turns the batch into single-row
>> writes internally, so it gets parallelized that way.
>>
>> If you can avoid waiting for One Big Batch and stream changes in as
>> they happen, that would help.
>>
>>
>>>
>>> I can see the sawtooth in the JVM only for Par Eden and Par Survivor
>>> space,
>>> the CMS Old Gen space just keeps on growing though.
>>>
>>
>> Is it actually filling up enough to trigger an old-gen CMS gc?
>>
>>
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: Batch writes getting slow
Posted by Stefan Reek <st...@unitedgames.com>.
Is it actually filling up enough to trigger an old-gen CMS gc?
Yes, it fills up to the 16G and then it starts doing the CMS gc's which
dramatically decreases the performance.
I'm still not sure why it does this, as a nodetool info states the load
as less than 4G.
Any ideas?
On 10/06/2011 06:15 PM, Jonathan Ellis wrote:
> On Thu, Oct 6, 2011 at 10:53 AM, Stefan Reek<st...@unitedgames.com> wrote:
>
>> We do have the commitlogs on separate devices, are there any other basics
>> that I could have forgotten, or
>> any parameters that are important for write performance?
>>
> 1.0 write performance is something like 30% better... I don't think
> there's anything else you'll find for "free."
>
>
>> As I understand it
>> the flush thresholds mainly
>> influence read performance instead of write performance.
>>
> It can affect write performance if you're flushing really small
> sstables, but I doubt that's the problem here.
>
>
>> Would it make any difference to write the data with more threads from the
>> client, as that's something we can easily tune.
>>
> Not in this case because Cassandra turns the batch into single-row
> writes internally, so it gets parallelized that way.
>
> If you can avoid waiting for One Big Batch and stream changes in as
> they happen, that would help.
>
>
>> I can see the sawtooth in the JVM only for Par Eden and Par Survivor space,
>> the CMS Old Gen space just keeps on growing though.
>>
> Is it actually filling up enough to trigger an old-gen CMS gc?
>
>
Re: Batch writes getting slow
Posted by Jonathan Ellis <jb...@gmail.com>.
On Thu, Oct 6, 2011 at 10:53 AM, Stefan Reek <st...@unitedgames.com> wrote:
> We do have the commitlogs on separate devices, are there any other basics
> that I could have forgotten, or
> any parameters that are important for write performance?
1.0 write performance is something like 30% better... I don't think
there's anything else you'll find for "free."
> As I understand it
> the flush thresholds mainly
> influence read performance instead of write performance.
It can affect write performance if you're flushing really small
sstables, but I doubt that's the problem here.
> Would it make any difference to write the data with more threads from the
> client, as that's something we can easily tune.
Not in this case because Cassandra turns the batch into single-row
writes internally, so it gets parallelized that way.
If you can avoid waiting for One Big Batch and stream changes in as
they happen, that would help.
> I can see the sawtooth in the JVM only for Par Eden and Par Survivor space,
> the CMS Old Gen space just keeps on growing though.
Is it actually filling up enough to trigger an old-gen CMS gc?
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: Batch writes getting slow
Posted by Stefan Reek <st...@unitedgames.com>.
On 10/06/2011 05:26 PM, Jonathan Ellis wrote:
> On Thu, Oct 6, 2011 at 10:09 AM, Stefan Reek<st...@unitedgames.com> wrote:
>
>> I can see that during the times the writing gets slow there are ~3000
>> pending tasks, but they disappear quickly.
>>
> Your best bet is to make the write load more constant and less bursty.
> If you really do need to handle bursts like that with low latency,
> then you probably do need more hardware. (But make sure you've
> covered the basics first, like putting commitlog on a separate
> device.)
>
>
We really need to make sure that all writes are succesfully written
before the next batch has to be written,
so the bursts are unavoidable I think.
We do have the commitlogs on separate devices, are there any other
basics that I could have forgotten, or
any parameters that are important for write performance? As I understand
it the flush thresholds mainly
influence read performance instead of write performance.
Would it make any difference to write the data with more threads from
the client, as that's something we can easily tune.
>> I can also see that Cassandra gradually takes more and more memory,
>> eventually filling up the 16GB that is assigned to it, although it
>> doesn't go out of memory.
>> Is this normal behaviour? I expected to see more of a sawtooth...
>>
> Yes, that is normal from the OS's view of the JVM. If you want to see
> the sawtooth you'd need to look at the JVM's internal metrics, e.g.,
> with jconsole.
>
>
I can see the sawtooth in the JVM only for Par Eden and Par Survivor
space, the CMS Old Gen space just keeps on growing though.
Anyway, thanks for the quick reply.
Regards,
Stefan
Re: Batch writes getting slow
Posted by Jonathan Ellis <jb...@gmail.com>.
On Thu, Oct 6, 2011 at 10:09 AM, Stefan Reek <st...@unitedgames.com> wrote:
> I can see that during the times the writing gets slow there are ~3000
> pending tasks, but they disappear quickly.
Your best bet is to make the write load more constant and less bursty.
If you really do need to handle bursts like that with low latency,
then you probably do need more hardware. (But make sure you've
covered the basics first, like putting commitlog on a separate
device.)
> I can also see that Cassandra gradually takes more and more memory,
> eventually filling up the 16GB that is assigned to it, although it
> doesn't go out of memory.
> Is this normal behaviour? I expected to see more of a sawtooth...
Yes, that is normal from the OS's view of the JVM. If you want to see
the sawtooth you'd need to look at the JVM's internal metrics, e.g.,
with jconsole.
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com