You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by onmstester onmstester <on...@zoho.com.INVALID> on 2020/06/02 06:30:40 UTC
Cassandra crashes when using offheap_objects for
memtable_allocation_type
I just changed these properties to increase flushed file size (decrease number of compactions):
memtable_allocation_type from heap_buffers to offheap_objects
memtable_offheap_space_in_mb: from default (2048) to 8192
Using default value for other memtable/compaction/commitlog configurations .
After a few hours some of nodes stopped to do any mutations (dropped mutaion increased) and also pending flushes increased, they were just up and running and there was only a single CPU core with 100% usage(other cores was 0%). other nodes on the cluster determines the node as DN. Could not access 7199 and also could not create thread dump even with jstack -F.
Restarting Cassandra service fixes the problem but after a while some other node would be DN.
Am i missing some configurations? What should i change in cassandra default configuration to maximize write throughput in single node/cluster in write-heavy scenario for the data model:
Data mode is a single table:
create table test(
text partition_key,
text clustering_key,
set<text> rows,
primary key ((partition_key, clustering_key))
vCPU: 12
Memory: 32GB
Node data size: 2TB
Apache cassandra 3.11.2
JVM heap size: 16GB, CMS, 1GB newgen
Sent using https://www.zoho.com/mail/
Re: Cassandra crashes when using offheap_objects for memtable_allocation_type
Posted by Aaron Ploetz <aa...@gmail.com>.
primary key ((partition_key, clustering_key))
Also, this primary key definition does not define a partitioning key and a
clustering key. It defines a *composite* partition key.
If you want it to instantiate both a partition and clustering key, get rid
of one set of parens.
primary key (partition_key, clustering_key)
On Tue, Jun 2, 2020 at 1:31 AM onmstester onmstester
<on...@zoho.com.invalid> wrote:
> I just changed these properties to increase flushed file size (decrease
> number of compactions):
>
> - memtable_allocation_type from heap_buffers to offheap_objects
> - memtable_offheap_space_in_mb: from default (2048) to 8192
>
> Using default value for other memtable/compaction/commitlog configurations
> .
>
> After a few hours some of nodes stopped to do any mutations (dropped
> mutaion increased) and also pending flushes increased, they were just up
> and running and there was only a single CPU core with 100% usage(other
> cores was 0%). other nodes on the cluster determines the node as DN. Could
> not access 7199 and also could not create thread dump even with jstack -F.
>
> Restarting Cassandra service fixes the problem but after a while some
> other node would be DN.
>
> Am i missing some configurations? What should i change in cassandra
> default configuration to maximize write throughput in single node/cluster
> in write-heavy scenario for the data model:
> Data mode is a single table:
> create table test(
> text partition_key,
> text clustering_key,
> set<text> rows,
> primary key ((partition_key, clustering_key))
>
>
> vCPU: 12
> Memory: 32GB
> Node data size: 2TB
> Apache cassandra 3.11.2
> JVM heap size: 16GB, CMS, 1GB newgen
>
> Sent using Zoho Mail <https://www.zoho.com/mail/>
>
>
>
>
Re: Cassandra crashes when using offheap_objects for
memtable_allocation_type
Posted by Reid Pinchback <rp...@tripadvisor.com>.
I’d also take a look at the O/S level. You might be queued up on flushing of dirty pages, which would also throttle your ability to write mempages. Once the I/O gets throttled badly, I’ve seen it push back into what you see in C*. To Aaron’s point, you want a balance in memory between C* and O/S buffer cache, because to write to disk you pass through buffer cache first.
From: Aaron Ploetz <aa...@gmail.com>
Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
Date: Tuesday, June 2, 2020 at 9:38 AM
To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
Subject: Re: Cassandra crashes when using offheap_objects for memtable_allocation_type
Message from External Sender
I would try running it with memtable_offheap_space_in_mb at the default for sure, but definitely lower than 8GB. With 32GB of RAM, you're already allocating half of that for your heap, and then halving the remainder for off heap memtables. What's left may not be enough for the OS, etc. Giving some of that back, will allow more to be used for page cache, which always helps.
"JVM heap size: 16GB, CMS, 1GB newgen"
For CMS GC with a 16GB heap, 1GB is way too small for new gen. You're going to want that to be at least 40% of the max heap size. Some folks here even advocate for setting Xmn as high as 50% of Xmx/s.
If you want to stick with CMS GC, take a look at https://issues.apache.org/jira/browse/CASSANDRA-8150<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_CASSANDRA-2D8150&d=DwMFaQ&c=9Hv6XPedRSA-5PSECC38X80c1h60_XWA4z1k_R1pROA&r=OIgB3poYhzp3_A7WgD7iBCnsJaYmspOa2okNpf6uqWc&m=eX5wEX3VXggU-3Y51C3hDF0XBO9xUZQZd12jW_Da8Qw&s=M9v_2_0-S3OhfynpBRwgk6dnPh_pJiNC9Fps-9Qu2S8&e=>. There's plenty of good info in there on CMS GC tuning. Make sure to read through the whole ticket, so that you understand what each setting does. You can't just pick-and-choose.
Regards,
Aaron
On Tue, Jun 2, 2020 at 1:31 AM onmstester onmstester <on...@zoho.com.invalid> wrote:
I just changed these properties to increase flushed file size (decrease number of compactions):
* memtable_allocation_type from heap_buffers to offheap_objects
* memtable_offheap_space_in_mb: from default (2048) to 8192
Using default value for other memtable/compaction/commitlog configurations .
After a few hours some of nodes stopped to do any mutations (dropped mutaion increased) and also pending flushes increased, they were just up and running and there was only a single CPU core with 100% usage(other cores was 0%). other nodes on the cluster determines the node as DN. Could not access 7199 and also could not create thread dump even with jstack -F.
Restarting Cassandra service fixes the problem but after a while some other node would be DN.
Am i missing some configurations? What should i change in cassandra default configuration to maximize write throughput in single node/cluster in write-heavy scenario for the data model:
Data mode is a single table:
create table test(
text partition_key,
text clustering_key,
set<text> rows,
primary key ((partition_key, clustering_key))
vCPU: 12
Memory: 32GB
Node data size: 2TB
Apache cassandra 3.11.2
JVM heap size: 16GB, CMS, 1GB newgen
Sent using Zoho Mail<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.zoho.com_mail_&d=DwMFaQ&c=9Hv6XPedRSA-5PSECC38X80c1h60_XWA4z1k_R1pROA&r=OIgB3poYhzp3_A7WgD7iBCnsJaYmspOa2okNpf6uqWc&m=eX5wEX3VXggU-3Y51C3hDF0XBO9xUZQZd12jW_Da8Qw&s=OpD4crlAdz_RSLV3zi8jcRzgpKzGjKn2up0lzd8mDlE&e=>
Re: Cassandra crashes when using offheap_objects for memtable_allocation_type
Posted by Aaron Ploetz <aa...@gmail.com>.
I would try running it with memtable_offheap_space_in_mb at the default for
sure, but definitely lower than 8GB. With 32GB of RAM, you're already
allocating half of that for your heap, and then halving the remainder for
off heap memtables. What's left may not be enough for the OS, etc. Giving
some of that back, will allow more to be used for page cache, which always
helps.
"JVM heap size: 16GB, CMS, 1GB newgen"
For CMS GC with a 16GB heap, 1GB is way too small for new gen. You're
going to want that to be at least 40% of the max heap size. Some folks
here even advocate for setting Xmn as high as 50% of Xmx/s.
If you want to stick with CMS GC, take a look at
https://issues.apache.org/jira/browse/CASSANDRA-8150. There's plenty of
good info in there on CMS GC tuning. Make sure to read through the whole
ticket, so that you understand what each setting does. You can't just
pick-and-choose.
Regards,
Aaron
On Tue, Jun 2, 2020 at 1:31 AM onmstester onmstester
<on...@zoho.com.invalid> wrote:
> I just changed these properties to increase flushed file size (decrease
> number of compactions):
>
> - memtable_allocation_type from heap_buffers to offheap_objects
> - memtable_offheap_space_in_mb: from default (2048) to 8192
>
> Using default value for other memtable/compaction/commitlog configurations
> .
>
> After a few hours some of nodes stopped to do any mutations (dropped
> mutaion increased) and also pending flushes increased, they were just up
> and running and there was only a single CPU core with 100% usage(other
> cores was 0%). other nodes on the cluster determines the node as DN. Could
> not access 7199 and also could not create thread dump even with jstack -F.
>
> Restarting Cassandra service fixes the problem but after a while some
> other node would be DN.
>
> Am i missing some configurations? What should i change in cassandra
> default configuration to maximize write throughput in single node/cluster
> in write-heavy scenario for the data model:
> Data mode is a single table:
> create table test(
> text partition_key,
> text clustering_key,
> set<text> rows,
> primary key ((partition_key, clustering_key))
>
>
> vCPU: 12
> Memory: 32GB
> Node data size: 2TB
> Apache cassandra 3.11.2
> JVM heap size: 16GB, CMS, 1GB newgen
>
> Sent using Zoho Mail <https://www.zoho.com/mail/>
>
>
>
>