You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Joseph Lynch (JIRA)" <ji...@apache.org> on 2018/09/12 23:32:00 UTC
[jira] [Comment Edited] (CASSANDRA-14747) Evaluate 200 node,
compression=none, encryption=none, coalescing=off
[ https://issues.apache.org/jira/browse/CASSANDRA-14747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16612856#comment-16612856 ]
Joseph Lynch edited comment on CASSANDRA-14747 at 9/12/18 11:31 PM:
--------------------------------------------------------------------
*Setup:*
* Cassandra: 192 (2*96) node i3.xlarge AWS instance (4 cpu cores, 30GB ram) running cassandra trunk f25a765b.
* Two datacenters with 100ms latency between them
* No compression, encryption, or coalescing turned on
*Test #1:*
ndbench sent 30k QPS at a coordinator level to one datacenter (RF=3*2 = 6 so 180k global replica QPS) of 40kb and then 4kb single partition BATCH mutations. This represents about 300 QPS per coordinator in the first datacenter or 75 per core.
*Result:*
We quickly overwhelmed the 4.0 cluster which started having high latencies and throwing errors while the 3.0 cluster remained healthy. 4.0 nodes were running out of heap within minutes during the 40kb test and a few minutes with the 4kb test. I've attached flamegraphs showing 4.0 spending half its time garbage collecting and logs indicating large on heap usage.
On the bright side the thread count was _way down:_ the 3.0 cluster had 1.2k threads and the 4.0 cluster only had 220 threads (and almost all of that reduction was the messaging thread reduction). Also the startup time was super fast (as in less than one second to handshake the entire cluster, vs 3.0 which took minutes. We didn't feel that proceeding with the test made sense given the instability until follow ups could be committed. We used heap dumps and {{jmap}} to determine the issue was the outgoing message queue retaining large numbers of mutations on heap rather than dropping them.
*Follow Ups:*
The outgoing queue holding mutations on heap appears to be the problem. Specifically the 3.x code would police the internode queues and ensure they did not get too large at enqueue and dequeue time (expiring messages and turning them into hints as needed), the 4.0 code took out the enqueue policing complexity in the hope that we wouldn't need it. It appears it is necessary. [~jasobrown] is including fixes to the queue policing in CASSANDRA-14503 and CASSANDRA-13630 and we will re-execute this test once those are merged to ensure that they fix the issue with large volume mutations.
was (Author: jolynch):
*Setup:*
* Cassandra: 192 (2*96) node i3.xlarge AWS instance (4 cpu cores, 30GB ram) running cassandra trunk f25a765b.
* Two datacenters with 100ms latency between them
* No compression, encryption, or coalescing turned on
*Test:*
ndbench sent 30k QPS at a coordinator level to one datacenter (RF=3*2 = 6 so 180k global replica QPS) of 40kb and then 4kb single partition BATCH mutations. This represents about 300 QPS per coordinator in the first datacenter or 75 per core.
*Result:*
We quickly overwhelmed the 4.0 cluster which started having high latencies and throwing errors while the 3.0 cluster remained healthy. 4.0 nodes were running out of heap within minutes during the 40kb test and a few minutes with the 4kb test. I've attached flamegraphs showing 4.0 spending half its time garbage collecting and logs indicating large on heap usage.
On the bright side the thread count was _way down:_ the 3.0 cluster had 1.2k threads and the 4.0 cluster only had 220 threads (and almost all of that reduction was the messaging thread reduction). Also the startup time was super fast (as in less than one second to handshake the entire cluster, vs 3.0 which took minutes. We didn't feel that proceeding with the test made sense given the instability until follow ups could be committed. We used heap dumps and {{jmap}} to determine the issue was the outgoing message queue retaining large numbers of mutations on heap rather than dropping them.
*Follow Ups:*
The outgoing queue holding mutations on heap appears to be the problem. Specifically the 3.x code would police the internode queues and ensure they did not get too large at enqueue and dequeue time (expiring messages and turning them into hints as needed), the 4.0 code took out the enqueue policing complexity in the hope that we wouldn't need it. It appears it is necessary. [~jasobrown] is including fixes to the queue policing in CASSANDRA-14503 and CASSANDRA-13630 and we will re-execute this test once those are merged to ensure that they fix the issue with large volume mutations.
> Evaluate 200 node, compression=none, encryption=none, coalescing=off
> ---------------------------------------------------------------------
>
> Key: CASSANDRA-14747
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14747
> Project: Cassandra
> Issue Type: Sub-task
> Reporter: Joseph Lynch
> Priority: Major
>
> Tracks evaluating a 200 node cluster with all internode settings off (no compression, no encryption, no coalescing).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org