[jira] [Comment Edited] (CASSANDRA-6357) Flush memtables to separate directory

["dan jatnieks (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-6357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13881433#comment-13881433 ] 

dan jatnieks edited comment on CASSANDRA-6357 at 1/24/14 9:20 PM:
------------------------------------------------------------------

I did some stress write testing on a single node physical machine with multiple rotating disks. The machine was a quad-core Intel Xeon E3-1230, 32Gb memory, and 4 disks (2Tb, 5400 rpm).

Two scenarios were measured using stress-write for 10 million records (cassandra-stress --num-keys 10000000 --columns=50 --operation=INSERT):

Base scenario: 
* O/S on separate device
* Commitlog on separate device
* Data and flush directories on the same device

Flush scenario: 
* O/S on separate device
* Commitlog on separate device
* Data directory on separate device
* Flush directory on separate device

The result of splitting the flush directory to another device on the stress-write results was:
* an elapsed time improvement of about 57%
* a much lower, and consistent, 99.9th percentile latency (3308 ms vs 15963 ms)

See attached graph: [Stress Write Latency 99.9th Percentile|^c6357-stress-write-latency-99th-1.png]



was (Author: djatnieks):
I did some stress write testing on a single node physical machine with multiple rotating disks. The machine was a quad-code Intel Xeon E3-1230, 32Gb memory, and 4 disks (2Tb, 5400 rpm).

Two scenarios were measured using stress-write for 10 million records (cassandra-stress --num-keys 10000000 --columns=50 --operation=INSERT):

Base scenario: 
* O/S on separate device
* Commitlog on separate device
* Data and flush directories on the same device

Flush scenario: 
* O/S on separate device
* Commitlog on separate device
* Data directory on separate device
* Flush directory on separate device

The result of splitting the flush directory to another device on the stress-write results was:
* an elapsed time improvement of about 57%
* a much lower, and consistent, 99.9th percentile latency (3308 ms vs 15963 ms)

See attached graph: [Stress Write Latency 99.9th Percentile|^c6357-stress-write-latency-99th-1.png]


> Flush memtables to separate directory
> -------------------------------------
>
>                 Key: CASSANDRA-6357
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6357
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Patrick McFadin
>            Assignee: Jonathan Ellis
>         Attachments: 6357.txt, c6357-stress-write-latency-99th-1.png
>
>
> Flush writers are a critical element for keeping a node healthy. When several compactions run on systems with low performing data directories, IO becomes a premium. Once the disk subsystem is saturated, write IO is blocked which will cause flush writer threads to backup. Since memtables are large blocks of memory in the JVM, too much blocking can cause excessive GC over time degrading performance. In the worst case causing an OOM.
> Since compaction is running on the data directories. My proposal is to create a separate directory for flushing memtables. Potentially we can use the same methodology of keeping the commit log separate and minimize disk contention against the critical function of the flushwriter. 



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)