[jira] [Updated] (CASSANDRA-3578) Multithreaded commitlog

["Benedict (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CASSANDRA-3578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Benedict updated CASSANDRA-3578:
--------------------------------

    Attachment: oprate.svg
                latency.svg

A patch for this is available for review at [3578-2|https://github.com/belliottsmith/cassandra/tree/iss-3578-2]

Already discussed:
- Chained headers
- Ensures commits are persistent, using the suggested synchronisation scheme (read/write lock)

Further changes:
- Writes are completely non-blocking unless the CLE is behind or you're using Batch CLE
- On activating a new CLS, we trigger a sync() of the log; so now we sync() ever pollInterval elapsed, OR commit_log_segment_size_in_mb written, whichever condition is met first after the previous sync. This allows us to stay a little ahead of pollInterval, giving us some breathing room during "brief" spikes in write load in excess of what the disk can handle.
- Once we've completely written a CLS we immediately close/unmap the buffer
- On any drop keyspace or column family command, or on a node drain, we force the recycling of any CLS in use at the time of the call (this addresses CASSANDRA-5911. I included it in this ticket as it was easier to think about both at once)

Some implementation detail changes:
- We maintain a separate cfDirty and cfClean set now, which we merge on demand, to avoid allocating/deallocating AtomicIntegers all of the time
- We now reject row mutations that are only HALF the size of the CL, as opposed to equal in size - this is to stop burning through lots of CLS if we try to switch to a new segment but then are beaten to allocating the first item in it.

Some future work:
- Could reasonably easily have a guaranteed non-blocking CL.add method, which yields a Future if blocking becomes necessary; this could allow us to short-circuit the write-path a little to reduce latency in the majority of cases where blocking doesn't happen
- Compressed CL to improve IO
- Need to improve error handling in CL in general

Note, Vijay, that I briefly switched to a simpler blocking approach to switching in a new segment, as you suggested you preferred the simpler approach, but I decided to revert to non-blocking, due to potential future dividends with this guarantee.

I've attached two graphs to demonstrate the effect of this patch in a real 4-node cluster. Note the latency graph has a logarithmic y-axis, so this patch looks to be an order of magnitude better at worst write latency measured; also variance in latency at the tail end is lower. This is also why there are fewer measurements, as the stderr of the measurements was smaller, so stress finished earlier. Also a roughly 12% increase in maximum throughput on this particular cluster.

> Multithreaded commitlog
> -----------------------
>
>                 Key: CASSANDRA-3578
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3578
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Jonathan Ellis
>            Assignee: Vijay
>            Priority: Minor
>              Labels: performance
>         Attachments: 0001-CASSANDRA-3578.patch, ComitlogStress.java, Current-CL.png, Multi-Threded-CL.png, latency.svg, oprate.svg, parallel_commit_log_2.patch
>
>
> Brian Aker pointed out a while ago that allowing multiple threads to modify the commitlog simultaneously (reserving space for each with a CAS first, the way we do in the SlabAllocator.Region.allocate) can improve performance, since you're not bottlenecking on a single thread to do all the copying and CRC computation.
> Now that we use mmap'd CommitLog segments (CASSANDRA-3411) this becomes doable.
> (moved from CASSANDRA-622, which was getting a bit muddled.)



--
This message was sent by Atlassian JIRA
(v6.1#6144)