[jira] [Commented] (CASSANDRA-7533) Let MAX_OUTSTANDING_REPLAY_COUNT be configurable

["Matt Byrd (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-7533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14061506#comment-14061506 ] 

Matt Byrd commented on CASSANDRA-7533:
--------------------------------------

Just to add a bit more context, we had a single instance of Cassandra get fairly stuck replaying commitlogs.
It was burning through 2000% cpu + for over four hours with no end in sight, so we killed it removed commit logs brought it up and ran repair. (This was in q.a thankfully)

The problem can easily be reproduce by just writing 100,000 cql row (range deletes) to the same partition key, stopping Cassandra and starting it again.
I admit this is somewhat of an anti-pattern, but still quite a dramatic effect from not very much data.
The problem exercised here is that:
1. We contend in the memtable to do this insert in a CAS loop.
2. the work done in this loop becomes ever more expensive as RangeTombstoneList.dataSize is iterated over to compute the size.

Point 2. effectively fixed in 2.1 with all the off-heap allocation, the dataSize calculation effectively becomes more online.
To resolve this problem in 2.0 you could also keep this tally of dataSize online, or maybe start keeping it online once the list is sufficiently big to cause a problem.
Doing this seemed to help a lot, but far simpler was just toggling the concurrency of the commitlog replay, which can be achieved by lowering MAX_OUTSTANDING_REPLAY_COUNT (in our case setting this to 1 seemed to help).

Thanks,
Matt


> Let MAX_OUTSTANDING_REPLAY_COUNT be configurable
> ------------------------------------------------
>
>                 Key: CASSANDRA-7533
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7533
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Jeremiah Jordan
>            Assignee: Yuki Morishita
>            Priority: Minor
>             Fix For: 2.0.10
>
>
> There are some workloads where commit log replay will run into contention issues with multiple things updating the same partition.  Through some testing it was found that lowering CommitLogReplayer.java MAX_OUTSTANDING_REPLAY_COUNT can help with this issue.
> The calculations added in CASSANDRA-6655 are one such place things get bottlenecked.



--
This message was sent by Atlassian JIRA
(v6.2#6252)