[jira] [Comment Edited] (CASSANDRA-8833) Stop opening compaction results early

["Benedict (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14327970#comment-14327970 ] 

Benedict edited comment on CASSANDRA-8833 at 2/19/15 7:27 PM:
--------------------------------------------------------------

bq. I suspect that memory mapping will eclipse performance gains
bq. I'd prefer hard #'s from test beds.

Make your mind up ;)

Joking aside, the two aren't mutually exclusive. We only rule out early reopening with mmap because of hard links, but they only live for the duration of the early reopening. We could delay mmapping until the final opening. The two are also very different performance profile implications; early opening mitigates worst case behaviour far more importantly than improving overall system performance (although it does that too).

bq.  Code-changes to stabilize this have been increasing quite a bit recently

They almost all happened within the past month, when I finished up my other commitments and decided to try and put a lid on this and the other correlated problems. So I'm not sure that's indicative of much.

bq. down to a 9% total ops improvement on the regression case

I'm not certain what you're referring to here, but we did compare against populate_io_cache_on_flush:true and saw "only" this level of improvement. But this option was seen as sufficiently pathological to be disabled by default, since it would damage page cache sufficiently badly to affect cluster performance. Users preferred the "no" side of the cliff to this option. The point of 6916 was to permit them to not have to choose either side.


was (Author: benedict):
bq. I suspect that memory mapping will eclipse performance gains
bq. I'd prefer hard #'s from test beds.

Make your mind up ;)

Joking aside, the two aren't mutually exclusive. We only rule out early reopening with mmap because of hard links, but they only live for the duration of the early reopening. We could delay mmapping until the final opening. The two are also very different performance profile implications; early opening mitigates worst case behaviour far more importantly than improving overall system performance (although it does that too).

bq.  Code-changes to stabilize this have been increasing quite a bit recently

They almost all happened within the past month, when I finished up my other commitments and decided to try and put a lid on this and the other correlated problems. So I'm not sure that's indicative of much.


> Stop opening compaction results early
> -------------------------------------
>
>                 Key: CASSANDRA-8833
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8833
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Marcus Eriksson
>             Fix For: 2.1.4
>
>
> We should simplify the code base by not doing early opening of compaction results. It makes it very hard to reason about sstable life cycles since they can be in many different states, "opened early", "starts moved", "shadowed", "final", instead of as before, basically just one (tmp files are not really 'live' yet so I don't count those). The ref counting of shared resources between sstables in these different states is also hard to reason about. This has caused quite a few issues since we released 2.1
> I think it all boils down to a performance vs code complexity issue, is opening compaction results early really 'worth it' wrt the performance gain? The results in CASSANDRA-6916 sure look like the benefits are big enough, but the difference should not be as big for people on SSDs (which most people who care about latencies are)
> WDYT [~benedict] [~jbellis] [~iamaleksey] [~JoshuaMcKenzie]?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)