[jira] [Commented] (CASSANDRA-8894) Our default buffer size for (uncompressed) buffered reads should be smaller, and based on the expected record size

["Albert P Tobey (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-8894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14726544#comment-14726544 ] 

Albert P Tobey commented on CASSANDRA-8894:
-------------------------------------------

Sorry I'm late to the thread, but the recently added patch seems very overcomplicated for little benefit.

I noticed the massive over-read in 2.1 and tracked it down independently by profiling with disk_access_mode: standard. I then ran a build with the DEFAULT_BUFFER_SIZE at 4K and saw an instant 4x increase in TXN/s on a simple -stress test with a 60% reduction in wasted disk IO. This over-read is causing performance problems on every Cassandra 2.1 cluster that isn't 100% writes. It doesn't always show up because of the massive amount of RAM a lot of folks are running, but under low memory situations it is killing even very fast SSDs.

Patch for 2.1: https://gist.github.com/tobert/10c307cf3709a585a7cf

In reading through the history, I think this is being overthought. If anything, the readahead and buffering in the read path should be *removed* and instead issue precise reads wherever it's possible. For now, the change to a 4K buffer size should be added to 2.1 in order to significantly speed up read workloads.

> Our default buffer size for (uncompressed) buffered reads should be smaller, and based on the expected record size
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-8894
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8894
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Benedict
>            Assignee: Stefania
>              Labels: benedict-to-commit
>             Fix For: 3.0 alpha 1
>
>         Attachments: 8894_25pct.yaml, 8894_5pct.yaml, 8894_tiny.yaml
>
>
> A large contributor to slower buffered reads than mmapped is likely that we read a full 64Kb at once, when average record sizes may be as low as 140 bytes on our stress tests. The TLB has only 128 entries on a modern core, and each read will touch 32 of these, meaning we are unlikely to almost ever be hitting the TLB, and will be incurring at least 30 unnecessary misses each time (as well as the other costs of larger than necessary accesses). When working with an SSD there is little to no benefit reading more than 4Kb at once, and in either case reading more data than we need is wasteful. So, I propose selecting a buffer size that is the next larger power of 2 than our average record size (with a minimum of 4Kb), so that we expect to read in one operation. I also propose that we create a pool of these buffers up-front, and that we ensure they are all exactly aligned to a virtual page, so that the source and target operations each touch exactly one virtual page per 4Kb of expected record size.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)