performance/benchmarking

[Sean Bowman <sb...@epochlabs.com>]

Hello,

We are evaluating Cassandra performance and would like to know if the
numbers we've gotten can be improved upon.  Right now we're just testing a
simple key/value store using the cassandra-stress tool.  A quick summary of
our setup:

* Using the datastax 2.2 distribution, Oracle HotSpot Java 1.8 default
config
  except as noted below
* 3 node cluster, each node with a 3.3GHz Xeon, 4 cores w/ 2 hyperthreads
  each, 32G memory and large SSDs
* stress.yaml files run with cassandra-stress on other machine(s) as
described
  below

First, vanilla cassandra-stress run like so

    cassandra-stress write duration=10m cl=QUORUM -mode native cql3 \
        -schema "replication(factor=3)" -node 10.40.252.5 -rate threads=700
-errors ignore

gives 98k ops/s, latency median 5.1, max 887, and

    cassandra-stress read duration=10m cl=QUORUM -mode native cql3 \
        -schema "replication(factor=3)" -node 10.40.252.5 -rate threads=700
-errors ignore

gives 105k ops/s, latency median 6.3, max 184.3.

Next, we run with our "user" profile.  The results are as follows, where
"small" denotes 5B keys/2B values and "medium" denotes 128B keys/512B
values.

one test machine:
small:  consistency QUORUM insert, 900 threads, 120k ops/s
small:  consistency QUORUM read,   800 threads, 132k ops/s

small:  consistency ONE insert,    600 threads, 185k ops/s
small:  consistency ONE read,      900 threads, 194k ops/s

medium: consistency QUORUM insert, 550 threads, 89k ops/s
medium: consistency QUORUM read,   880 threads, 55k ops/s

two test machines, total number threads listed:
medium: consistency ONE insert,    800 threads, 108k ops/s
medium: consistency ONE read,     1200 threads, 136k ops/s

Do these numbers seem reasonable?  Is there any low hanging fruit that would
change them dramatically?  Thanks!


More details: the stress.yaml user configuration is below.  The numbers in
the
"columnspec" key change with the above key/value sizes, but that's the only
change.

---
keyspace: smstress

keyspace_definition: |
    CREATE KEYSPACE smstress WITH replication = {'class': 'SimpleStrategy',
'replication_factor': 3};
table: kvtestsmkey

table_definition: |
    CREATE TABLE kvtestsmkey (
        key blob PRIMARY KEY,
        value blob
    ) WITH COMPACT STORAGE;
columnspec:
    - name: key
      size: fixed(5)
    - name: value
      size: fixed(2)

insert:
    partitions: fixed(1)
    select: fixed(1)/1
    batchtype: UNLOGGED

queries:
    read:
        cql: select * from kvtestsmkey where key = ? LIMIT 1
        fields: samerow
---

The default config is used except for the following changes:

* concurrent_reads and _writes is set to 64,
* memtable_allocation_type is set to offheap_objects,
* memtable_flush_writers is set to 8,
* request timeouts have been increased somewhat

cassandra-stress is run like this, mutatis mutandis, for user profiles:

    cassandra-stress user duration=10m cl=ONE profile=stress.yaml
ops\(insert=1\) \
        -mode native cql3 -node (ip address) -rate threads=(number) -errors
ignore

I have system data that seem to indicate that net and disk IO are not
bottlenecks.

Thanks very much for your help!

--
Sean Bowman
Software Developer, Epoch Labs