[jira] [Created] (CASSANDRA-15717) Benchmark performance difference between Docker and Kubernetes when running Cassandra:2.2.16 official Docker image

["Eddy Truyen (Jira)" <ji...@apache.org>]

Eddy Truyen created CASSANDRA-15717:
---------------------------------------

             Summary: Benchmark performance difference between Docker and Kubernetes when running Cassandra:2.2.16 official Docker image
                 Key: CASSANDRA-15717
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-15717
             Project: Cassandra
          Issue Type: Bug
          Components: Test/benchmark
            Reporter: Eddy Truyen
         Attachments: docker-cassandra-nodetool-tablestats, kube-adm-cassandra-nodetool-tablestats

This is my first JIRA issue. Sorry if I do something  wrong in the reporting.

I experienced a performance degradation when running a single Cassandra Docker container  inside Kubernetes in comparison with running the Docker container stand-alone. I used the following image decomads/cassandra:2.2.16, which uses cassandra:2.2.16 as base image and adds a readinessProbe to it.

I used identical Docker configuration parameters by ensuring that the output of docker inspect is as much as possible the same.  Docker runs in bridged mode

 Experiment (repeated on minikube+VirtualBox (12GB, 4 CPU cores, 30 GB) on physical laptop with 4 cores/8 logical processors and 16GB RAM on and Openstack VM Ubuntu 16:04  (4GB, 4 CPU cores, 50GB), that runs on a physical nodes with 16 CPU cores. Storage is Ceph. 
 * A write-only workload (YCSB benchmark workload A - Load phase) using the following user table: 
 cqlsh> create keyspace ycsb
 WITH REPLICATION = \{'class' : 'SimpleStrategy', 'replication_factor': 3 }
;
 cqlsh> USE ycsb;
 cqlsh> create table usertable (
 y_id varchar primary key,
 field0 varchar,
 field1 varchar,
 field2 varchar,
 field3 varchar,
 field4 varchar,
 field5 varchar,
 field6 varchar,
 field7 varchar,
 field8 varchar,
 field9 varchar);



 * And using the following script: python ./bin/ycsb load cassandra2-cql -P workloads/workloada -p recordcount=1500000 -p operationcount=1500000 -p measurementtype=raw -p cassandra.connecttimeoutmillis=60000 -p cassandra.readtimeoutmillis=60000 -target 1500 -threads 20 -p hosts=localhost > results/cassandra-docker/cassandra-docker-load-workloada-1-records-1500000-rnd-1762034446.txt
sleep 15

Observations:
 * Docker:
 ** Mean average  response latency: 1500
 ** Average CPU usage of cassandra instances (wrt 2 cores): 42
 * Kubernetes
 ** Mean average response latency: 2700
 ** Average CPU usage of cassandra instance (wrt 2 cores): 32%
 * Nodetool tablestats
 ** There are little difference for the usertable, with an almost identical  write latency (difference < 0.002 ms).
 ** However for the system keyspace there are quite some differences in read/write count and read latency (difference = 2.5 ms). More specifically,  compaction history (see attachment the 2 tablestats output)
 *** Table: compaction_history Kubernetes
 SSTable count: 1
 Space used (live): 12049
 Space used (total): 12049
 Space used by snapshots (total): 0
 Off heap memory used (total): 108
 SSTable Compression Ratio: 0.25466231166368136
 Number of keys (estimate): 54 
 Memtable data size: 0
 Memtable off heap memory used: 0
 Memtable switch count: 12
 *** Table: compaction_history Docker
 SSTable count: 1
 Space used (live): 8921
 Space used (total): 8921
 Space used by snapshots (total): 0
 Off heap memory used (total): 76
 SSTable Compression Ratio: 0.2603359822955437
 Number of keys (estimate): 25
Memtable data size: 0
 Memtable off heap memory used: 0
 Memtable switch count: 1
 * Cassandra Logs:
 * 
|Docker|Kubernetes|
|ubuntu@k8-test-2:/data/ycsb/cassandra-docker$ grep -c 'Writing Memtable-compaction' docker-cassandra-logs
27
ubuntu@k8-test-2:/data/ycsb/cassandra-docker$ grep -c 'Writing Memtable-size' docker-cassandra-logs
1
ubuntu@k8-test-2:/data/ycsb/cassandra-docker$ grep -c ' Writing Memtable-sstable' docker-cassandra-logs
1
ubuntu@k8-test-2:/data/ycsb/cassandra-docker$ grep -c 'Writing Memtable-usertable' docker-cassandra-logs
45
ubuntu@k8-test-2:/data/ycsb/cassandra-docker$ grep -c 'Writing Memtable-compactions_in_progress' docker-cassandra-logs
26
ubuntu@k8-test-2:/data/ycsb/cassandra-docker$ grep -c 'Writing Memtable-peers' docker-cassandra-logs
1
ubuntu@k8-test-2:/data/ycsb/cassandra-docker$ grep -c 'Writing Memtable-schema' docker-cassandra-logs
24
ubuntu@k8-test-2:/data/ycsb/cassandra-docker$ grep -c 'Writing Memtable-local' docker-cassandra-logs
6|ubuntu@k8-test-2:/data/ycsb/cassandra-kube$ grep -c  'Writing Memtable-compaction' kubeadm-cassandra-logs \| more
32
ubuntu@k8-test-2:/data/ycsb/cassandra-kube$ grep -c  'Writing Memtable-size' kubeadm-cassandra-logs \| more
7
ubuntu@k8-test-2:/data/ycsb/cassandra-kube$ grep -c  ' Writing Memtable-sstable' kubeadm-cassandra-logs \| more
7
ubuntu@k8-test-2:/data/ycsb/cassandra-kube$ grep -c  'Writing Memtable-usertable' kubeadm-cassandra-logs \| more
45
ubuntu@k8-test-2:/data/ycsb/cassandra-kube$ grep -c  'Writing Memtable-compactions_in_progress' kubeadm-cassandra-logs \| more
26
ubuntu@k8-test-2:/data/ycsb/cassandra-kube$ grep -c  'Writing Memtable-peers' kubeadm-cassandra-logs \| more
2
ubuntu@k8-test-2:/data/ycsb/cassandra-kube$ grep -c  'Writing Memtable-schema' kubeadm-cassandra-logs
17
ubuntu@k8-test-2:/data/ycsb/cassandra-kube$ grep -c 'Writing Memtable-local' kubeadm-cassandra-logs
5|

Can the higher compaction rate of Kubernetes-based Cassandra instance be the cause of the latency for YCSB benchmark? And how could this be resolved?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org