You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Arsen Vladimirskiy <ar...@outlook.com> on 2019/06/17 02:35:53 UTC
Write perf better when commitlog is on ext4 vs xfs
Hi,
On CentOS 7.5 (3.10.0-957.21.2.el7.x86_64), I am seeing better write performance when commitlog is using "ext4" versus "xfs" filesystem. It is the same local/ephemeral disk within an Azure VM and the only change I am making is switching the filesystem on the disk from "ext4" to "xfs".
Is this expected? Any ideas on how to troubleshoot and tune the filesystem (or Linux or Cassandra) so that xfs performs same or better than ext4?
Details below.
OpenJDK 1.8.0 (same with Oracle JDK 1.8.0)
Cassandra version: 3.11.4
cassandra.yaml has this default config:
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
commitlog_segment_size_in_mb: 32
I am testing write performance using the following cassandra-stress command:
cqlsh --execute "DROP KEYSPACE IF EXISTS keyspace1;" 10.0.0.4
cassandra-stress write no-warmup n=2000000 cl=LOCAL_QUORUM \
-rate threads=32 throttle=0/s \
-pop seq="1..10000000" -col size=FIXED\(818\) \
-node "10.0.0.4" \
-schema keyspace=keyspace1 \
-log interval=1 -mode native cql3 connectionsPerHost=128
ext4 mount and ops/s look like this:
/dev/sdb1 on /mnt/resource type ext4 (rw,relatime,seclabel,data=ordered)
Connected to cluster: avcassandra, max pending requests per connection 128, max connections per host 128
Datatacenter: dc1; Host: /10.0.0.4; Rack: rack0
Created keyspaces. Sleeping 1s for propagation.
Sleeping 2s...
Running WRITE with 32 threads for 2000000 iteration
Failed to connect over JMX; not collecting these stats
type total ops, op/s, pk/s, row/s, mean, med, .95, .99, .999, max, time, stderr, errors, gc: #, max ms, sum ms, sdv ms, mb
total, 749, 749, 749, 749, 14.4, 8.4, 43.5, 79.7, 107.0, 131.9, 1.0, 0.00000, 0, 0, 0, 0, 0, 0
total, 4776, 4027, 4027, 4027, 7.9, 4.9, 24.6, 44.8, 65.4, 76.7, 2.0, 0.47357, 0, 0, 0, 0, 0, 0
total, 11129, 6353, 6353, 6353, 5.0, 2.5, 16.4, 26.6, 40.8, 46.5, 3.0, 0.34745, 0, 0, 0, 0, 0, 0
total, 17547, 6418, 6418, 6418, 4.9, 2.2, 16.4, 31.8, 74.1, 83.6, 4.0, 0.25930, 0, 0, 0, 0, 0, 0
total, 28396, 10849, 10849, 10849, 3.0, 1.5, 11.4, 21.2, 37.9, 49.0, 5.0, 0.25513, 0, 0, 0, 0, 0, 0
total, 40808, 12412, 12412, 12412, 2.5, 1.5, 9.0, 19.5, 33.8, 68.0, 6.0, 0.23341, 0, 0, 0, 0, 0, 0
total, 53080, 12272, 12272, 12272, 2.6, 1.3, 10.0, 19.0, 60.8, 73.5, 7.0, 0.20377, 0, 0, 0, 0, 0, 0
total, 67025, 13945, 13945, 13945, 2.3, 1.3, 9.1, 17.7, 30.5, 40.8, 8.0, 0.18276, 0, 0, 0, 0, 0, 0
total, 86886, 19861, 19861, 19861, 1.6, 1.2, 2.9, 9.3, 42.7, 56.5, 9.0, 0.18765, 0, 0, 0, 0, 0, 0
total, 106184, 19298, 19298, 19298, 1.6, 1.2, 3.1, 10.8, 57.8, 61.5, 10.0, 0.17614, 0, 0, 0, 0, 0, 0
total, 128877, 22693, 22693, 22693, 1.4, 1.2, 2.6, 4.1, 14.7, 30.2, 11.0, 0.16866, 0, 0, 0, 0, 0, 0
total, 148331, 19454, 19454, 19454, 1.6, 1.3, 3.0, 5.7, 45.3, 47.7, 12.0, 0.15466, 0, 0, 0, 0, 0, 0
total, 156974, 8643, 8643, 8643, 1.6, 1.2, 3.3, 10.2, 24.6, 41.2, 13.0, 0.14813, 0, 0, 0, 0, 0, 0
total, 170654, 13680, 13680, 13680, 3.6, 1.3, 3.1, 9.2, 890.8, 896.5, 14.0, 0.39489, 0, 0, 0, 0, 0, 0
total, 187715, 17061, 17061, 17061, 1.9, 1.2, 4.5, 13.0, 27.2, 44.2, 15.0, 0.37273, 0, 0, 0, 0, 0, 0
total, 205718, 18003, 18003, 18003, 1.7, 1.3, 3.7, 11.3, 32.8, 53.4, 16.0, 0.35161, 0, 0, 0, 0, 0, 0
total, 212945, 7227, 7227, 7227, 1.8, 1.2, 3.2, 11.2, 71.3, 88.5, 17.0, 0.34545, 0, 0, 0, 0, 0, 0
total, 220825, 7880, 7880, 7880, 6.4, 1.2, 3.1, 11.1, 1200.6, 1209.0, 18.0, 0.42966, 0, 0, 0, 0, 0, 0
total, 241273, 20448, 20448, 20448, 1.5, 1.2, 3.1, 7.3, 23.0, 60.4, 19.0, 0.40131, 0, 0, 0, 0, 0, 0
total, 261889, 20616, 20616, 20616, 1.5, 1.2, 2.9, 8.0, 25.0, 33.5, 20.0, 0.37691, 0, 0, 0, 0, 0, 0
total, 282464, 20575, 20575, 20575, 1.5, 1.2, 2.7, 5.2, 68.6, 72.7, 21.0, 0.35456, 0, 0, 0, 0, 0, 0
total, 305429, 22965, 22965, 22965, 1.4, 1.2, 2.5, 3.8, 17.2, 23.8, 22.0, 0.33370, 0, 0, 0, 0, 0, 0
total, 326504, 21075, 21075, 21075, 1.5, 1.2, 2.9, 5.6, 17.9, 31.5, 23.0, 0.31637, 0, 0, 0, 0, 0, 0
total, 342653, 16149, 16149, 16149, 1.9, 1.3, 3.8, 13.2, 107.0, 124.4, 24.0, 0.30293, 0, 0, 0, 0, 0, 0
total, 360972, 18319, 18319, 18319, 1.7, 1.2, 3.8, 11.5, 36.1, 63.7, 25.0, 0.28988, 0, 0, 0, 0, 0, 0
total, 379092, 18120, 18120, 18120, 1.7, 1.2, 2.9, 8.2, 133.5, 161.1, 26.0, 0.27680, 0, 0, 0, 0, 0, 0
total, 400379, 21287, 21287, 21287, 1.5, 1.2, 2.9, 6.4, 30.8, 43.2, 27.0, 0.26454, 0, 0, 0, 0, 0, 0
total, 422330, 21951, 21951, 21951, 1.4, 1.2, 2.6, 4.3, 21.9, 35.3, 28.0, 0.25311, 0, 0, 0, 0, 0, 0
total, 442491, 20161, 20161, 20161, 1.4, 1.2, 2.7, 5.0, 21.6, 41.5, 29.0, 0.24331, 0, 0, 0, 0, 0, 0
total, 456035, 13544, 13544, 13544, 11.9, 1.3, 3.4, 9.7, 4366.3, 4366.3, 34.0, 0.25639, 0, 0, 0, 0, 0, 0
total, 458788, 2753, 2753, 2753, 1.5, 1.3, 2.9, 4.2, 32.8, 40.9, 35.0, 0.25612, 0, 0, 0, 0, 0, 0
total, 468346, 9558, 9558, 9558, 46.4, 1.4, 7.1, 15.6, 13203.7, 13228.8, 48.0, 0.27131, 0, 0, 0, 0, 0, 0
total, 489463, 21117, 21117, 21117, 1.5, 1.2, 2.8, 8.5, 19.7, 32.8, 49.0, 0.26143, 0, 0, 0, 0, 0, 0
total, 510484, 21021, 21021, 21021, 1.5, 1.2, 2.9, 5.7, 20.2, 28.0, 50.0, 0.25232, 0, 0, 0, 0, 0, 0
total, 532895, 22411, 22411, 22411, 1.4, 1.2, 2.5, 4.8, 28.4, 40.7, 51.0, 0.24335, 0, 0, 0, 0, 0, 0
total, 553656, 20761, 20761, 20761, 1.5, 1.2, 3.0, 9.0, 18.2, 33.0, 52.0, 0.23551, 0, 0, 0, 0, 0, 0
xfs mount and ops/s look like this:
/dev/sdb1 on /mnt/resource type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
Connected to cluster: avcassandra, max pending requests per connection 128, max connections per host 128
Datatacenter: dc1; Host: /10.0.0.4; Rack: rack0
Created keyspaces. Sleeping 1s for propagation.
Sleeping 2s...
Running WRITE with 32 threads for 2000000 iteration
Failed to connect over JMX; not collecting these stats
type total ops, op/s, pk/s, row/s, mean, med, .95, .99, .999, max, time, stderr, errors, gc: #, max ms, sum ms, sdv ms, mb
total, 1550, 1550, 1550, 1550, 14.9, 12.2, 37.8, 61.5, 75.6, 82.2, 1.0, 0.00000, 0, 0, 0, 0, 0, 0
total, 5742, 4192, 4192, 4192, 7.6, 4.6, 23.4, 36.7, 63.0, 78.8, 2.0, 0.32434, 0, 0, 0, 0, 0, 0
total, 10026, 4284, 4284, 4284, 7.4, 5.1, 20.4, 33.5, 95.2, 112.1, 3.0, 0.22200, 0, 0, 0, 0, 0, 0
total, 15802, 5776, 5776, 5776, 5.5, 3.2, 17.0, 32.8, 50.5, 62.6, 4.0, 0.18920, 0, 0, 0, 0, 0, 0
total, 23728, 7926, 7926, 7926, 4.0, 2.6, 11.6, 23.2, 54.7, 90.6, 5.0, 0.19722, 0, 0, 0, 0, 0, 0
total, 32143, 8415, 8415, 8415, 3.7, 2.5, 11.8, 20.4, 40.8, 48.1, 6.0, 0.17557, 0, 0, 0, 0, 0, 0
total, 41506, 9363, 9363, 9363, 3.4, 2.0, 11.0, 26.1, 72.2, 84.2, 7.0, 0.16341, 0, 0, 0, 0, 0, 0
total, 52696, 11190, 11190, 11190, 2.8, 2.6, 5.0, 10.8, 20.1, 32.3, 8.0, 0.15527, 0, 0, 0, 0, 0, 0
total, 63707, 11011, 11011, 11011, 2.9, 2.6, 4.6, 11.2, 20.7, 62.6, 9.0, 0.14386, 0, 0, 0, 0, 0, 0
total, 74246, 10539, 10539, 10539, 3.0, 2.7, 5.2, 11.9, 41.4, 46.3, 10.0, 0.13055, 0, 0, 0, 0, 0, 0
total, 84693, 10447, 10447, 10447, 3.0, 2.8, 5.0, 7.2, 13.6, 30.0, 11.0, 0.11879, 0, 0, 0, 0, 0, 0
total, 95304, 10611, 10611, 10611, 3.0, 2.8, 5.0, 7.2, 14.6, 32.6, 12.0, 0.10912, 0, 0, 0, 0, 0, 0
total, 105865, 10561, 10561, 10561, 3.0, 2.7, 4.8, 10.4, 53.1, 60.6, 13.0, 0.10111, 0, 0, 0, 0, 0, 0
total, 116431, 10566, 10566, 10566, 3.0, 2.8, 5.0, 8.9, 14.8, 25.2, 14.0, 0.09380, 0, 0, 0, 0, 0, 0
total, 127856, 11425, 11425, 11425, 2.8, 2.6, 4.5, 6.8, 16.6, 24.4, 15.0, 0.08816, 0, 0, 0, 0, 0, 0
total, 139136, 11280, 11280, 11280, 2.8, 2.6, 4.4, 6.5, 17.6, 20.6, 16.0, 0.08292, 0, 0, 0, 0, 0, 0
total, 149943, 10807, 10807, 10807, 2.9, 2.7, 4.5, 6.1, 50.0, 52.2, 17.0, 0.07817, 0, 0, 0, 0, 0, 0
total, 161338, 11395, 11395, 11395, 2.8, 2.6, 4.8, 10.1, 18.4, 30.0, 18.0, 0.07411, 0, 0, 0, 0, 0, 0
total, 172127, 10789, 10789, 10789, 2.9, 2.6, 5.3, 12.3, 41.9, 53.9, 19.0, 0.07025, 0, 0, 0, 0, 0, 0
total, 182078, 9951, 9951, 9951, 3.2, 2.1, 10.9, 20.9, 36.3, 80.7, 20.0, 0.06662, 0, 0, 0, 0, 0, 0
total, 193140, 11062, 11062, 11062, 2.9, 2.5, 5.3, 12.0, 29.0, 44.7, 21.0, 0.06354, 0, 0, 0, 0, 0, 0
total, 203536, 10396, 10396, 10396, 3.1, 2.6, 5.2, 11.7, 60.6, 68.7, 22.0, 0.06061, 0, 0, 0, 0, 0, 0
total, 214575, 11039, 11039, 11039, 2.9, 2.6, 5.0, 10.1, 22.5, 36.2, 23.0, 0.05801, 0, 0, 0, 0, 0, 0
total, 225072, 10497, 10497, 10497, 3.0, 2.6, 5.6, 13.2, 29.2, 48.2, 24.0, 0.05553, 0, 0, 0, 0, 0, 0
total, 235155, 10083, 10083, 10083, 3.1, 2.7, 4.8, 9.9, 89.3, 102.6, 25.0, 0.05329, 0, 0, 0, 0, 0, 0
total, 245257, 10102, 10102, 10102, 3.1, 2.6, 5.2, 11.4, 65.6, 67.3, 26.0, 0.05118, 0, 0, 0, 0, 0, 0
total, 256802, 11545, 11545, 11545, 2.7, 2.6, 4.5, 8.2, 18.6, 36.6, 27.0, 0.04946, 0, 0, 0, 0, 0, 0
total, 267959, 11157, 11157, 11157, 2.8, 2.6, 4.6, 9.2, 20.9, 69.1, 28.0, 0.04784, 0, 0, 0, 0, 0, 0
total, 279085, 11126, 11126, 11126, 2.8, 2.6, 4.6, 8.2, 16.0, 30.8, 29.0, 0.04620, 0, 0, 0, 0, 0, 0
total, 289773, 10688, 10688, 10688, 3.0, 2.7, 4.9, 7.9, 19.0, 35.5, 30.0, 0.04463, 0, 0, 0, 0, 0, 0
total, 299929, 10156, 10156, 10156, 3.1, 2.7, 4.7, 10.8, 87.3, 90.8, 31.0, 0.04316, 0, 0, 0, 0, 0, 0
total, 310716, 10787, 10787, 10787, 2.9, 2.7, 5.0, 8.9, 16.3, 29.2, 32.0, 0.04179, 0, 0, 0, 0, 0, 0
total, 320931, 10215, 10215, 10215, 2.8, 2.6, 4.5, 7.2, 18.4, 25.5, 33.0, 0.04050, 0, 0, 0, 0, 0, 0
total, 322021, 1090, 1090, 1090, 31.9, 3.1, 11.0, 974.1, 984.6, 985.1, 34.0, 0.15458, 0, 0, 0, 0, 0, 0
total, 333309, 11288, 11288, 11288, 2.8, 2.6, 4.7, 9.9, 21.0, 25.4, 35.0, 0.15035, 0, 0, 0, 0, 0, 0
total, 344239, 10930, 10930, 10930, 2.9, 2.6, 5.0, 9.9, 39.6, 54.8, 36.0, 0.14636, 0, 0, 0, 0, 0, 0
total, 355560, 11321, 11321, 11321, 2.8, 2.6, 4.7, 9.3, 22.8, 41.4, 37.0, 0.14249, 0, 0, 0, 0, 0, 0
total, 364990, 9430, 9430, 9430, 3.4, 2.6, 5.0, 12.3, 141.3, 158.2, 38.0, 0.13902, 0, 0, 0, 0, 0, 0
total, 376297, 11307, 11307, 11307, 2.8, 2.6, 4.5, 8.6, 21.8, 24.0, 39.0, 0.13559, 0, 0, 0, 0, 0, 0
total, 387651, 11354, 11354, 11354, 2.8, 2.5, 4.7, 8.9, 20.3, 26.4, 40.0, 0.13230, 0, 0, 0, 0, 0, 0
total, 394128, 6477, 6477, 6477, 2.7, 2.6, 4.3, 5.8, 12.0, 14.0, 41.0, 0.13099, 0, 0, 0, 0, 0, 0
total, 396263, 2135, 2135, 2135, 21.6, 2.7, 4.7, 1256.2, 1258.3, 1259.3, 42.0, 0.13955, 0, 0, 0, 0, 0, 0
total, 407132, 10869, 10869, 10869, 2.9, 2.6, 4.7, 10.1, 18.8, 24.2, 43.0, 0.13638, 0, 0, 0, 0, 0, 0
total, 418276, 11144, 11144, 11144, 2.8, 2.6, 4.6, 9.0, 15.6, 26.1, 44.0, 0.13327, 0, 0, 0, 0, 0, 0
total, 429779, 11503, 11503, 11503, 2.8, 2.6, 4.3, 5.6, 13.4, 22.0, 45.0, 0.13022, 0, 0, 0, 0, 0, 0
total, 439888, 10109, 10109, 10109, 3.1, 2.6, 4.5, 9.7, 106.7, 108.4, 46.0, 0.12741, 0, 0, 0, 0, 0, 0
total, 450854, 10966, 10966, 10966, 2.9, 2.6, 4.7, 8.5, 21.0, 40.6, 47.0, 0.12470, 0, 0, 0, 0, 0, 0
total, 462404, 11550, 11550, 11550, 2.7, 2.6, 4.3, 5.9, 11.4, 19.3, 48.0, 0.12202, 0, 0, 0, 0, 0, 0
total, 472649, 10245, 10245, 10245, 2.7, 2.5, 4.3, 8.8, 13.8, 43.7, 49.0, 0.11969, 0, 0, 0, 0, 0, 0
total, 476297, 3648, 3648, 3648, 9.8, 2.7, 4.3, 12.1, 804.3, 805.3, 50.0, 0.11649, 0, 0, 0, 0, 0, 0
total, 487422, 11125, 11125, 11125, 2.8, 2.7, 4.5, 6.5, 16.0, 32.0, 51.0, 0.11421, 0, 0, 0, 0, 0, 0
total, 498707, 11285, 11285, 11285, 2.8, 2.6, 4.4, 8.9, 22.1, 32.6, 52.0, 0.11200, 0, 0, 0, 0, 0, 0
total, 509903, 11196, 11196, 11196, 2.8, 2.6, 4.5, 9.0, 16.5, 33.3, 53.0, 0.10988, 0, 0, 0, 0, 0, 0
Fw: Write perf better when commitlog is on ext4 vs xfs
Posted by Arsen Vlad <ar...@outlook.com>.
Hi,
I previously sent the question below to the user list, but didn't hear back, and after more testing I think it maybe a dev question... potentially specific to MemoryMappedSegment behavior.
Also, I see similar ext4 vs xfs performance difference on both Azure and GCP VMs running CentOS 7.5.
From additional tests, seeing that when using commitlog_compression (e.g. LZ4Compressor) the performance of xfs and ext4 is equivalent and good (i.e. compressed xfs appears to perform as well as compressed and non-compressed ext4). I see that the write code-path in non-compressed scenario is using MappedByteBuffer<https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/db/commitlog/MemoryMappedSegment.java> (i.e. memory mapped file) while compressed code-path<https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/db/commitlog/CompressedSegment.java> is using regular FileChannel and ByteBuffer. Usually MappedByteBuffer should perform better, but seems to perform worse in a multi-threaded scenario with xfs on CentOS 7.5
Here is an example of what it looks like with "rate threads=64"
[cid:99c00a63-7d7e-49cb-97a9-b5ed19951942]
Any ideas on what can be causing this difference in performance for XFS when using the default not compressed commitlog (i.e. MappedByteBuffer code path)?
Thanks,
Arsen
________________________________
From: arsenvlad@outlook.com
Sent: Sunday, June 16, 2019 9:35 PM
To: user@cassandra.apache.org
Subject: Write perf better when commitlog is on ext4 vs xfs
Hi,
On CentOS 7.5 (3.10.0-957.21.2.el7.x86_64), I am seeing better write performance when commitlog is using "ext4" versus "xfs" filesystem. It is the same local/ephemeral disk within an Azure VM and the only change I am making is switching the filesystem on the disk from "ext4" to "xfs".
Is this expected? Any ideas on how to troubleshoot and tune the filesystem (or Linux or Cassandra) so that xfs performs same or better than ext4?
Details below.
OpenJDK 1.8.0 (same with Oracle JDK 1.8.0)
Cassandra version: 3.11.4
cassandra.yaml has this default config:
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
commitlog_segment_size_in_mb: 32
I am testing write performance using the following cassandra-stress command:
cqlsh --execute "DROP KEYSPACE IF EXISTS keyspace1;" 10.0.0.4
cassandra-stress write no-warmup n=2000000 cl=LOCAL_QUORUM \
-rate threads=32 throttle=0/s \
-pop seq="1..10000000" -col size=FIXED\(818\) \
-node "10.0.0.4" \
-schema keyspace=keyspace1 \
-log interval=1 -mode native cql3 connectionsPerHost=128
ext4 mount and ops/s look like this:
/dev/sdb1 on /mnt/resource type ext4 (rw,relatime,seclabel,data=ordered)
Connected to cluster: avcassandra, max pending requests per connection 128, max connections per host 128
Datatacenter: dc1; Host: /10.0.0.4; Rack: rack0
Created keyspaces. Sleeping 1s for propagation.
Sleeping 2s...
Running WRITE with 32 threads for 2000000 iteration
Failed to connect over JMX; not collecting these stats
type total ops, op/s, pk/s, row/s, mean, med, .95, .99, .999, max, time, stderr, errors, gc: #, max ms, sum ms, sdv ms, mb
total, 749, 749, 749, 749, 14.4, 8.4, 43.5, 79.7, 107.0, 131.9, 1.0, 0.00000, 0, 0, 0, 0, 0, 0
total, 4776, 4027, 4027, 4027, 7.9, 4.9, 24.6, 44.8, 65.4, 76.7, 2.0, 0.47357, 0, 0, 0, 0, 0, 0
total, 11129, 6353, 6353, 6353, 5.0, 2.5, 16.4, 26.6, 40.8, 46.5, 3.0, 0.34745, 0, 0, 0, 0, 0, 0
total, 17547, 6418, 6418, 6418, 4.9, 2.2, 16.4, 31.8, 74.1, 83.6, 4.0, 0.25930, 0, 0, 0, 0, 0, 0
total, 28396, 10849, 10849, 10849, 3.0, 1.5, 11.4, 21.2, 37.9, 49.0, 5.0, 0.25513, 0, 0, 0, 0, 0, 0
total, 40808, 12412, 12412, 12412, 2.5, 1.5, 9.0, 19.5, 33.8, 68.0, 6.0, 0.23341, 0, 0, 0, 0, 0, 0
total, 53080, 12272, 12272, 12272, 2.6, 1.3, 10.0, 19.0, 60.8, 73.5, 7.0, 0.20377, 0, 0, 0, 0, 0, 0
total, 67025, 13945, 13945, 13945, 2.3, 1.3, 9.1, 17.7, 30.5, 40.8, 8.0, 0.18276, 0, 0, 0, 0, 0, 0
total, 86886, 19861, 19861, 19861, 1.6, 1.2, 2.9, 9.3, 42.7, 56.5, 9.0, 0.18765, 0, 0, 0, 0, 0, 0
total, 106184, 19298, 19298, 19298, 1.6, 1.2, 3.1, 10.8, 57.8, 61.5, 10.0, 0.17614, 0, 0, 0, 0, 0, 0
total, 128877, 22693, 22693, 22693, 1.4, 1.2, 2.6, 4.1, 14.7, 30.2, 11.0, 0.16866, 0, 0, 0, 0, 0, 0
total, 148331, 19454, 19454, 19454, 1.6, 1.3, 3.0, 5.7, 45.3, 47.7, 12.0, 0.15466, 0, 0, 0, 0, 0, 0
total, 156974, 8643, 8643, 8643, 1.6, 1.2, 3.3, 10.2, 24.6, 41.2, 13.0, 0.14813, 0, 0, 0, 0, 0, 0
total, 170654, 13680, 13680, 13680, 3.6, 1.3, 3.1, 9.2, 890.8, 896.5, 14.0, 0.39489, 0, 0, 0, 0, 0, 0
total, 187715, 17061, 17061, 17061, 1.9, 1.2, 4.5, 13.0, 27.2, 44.2, 15.0, 0.37273, 0, 0, 0, 0, 0, 0
total, 205718, 18003, 18003, 18003, 1.7, 1.3, 3.7, 11.3, 32.8, 53.4, 16.0, 0.35161, 0, 0, 0, 0, 0, 0
total, 212945, 7227, 7227, 7227, 1.8, 1.2, 3.2, 11.2, 71.3, 88.5, 17.0, 0.34545, 0, 0, 0, 0, 0, 0
total, 220825, 7880, 7880, 7880, 6.4, 1.2, 3.1, 11.1, 1200.6, 1209.0, 18.0, 0.42966, 0, 0, 0, 0, 0, 0
total, 241273, 20448, 20448, 20448, 1.5, 1.2, 3.1, 7.3, 23.0, 60.4, 19.0, 0.40131, 0, 0, 0, 0, 0, 0
total, 261889, 20616, 20616, 20616, 1.5, 1.2, 2.9, 8.0, 25.0, 33.5, 20.0, 0.37691, 0, 0, 0, 0, 0, 0
total, 282464, 20575, 20575, 20575, 1.5, 1.2, 2.7, 5.2, 68.6, 72.7, 21.0, 0.35456, 0, 0, 0, 0, 0, 0
total, 305429, 22965, 22965, 22965, 1.4, 1.2, 2.5, 3.8, 17.2, 23.8, 22.0, 0.33370, 0, 0, 0, 0, 0, 0
total, 326504, 21075, 21075, 21075, 1.5, 1.2, 2.9, 5.6, 17.9, 31.5, 23.0, 0.31637, 0, 0, 0, 0, 0, 0
total, 342653, 16149, 16149, 16149, 1.9, 1.3, 3.8, 13.2, 107.0, 124.4, 24.0, 0.30293, 0, 0, 0, 0, 0, 0
total, 360972, 18319, 18319, 18319, 1.7, 1.2, 3.8, 11.5, 36.1, 63.7, 25.0, 0.28988, 0, 0, 0, 0, 0, 0
total, 379092, 18120, 18120, 18120, 1.7, 1.2, 2.9, 8.2, 133.5, 161.1, 26.0, 0.27680, 0, 0, 0, 0, 0, 0
total, 400379, 21287, 21287, 21287, 1.5, 1.2, 2.9, 6.4, 30.8, 43.2, 27.0, 0.26454, 0, 0, 0, 0, 0, 0
total, 422330, 21951, 21951, 21951, 1.4, 1.2, 2.6, 4.3, 21.9, 35.3, 28.0, 0.25311, 0, 0, 0, 0, 0, 0
total, 442491, 20161, 20161, 20161, 1.4, 1.2, 2.7, 5.0, 21.6, 41.5, 29.0, 0.24331, 0, 0, 0, 0, 0, 0
total, 456035, 13544, 13544, 13544, 11.9, 1.3, 3.4, 9.7, 4366.3, 4366.3, 34.0, 0.25639, 0, 0, 0, 0, 0, 0
total, 458788, 2753, 2753, 2753, 1.5, 1.3, 2.9, 4.2, 32.8, 40.9, 35.0, 0.25612, 0, 0, 0, 0, 0, 0
total, 468346, 9558, 9558, 9558, 46.4, 1.4, 7.1, 15.6, 13203.7, 13228.8, 48.0, 0.27131, 0, 0, 0, 0, 0, 0
total, 489463, 21117, 21117, 21117, 1.5, 1.2, 2.8, 8.5, 19.7, 32.8, 49.0, 0.26143, 0, 0, 0, 0, 0, 0
total, 510484, 21021, 21021, 21021, 1.5, 1.2, 2.9, 5.7, 20.2, 28.0, 50.0, 0.25232, 0, 0, 0, 0, 0, 0
total, 532895, 22411, 22411, 22411, 1.4, 1.2, 2.5, 4.8, 28.4, 40.7, 51.0, 0.24335, 0, 0, 0, 0, 0, 0
total, 553656, 20761, 20761, 20761, 1.5, 1.2, 3.0, 9.0, 18.2, 33.0, 52.0, 0.23551, 0, 0, 0, 0, 0, 0
xfs mount and ops/s look like this:
/dev/sdb1 on /mnt/resource type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
Connected to cluster: avcassandra, max pending requests per connection 128, max connections per host 128
Datatacenter: dc1; Host: /10.0.0.4; Rack: rack0
Created keyspaces. Sleeping 1s for propagation.
Sleeping 2s...
Running WRITE with 32 threads for 2000000 iteration
Failed to connect over JMX; not collecting these stats
type total ops, op/s, pk/s, row/s, mean, med, .95, .99, .999, max, time, stderr, errors, gc: #, max ms, sum ms, sdv ms, mb
total, 1550, 1550, 1550, 1550, 14.9, 12.2, 37.8, 61.5, 75.6, 82.2, 1.0, 0.00000, 0, 0, 0, 0, 0, 0
total, 5742, 4192, 4192, 4192, 7.6, 4.6, 23.4, 36.7, 63.0, 78.8, 2.0, 0.32434, 0, 0, 0, 0, 0, 0
total, 10026, 4284, 4284, 4284, 7.4, 5.1, 20.4, 33.5, 95.2, 112.1, 3.0, 0.22200, 0, 0, 0, 0, 0, 0
total, 15802, 5776, 5776, 5776, 5.5, 3.2, 17.0, 32.8, 50.5, 62.6, 4.0, 0.18920, 0, 0, 0, 0, 0, 0
total, 23728, 7926, 7926, 7926, 4.0, 2.6, 11.6, 23.2, 54.7, 90.6, 5.0, 0.19722, 0, 0, 0, 0, 0, 0
total, 32143, 8415, 8415, 8415, 3.7, 2.5, 11.8, 20.4, 40.8, 48.1, 6.0, 0.17557, 0, 0, 0, 0, 0, 0
total, 41506, 9363, 9363, 9363, 3.4, 2.0, 11.0, 26.1, 72.2, 84.2, 7.0, 0.16341, 0, 0, 0, 0, 0, 0
total, 52696, 11190, 11190, 11190, 2.8, 2.6, 5.0, 10.8, 20.1, 32.3, 8.0, 0.15527, 0, 0, 0, 0, 0, 0
total, 63707, 11011, 11011, 11011, 2.9, 2.6, 4.6, 11.2, 20.7, 62.6, 9.0, 0.14386, 0, 0, 0, 0, 0, 0
total, 74246, 10539, 10539, 10539, 3.0, 2.7, 5.2, 11.9, 41.4, 46.3, 10.0, 0.13055, 0, 0, 0, 0, 0, 0
total, 84693, 10447, 10447, 10447, 3.0, 2.8, 5.0, 7.2, 13.6, 30.0, 11.0, 0.11879, 0, 0, 0, 0, 0, 0
total, 95304, 10611, 10611, 10611, 3.0, 2.8, 5.0, 7.2, 14.6, 32.6, 12.0, 0.10912, 0, 0, 0, 0, 0, 0
total, 105865, 10561, 10561, 10561, 3.0, 2.7, 4.8, 10.4, 53.1, 60.6, 13.0, 0.10111, 0, 0, 0, 0, 0, 0
total, 116431, 10566, 10566, 10566, 3.0, 2.8, 5.0, 8.9, 14.8, 25.2, 14.0, 0.09380, 0, 0, 0, 0, 0, 0
total, 127856, 11425, 11425, 11425, 2.8, 2.6, 4.5, 6.8, 16.6, 24.4, 15.0, 0.08816, 0, 0, 0, 0, 0, 0
total, 139136, 11280, 11280, 11280, 2.8, 2.6, 4.4, 6.5, 17.6, 20.6, 16.0, 0.08292, 0, 0, 0, 0, 0, 0
total, 149943, 10807, 10807, 10807, 2.9, 2.7, 4.5, 6.1, 50.0, 52.2, 17.0, 0.07817, 0, 0, 0, 0, 0, 0
total, 161338, 11395, 11395, 11395, 2.8, 2.6, 4.8, 10.1, 18.4, 30.0, 18.0, 0.07411, 0, 0, 0, 0, 0, 0
total, 172127, 10789, 10789, 10789, 2.9, 2.6, 5.3, 12.3, 41.9, 53.9, 19.0, 0.07025, 0, 0, 0, 0, 0, 0
total, 182078, 9951, 9951, 9951, 3.2, 2.1, 10.9, 20.9, 36.3, 80.7, 20.0, 0.06662, 0, 0, 0, 0, 0, 0
total, 193140, 11062, 11062, 11062, 2.9, 2.5, 5.3, 12.0, 29.0, 44.7, 21.0, 0.06354, 0, 0, 0, 0, 0, 0
total, 203536, 10396, 10396, 10396, 3.1, 2.6, 5.2, 11.7, 60.6, 68.7, 22.0, 0.06061, 0, 0, 0, 0, 0, 0
total, 214575, 11039, 11039, 11039, 2.9, 2.6, 5.0, 10.1, 22.5, 36.2, 23.0, 0.05801, 0, 0, 0, 0, 0, 0
total, 225072, 10497, 10497, 10497, 3.0, 2.6, 5.6, 13.2, 29.2, 48.2, 24.0, 0.05553, 0, 0, 0, 0, 0, 0
total, 235155, 10083, 10083, 10083, 3.1, 2.7, 4.8, 9.9, 89.3, 102.6, 25.0, 0.05329, 0, 0, 0, 0, 0, 0
total, 245257, 10102, 10102, 10102, 3.1, 2.6, 5.2, 11.4, 65.6, 67.3, 26.0, 0.05118, 0, 0, 0, 0, 0, 0
total, 256802, 11545, 11545, 11545, 2.7, 2.6, 4.5, 8.2, 18.6, 36.6, 27.0, 0.04946, 0, 0, 0, 0, 0, 0
total, 267959, 11157, 11157, 11157, 2.8, 2.6, 4.6, 9.2, 20.9, 69.1, 28.0, 0.04784, 0, 0, 0, 0, 0, 0
total, 279085, 11126, 11126, 11126, 2.8, 2.6, 4.6, 8.2, 16.0, 30.8, 29.0, 0.04620, 0, 0, 0, 0, 0, 0
total, 289773, 10688, 10688, 10688, 3.0, 2.7, 4.9, 7.9, 19.0, 35.5, 30.0, 0.04463, 0, 0, 0, 0, 0, 0
total, 299929, 10156, 10156, 10156, 3.1, 2.7, 4.7, 10.8, 87.3, 90.8, 31.0, 0.04316, 0, 0, 0, 0, 0, 0
total, 310716, 10787, 10787, 10787, 2.9, 2.7, 5.0, 8.9, 16.3, 29.2, 32.0, 0.04179, 0, 0, 0, 0, 0, 0
total, 320931, 10215, 10215, 10215, 2.8, 2.6, 4.5, 7.2, 18.4, 25.5, 33.0, 0.04050, 0, 0, 0, 0, 0, 0
total, 322021, 1090, 1090, 1090, 31.9, 3.1, 11.0, 974.1, 984.6, 985.1, 34.0, 0.15458, 0, 0, 0, 0, 0, 0
total, 333309, 11288, 11288, 11288, 2.8, 2.6, 4.7, 9.9, 21.0, 25.4, 35.0, 0.15035, 0, 0, 0, 0, 0, 0
total, 344239, 10930, 10930, 10930, 2.9, 2.6, 5.0, 9.9, 39.6, 54.8, 36.0, 0.14636, 0, 0, 0, 0, 0, 0
total, 355560, 11321, 11321, 11321, 2.8, 2.6, 4.7, 9.3, 22.8, 41.4, 37.0, 0.14249, 0, 0, 0, 0, 0, 0
total, 364990, 9430, 9430, 9430, 3.4, 2.6, 5.0, 12.3, 141.3, 158.2, 38.0, 0.13902, 0, 0, 0, 0, 0, 0
total, 376297, 11307, 11307, 11307, 2.8, 2.6, 4.5, 8.6, 21.8, 24.0, 39.0, 0.13559, 0, 0, 0, 0, 0, 0
total, 387651, 11354, 11354, 11354, 2.8, 2.5, 4.7, 8.9, 20.3, 26.4, 40.0, 0.13230, 0, 0, 0, 0, 0, 0
total, 394128, 6477, 6477, 6477, 2.7, 2.6, 4.3, 5.8, 12.0, 14.0, 41.0, 0.13099, 0, 0, 0, 0, 0, 0
total, 396263, 2135, 2135, 2135, 21.6, 2.7, 4.7, 1256.2, 1258.3, 1259.3, 42.0, 0.13955, 0, 0, 0, 0, 0, 0
total, 407132, 10869, 10869, 10869, 2.9, 2.6, 4.7, 10.1, 18.8, 24.2, 43.0, 0.13638, 0, 0, 0, 0, 0, 0
total, 418276, 11144, 11144, 11144, 2.8, 2.6, 4.6, 9.0, 15.6, 26.1, 44.0, 0.13327, 0, 0, 0, 0, 0, 0
total, 429779, 11503, 11503, 11503, 2.8, 2.6, 4.3, 5.6, 13.4, 22.0, 45.0, 0.13022, 0, 0, 0, 0, 0, 0
total, 439888, 10109, 10109, 10109, 3.1, 2.6, 4.5, 9.7, 106.7, 108.4, 46.0, 0.12741, 0, 0, 0, 0, 0, 0
total, 450854, 10966, 10966, 10966, 2.9, 2.6, 4.7, 8.5, 21.0, 40.6, 47.0, 0.12470, 0, 0, 0, 0, 0, 0
total, 462404, 11550, 11550, 11550, 2.7, 2.6, 4.3, 5.9, 11.4, 19.3, 48.0, 0.12202, 0, 0, 0, 0, 0, 0
total, 472649, 10245, 10245, 10245, 2.7, 2.5, 4.3, 8.8, 13.8, 43.7, 49.0, 0.11969, 0, 0, 0, 0, 0, 0
total, 476297, 3648, 3648, 3648, 9.8, 2.7, 4.3, 12.1, 804.3, 805.3, 50.0, 0.11649, 0, 0, 0, 0, 0, 0
total, 487422, 11125, 11125, 11125, 2.8, 2.7, 4.5, 6.5, 16.0, 32.0, 51.0, 0.11421, 0, 0, 0, 0, 0, 0
total, 498707, 11285, 11285, 11285, 2.8, 2.6, 4.4, 8.9, 22.1, 32.6, 52.0, 0.11200, 0, 0, 0, 0, 0, 0
total, 509903, 11196, 11196, 11196, 2.8, 2.6, 4.5, 9.0, 16.5, 33.3, 53.0, 0.10988, 0, 0, 0, 0, 0, 0