You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Philip Thompson (JIRA)" <ji...@apache.org> on 2016/06/23 18:23:18 UTC
[jira] [Commented] (CASSANDRA-7019) Improve tombstone compactions
[ https://issues.apache.org/jira/browse/CASSANDRA-7019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15346911#comment-15346911 ]
Philip Thompson commented on CASSANDRA-7019:
--------------------------------------------
Here is the relevant portion of the stress outputs:
CONTROL:
{code}
Results:
Op rate : 21,050 op/s [columndelete: 1,403 op/s, delete: 702 op/s, insert: 10,524 op/s, read: 7,017 op/s, rowdelete: 1,404 op/s]
Partition rate : 17,540 pk/s [columndelete: 0 pk/s, delete: 0 pk/s, insert: 10,524 pk/s, read: 7,016 pk/s, rowdelete: 0 pk/s]
Row rate : 43,872 row/s [columndelete: 0 row/s, delete: 0 row/s, insert: 26,324 row/s, read: 17,548 row/s, rowdelete: 0 row/s]
Latency mean : 2.4 ms [columndelete: 2.2 ms, delete: 2.1 ms, insert: 2.2 ms, read: 2.7 ms, rowdelete: 2.1 ms]
Latency median : 2.0 ms [columndelete: 1.8 ms, delete: 1.8 ms, insert: 1.9 ms, read: 2.3 ms, rowdelete: 1.8 ms]
Latency 95th percentile : 3.6 ms [columndelete: 3.3 ms, delete: 3.3 ms, insert: 3.4 ms, read: 4.1 ms, rowdelete: 3.3 ms]
Latency 99th percentile : 5.5 ms [columndelete: 4.8 ms, delete: 4.8 ms, insert: 4.9 ms, read: 7.2 ms, rowdelete: 4.7 ms]
Latency 99.9th percentile : 75.2 ms [columndelete: 69.9 ms, delete: 69.9 ms, insert: 72.3 ms, read: 78.6 ms, rowdelete: 69.3 ms]
Latency max : 1032.3 ms [columndelete: 1,004.5 ms, delete: 1,004.5 ms, insert: 1,031.8 ms, read: 1,032.3 ms, rowdelete: 1,003.5 ms]
Total partitions : 378,840,394 [columndelete: 0, delete: 0, insert: 227,304,928, read: 151,535,466, rowdelete: 0]
Total errors : 0 [columndelete: 0, delete: 0, insert: 0, read: 0, rowdelete: 0]
Total GC count : 13,090
Total GC memory : 15900.717 GiB
Total GC time : 998.3 seconds
Avg GC time : 76.3 ms
StdDev GC time : 16.1 ms
Total operation time : 05:59:58
{code}
NONE:
{code}
Results:
Op rate : 20,729 op/s [columndelete: 1,382 op/s, delete: 690 op/s, insert: 10,366 op/s, read: 6,909 op/s, rowdelete: 1,382 op/s]
Partition rate : 17,274 pk/s [columndelete: 0 pk/s, delete: 0 pk/s, insert: 10,366 pk/s, read: 6,908 pk/s, rowdelete: 0 pk/s]
Row rate : 43,206 row/s [columndelete: 0 row/s, delete: 0 row/s, insert: 25,929 row/s, read: 17,277 row/s, rowdelete: 0 row/s]
Latency mean : 2.4 ms [columndelete: 2.1 ms, delete: 2.1 ms, insert: 2.2 ms, read: 2.9 ms, rowdelete: 2.1 ms]
Latency median : 1.9 ms [columndelete: 1.7 ms, delete: 1.7 ms, insert: 1.8 ms, read: 2.2 ms, rowdelete: 1.7 ms]
Latency 95th percentile : 3.3 ms [columndelete: 2.9 ms, delete: 2.9 ms, insert: 3.0 ms, read: 3.7 ms, rowdelete: 2.9 ms]
Latency 99th percentile : 4.7 ms [columndelete: 4.1 ms, delete: 4.1 ms, insert: 4.2 ms, read: 6.0 ms, rowdelete: 4.1 ms]
Latency 99.9th percentile : 47.6 ms [columndelete: 12.0 ms, delete: 13.0 ms, insert: 14.7 ms, read: 67.9 ms, rowdelete: 13.4 ms]
Latency max : 1055.6 ms [columndelete: 1,006.0 ms, delete: 1,004.4 ms, insert: 1,055.6 ms, read: 1,055.6 ms, rowdelete: 1,031.9 ms]
Total partitions : 373,111,699 [columndelete: 0, delete: 0, insert: 223,905,059, read: 149,206,640, rowdelete: 0]
Total errors : 0 [columndelete: 0, delete: 0, insert: 0, read: 0, rowdelete: 0]
Total GC count : 14,082
Total GC memory : 17120.316 GiB
Total GC time : 1,005.7 seconds
Avg GC time : 71.4 ms
StdDev GC time : 13.5 ms
Total operation time : 06:00:00
{code}
ROW:
Results:
Op rate : 16,121 op/s [columndelete: 1,075 op/s, delete: 538 op/s, insert: 8,061 op/s, read: 5,372 op/s, rowdelete: 1,074 op/s]
Partition rate : 13,432 pk/s [columndelete: 0 pk/s, delete: 0 pk/s, insert: 8,061 pk/s, read: 5,371 pk/s, rowdelete: 0 pk/s]
Row rate : 33,597 row/s [columndelete: 0 row/s, delete: 0 row/s, insert: 20,165 row/s, read: 13,433 row/s, rowdelete: 0 row/s]
Latency mean : 3.1 ms [columndelete: 2.3 ms, delete: 2.3 ms, insert: 2.4 ms, read: 4.5 ms, rowdelete: 2.3 ms]
Latency median : 2.3 ms [columndelete: 1.8 ms, delete: 1.7 ms, insert: 1.8 ms, read: 3.5 ms, rowdelete: 1.8 ms]
Latency 95th percentile : 5.3 ms [columndelete: 3.8 ms, delete: 3.8 ms, insert: 3.8 ms, read: 6.8 ms, rowdelete: 3.8 ms]
Latency 99th percentile : 8.4 ms [columndelete: 6.1 ms, delete: 6.2 ms, insert: 6.2 ms, read: 10.9 ms, rowdelete: 6.3 ms]
Latency 99.9th percentile : 51.4 ms [columndelete: 16.4 ms, delete: 15.7 ms, insert: 14.7 ms, read: 58.7 ms, rowdelete: 41.4 ms]
Latency max : 1053.8 ms [columndelete: 1,003.5 ms, delete: 1,053.7 ms, insert: 1,053.7 ms, read: 1,053.8 ms, rowdelete: 1,006.9 ms]
Total partitions : 290,123,117 [columndelete: 0, delete: 0, insert: 174,120,120, read: 116,002,997, rowdelete: 0]
Total errors : 0 [columndelete: 0, delete: 0, insert: 0, read: 0, rowdelete: 0]
Total GC count : 16,954
Total GC memory : 20848.789 GiB
Total GC time : 819.7 seconds
Avg GC time : 48.3 ms
StdDev GC time : 10.9 ms
Total operation time : 05:59:58
{code}
CELL:
{code}
Results:
Op rate : 5,876 op/s [columndelete: 392 op/s, delete: 196 op/s, insert: 2,937 op/s, read: 1,959 op/s, rowdelete: 392 op/s]
Partition rate : 4,896 pk/s [columndelete: 0 pk/s, delete: 0 pk/s, insert: 2,937 pk/s, read: 1,959 pk/s, rowdelete: 0 pk/s]
Row rate : 12,247 row/s [columndelete: 0 row/s, delete: 0 row/s, insert: 7,348 row/s, read: 4,899 row/s, rowdelete: 0 row/s]
Latency mean : 8.5 ms [columndelete: 3.3 ms, delete: 3.3 ms, insert: 3.4 ms, read: 18.8 ms, rowdelete: 3.3 ms]
Latency median : 3.2 ms [columndelete: 1.7 ms, delete: 1.7 ms, insert: 1.7 ms, read: 14.7 ms, rowdelete: 1.7 ms]
Latency 95th percentile : 25.6 ms [columndelete: 8.0 ms, delete: 7.9 ms, insert: 8.2 ms, read: 34.5 ms, rowdelete: 8.0 ms]
Latency 99th percentile : 39.6 ms [columndelete: 13.5 ms, delete: 13.6 ms, insert: 14.2 ms, read: 49.2 ms, rowdelete: 13.6 ms]
Latency 99.9th percentile : 65.2 ms [columndelete: 35.2 ms, delete: 34.4 ms, insert: 34.5 ms, read: 76.2 ms, rowdelete: 31.3 ms]
Latency max : 1037.8 ms [columndelete: 208.5 ms, delete: 1,002.7 ms, insert: 1,006.2 ms, read: 1,037.8 ms, rowdelete: 231.9 ms]
Total partitions : 105,729,389 [columndelete: 0, delete: 0, insert: 63,434,249, read: 42,295,140, rowdelete: 0]
Total errors : 0 [columndelete: 0, delete: 0, insert: 0, read: 0, rowdelete: 0]
Total GC count : 14,437
Total GC memory : 17937.624 GiB
Total GC time : 348.4 seconds
Avg GC time : 24.1 ms
StdDev GC time : 7.1 ms
Total operation time : 05:59:54
{code}
All were generated with the following stress command: {{cassandra-stress user duration=6h cl=LOCAL_ONE profile=https://gist.githubusercontent.com/ptnapoleon/ce59cc5a2a42ecfd3d27657f311e4615/raw/fc85d9d715eb6ed87dba10796e505e2b5ef2c7f8/7019-none.yaml ops\(insert=15,read=10,delete=1,rowdelete=2,columndelete=2\) -rate threads=50 -pop seq=1..750000000 -node 172.31.9.18 -graph file=/home/automaton/fallout_artifacts/delete_heavy-graph.html | tee /home/automaton/fallout_artifacts/stress-delete_heavy.log "}} with the obvious disparity that the linked yaml file changes the relevant compaction option in the schema.
As you can see, the {{none}} and {{control}} options are very similar. {{row}} sees a ~20% dip in perf, and {{cell}} sees an incredibly large one, roughly 75% of throughput is lost.
> Improve tombstone compactions
> -----------------------------
>
> Key: CASSANDRA-7019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7019
> Project: Cassandra
> Issue Type: Improvement
> Components: Compaction
> Reporter: Marcus Eriksson
> Assignee: Branimir Lambov
> Labels: compaction, fallout
> Fix For: 3.x
>
> Attachments: 7019-2-system.log, 7019-debug.log, cell.tar.gz, control.tar.gz, none.tar.gz, row.tar.gz, temp-plot.html
>
>
> When there are no other compactions to do, we trigger a single-sstable compaction if there is more than X% droppable tombstones in the sstable.
> In this ticket we should try to include overlapping sstables in those compactions to be able to actually drop the tombstones. Might only be doable with LCS (with STCS we would probably end up including all sstables)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)