[jira] [Commented] (CASSANDRA-15512) Add compaction-related JMH tests

["Blake Eggleston (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-15512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041353#comment-17041353 ] 

Blake Eggleston commented on CASSANDRA-15512:
---------------------------------------------

This is really cool. I’ve spent some time working on the parameter space that these benchmarks work on, the run time, and playing around with ways to organize the large output into something that can be summarized and explored a bit more easily.

My branch is here: https://github.com/bdeggleston/cassandra/tree/15512-trunk, let me know what you think. Changes described below.

*Workload Normalization*

Instead of individual parameter ranges for partition, row, and column counts, there’s now a single parameter (concentration) that sweeps from 0.0 to 1.0, and the partition/row/column counts are derived from that. As this parameter changes, the values of partitions, rows, and columns change, but the result of (partitions * rows * columns) stays about the same, so you’re (in theory) comparing the same volume of work, just done by different parts of the read path.

This has been really useful for getting a high level picture of the performance impact of a set of changes over a variety of workloads. That said, it does this at the expense of seeing how more extreme workloads perform, or comparing the throughput of workloads moving different amounts of data (measuring column index overhead, for example), but that seems like something we should add later, instead of trying to also do it here.

*Result Visualization*

I added a script that generates a heat map of the relative throughput values for each parameter permutation, as well as the difference between 2 runs for a quick before and after comparison. By graphing the concentration on the x axis, and column_type -> value size -> overlap on the y axis, you can quickly get an idea of how things change, both per run and between them.

*Other Stuff*

I added a micro bench test script that sets up the JVM in about the same way as the server config. There may be a better way to do this, but this has worked fine for me.

There's also a handful of review fix style changes:
 * I fixed a few bugs I found
 * Switched assert in CompactionController ctor to use Preconditions.
 * added @VisibleForTesting annotation to PartitionUpdate#unsafeConstruct
 * cleaned up some imports

*Misc WIP*

There are also a handful of tools that aren't really at ready-to-commit levels of polish, but that have been useful. Some of these may be worth committing in their incomplete state so that they will continue to improve as part of our performance tooling. I might also be able to put them in a repo where they could be worked on until they're "good enough"
 * A profiler that includes jvm methods (ie: gc), and kernel activity in the results. It uses the profile tool from bcc-tools, modified to extract symbols for jit compiled methods from the jvm before dumping its data. No idea if it would be possible/practical to eventually be able to map interpreter stack frames to their java class/method. Anyway, it's still pretty rough, and needs a bit of setup, but helps put things in perspective when working on things that are focused on reducing gc time. It's basically like perf record, except the sampled data is aggregated in the kernel, and data is only sent to user space at the end of the profiling session.
 * A perf profiler that profiles cpu performance counters, which I used to profile branch misses. I'd like to rewrite this as a bpf script, possibly as an extension to the profiler I was just talking about, or modify stackcount from bcc-tools.
 * Flame graph generator. I'd originally written this so I could embed comparative flame graphs in the heatmap for each run. Unfortunately the generator is too slow, and it's output is too large for this to be currently practical. The comparative part also wasn't as useful as I'd hoped.

> Add compaction-related JMH tests
> --------------------------------
>
>                 Key: CASSANDRA-15512
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-15512
>             Project: Cassandra
>          Issue Type: Sub-task
>          Components: Local/Compaction
>            Reporter: Benedict Elliott Smith
>            Assignee: Benedict Elliott Smith
>            Priority: Normal
>
> Following up CASSANDRA-15388, we should introduce some isolated JMH tests covering at least the compaction merge path and partition deserialization, covering a wide range of partition shapes and characteristics.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org