[jira] [Commented] (CASSANDRA-8986) Major cassandra-stress refactor

["Benedict (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14727218#comment-14727218 ] 

Benedict commented on CASSANDRA-8986:
-------------------------------------

I think for consistency of testing, and for realism of read workloads, we need stress to be able to build sstables directly to serve to each node when bootstrapping a test cluster, which has its compactions disabled. We can then specify the distribution of data amongst these sstables, so that we produce data that looks like a real cluster with live data might produce. Currently it is very hard to get a consistent state across different versions of a cluster for performing read tests, and replicating a realistic DTCS cluster state (for instance) is hard since we don't run for days, weeks or months (and artificially lowering the windows doesn't necessarily give us a realistic cluster state).

I don't propose that this all be delivered as part of the rewrite, but I note it here to make certain it is considered at the same time, hopefully to be delivered soon after.

> Major cassandra-stress refactor
> -------------------------------
>
>                 Key: CASSANDRA-8986
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8986
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Tools
>            Reporter: Benedict
>            Assignee: Benedict
>
> We need a tool for both stressing _and_ validating more complex workloads than stress currently supports. Stress needs a raft of changes, and I think it would be easier to deliver many of these as a single major endeavour which I think is justifiable given its audience. The rough behaviours I want stress to support are:
> * Ability to know exactly how many rows it will produce, for any clustering prefix, without generating those prefixes
> * Ability to generate an amount of data proportional to the amount it will produce to the server (or consume from the server), rather than proportional to the variation in clustering columns
> * Ability to reliably produce near identical behaviour each run
> * Ability to understand complex overlays of operation types (LWT, Delete, Expiry, although perhaps not all implemented immediately, the framework for supporting them easily)
> * Ability to (with minimal internal state) understand the complete cluster state through overlays of multiple procedural generations
> * Ability to understand the in-flight state of in-progress operations (i.e. if we're applying a delete, understand that the delete may have been applied, and may not have been, for potentially multiple conflicting in flight operations)
> I think the necessary changes to support this would give us the _functional_ base to support all the functionality I can currently envisage stress needing. Before embarking on this (which I may attempt very soon), it would be helpful to get input from others as to features missing from stress that I haven't covered here that we will certainly want in the future, so that they can be factored in to the overall design and hopefully avoid another refactor one year from now, as its complexity is scaling each time, and each time it is a higher sunk cost. [~jbellis] [~iamaleksey] [~slebresne] [~tjake] [~enigmacurry] [~aweisberg] [~blambov] [~jshook] ... and @everyone else :) 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)