You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Joey Lynch (Jira)" <ji...@apache.org> on 2022/02/14 15:36:00 UTC
[jira] [Updated] (CASSANDRA-17381) Produce and verify BoundedReadCompactionStrategy as a unified general purpose compaction algorithm
[ https://issues.apache.org/jira/browse/CASSANDRA-17381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joey Lynch updated CASSANDRA-17381:
-----------------------------------
Description:
The existing compaction strategies have a number of drawbacks that make all three unsuitable as a general use compaction strategy, for example STCS creates giant files that are hard to back up, mess with the page cache, and led to many of the early re-open bugs. LCS improved dramatically on this but also has various issues e.g. lack of performant full compaction or due to the strict leveling with e.g. bulk loading when writes exceed the rate we can do the L0 - L1 promotion.
In this [talk|https://github.com/ngcc/ngcc2019/blob/master/NextGenerationCassandraCompactionGoingBeyondLCS.pdf] I introduced a novel compaction strategy that aims to expose a single tunable that the user can control for the read amplification. Raise the max_read_per_read and you tradeoff read/space performance for write performance. Since then a proof of concept [patch |https://github.com/jolynch/cassandra/tree/jolynch_bounded_read_final]has been published along with some rudimentary [documentation |https://gist.github.com/jolynch/9118465f32ad5298b4e39d03ccd4370e] but this is still not tracked in Jira.
The remaining work here is
1. Validate the algorithm is correct via test suites and performance testing stress testing and benchmarking with OSS tools (e.g. cassandra-stress, [tlp-stress|https://github.com/thelastpickle/tlp-stress], or [ndbench|https://github.com/Netflix/ndbench]). When issues are found (there likely will be issues as the patch is a PoC), devise how to adjust the algorithm and implementation appropriately. Key metric of success is we can run Cassandra stably for more than 24 hours while applying sustained load, and compaction can keep up.
2. Do more in depth experiments measuring performance across a wide range of workloads (e.g. write heavy, read heavy, balanced, time series, register update, etc ...) and in comparison with LCS (leveled), STCS (size tiered), and TWCS (time window). Key metrics of success are establishing that as we tune max_read_per_read we should get more predictable read latency under low system load (ρ < 30%) while not degrading at high system load (ρ > 70%), and we should match LCS performance under low load while doing better at high load.
Once this is validated a Cassandra blog post reporting on the findings (positive or negative) may be advisable.
was:
The existing compaction strategies have a number of drawbacks that make all three unsuitable as a general use compaction strategy, for example STCS creates giant files that are hard to back up, mess with the page cache, and led to many of the early re-open bugs. LCS improved dramatically on this but also has various issues e.g. lack of performant full compaction or due to the strict leveling with e.g. bulk loading when writes exceed the rate we can do the L0 - L1 promotion.
In this [talk|https://github.com/ngcc/ngcc2019/blob/master/NextGenerationCassandraCompactionGoingBeyondLCS.pdf] I introduced a novel compaction strategy that aims to expose a single tunable that the user can control for the read amplification. Raise the max_read_per_read and you tradeoff read/space performance for write performance. Since then a proof of concept [patch |https://github.com/jolynch/cassandra/tree/jolynch_bounded_read_final]has been published along with some rudimentary [documentation |https://gist.github.com/jolynch/9118465f32ad5298b4e39d03ccd4370e] but this is still not tracked in Jira.
The remaining work here is
1. Validate the algorithm is correct via test suites and performance testing stress testing and benchmarking with OSS tools (e.g. cassandra-stress, [tlp-stress|https://github.com/thelastpickle/tlp-stress], or [ndbench|https://github.com/Netflix/ndbench]). When issues are found (there likely will be issues as the patch is a PoC), devise how to adjust the algorithm and implementation appropriately. Key metric of success is we can run Cassandra stably for more than 24 hours while applying sustained load, and compaction can keep up.
2. Do more in depth experiments measuring performance across a wide range of workloads (e.g. write heavy, read heavy, balanced, time series, register update, etc ...) and in comparison with LCS (leveled), STCS (size tiered), and TWCS (time window). Key metrics of success are establishing that as we tune max_read_per_read we should get more predictable read latency under low system load (ρ < 30%) while not degrading at high system load (ρ > 70%), and we should match LCS performance under low load while doing better at high load.
> Produce and verify BoundedReadCompactionStrategy as a unified general purpose compaction algorithm
> --------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-17381
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17381
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Compaction
> Reporter: Joey Lynch
> Assignee: Joey Lynch
> Priority: Normal
>
> The existing compaction strategies have a number of drawbacks that make all three unsuitable as a general use compaction strategy, for example STCS creates giant files that are hard to back up, mess with the page cache, and led to many of the early re-open bugs. LCS improved dramatically on this but also has various issues e.g. lack of performant full compaction or due to the strict leveling with e.g. bulk loading when writes exceed the rate we can do the L0 - L1 promotion.
> In this [talk|https://github.com/ngcc/ngcc2019/blob/master/NextGenerationCassandraCompactionGoingBeyondLCS.pdf] I introduced a novel compaction strategy that aims to expose a single tunable that the user can control for the read amplification. Raise the max_read_per_read and you tradeoff read/space performance for write performance. Since then a proof of concept [patch |https://github.com/jolynch/cassandra/tree/jolynch_bounded_read_final]has been published along with some rudimentary [documentation |https://gist.github.com/jolynch/9118465f32ad5298b4e39d03ccd4370e] but this is still not tracked in Jira.
> The remaining work here is
> 1. Validate the algorithm is correct via test suites and performance testing stress testing and benchmarking with OSS tools (e.g. cassandra-stress, [tlp-stress|https://github.com/thelastpickle/tlp-stress], or [ndbench|https://github.com/Netflix/ndbench]). When issues are found (there likely will be issues as the patch is a PoC), devise how to adjust the algorithm and implementation appropriately. Key metric of success is we can run Cassandra stably for more than 24 hours while applying sustained load, and compaction can keep up.
> 2. Do more in depth experiments measuring performance across a wide range of workloads (e.g. write heavy, read heavy, balanced, time series, register update, etc ...) and in comparison with LCS (leveled), STCS (size tiered), and TWCS (time window). Key metrics of success are establishing that as we tune max_read_per_read we should get more predictable read latency under low system load (ρ < 30%) while not degrading at high system load (ρ > 70%), and we should match LCS performance under low load while doing better at high load.
> Once this is validated a Cassandra blog post reporting on the findings (positive or negative) may be advisable.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org