[jira] [Commented] (CASSANDRA-17781) Secondary index building should not block compaction progress

["Josh McKenzie (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-17781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572034#comment-17572034 ] 

Josh McKenzie commented on CASSANDRA-17781:
-------------------------------------------

||Item|Link|
|PR|[link|https://github.com/apache/cassandra/pull/1761]|
|JDK8 CI|[link|https://app.circleci.com/pipelines/github/josh-mckenzie/cassandra/251/workflows/6d539e3e-4517-4c13-9442-9cef083086d1]|
|JDK11 CI|[link|https://app.circleci.com/pipelines/github/josh-mckenzie/cassandra/251/workflows/2c8859be-2e35-40e0-a392-2efa132e5259]|

No new testing added; relying on existing 2i building tests (unit, in-jvm, and python dtests) to exercise this code sufficiently. Also pretty straightforward patch and has been running for a good long while without issue.

> Secondary index building should not block compaction progress
> -------------------------------------------------------------
>
>                 Key: CASSANDRA-17781
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-17781
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Feature/2i Index
>            Reporter: Josh McKenzie
>            Assignee: Josh McKenzie
>            Priority: Normal
>
> While CASSANDRA-11218 should also address this same problem, that ticket's a pretty big beautiful hammer and there's a fairly surgical modification we can make to the 2i building process to mitigate 2i building's ability to block compaction.
> While 11218 approaches this from a "lots of compactions can starve 2i build and things never finish", what I'd like to address here is the inverse that building multiple secondary indices in parallel can occupy all available compactors to a point where we don't have any left to do actual compactions. Completing 2i build can take several hours and the backlog of compactions from that time may result in too many uncompacted sstables left to catch up.
> A fairly simple fix is to break 2i building out to its own threadpool w/a hot proppable param you can use to change the # of threads for that build in the extreme cases where the default of 2 is indufficient.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org