You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Vincent White (JIRA)" <ji...@apache.org> on 2018/03/01 23:18:00 UTC
[jira] [Commented] (CASSANDRA-13797) RepairJob blocks on syncTasks
[ https://issues.apache.org/jira/browse/CASSANDRA-13797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16382839#comment-16382839 ]
Vincent White commented on CASSANDRA-13797:
-------------------------------------------
Now that we don't wait for the validations of each repair job to finish before moving onto the next one, I don't see anything to stop the repair coordinator from spinning through all the token ranges and effectively triggering all the validations tasks at once, which could be a significant amount of validation compactions on each node depending on your topology and common ranges for that keyspace. I'm also not sure of the overhead of creating all the futures/listeners on the coordinator at once in this case.
In 3 the validation executor thread pool has no size limit so a new validation is started as soon as a validation request is received. I admit I haven't caught up on the changes to repair in trunk, and while the validation executor pool size is configurable in trunk, its default is still Integer.MAX_VALUE.
I understand this same affect (hundreds of concurrent validations) can still happen if you trigger a repair across a keyspace with a large number of column families but with this change there is no way of avoiding it without using subrange repairs on a single column family (if you have a topology/replication that cant be merged into a small number of common ranges) .
> RepairJob blocks on syncTasks
> -----------------------------
>
> Key: CASSANDRA-13797
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13797
> Project: Cassandra
> Issue Type: Bug
> Components: Repair
> Reporter: Blake Eggleston
> Assignee: Blake Eggleston
> Priority: Major
> Fix For: 3.0.15, 3.11.1, 4.0
>
>
> The thread running {{RepairJob}} blocks while it waits for the validations it starts to complete ([see here|https://github.com/bdeggleston/cassandra/blob/9fdec0a82851f5c35cd21d02e8c4da8fc685edb2/src/java/org/apache/cassandra/repair/RepairJob.java#L185]). However, the downstream callbacks (ie: the post-repair cleanup stuff) aren't waiting for {{RepairJob#run}} to return, they're waiting for a result to be set on RepairJob the future, which happens after the sync tasks have completed. This post repair cleanup stuff also immediately shuts down the executor {{RepairJob#run}} is running in. So in noop repair sessions, where there's nothing to stream, I'm seeing the callbacks sometimes fire before {{RepairJob#run}} wakes up, and causing an {{InterruptedException}} is thrown.
> I'm pretty sure this can just be removed, but I'd like a second opinion. This appears to just be a holdover from before repair coordination became async. I thought it might be doing some throttling by blocking, but each repair session gets it's own executor, and validation is throttled by the fixed size executors doing the actual work of validation, so I don't think we need to keep this around.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org