You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Sam Lightfoot (Jira)" <ji...@apache.org> on 2025/04/15 20:16:00 UTC
[jira] [Created] (CASSANDRA-20560) Incremental Repair stuck with 'STARTED' repair sessions due to dropped messages
Sam Lightfoot created CASSANDRA-20560:
-----------------------------------------
Summary: Incremental Repair stuck with 'STARTED' repair sessions due to dropped messages
Key: CASSANDRA-20560
URL: https://issues.apache.org/jira/browse/CASSANDRA-20560
Project: Apache Cassandra
Issue Type: Improvement
Components: Consistency/Repair
Reporter: Sam Lightfoot
On running initial incremental repairs via the Reaper we were repeatedly seeing repairs hang indefinitely with sessions in 'STARTED' state. Per logs, the issue occurred due to dropped messages between a remote replica whilst it was sending "large messages" to the repair coordinator on the AntiEntropyStage executor.
{code:java}
"WARNÂ [AntiEntropyStage:1] 2024-02-07 14:46:51,256 NoSpamLogger.java:95 - x.63.94.y:10700->x.68.75.z:10700-LARGE_MESSAGES-adbea270 overloaded; dropping 14.097MiB message (queue: 131.855MiB local, 127.855MiB endpoint, 127.855MiB global)" {code}
We were able to resolve these issues by setting concurrent Merkle tree requests to 1, but I assume there's a gap in the recovery logic that meant this repair could not be recovered. Perhaps a missing condition within the retry logic in RepairMessage.
This happens whilst running v5.0.3 on all 72 nodes across 4 DCs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org