You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Sam Lightfoot (Jira)" <ji...@apache.org> on 2025/04/15 20:16:00 UTC

[jira] [Created] (CASSANDRA-20560) Incremental Repair stuck with 'STARTED' repair sessions due to dropped messages

Sam Lightfoot created CASSANDRA-20560:
-----------------------------------------

             Summary: Incremental Repair stuck with 'STARTED' repair sessions due to dropped messages
                 Key: CASSANDRA-20560
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-20560
             Project: Apache Cassandra
          Issue Type: Improvement
          Components: Consistency/Repair
            Reporter: Sam Lightfoot


On running initial incremental repairs via the Reaper we were repeatedly seeing repairs hang indefinitely with sessions in 'STARTED' state. Per logs, the issue occurred due to dropped messages between a remote replica whilst it was sending "large messages" to the repair coordinator on the AntiEntropyStage executor.
{code:java}
"WARN  [AntiEntropyStage:1] 2024-02-07 14:46:51,256 NoSpamLogger.java:95 - x.63.94.y:10700->x.68.75.z:10700-LARGE_MESSAGES-adbea270 overloaded; dropping 14.097MiB message (queue: 131.855MiB local, 127.855MiB endpoint, 127.855MiB global)" {code}
We were able to resolve these issues by setting concurrent Merkle tree requests to 1, but I assume there's a gap in the recovery logic that meant this repair could not be recovered. Perhaps a missing condition within the retry logic in RepairMessage.

This happens whilst running v5.0.3 on all 72 nodes across 4 DCs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org