You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pr@cassandra.apache.org by "tommystendahl (via GitHub)" <gi...@apache.org> on 2026/09/10 09:56:06 UTC

[PR] CASSANDRA-21663 Failed force repair should clear force repair flag [cassandra]

tommystendahl opened a new pull request, #5145:
URL: https://github.com/apache/cassandra/pull/5145

   patch by Tommy Stendahl; reviewed by   for CASSANDRA-21663
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] CASSANDRA-21663 Failed force repair should clear force repair flag [cassandra]

Posted by "tommystendahl (via GitHub)" <gi...@apache.org>.
tommystendahl commented on code in PR #5145:
URL: https://github.com/apache/cassandra/pull/5145#discussion_r4335577149


##########
src/java/org/apache/cassandra/repair/autorepair/AutoRepair.java:
##########
@@ -200,71 +200,112 @@ public void repair(AutoRepairConfig.RepairType repairType)
             RepairTurn turn = AutoRepairUtils.myTurnToRunRepair(repairType, myId);
             if (turn == MY_TURN || turn == MY_TURN_DUE_TO_PRIORITY || turn == MY_TURN_FORCE_REPAIR)
             {

Review Comment:
   What do you think of this?
   [cassandra21663-5.0-alt1](https://github.com/tommystendahl/cassandra/tree/cassandra21663-5.0-alt1)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] CASSANDRA-21663 Failed force repair should clear force repair flag [cassandra]

Posted by "gandherva (via GitHub)" <gi...@apache.org>.
gandherva commented on code in PR #5145:
URL: https://github.com/apache/cassandra/pull/5145#discussion_r4235946140


##########
src/java/org/apache/cassandra/repair/autorepair/AutoRepair.java:
##########
@@ -200,71 +200,112 @@ public void repair(AutoRepairConfig.RepairType repairType)
             RepairTurn turn = AutoRepairUtils.myTurnToRunRepair(repairType, myId);
             if (turn == MY_TURN || turn == MY_TURN_DUE_TO_PRIORITY || turn == MY_TURN_FORCE_REPAIR)
             {
-                repairState.recordTurn(turn);
-                repairState.setBytesAlreadyRepaired(0L);
-                repairState.setKeyspaceRepairPlansAlreadyRepaired(0);
-                // For normal auto repair, we will use primary range only repairs (Repair with -pr option).
-                // For some cases, we may set the auto_repair_primary_token_range_only flag to false then we will do repair
-                // without -pr. We may also do force repair for certain node that we want to repair all the data on one node
-                // When doing force repair, we want to repair without -pr.
-                boolean primaryRangeOnly = config.getRepairPrimaryTokenRangeOnly(repairType)
-                                           && turn != MY_TURN_FORCE_REPAIR;
-
-                long startTimeInMillis = timeFunc.get();
-                logger.info("My host id: {}, my turn to run repair...repair primary-ranges only? {}", myId,
-                            config.getRepairPrimaryTokenRangeOnly(repairType));
-                AutoRepairUtils.updateStartAutoRepairHistory(repairType, myId, startTimeInMillis, turn);

Review Comment:
   I wanted to ask if the removal of these two lines are intentional in 6.0 and trunk branch. (These two lines don't exist in 5.0)
   
   AutoRepairUtils.updateStartAutoRepairHistory(repairType, myId, startTimeInMillis, turn);
   repairState.setLastRepairStartTime(startTimeInMillis);
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] CASSANDRA-21663 Failed force repair should clear force repair flag [cassandra]

Posted by "jaydeepkumar1984 (via GitHub)" <gi...@apache.org>.
jaydeepkumar1984 commented on code in PR #5145:
URL: https://github.com/apache/cassandra/pull/5145#discussion_r4239455819


##########
src/java/org/apache/cassandra/repair/autorepair/AutoRepair.java:
##########
@@ -200,71 +200,112 @@ public void repair(AutoRepairConfig.RepairType repairType)
             RepairTurn turn = AutoRepairUtils.myTurnToRunRepair(repairType, myId);
             if (turn == MY_TURN || turn == MY_TURN_DUE_TO_PRIORITY || turn == MY_TURN_FORCE_REPAIR)
             {

Review Comment:
   I am trying to understand why we need so many changes for this improvement. Isn't the following change sufficient?
   ```
   jchovati@jchovati-go:~/cassandra_open/cassandra$ git diff -w
   diff --git a/src/java/org/apache/cassandra/repair/autorepair/AutoRepair.java b/src/java/org/apache/cassandra/repair/autorepair/AutoRepair.java
   index 05f9c46f75..ebafe93e79 100644
   --- a/src/java/org/apache/cassandra/repair/autorepair/AutoRepair.java
   +++ b/src/java/org/apache/cassandra/repair/autorepair/AutoRepair.java
   @@ -204,6 +204,8 @@ public class AutoRepair
    
                RepairTurn turn = AutoRepairUtils.myTurnToRunRepair(repairType, myId);
                if (turn == MY_TURN || turn == MY_TURN_DUE_TO_PRIORITY || turn == MY_TURN_FORCE_REPAIR)
   +            {
   +                try
                    {
                    repairState.recordTurn(turn);
                    repairState.setBytesAlreadyRepaired(0L);
   @@ -270,6 +272,14 @@ public class AutoRepair
    
                    cleanupAndUpdateStats(turn, repairType, repairState, myId, startTimeInMillis, collectedRepairStats);
                }
   +            catch (Exception e)
   +            {
   +                if (turn == MY_TURN_FORCE_REPAIR)
   +                {
   +                    AutoRepairUtils.updateFinishAutoRepairHistory(repairType, myId, timeFunc.get());
   +                }
   +                logger.error("Exception in autorepair:", e);
   +            }
                else
                {
                    logger.info("Waiting for my turn...");
   jchovati@jchovati-go:~/cassandra_open/cassandra$ 
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] CASSANDRA-21663 Failed force repair should clear force repair flag [cassandra]

Posted by "tommystendahl (via GitHub)" <gi...@apache.org>.
tommystendahl commented on code in PR #5145:
URL: https://github.com/apache/cassandra/pull/5145#discussion_r4247480292


##########
src/java/org/apache/cassandra/repair/autorepair/AutoRepair.java:
##########
@@ -200,71 +200,112 @@ public void repair(AutoRepairConfig.RepairType repairType)
             RepairTurn turn = AutoRepairUtils.myTurnToRunRepair(repairType, myId);
             if (turn == MY_TURN || turn == MY_TURN_DUE_TO_PRIORITY || turn == MY_TURN_FORCE_REPAIR)
             {
-                repairState.recordTurn(turn);
-                repairState.setBytesAlreadyRepaired(0L);
-                repairState.setKeyspaceRepairPlansAlreadyRepaired(0);
-                // For normal auto repair, we will use primary range only repairs (Repair with -pr option).
-                // For some cases, we may set the auto_repair_primary_token_range_only flag to false then we will do repair
-                // without -pr. We may also do force repair for certain node that we want to repair all the data on one node
-                // When doing force repair, we want to repair without -pr.
-                boolean primaryRangeOnly = config.getRepairPrimaryTokenRangeOnly(repairType)
-                                           && turn != MY_TURN_FORCE_REPAIR;
-
-                long startTimeInMillis = timeFunc.get();
-                logger.info("My host id: {}, my turn to run repair...repair primary-ranges only? {}", myId,
-                            config.getRepairPrimaryTokenRangeOnly(repairType));
-                AutoRepairUtils.updateStartAutoRepairHistory(repairType, myId, startTimeInMillis, turn);

Review Comment:
   Yes, this is a merge error. I have fixed it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] CASSANDRA-21663 Failed force repair should clear force repair flag [cassandra]

Posted by "tommystendahl (via GitHub)" <gi...@apache.org>.
tommystendahl commented on code in PR #5145:
URL: https://github.com/apache/cassandra/pull/5145#discussion_r4334981324


##########
src/java/org/apache/cassandra/repair/autorepair/AutoRepair.java:
##########
@@ -200,71 +200,112 @@ public void repair(AutoRepairConfig.RepairType repairType)
             RepairTurn turn = AutoRepairUtils.myTurnToRunRepair(repairType, myId);
             if (turn == MY_TURN || turn == MY_TURN_DUE_TO_PRIORITY || turn == MY_TURN_FORCE_REPAIR)
             {

Review Comment:
   This is smaller and cleaner, that's good. But if force_repair is set during a ongoing repair a successful normal repair will clear the force_repair and the force repair will never be done.
   If force_repair is set during an ongoing force repair that will also be lost but that's probably a minor issue.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] CASSANDRA-21663 Failed force repair should clear force repair flag [cassandra]

Posted by "tommystendahl (via GitHub)" <gi...@apache.org>.
tommystendahl commented on code in PR #5145:
URL: https://github.com/apache/cassandra/pull/5145#discussion_r4247592210


##########
src/java/org/apache/cassandra/repair/autorepair/AutoRepair.java:
##########
@@ -200,71 +200,112 @@ public void repair(AutoRepairConfig.RepairType repairType)
             RepairTurn turn = AutoRepairUtils.myTurnToRunRepair(repairType, myId);
             if (turn == MY_TURN || turn == MY_TURN_DUE_TO_PRIORITY || turn == MY_TURN_FORCE_REPAIR)
             {

Review Comment:
   The main reasons are:
   - if we set the finish time to now it looks like this was a successful repair and the next normal repair will be started at now+min_repair_interval instead of last_successful_finish+min_repair_interval.
   - A successful repair always cleared force_repair so if force_repair was set during an ongoing repair it was cleared without being done.
   - If the node crash or restarts during a ongoing force repair it was restarted as a normal repair, not as a force repair.
   
   But doing the work in catch instead of using finally is a bit cleaner. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] CASSANDRA-21663 Failed force repair should clear force repair flag [cassandra]

Posted by "jaydeepkumar1984 (via GitHub)" <gi...@apache.org>.
jaydeepkumar1984 commented on code in PR #5145:
URL: https://github.com/apache/cassandra/pull/5145#discussion_r4329836396


##########
src/java/org/apache/cassandra/repair/autorepair/AutoRepair.java:
##########
@@ -200,71 +200,112 @@ public void repair(AutoRepairConfig.RepairType repairType)
             RepairTurn turn = AutoRepairUtils.myTurnToRunRepair(repairType, myId);
             if (turn == MY_TURN || turn == MY_TURN_DUE_TO_PRIORITY || turn == MY_TURN_FORCE_REPAIR)
             {

Review Comment:
   How about this?
   ```
   diff --git a/src/java/org/apache/cassandra/repair/autorepair/AutoRepair.java b/src/java/org/apache/cassandra/repair/autorepair/AutoRepair.java
   index 05f9c46f75..2697a8ee1b 100644
   --- a/src/java/org/apache/cassandra/repair/autorepair/AutoRepair.java
   +++ b/src/java/org/apache/cassandra/repair/autorepair/AutoRepair.java
   @@ -180,6 +180,8 @@ public class AutoRepair
            }
            AutoRepairService.instance.checkCanRun(repairType);
            AutoRepairState repairState = repairStates.get(repairType);
   +        UUID myId = null;
   +        RepairTurn turn = null;
            try
            {
                String localDC = DatabaseDescriptor.getLocalDataCenter();
   @@ -193,7 +195,7 @@ public class AutoRepair
                repairState.setLongestUnrepairedNode(AutoRepairUtils.getHostWithLongestUnrepairTime(repairType));
    
                //consistency level to use for local query
   -            UUID myId = Gossiper.instance.getHostId(FBUtilities.getBroadcastAddressAndPort());
   +            myId = Gossiper.instance.getHostId(FBUtilities.getBroadcastAddressAndPort());
    
                // Skip this repair cycle if the minimum interval since the last repair has not elapsed,
                // unless force repair is set for this node, which bypasses the interval check
   @@ -202,7 +204,7 @@ public class AutoRepair
                    return;
                }
    
   -            RepairTurn turn = AutoRepairUtils.myTurnToRunRepair(repairType, myId);
   +            turn = AutoRepairUtils.myTurnToRunRepair(repairType, myId);
                if (turn == MY_TURN || turn == MY_TURN_DUE_TO_PRIORITY || turn == MY_TURN_FORCE_REPAIR)
                {
                    repairState.recordTurn(turn);
   @@ -277,6 +279,19 @@ public class AutoRepair
            }
            catch (Exception e)
            {
   +            if (turn == MY_TURN_FORCE_REPAIR)
   +            {
   +                // A failed force repair must not leave force_repair set, otherwise the node would force repair on
   +                // every cycle. Do not record it as finished, as that would make the failed run look successful.
   +                try
   +                {
   +                    AutoRepairUtils.resetForcedRepair(repairType, myId);
   +                }
   +                catch (Exception resetError)
   +                {
   +                    e.addSuppressed(resetError);
   +                }
   +            }
                logger.error("Exception in autorepair:", e);
            }
        }
   diff --git a/src/java/org/apache/cassandra/repair/autorepair/AutoRepairUtils.java b/src/java/org/apache/cassandra/repair/autorepair/AutoRepairUtils.java
   index 1c788fe660..a57aee727a 100644
   --- a/src/java/org/apache/cassandra/repair/autorepair/AutoRepairUtils.java
   +++ b/src/java/org/apache/cassandra/repair/autorepair/AutoRepairUtils.java
   @@ -170,6 +170,13 @@ public class AutoRepairUtils
        , SchemaConstants.DISTRIBUTED_KEYSPACE_NAME, SystemDistributedKeyspace.AUTO_REPAIR_HISTORY, COL_FORCE_REPAIR,
        COL_REPAIR_TYPE, COL_HOST_ID);
    
   +    // Used to abandon a failed force repair: rewinds repair_start_ts (so the record is no longer "ongoing")
   +    // and clears force_repair, in a single row update. repair_finish_ts is left untouched.
   +    final static String RESET_FORCE_REPAIR = String.format(
   +    "UPDATE %s.%s SET %s = ?, %s = false WHERE %s = ? AND %s = ?"
   +    , SchemaConstants.DISTRIBUTED_KEYSPACE_NAME, SystemDistributedKeyspace.AUTO_REPAIR_HISTORY, COL_REPAIR_START_TS,
   +    COL_FORCE_REPAIR, COL_REPAIR_TYPE, COL_HOST_ID);
   +
        final static String SELECT_LAST_REPAIR_TIME_FOR_NODE = String.format(
        "SELECT %s FROM %s.%s WHERE %s = ? AND %s = ?", COL_REPAIR_FINISH_TS, SchemaConstants.DISTRIBUTED_KEYSPACE_NAME,
        SystemDistributedKeyspace.AUTO_REPAIR_HISTORY, COL_REPAIR_TYPE, COL_HOST_ID);
   @@ -191,6 +198,7 @@ public class AutoRepairUtils
        static ModificationStatement addHostIDToDeleteHostsStatement;
        static ModificationStatement clearDeleteHostsStatement;
        static ModificationStatement setForceRepairStatement;
   +    static ModificationStatement resetForceRepairStatement;
        static ConsistencyLevel internalQueryCL;
    
        public enum RepairTurn
   @@ -225,6 +233,8 @@ public class AutoRepairUtils
                                                                                                                               .forInternalCalls());
            setForceRepairStatement = (ModificationStatement) QueryProcessor.getStatement(SET_FORCE_REPAIR, ClientState
                                                                                                            .forInternalCalls());
   +        resetForceRepairStatement = (ModificationStatement) QueryProcessor.getStatement(RESET_FORCE_REPAIR, ClientState
   +                                                                                                            .forInternalCalls());
            clearDeleteHostsStatement = (ModificationStatement) QueryProcessor.getStatement(CLEAR_DELETE_HOSTS, ClientState
                                                                                                                .forInternalCalls());
            delStatementRepairHistory = (ModificationStatement) QueryProcessor.getStatement(DEL_AUTO_REPAIR_HISTORY, ClientState
   @@ -411,6 +421,27 @@ public class AutoRepairUtils
            logger.info("Set force repair repair type: {}, node: {}", repairType, hostId);
        }
    
   +    /**
   +     * Abandon a failed force repair for the given node: clears the force repair flag and rewinds
   +     * {@code repair_start_ts} to the existing {@code repair_finish_ts}, so the record is no longer considered
   +     * ongoing. {@code repair_finish_ts} is not modified, so the failed run is not recorded as a successful repair.
   +     *
   +     * @param repairType the repair type
   +     * @param hostId the host id whose failed force repair should be reset
   +     */
   +    public static void resetForcedRepair(RepairType repairType, UUID hostId)
   +    {
   +        long lastFinishTime = getLastRepairTimeForNode(repairType, hostId);
   +        resetForceRepairStatement.execute(QueryState.forInternalCalls(),
   +                                          QueryOptions.forInternalCalls(internalQueryCL,
   +                                                                        Lists.newArrayList(ByteBufferUtil.bytes(lastFinishTime),
   +                                                                                           ByteBufferUtil.bytes(repairType.toString()),
   +                                                                                           ByteBufferUtil.bytes(hostId))),
   +                                          Dispatcher.RequestTime.forImmediateExecution());
   +
   +        logger.info("Reset failed force repair repair type: {}, node: {}", repairType, hostId);
   +    }
   +
        /**
         * Check if force repair is set for the given node.
         *
   @@ -447,7 +478,8 @@ public class AutoRepairUtils
            {
                return 0;
            }
   -        return repairTime.one().getLong(COL_REPAIR_FINISH_TS);
   +        UntypedResultSet.Row one = repairTime.one();
   +        return one.has(COL_REPAIR_FINISH_TS) ? one.getLong(COL_REPAIR_FINISH_TS) : 0;
        }
    
   
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org