You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pr@cassandra.apache.org by "jaydeepkumar1984 (via GitHub)" <gi...@apache.org> on 2024/06/11 19:28:50 UTC
[PR] autorepair v2 framework [cassandra]
jaydeepkumar1984 opened a new pull request, #3367:
URL: https://github.com/apache/cassandra/pull/3367
AutoRepir v2 framework that covers both FR & IR
An interface with a default implementation for TokenRange calculation
Enable/Disable flag as a CQL Table property
More details to this design discussion [doc](https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] autorepair v2 framework (against 4.1.6) [cassandra]
Posted by "clohfink (via GitHub)" <gi...@apache.org>.
clohfink commented on code in PR #3367:
URL: https://github.com/apache/cassandra/pull/3367#discussion_r1794022962
##########
src/java/org/apache/cassandra/repair/autorepair/AutoRepairState.java:
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.repair.autorepair;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import org.apache.cassandra.config.DurationSpec;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.view.TableViews;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.metrics.AutoRepairMetricsManager;
+import org.apache.cassandra.metrics.AutoRepairMetrics;
+import org.apache.cassandra.repair.autorepair.AutoRepairConfig.RepairType;
+import org.apache.cassandra.repair.autorepair.AutoRepairUtils.AutoRepairHistory;
+import org.apache.cassandra.repair.RepairParallelism;
+import org.apache.cassandra.repair.RepairRunnable;
+import org.apache.cassandra.repair.messages.RepairOption;
+import org.apache.cassandra.service.AutoRepairService;
+import org.apache.cassandra.service.StorageService;
+import org.apache.cassandra.streaming.PreviewKind;
+import org.apache.cassandra.utils.concurrent.Condition;
+import org.apache.cassandra.utils.progress.ProgressEvent;
+import org.apache.cassandra.utils.progress.ProgressEventType;
+import org.apache.cassandra.utils.progress.ProgressListener;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.text.SimpleDateFormat;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+import static org.apache.cassandra.utils.Clock.Global.currentTimeMillis;
+import static org.apache.cassandra.utils.concurrent.Condition.newOneTimeCondition;
+
+// AutoRepairState represents the state of automated repair for a given repair type.
+public abstract class AutoRepairState implements ProgressListener
+{
+ protected static final Logger logger = LoggerFactory.getLogger(AutoRepairState.class);
+ private final SimpleDateFormat format = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss,SSS");
+ @VisibleForTesting
+ protected static Supplier<Long> timeFunc = System::currentTimeMillis;
Review Comment:
org.apache.cassandra.utils.Clock was made to handle this kinda thing - theres normally a checkstyles thing that would fail on this method call but because its a lambda reference it misses it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] autorepair v2 framework (against 4.1.6) [cassandra]
Posted by "clohfink (via GitHub)" <gi...@apache.org>.
clohfink commented on code in PR #3367:
URL: https://github.com/apache/cassandra/pull/3367#discussion_r1683366109
##########
test/unit/org/apache/cassandra/repair/autorepair/AutoRepairParameterizedTest.java:
##########
@@ -0,0 +1,581 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.repair.autorepair;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Sets;
+
+import org.apache.cassandra.cql3.statements.schema.TableAttributes;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.repair.RepairRunnable;
+import org.apache.cassandra.schema.AutoRepairParams;
+import org.apache.cassandra.schema.TableParams;
+import org.apache.cassandra.service.StorageService;
+import org.apache.cassandra.utils.Pair;
+
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import junit.framework.Assert;
Review Comment:
change to org.junit.Assert
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] autorepair v2 framework (against 4.1.6) [cassandra]
Posted by "clohfink (via GitHub)" <gi...@apache.org>.
clohfink commented on code in PR #3367:
URL: https://github.com/apache/cassandra/pull/3367#discussion_r1683354596
##########
src/java/org/apache/cassandra/repair/autorepair/AutoRepair.java:
##########
@@ -0,0 +1,401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.repair.autorepair;
+
+import java.util.ArrayList;
+import java.util.EnumMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.FutureTask;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableList;
+
+import org.apache.cassandra.repair.RepairRunnable;
+import org.apache.cassandra.utils.Pair;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.concurrent.NamedThreadFactory;
+import org.apache.cassandra.concurrent.ScheduledExecutorPlus;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.exceptions.ConfigurationException;
+import org.apache.cassandra.gms.Gossiper;
+import org.apache.cassandra.schema.TableMetadata;
+import org.apache.cassandra.schema.Tables;
+import org.apache.cassandra.service.AutoRepairService;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.repair.autorepair.AutoRepairUtils.RepairTurn;
+
+import static org.apache.cassandra.concurrent.ExecutorFactory.Global.executorFactory;
+import static org.apache.cassandra.repair.autorepair.AutoRepairUtils.RepairTurn.MY_TURN;
+import static org.apache.cassandra.repair.autorepair.AutoRepairUtils.RepairTurn.MY_TURN_DUE_TO_PRIORITY;
+import static org.apache.cassandra.repair.autorepair.AutoRepairUtils.RepairTurn.MY_TURN_FORCE_REPAIR;
+
+public class AutoRepair
+{
+ // Initial delay for repair session to start after setup
+ final static long INITIAL_REPAIR_DELAY_SEC = 30;
+
+ private static final Logger logger = LoggerFactory.getLogger(AutoRepair.class);
+
+ @VisibleForTesting
+ protected static Supplier<Long> timeFunc = System::currentTimeMillis;
+
+ public static AutoRepair instance = new AutoRepair();
+
+ @VisibleForTesting
+ protected final Map<AutoRepairConfig.RepairType, ScheduledExecutorPlus> repairExecutors;
+ @VisibleForTesting
+ protected final Map<AutoRepairConfig.RepairType, AutoRepairState> repairStates;
+
+ protected final Map<AutoRepairConfig.RepairType, IAutoRepairTokenRangeSplitter> tokenRangeSplitters = new EnumMap<>(AutoRepairConfig.RepairType.class);
+
+
+ @VisibleForTesting
+ protected AutoRepair()
+ {
+ AutoRepairConfig config = DatabaseDescriptor.getAutoRepairConfig();
+ repairExecutors = new EnumMap<>(AutoRepairConfig.RepairType.class);
+ repairStates = new EnumMap<>(AutoRepairConfig.RepairType.class);
+ for (AutoRepairConfig.RepairType repairType : AutoRepairConfig.RepairType.values())
+ {
+ repairExecutors.put(repairType, executorFactory().scheduled(false, "AutoRepair-Repair-" + repairType, Thread.NORM_PRIORITY));
+ repairStates.put(repairType, AutoRepairConfig.RepairType.getAutoRepairState(repairType));
+ tokenRangeSplitters.put(repairType, FBUtilities.newAutoRepairTokenRangeSplitter(config.getTokenRangeSplitter(repairType)));
+ }
+ }
+
+ public void setup()
+ {
+ verifyIsSafeToEnable();
+
+ AutoRepairConfig config = DatabaseDescriptor.getAutoRepairConfig();
+ AutoRepairService.setup();
+ AutoRepairUtils.setup();
+
+ for (AutoRepairConfig.RepairType repairType : AutoRepairConfig.RepairType.values())
+ {
+ repairExecutors.get(repairType).scheduleWithFixedDelay(
+ () -> repair(repairType, 60000),
+ INITIAL_REPAIR_DELAY_SEC,
+ config.getRepairCheckInterval().toSeconds(),
+ TimeUnit.SECONDS);
+ }
+ }
+
+ @VisibleForTesting
+ protected void verifyIsSafeToEnable()
+ {
+ AutoRepairConfig config = DatabaseDescriptor.getAutoRepairConfig();
+ if (config.isAutoRepairEnabled(AutoRepairConfig.RepairType.incremental) &&
+ (DatabaseDescriptor.getMaterializedViewsEnabled() || DatabaseDescriptor.isCDCEnabled()))
+ throw new ConfigurationException("Cannot enable incremental repair with materialized views or CDC enabled");
+ }
+
+ // repairAsync runs a repair session of the given type asynchronously.
+ public void repairAsync(AutoRepairConfig.RepairType repairType, long millisToWait)
+ {
+ repairExecutors.get(repairType).submit(() -> repair(repairType, millisToWait));
+ }
+
+ // repair runs a repair session of the given type synchronously.
+ public void repair(AutoRepairConfig.RepairType repairType, long millisToWait)
+ {
+ AutoRepairConfig config = AutoRepairService.instance.getAutoRepairConfig();
+ if (!config.isAutoRepairEnabled(repairType))
+ {
+ logger.debug("Auto-repair is disabled for repair type {}", repairType);
+ return;
+ }
+
+
+ AutoRepairState repairState = repairStates.get(repairType);
+
+ try
+ {
+ String localDC = DatabaseDescriptor.getLocalDataCenter();
+ if (config.getIgnoreDCs(repairType).contains(localDC))
+ {
+ logger.info("Not running repair as this node belongs to datacenter {}", localDC);
+ return;
+ }
+
+ // refresh the longest unrepaired node
+ repairState.setLongestUnrepairedNode(AutoRepairUtils.getHostWithLongestUnrepairTime(repairType));
+
+ //consistency level to use for local query
+ UUID myId = Gossiper.instance.getHostId(FBUtilities.getBroadcastAddressAndPort());
+ RepairTurn turn = AutoRepairUtils.myTurnToRunRepair(repairType, myId);
+ if (turn == MY_TURN || turn == MY_TURN_DUE_TO_PRIORITY || turn == MY_TURN_FORCE_REPAIR)
+ {
+ repairState.recordTurn(turn);
+ // For normal auto repair, we will use primary range only repairs (Repair with -pr option).
+ // For some cases, we may set the auto_repair_primary_token_range_only flag to false then we will do repair
+ // without -pr. We may also do force repair for certain node that we want to repair all the data on one node
+ // When doing force repair, we want to repair without -pr.
+ boolean primaryRangeOnly = config.getRepairPrimaryTokenRangeOnly(repairType)
+ && turn != MY_TURN_FORCE_REPAIR;
+ repairState.setTotalTablesConsideredForRepair(0);
+ if (repairState.getLastRepairTime() != 0)
+ {
+ /** check if it is too soon to run repair. one of the reason we
+ * should not run frequent repair is because repair triggers
+ * memtable flush
+ */
+ long timeElapsedSinceLastRepair = TimeUnit.MILLISECONDS.toSeconds(timeFunc.get() - repairState.getLastRepairTime());
+ if (timeElapsedSinceLastRepair < config.getRepairMinInterval(repairType).toSeconds())
+ {
+ logger.info("Too soon to run repair, last repair was done {} seconds ago",
+ timeElapsedSinceLastRepair);
+ return;
+ }
+ }
+
+ long startTime = timeFunc.get();
+ logger.info("My host id: {}, my turn to run repair...repair primary-ranges only? {}", myId,
+ config.getRepairPrimaryTokenRangeOnly(repairType));
+ AutoRepairUtils.updateStartAutoRepairHistory(repairType, myId, timeFunc.get(), turn);
+
+ repairState.setRepairKeyspaceCount(0);
+ repairState.setRepairTableSuccessCount(0);
+ repairState.setRepairFailedTablesCount(0);
+ repairState.setRepairSkippedTablesCount(0);
+ repairState.setRepairInProgress(true);
+ repairState.setTotalMVTablesConsideredForRepair(0);
+ for (Keyspace keyspace : Keyspace.all())
+ {
+ Tables tables = keyspace.getMetadata().tables;
+ Iterator<TableMetadata> iter = tables.iterator();
+ String keyspaceName = keyspace.getName();
+ if (!AutoRepairUtils.checkNodeContainsKeyspaceReplica(keyspace))
+ {
+ continue;
+ }
+
+ repairState.setRepairKeyspaceCount(repairState.getRepairKeyspaceCount() + 1);
+ List<String> tablesToBeRepaired = new ArrayList<>();
+ while (iter.hasNext())
+ {
+ repairState.setTotalTablesConsideredForRepair(repairState.getTotalTablesConsideredForRepair() + 1);
+ TableMetadata tableMetadata = iter.next();
+ String tableName = tableMetadata.name;
+ tablesToBeRepaired.add(tableName);
+
+ // See if we should repair MVs as well that are associated with this given table
+ List<String> mvs = AutoRepairUtils.getAllMVs(repairType, keyspace, tableMetadata);
+ if (!mvs.isEmpty())
+ {
+ tablesToBeRepaired.addAll(mvs);
+ repairState.setTotalMVTablesConsideredForRepair(repairState.getTotalMVTablesConsideredForRepair() + mvs.size());
+ }
+ }
+
+ for (String tableName : tablesToBeRepaired)
+ {
+ try
+ {
+ ColumnFamilyStore columnFamilyStore = keyspace.getColumnFamilyStore(tableName);
+ if (!columnFamilyStore.metadata().params.automatedRepair.get(repairType).repairEnabled())
+ {
+ logger.info("Repair is disabled for keyspace {} for tables: {}", keyspaceName, tableName);
+ repairState.setTotalDisabledTablesRepairCount(repairState.getTotalDisabledTablesRepairCount() + 1);
+ continue;
+ }
+ // this is done to make autorepair safe as running repair on table with more sstables
+ // may have its own challenges
+ int size = columnFamilyStore.getLiveSSTables().size();
+ if (size > config.getRepairSSTableCountHigherThreshold(repairType))
+ {
+ logger.info("Too many SSTables for repair, not doing repair on table {}.{} " +
+ "totalSSTables {}", keyspaceName, tableName, columnFamilyStore.getLiveSSTables().size());
+ repairState.setRepairSkippedTablesCount(repairState.getRepairSkippedTablesCount() + 1);
+ continue;
+ }
+
+ if (config.getRepairByKeyspace(repairType))
+ {
+ logger.info("Repair keyspace {} for tables: {}", keyspaceName, tablesToBeRepaired);
+ }
+ else
+ {
+ logger.info("Repair table {}.{}", keyspaceName, tableName);
+ }
+ long tableStartTime = timeFunc.get();
+ boolean repairSuccess = true;
+ Set<Range<Token>> ranges = new HashSet<>();
+ List<Pair<Token, Token>> subRangesToBeRepaired = tokenRangeSplitters.get(repairType).getRange(repairType, primaryRangeOnly, keyspaceName, tableName);
+ int totalSubRanges = subRangesToBeRepaired.size();
+ int totalProcessedSubRanges = 0;
+ for (Pair<Token, Token> token : subRangesToBeRepaired)
+ {
+ if (!config.isAutoRepairEnabled(repairType))
+ {
+ logger.error("Auto-repair for type {} is disabled hence not running repair", repairType);
+ repairState.setRepairInProgress(false);
+ return;
+ }
+
+ if (config.getRepairByKeyspace(repairType))
+ {
+ if (AutoRepairUtils.keyspaceMaxRepairTimeExceeded(repairType, tableStartTime, tablesToBeRepaired.size()))
+ {
+ repairState.setRepairSkippedTablesCount(repairState.getRepairSkippedTablesCount() + tablesToBeRepaired.size());
+ logger.info("Keyspace took too much time to repair hence skipping it {}",
+ keyspaceName);
+ break;
+ }
+ }
+ else
+ {
+ if (AutoRepairUtils.tableMaxRepairTimeExceeded(repairType, tableStartTime))
+ {
+ repairState.setRepairSkippedTablesCount(repairState.getRepairSkippedTablesCount() + 1);
+ logger.info("Table took too much time to repair hence skipping it {}.{}",
+ keyspaceName, tableName);
+ break;
+ }
+ }
+ Token childStartToken = token.left;
+ Token childEndToken = token.right;
+ logger.debug("Current Token Left side {}, right side {}", childStartToken
+ .toString(), childEndToken.toString());
+
+ ranges.add(new Range<>(childStartToken, childEndToken));
+ totalProcessedSubRanges++;
+ if ((totalProcessedSubRanges % config.getRepairThreads(repairType) == 0) ||
+ (totalProcessedSubRanges == totalSubRanges))
+ {
+ RepairRunnable task = repairState.getRepairRunnable(keyspaceName,
+ config.getRepairByKeyspace(repairType) ? tablesToBeRepaired : ImmutableList.of(tableName),
+ ranges, primaryRangeOnly);
+ repairState.resetWaitCondition();
+ new Thread(NamedThreadFactory.createAnonymousThread(new FutureTask<>(task, null))).start();
Review Comment:
This fails checkstyles requirements. Should create a single thread pool for this or reuse one of existing ones
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] autorepair v2 framework (against 4.1.6) [cassandra]
Posted by "clohfink (via GitHub)" <gi...@apache.org>.
clohfink commented on code in PR #3367:
URL: https://github.com/apache/cassandra/pull/3367#discussion_r1683365056
##########
test/unit/org/apache/cassandra/repair/autorepair/AutoRepairParameterizedTest.java:
##########
@@ -0,0 +1,581 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.repair.autorepair;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Sets;
+
+import org.apache.cassandra.cql3.statements.schema.TableAttributes;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.repair.RepairRunnable;
+import org.apache.cassandra.schema.AutoRepairParams;
+import org.apache.cassandra.schema.TableParams;
+import org.apache.cassandra.service.StorageService;
+import org.apache.cassandra.utils.Pair;
+
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import junit.framework.Assert;
+import org.apache.cassandra.SchemaLoader;
+import org.apache.cassandra.concurrent.ScheduledExecutorPlus;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.cql3.CQLTester;
+import org.apache.cassandra.cql3.QueryProcessor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.marshal.IntegerType;
+import org.apache.cassandra.db.marshal.UTF8Type;
+import org.apache.cassandra.gms.Gossiper;
+import org.apache.cassandra.io.sstable.format.SSTableReader;
+import org.apache.cassandra.metrics.AutoRepairMetricsManager;
+import org.apache.cassandra.metrics.AutoRepairMetrics;
+import org.apache.cassandra.schema.KeyspaceParams;
+import org.apache.cassandra.schema.Schema;
+import org.apache.cassandra.schema.SchemaConstants;
+import org.apache.cassandra.schema.TableMetadata;
+import org.apache.cassandra.service.AutoRepairService;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.utils.progress.ProgressEvent;
+import org.mockito.Mock;
+import org.mockito.Mockito;
+import org.mockito.MockitoAnnotations;
+
+import static org.apache.cassandra.Util.setAutoRepairEnabled;
+import static org.apache.cassandra.Util.token;
Review Comment:
unused import breaks build
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] autorepair v2 framework (against 4.1.6) [cassandra]
Posted by "jaydeepkumar1984 (via GitHub)" <gi...@apache.org>.
jaydeepkumar1984 commented on code in PR #3367:
URL: https://github.com/apache/cassandra/pull/3367#discussion_r1695735326
##########
src/java/org/apache/cassandra/schema/TableParams.java:
##########
@@ -105,6 +112,13 @@ private TableParams(Builder builder)
extensions = builder.extensions;
cdc = builder.cdc;
readRepair = builder.readRepair;
+ automatedRepair = new HashMap<>()
Review Comment:
Incorporated
##########
src/java/org/apache/cassandra/schema/TableParams.java:
##########
@@ -84,12 +89,14 @@ public String toString()
public final boolean cdc;
public final ReadRepairStrategy readRepair;
+ public final Map<AutoRepairConfig.RepairType, AutoRepairParams> automatedRepair;
Review Comment:
Incorporated
##########
src/java/org/apache/cassandra/repair/autorepair/AutoRepairUtils.java:
##########
@@ -0,0 +1,806 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.repair.autorepair;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.MoreObjects;
+import com.google.common.collect.Lists;
+
+import org.apache.cassandra.locator.LocalStrategy;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.cql3.QueryOptions;
+import org.apache.cassandra.cql3.QueryProcessor;
+import org.apache.cassandra.cql3.UntypedResultSet;
+import org.apache.cassandra.cql3.statements.ModificationStatement;
+import org.apache.cassandra.cql3.statements.SelectStatement;
+import org.apache.cassandra.db.ConsistencyLevel;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.marshal.UTF8Type;
+import org.apache.cassandra.db.marshal.UUIDType;
+import org.apache.cassandra.gms.Gossiper;
+import org.apache.cassandra.locator.AbstractReplicationStrategy;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.locator.NetworkTopologyStrategy;
+import org.apache.cassandra.schema.Schema;
+import org.apache.cassandra.schema.SchemaConstants;
+import org.apache.cassandra.schema.TableMetadata;
+import org.apache.cassandra.schema.ViewMetadata;
+import org.apache.cassandra.serializers.SetSerializer;
+import org.apache.cassandra.serializers.UUIDSerializer;
+import org.apache.cassandra.service.AutoRepairService;
+import org.apache.cassandra.service.ClientState;
+import org.apache.cassandra.service.QueryState;
+import org.apache.cassandra.service.StorageService;
+import org.apache.cassandra.transport.Dispatcher;
+import org.apache.cassandra.transport.ProtocolVersion;
+import org.apache.cassandra.transport.messages.ResultMessage;
+import org.apache.cassandra.utils.ByteBufferUtil;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.repair.autorepair.AutoRepairConfig.RepairType;
+
+import static org.apache.cassandra.repair.autorepair.AutoRepairUtils.RepairTurn.MY_TURN;
+import static org.apache.cassandra.repair.autorepair.AutoRepairUtils.RepairTurn.MY_TURN_DUE_TO_PRIORITY;
+import static org.apache.cassandra.repair.autorepair.AutoRepairUtils.RepairTurn.NOT_MY_TURN;
+import static org.apache.cassandra.repair.autorepair.AutoRepairUtils.RepairTurn.MY_TURN_FORCE_REPAIR;
+
+/**
+ * This class serves as a utility class for AutoRepair. It contains various helper APIs
+ * to store/retrieve repair status, decide whose turn is next, etc.
+ */
+public class AutoRepairUtils
+{
+ private static final Logger logger = LoggerFactory.getLogger(AutoRepairUtils.class);
+ static final String COL_REPAIR_TYPE = "repair_type";
+ static final String COL_HOST_ID = "host_id";
+ static final String COL_REPAIR_START_TS = "repair_start_ts";
+ static final String COL_REPAIR_FINISH_TS = "repair_finish_ts";
+ static final String COL_REPAIR_PRIORITY = "repair_priority";
+ static final String COL_DELETE_HOSTS = "delete_hosts"; // this set stores the host ids which think the row should be deleted
+ static final String COL_REPAIR_TURN = "repair_turn"; // this record the last repair turn. Normal turn or turn due to priority
+ static final String COL_DELETE_HOSTS_UPDATE_TIME = "delete_hosts_update_time"; // the time when delete hosts are upated
+ static final String COL_FORCE_REPAIR = "force_repair"; // if set to true, the node will do non-primary range rapair
+
+ final static String SELECT_REPAIR_HISTORY = String.format(
+ "SELECT * FROM %s.%s WHERE %s = ?", SchemaConstants.DISTRIBUTED_KEYSPACE_NAME,
+ AutoRepairKeyspace.AUTO_REPAIR_HISTORY, COL_REPAIR_TYPE);
+ final static String SELECT_REPAIR_PRIORITY = String.format(
+ "SELECT * FROM %s.%s WHERE %s = ?", SchemaConstants.DISTRIBUTED_KEYSPACE_NAME,
+ AutoRepairKeyspace.AUTO_REPAIR_PRIORITY, COL_REPAIR_TYPE);
+ final static String DEL_REPAIR_PRIORITY = String.format(
+ "DELETE %s[?] FROM %s.%s WHERE %s = ?", COL_REPAIR_PRIORITY, SchemaConstants.DISTRIBUTED_KEYSPACE_NAME,
+ AutoRepairKeyspace.AUTO_REPAIR_PRIORITY, COL_REPAIR_TYPE);
+ final static String ADD_PRIORITY_HOST = String.format(
+ "UPDATE %s.%s SET %s = %s + ? WHERE %s = ?", SchemaConstants.DISTRIBUTED_KEYSPACE_NAME,
+ AutoRepairKeyspace.AUTO_REPAIR_PRIORITY, COL_REPAIR_PRIORITY, COL_REPAIR_PRIORITY, COL_REPAIR_TYPE);
+
+ final static String INSERT_NEW_REPAIR_HISTORY = String.format(
+ "INSERT INTO %s.%s (%s, %s, %s, %s, %s, %s) values (?, ? ,?, ?, {}, ?) IF NOT EXISTS",
+ SchemaConstants.DISTRIBUTED_KEYSPACE_NAME, AutoRepairKeyspace.AUTO_REPAIR_HISTORY, COL_REPAIR_TYPE,
+ COL_HOST_ID, COL_REPAIR_START_TS, COL_REPAIR_FINISH_TS, COL_DELETE_HOSTS, COL_DELETE_HOSTS_UPDATE_TIME);
+
+ final static String ADD_HOST_ID_TO_DELETE_HOSTS = String.format(
+ "UPDATE %s.%s SET %s = %s + ?, %s = ? WHERE %s = ? AND %s = ? IF EXISTS"
+ , SchemaConstants.DISTRIBUTED_KEYSPACE_NAME, AutoRepairKeyspace.AUTO_REPAIR_HISTORY, COL_DELETE_HOSTS,
+ COL_DELETE_HOSTS, COL_DELETE_HOSTS_UPDATE_TIME, COL_REPAIR_TYPE, COL_HOST_ID);
+
+ final static String DEL_AUTO_REPAIR_HISTORY = String.format(
+ "DELETE FROM %s.%s WHERE %s = ? AND %s = ?"
+ , SchemaConstants.DISTRIBUTED_KEYSPACE_NAME, AutoRepairKeyspace.AUTO_REPAIR_HISTORY, COL_REPAIR_TYPE,
+ COL_HOST_ID);
+
+ final static String RECORD_START_REPAIR_HISTORY = String.format(
+ "UPDATE %s.%s SET %s= ?, repair_turn = ? WHERE %s = ? AND %s = ?"
+ , SchemaConstants.DISTRIBUTED_KEYSPACE_NAME, AutoRepairKeyspace.AUTO_REPAIR_HISTORY, COL_REPAIR_START_TS,
+ COL_REPAIR_TYPE, COL_HOST_ID);
+
+ final static String RECORD_FINISH_REPAIR_HISTORY = String.format(
+
+ "UPDATE %s.%s SET %s= ?, %s=false WHERE %s = ? AND %s = ?"
+ , SchemaConstants.DISTRIBUTED_KEYSPACE_NAME, AutoRepairKeyspace.AUTO_REPAIR_HISTORY, COL_REPAIR_FINISH_TS,
+ COL_FORCE_REPAIR, COL_REPAIR_TYPE, COL_HOST_ID);
+
+ final static String CLEAR_DELETE_HOSTS = String.format(
+ "UPDATE %s.%s SET %s= {} WHERE %s = ? AND %s = ?"
+ , SchemaConstants.DISTRIBUTED_KEYSPACE_NAME, AutoRepairKeyspace.AUTO_REPAIR_HISTORY, COL_DELETE_HOSTS,
+ COL_REPAIR_TYPE, COL_HOST_ID);
+
+ final static String SET_FORCE_REPAIR = String.format(
+ "UPDATE %s.%s SET %s=true WHERE %s = ? AND %s = ?"
+ , SchemaConstants.DISTRIBUTED_KEYSPACE_NAME, AutoRepairKeyspace.AUTO_REPAIR_HISTORY, COL_FORCE_REPAIR,
+ COL_REPAIR_TYPE, COL_HOST_ID);
+
+ static ModificationStatement delStatementRepairHistory;
+ static SelectStatement selectStatementRepairHistory;
+ static ModificationStatement delStatementPriorityStatus;
+ static SelectStatement selectStatementRepairPriority;
+ static ModificationStatement addPriorityHost;
+ static ModificationStatement insertNewRepairHistoryStatement;
+ static ModificationStatement recordStartRepairHistoryStatement;
+ static ModificationStatement recordFinishRepairHistoryStatement;
+ static ModificationStatement addHostIDToDeleteHostsStatement;
+ static ModificationStatement clearDeleteHostsStatement;
+ static ModificationStatement setForceRepairStatement;
+ static ConsistencyLevel internalQueryCL;
+
+ public enum RepairTurn
+ {
+ MY_TURN,
+ NOT_MY_TURN,
+ MY_TURN_DUE_TO_PRIORITY,
+ MY_TURN_FORCE_REPAIR
+ }
+
+ public static void setup()
+ {
+ selectStatementRepairHistory = (SelectStatement) QueryProcessor.getStatement(SELECT_REPAIR_HISTORY, ClientState
+ .forInternalCalls());
+ selectStatementRepairPriority = (SelectStatement) QueryProcessor.getStatement(SELECT_REPAIR_PRIORITY, ClientState
+ .forInternalCalls());
+ delStatementPriorityStatus = (ModificationStatement) QueryProcessor.getStatement(DEL_REPAIR_PRIORITY, ClientState
+ .forInternalCalls());
+ addPriorityHost = (ModificationStatement) QueryProcessor.getStatement(ADD_PRIORITY_HOST, ClientState
+ .forInternalCalls());
+ insertNewRepairHistoryStatement = (ModificationStatement) QueryProcessor.getStatement(INSERT_NEW_REPAIR_HISTORY, ClientState
+ .forInternalCalls());
+ recordStartRepairHistoryStatement = (ModificationStatement) QueryProcessor.getStatement(RECORD_START_REPAIR_HISTORY, ClientState
+ .forInternalCalls());
+ recordFinishRepairHistoryStatement = (ModificationStatement) QueryProcessor.getStatement(RECORD_FINISH_REPAIR_HISTORY, ClientState
+ .forInternalCalls());
+ addHostIDToDeleteHostsStatement = (ModificationStatement) QueryProcessor.getStatement(ADD_HOST_ID_TO_DELETE_HOSTS, ClientState
+ .forInternalCalls());
+ setForceRepairStatement = (ModificationStatement) QueryProcessor.getStatement(SET_FORCE_REPAIR, ClientState
+ .forInternalCalls());
+ clearDeleteHostsStatement = (ModificationStatement) QueryProcessor.getStatement(CLEAR_DELETE_HOSTS, ClientState
+ .forInternalCalls());
+ delStatementRepairHistory = (ModificationStatement) QueryProcessor.getStatement(DEL_AUTO_REPAIR_HISTORY, ClientState
+ .forInternalCalls());
+ Keyspace autoRepairKS = Schema.instance.getKeyspaceInstance(SchemaConstants.DISTRIBUTED_KEYSPACE_NAME);
+ internalQueryCL = autoRepairKS.getReplicationStrategy().getClass() == NetworkTopologyStrategy.class ?
+ ConsistencyLevel.LOCAL_QUORUM : ConsistencyLevel.ONE;
+ }
+
+ public static class AutoRepairHistory
+ {
+ UUID hostId;
+ String repairTurn;
+ long lastRepairStartTime;
+ long lastRepairFinishTime;
+ Set<UUID> deleteHosts;
+ long deleteHostsUpdateTime;
+ boolean forceRepair;
+
+ public AutoRepairHistory(UUID hostId, String repairTurn, long lastRepairStartTime, long lastRepairFinishTime,
+ Set<UUID> deleteHosts, long deleteHostsUpateTime, boolean forceRepair)
+ {
+ this.hostId = hostId;
+ this.repairTurn = repairTurn;
+ this.lastRepairStartTime = lastRepairStartTime;
+ this.lastRepairFinishTime = lastRepairFinishTime;
+ this.deleteHosts = deleteHosts;
+ if (this.deleteHosts == null)
+ {
+ this.deleteHosts = new HashSet<>();
+ }
+ this.deleteHostsUpdateTime = deleteHostsUpateTime;
+ this.forceRepair = forceRepair;
+ }
+
+ public String toString()
+ {
+ return MoreObjects.toStringHelper(this).
+ add("hostId", hostId).
+ add("repairTurn", repairTurn).
+ add("lastRepairStartTime", lastRepairStartTime).
+ add("lastRepairFinishTime", lastRepairFinishTime).
+ add("deleteHosts", deleteHosts).
+ toString();
+ }
+
+ public boolean isRepairRunning()
+ {
+ // if a repair history record has start time laster than finish time, it means the repair is running
+ return lastRepairStartTime > lastRepairFinishTime;
+ }
+
+ public long getLastRepairFinishTime()
+ {
+ return lastRepairFinishTime;
+ }
+ }
+
+ public static class CurrentRepairStatus
+ {
+ public Set<UUID> hostIdsWithOnGoingRepair; // hosts that is running repair
+ public Set<UUID> hostIdsWithOnGoingForceRepair; // hosts that is running repair because of force repair
+ Set<UUID> priority;
+ List<AutoRepairHistory> historiesWithoutOnGoingRepair; // hosts that is NOT running repair
+
+ public CurrentRepairStatus(List<AutoRepairHistory> repairHistories, Set<UUID> priority)
+ {
+ hostIdsWithOnGoingRepair = new HashSet<>();
+ hostIdsWithOnGoingForceRepair = new HashSet<>();
+ historiesWithoutOnGoingRepair = new ArrayList<>();
+
+ for (AutoRepairHistory history : repairHistories)
+ {
+ if (history.isRepairRunning())
+ {
+ if (history.forceRepair)
+ {
+ hostIdsWithOnGoingForceRepair.add(history.hostId);
+ }
+ else
+ {
+ hostIdsWithOnGoingRepair.add(history.hostId);
+ }
+ }
+ else
+ {
+ historiesWithoutOnGoingRepair.add(history);
+ }
+ }
+ this.priority = priority;
+ }
+
+ public String toString()
+ {
+ return MoreObjects.toStringHelper(this).
+ add("hostIdsWithOnGoingRepair", hostIdsWithOnGoingRepair).
+ add("hostIdsWithOnGoingForceRepair", hostIdsWithOnGoingForceRepair).
+ add("historiesWithoutOnGoingRepair", historiesWithoutOnGoingRepair).
+ add("priority", priority).
+ toString();
+ }
+ }
+
+ @VisibleForTesting
+ public static List<AutoRepairHistory> getAutoRepairHistoryByGroupID(RepairType repairType)
+ {
+ UntypedResultSet repairHistoryResult;
+
+ ResultMessage.Rows repairStatusRows = selectStatementRepairHistory.execute(QueryState.forInternalCalls(),
+ QueryOptions.forInternalCalls(internalQueryCL, Lists.newArrayList(ByteBufferUtil.bytes(repairType.toString()))), Dispatcher.RequestTime.forImmediateExecution());
+ repairHistoryResult = UntypedResultSet.create(repairStatusRows.result);
+
+ List<AutoRepairHistory> repairHistories = new ArrayList<>();
+ if (repairHistoryResult.size() > 0)
+ {
+ for (UntypedResultSet.Row row : repairHistoryResult)
+ {
+ UUID hostId = row.getUUID(COL_HOST_ID);
+ String repairTurn = null;
+ if (row.has(COL_REPAIR_TURN))
+ repairTurn = row.getString(COL_REPAIR_TURN);
+ long lastRepairStartTime = row.getLong(COL_REPAIR_START_TS, 0);
+ long lastRepairFinishTime = row.getLong(COL_REPAIR_FINISH_TS, 0);
+ Set<UUID> deleteHosts = row.getSet(COL_DELETE_HOSTS, UUIDType.instance);
+ long deleteHostsUpdateTime = row.getLong(COL_DELETE_HOSTS_UPDATE_TIME, 0);
+ Boolean forceRepair = row.has(COL_FORCE_REPAIR) ? row.getBoolean(COL_FORCE_REPAIR) : false;
+ repairHistories.add(new AutoRepairHistory(hostId, repairTurn, lastRepairStartTime, lastRepairFinishTime,
+ deleteHosts, deleteHostsUpdateTime, forceRepair));
+ }
+ return repairHistories;
+ }
+ logger.info("No repair history found");
+ return null;
+ }
+
+ public static List<AutoRepairHistory> getAutoRepairHistoryForLocalGroup(RepairType repairType)
+ {
+ return getAutoRepairHistoryByGroupID(repairType);
+ }
+
+ // A host may add itself in delete hosts for some other hosts due to restart or some temp gossip issue. If a node's record
+ // delete_hosts is not growing for more than 2 hours, we consider it as a normal node so we clear the delete_hosts for that node
+ public static void clearDeleteHosts(RepairType repairType, UUID hostId)
+ {
+ clearDeleteHostsStatement.execute(QueryState.forInternalCalls(),
+ QueryOptions.forInternalCalls(internalQueryCL,
+ Lists.newArrayList(ByteBufferUtil.bytes(repairType.toString()),
+ ByteBufferUtil.bytes(hostId))), Dispatcher.RequestTime.forImmediateExecution());
+ }
+
+ public static void setForceRepairNewNode(RepairType repairType)
+ {
+ // this function will be called when a node bootstrap finished
+ UUID hostId = Gossiper.instance.getHostId(FBUtilities.getBroadcastAddressAndPort());
+ // insert the data first
+ insertNewRepairHistory(repairType, System.currentTimeMillis(), System.currentTimeMillis());
Review Comment:
Incorporated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] autorepair v2 framework (against 4.1.6) [cassandra]
Posted by "tolbertam (via GitHub)" <gi...@apache.org>.
tolbertam commented on code in PR #3367:
URL: https://github.com/apache/cassandra/pull/3367#discussion_r1775208068
##########
src/java/org/apache/cassandra/repair/autorepair/AutoRepair.java:
##########
@@ -0,0 +1,457 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.repair.autorepair;
+
+import java.util.ArrayList;
+import java.util.EnumMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+import java.util.function.BiConsumer;
+import java.util.function.Consumer;
+import java.util.function.Supplier;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableList;
+import com.google.common.util.concurrent.Uninterruptibles;
+
+import org.apache.cassandra.repair.RepairRunnable;
+import org.apache.cassandra.utils.Pair;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.concurrent.ScheduledExecutorPlus;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.exceptions.ConfigurationException;
+import org.apache.cassandra.gms.Gossiper;
+import org.apache.cassandra.schema.TableMetadata;
+import org.apache.cassandra.schema.Tables;
+import org.apache.cassandra.service.AutoRepairService;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.repair.autorepair.AutoRepairUtils.RepairTurn;
+import org.apache.cassandra.utils.concurrent.Future;
+
+import static org.apache.cassandra.concurrent.ExecutorFactory.Global.executorFactory;
+import static org.apache.cassandra.repair.autorepair.AutoRepairUtils.RepairTurn.MY_TURN;
+import static org.apache.cassandra.repair.autorepair.AutoRepairUtils.RepairTurn.MY_TURN_DUE_TO_PRIORITY;
+import static org.apache.cassandra.repair.autorepair.AutoRepairUtils.RepairTurn.MY_TURN_FORCE_REPAIR;
+
+public class AutoRepair
+{
+ private static final Logger logger = LoggerFactory.getLogger(AutoRepair.class);
+
+ @VisibleForTesting
+ protected static Supplier<Long> timeFunc = System::currentTimeMillis;
+
+ public static AutoRepair instance = new AutoRepair();
+
+ @VisibleForTesting
+ protected final Map<AutoRepairConfig.RepairType, ScheduledExecutorPlus> repairExecutors;
+
+ protected final Map<AutoRepairConfig.RepairType, ScheduledExecutorPlus> repairRunnableExecutors;
+
+ @VisibleForTesting
+ protected final Map<AutoRepairConfig.RepairType, AutoRepairState> repairStates;
+
+ @VisibleForTesting
+ protected static Consumer<List<?>> shuffleFunc = java.util.Collections::shuffle;
+
+ @VisibleForTesting
+ protected static BiConsumer<Long, TimeUnit> sleepFunc = Uninterruptibles::sleepUninterruptibly;
+
+ protected final Map<AutoRepairConfig.RepairType, IAutoRepairTokenRangeSplitter> tokenRangeSplitters = new EnumMap<>(AutoRepairConfig.RepairType.class);
+
+
+ @VisibleForTesting
+ protected AutoRepair()
+ {
+ AutoRepairConfig config = DatabaseDescriptor.getAutoRepairConfig();
+ repairExecutors = new EnumMap<>(AutoRepairConfig.RepairType.class);
+ repairRunnableExecutors = new EnumMap<>(AutoRepairConfig.RepairType.class);
+ repairStates = new EnumMap<>(AutoRepairConfig.RepairType.class);
+ for (AutoRepairConfig.RepairType repairType : AutoRepairConfig.RepairType.values())
+ {
+ repairExecutors.put(repairType, executorFactory().scheduled(false, "AutoRepair-Repair-" + repairType, Thread.NORM_PRIORITY));
+ repairRunnableExecutors.put(repairType, executorFactory().scheduled(false, "AutoRepair-RepairRunnable-" + repairType, Thread.NORM_PRIORITY));
+ repairStates.put(repairType, AutoRepairConfig.RepairType.getAutoRepairState(repairType));
+ tokenRangeSplitters.put(repairType, FBUtilities.newAutoRepairTokenRangeSplitter(config.getTokenRangeSplitter(repairType)));
+ }
+ }
+
+ public void setup()
+ {
+ verifyIsSafeToEnable();
+
+ AutoRepairConfig config = DatabaseDescriptor.getAutoRepairConfig();
+ AutoRepairService.setup();
+ AutoRepairUtils.setup();
+
+ for (AutoRepairConfig.RepairType repairType : AutoRepairConfig.RepairType.values())
+ {
+ repairExecutors.get(repairType).scheduleWithFixedDelay(
+ () -> repair(repairType, 5000),
+ config.getInitialSchedulerDelay(repairType).toSeconds(),
+ config.getRepairCheckInterval().toSeconds(),
+ TimeUnit.SECONDS);
+ }
+ }
+
+ @VisibleForTesting
+ protected void verifyIsSafeToEnable()
+ {
+ AutoRepairConfig config = DatabaseDescriptor.getAutoRepairConfig();
+ if (config.isAutoRepairEnabled(AutoRepairConfig.RepairType.incremental) &&
+ (DatabaseDescriptor.getMaterializedViewsEnabled() || DatabaseDescriptor.isCDCEnabled()))
+ throw new ConfigurationException("Cannot enable incremental repair with materialized views or CDC enabled");
+ }
+
+ // repairAsync runs a repair session of the given type asynchronously.
+ public void repairAsync(AutoRepairConfig.RepairType repairType, long millisToWait)
+ {
+ repairExecutors.get(repairType).submit(() -> repair(repairType, millisToWait));
+ }
+
+ // repair runs a repair session of the given type synchronously.
+ public void repair(AutoRepairConfig.RepairType repairType, long millisToWait)
+ {
+ AutoRepairConfig config = AutoRepairService.instance.getAutoRepairConfig();
+ if (!config.isAutoRepairEnabled(repairType))
+ {
+ logger.debug("Auto-repair is disabled for repair type {}", repairType);
+ return;
+ }
+ AutoRepairState repairState = repairStates.get(repairType);
+ try
+ {
+ String localDC = DatabaseDescriptor.getLocalDataCenter();
+ if (config.getIgnoreDCs(repairType).contains(localDC))
+ {
+ logger.info("Not running repair as this node belongs to datacenter {}", localDC);
+ return;
+ }
+
+ // refresh the longest unrepaired node
+ repairState.setLongestUnrepairedNode(AutoRepairUtils.getHostWithLongestUnrepairTime(repairType));
+
+ //consistency level to use for local query
+ UUID myId = Gossiper.instance.getHostId(FBUtilities.getBroadcastAddressAndPort());
+ RepairTurn turn = AutoRepairUtils.myTurnToRunRepair(repairType, myId);
+ if (turn == MY_TURN || turn == MY_TURN_DUE_TO_PRIORITY || turn == MY_TURN_FORCE_REPAIR)
+ {
+ repairState.recordTurn(turn);
+ // For normal auto repair, we will use primary range only repairs (Repair with -pr option).
+ // For some cases, we may set the auto_repair_primary_token_range_only flag to false then we will do repair
+ // without -pr. We may also do force repair for certain node that we want to repair all the data on one node
+ // When doing force repair, we want to repair without -pr.
+ boolean primaryRangeOnly = config.getRepairPrimaryTokenRangeOnly(repairType)
+ && turn != MY_TURN_FORCE_REPAIR;
+ if (tooSoonToRunRepair(repairType, repairState, config, myId))
+ {
+ return;
+ }
+
+ long startTime = timeFunc.get();
+ logger.info("My host id: {}, my turn to run repair...repair primary-ranges only? {}", myId,
+ config.getRepairPrimaryTokenRangeOnly(repairType));
+ AutoRepairUtils.updateStartAutoRepairHistory(repairType, myId, timeFunc.get(), turn);
+
+ repairState.setRepairKeyspaceCount(0);
+ repairState.setRepairInProgress(true);
+ repairState.setTotalTablesConsideredForRepair(0);
+ repairState.setTotalMVTablesConsideredForRepair(0);
+
+ int failedTokenRanges = 0;
+ int succeededTokenRanges = 0;
+ int skippedTokenRanges = 0;
+
+ List<Keyspace> keyspaces = new ArrayList<>();
+ Keyspace.all().forEach(keyspaces::add);
+ // Auto-repair is likely to be run on multiple nodes independently, we want to avoid running multiple repair
+ // sessions on overlapping datasets at the same time. Shuffling keyspaces reduces the likelihood of this happening.
+ shuffleFunc.accept(keyspaces);
+
+ for (Keyspace keyspace : keyspaces)
+ {
+ if (!AutoRepairUtils.checkNodeContainsKeyspaceReplica(keyspace))
+ {
+ continue;
+ }
+
+ repairState.setRepairKeyspaceCount(repairState.getRepairKeyspaceCount() + 1);
+ List<String> tablesToBeRepaired = retrieveTablesToBeRepaired(keyspace, repairType, repairState);
+ shuffleFunc.accept(tablesToBeRepaired);
+ for (String tableName : tablesToBeRepaired)
+ {
+ String keyspaceName = keyspace.getName();
+ try
+ {
+ List<Pair<Token, Token>> subRangesToBeRepaired = tokenRangeSplitters.get(repairType).getRange(repairType, primaryRangeOnly, keyspaceName, tableName);
+ int totalSubRanges = subRangesToBeRepaired.size();
+
+ ColumnFamilyStore columnFamilyStore = keyspace.getColumnFamilyStore(tableName);
+ if (!columnFamilyStore.metadata().params.automatedRepair.get(repairType).repairEnabled())
+ {
+ logger.info("Repair is disabled for keyspace {} for tables: {}", keyspaceName, tableName);
+ repairState.setTotalDisabledTablesRepairCount(repairState.getTotalDisabledTablesRepairCount() + 1);
+ continue;
+ }
+ // this is done to make autorepair safe as running repair on table with more sstables
+ // may have its own challenges
+ int size = columnFamilyStore.getLiveSSTables().size();
+ if (size > config.getRepairSSTableCountHigherThreshold(repairType))
+ {
+ logger.info("Too many SSTables for repair, not doing repair on table {}.{} " +
+ "totalSSTables {}", keyspaceName, tableName, columnFamilyStore.getLiveSSTables().size());
+ skippedTokenRanges += totalSubRanges;
+ continue;
+ }
+
+ if (config.getRepairByKeyspace(repairType))
+ {
+ logger.info("Repair keyspace {} for tables: {}", keyspaceName, tablesToBeRepaired);
+ }
+ else
+ {
+ logger.info("Repair table {}.{}", keyspaceName, tableName);
+ }
+ long tableStartTime = timeFunc.get();
+ Set<Range<Token>> ranges = new HashSet<>();
+ int totalProcessedSubRanges = 0;
+ for (Pair<Token, Token> token : subRangesToBeRepaired)
+ {
+ if (!config.isAutoRepairEnabled(repairType))
+ {
+ logger.error("Auto-repair for type {} is disabled hence not running repair", repairType);
+ repairState.setRepairInProgress(false);
+ return;
+ }
+
+ if (config.getRepairByKeyspace(repairType))
+ {
+ if (AutoRepairUtils.keyspaceMaxRepairTimeExceeded(repairType, tableStartTime, tablesToBeRepaired.size()))
+ {
+ skippedTokenRanges += totalSubRanges - totalProcessedSubRanges;
+ logger.info("Keyspace took too much time to repair hence skipping it {}",
+ keyspaceName);
+ break;
+ }
+ }
+ else
+ {
+ if (AutoRepairUtils.tableMaxRepairTimeExceeded(repairType, tableStartTime))
+ {
+ skippedTokenRanges += totalSubRanges - totalProcessedSubRanges;
+ logger.info("Table took too much time to repair hence skipping it {}.{}",
+ keyspaceName, tableName);
+ break;
+ }
+ }
+ Token childStartToken = token.left;
+ Token childEndToken = token.right;
+ logger.debug("Current Token Left side {}, right side {}", childStartToken
+ .toString(), childEndToken.toString());
+
+ ranges.add(new Range<>(childStartToken, childEndToken));
+ totalProcessedSubRanges++;
+ if ((totalProcessedSubRanges % config.getRepairThreads(repairType) == 0) ||
+ (totalProcessedSubRanges == totalSubRanges))
+ {
+ int retryCount = 0;
+ Future<?> f = null;
+ while (retryCount <= config.getRepairMaxRetries())
+ {
+ RepairRunnable task = repairState.getRepairRunnable(keyspaceName,
+ config.getRepairByKeyspace(repairType) ? tablesToBeRepaired : ImmutableList.of(tableName),
Review Comment:
Ah nevermind, I see there is a break at line 340 that breaks out after evaluating over the first iteration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] autorepair v2 framework (against 4.1.6) [cassandra]
Posted by "tolbertam (via GitHub)" <gi...@apache.org>.
tolbertam commented on code in PR #3367:
URL: https://github.com/apache/cassandra/pull/3367#discussion_r1775200138
##########
src/java/org/apache/cassandra/repair/autorepair/AutoRepair.java:
##########
@@ -0,0 +1,457 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.repair.autorepair;
+
+import java.util.ArrayList;
+import java.util.EnumMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+import java.util.function.BiConsumer;
+import java.util.function.Consumer;
+import java.util.function.Supplier;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableList;
+import com.google.common.util.concurrent.Uninterruptibles;
+
+import org.apache.cassandra.repair.RepairRunnable;
+import org.apache.cassandra.utils.Pair;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.concurrent.ScheduledExecutorPlus;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.exceptions.ConfigurationException;
+import org.apache.cassandra.gms.Gossiper;
+import org.apache.cassandra.schema.TableMetadata;
+import org.apache.cassandra.schema.Tables;
+import org.apache.cassandra.service.AutoRepairService;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.repair.autorepair.AutoRepairUtils.RepairTurn;
+import org.apache.cassandra.utils.concurrent.Future;
+
+import static org.apache.cassandra.concurrent.ExecutorFactory.Global.executorFactory;
+import static org.apache.cassandra.repair.autorepair.AutoRepairUtils.RepairTurn.MY_TURN;
+import static org.apache.cassandra.repair.autorepair.AutoRepairUtils.RepairTurn.MY_TURN_DUE_TO_PRIORITY;
+import static org.apache.cassandra.repair.autorepair.AutoRepairUtils.RepairTurn.MY_TURN_FORCE_REPAIR;
+
+public class AutoRepair
+{
+ private static final Logger logger = LoggerFactory.getLogger(AutoRepair.class);
+
+ @VisibleForTesting
+ protected static Supplier<Long> timeFunc = System::currentTimeMillis;
+
+ public static AutoRepair instance = new AutoRepair();
+
+ @VisibleForTesting
+ protected final Map<AutoRepairConfig.RepairType, ScheduledExecutorPlus> repairExecutors;
+
+ protected final Map<AutoRepairConfig.RepairType, ScheduledExecutorPlus> repairRunnableExecutors;
+
+ @VisibleForTesting
+ protected final Map<AutoRepairConfig.RepairType, AutoRepairState> repairStates;
+
+ @VisibleForTesting
+ protected static Consumer<List<?>> shuffleFunc = java.util.Collections::shuffle;
+
+ @VisibleForTesting
+ protected static BiConsumer<Long, TimeUnit> sleepFunc = Uninterruptibles::sleepUninterruptibly;
+
+ protected final Map<AutoRepairConfig.RepairType, IAutoRepairTokenRangeSplitter> tokenRangeSplitters = new EnumMap<>(AutoRepairConfig.RepairType.class);
+
+
+ @VisibleForTesting
+ protected AutoRepair()
+ {
+ AutoRepairConfig config = DatabaseDescriptor.getAutoRepairConfig();
+ repairExecutors = new EnumMap<>(AutoRepairConfig.RepairType.class);
+ repairRunnableExecutors = new EnumMap<>(AutoRepairConfig.RepairType.class);
+ repairStates = new EnumMap<>(AutoRepairConfig.RepairType.class);
+ for (AutoRepairConfig.RepairType repairType : AutoRepairConfig.RepairType.values())
+ {
+ repairExecutors.put(repairType, executorFactory().scheduled(false, "AutoRepair-Repair-" + repairType, Thread.NORM_PRIORITY));
+ repairRunnableExecutors.put(repairType, executorFactory().scheduled(false, "AutoRepair-RepairRunnable-" + repairType, Thread.NORM_PRIORITY));
+ repairStates.put(repairType, AutoRepairConfig.RepairType.getAutoRepairState(repairType));
+ tokenRangeSplitters.put(repairType, FBUtilities.newAutoRepairTokenRangeSplitter(config.getTokenRangeSplitter(repairType)));
+ }
+ }
+
+ public void setup()
+ {
+ verifyIsSafeToEnable();
+
+ AutoRepairConfig config = DatabaseDescriptor.getAutoRepairConfig();
+ AutoRepairService.setup();
+ AutoRepairUtils.setup();
+
+ for (AutoRepairConfig.RepairType repairType : AutoRepairConfig.RepairType.values())
+ {
+ repairExecutors.get(repairType).scheduleWithFixedDelay(
+ () -> repair(repairType, 5000),
+ config.getInitialSchedulerDelay(repairType).toSeconds(),
+ config.getRepairCheckInterval().toSeconds(),
+ TimeUnit.SECONDS);
+ }
+ }
+
+ @VisibleForTesting
+ protected void verifyIsSafeToEnable()
+ {
+ AutoRepairConfig config = DatabaseDescriptor.getAutoRepairConfig();
+ if (config.isAutoRepairEnabled(AutoRepairConfig.RepairType.incremental) &&
+ (DatabaseDescriptor.getMaterializedViewsEnabled() || DatabaseDescriptor.isCDCEnabled()))
+ throw new ConfigurationException("Cannot enable incremental repair with materialized views or CDC enabled");
+ }
+
+ // repairAsync runs a repair session of the given type asynchronously.
+ public void repairAsync(AutoRepairConfig.RepairType repairType, long millisToWait)
+ {
+ repairExecutors.get(repairType).submit(() -> repair(repairType, millisToWait));
+ }
+
+ // repair runs a repair session of the given type synchronously.
+ public void repair(AutoRepairConfig.RepairType repairType, long millisToWait)
+ {
+ AutoRepairConfig config = AutoRepairService.instance.getAutoRepairConfig();
+ if (!config.isAutoRepairEnabled(repairType))
+ {
+ logger.debug("Auto-repair is disabled for repair type {}", repairType);
+ return;
+ }
+ AutoRepairState repairState = repairStates.get(repairType);
+ try
+ {
+ String localDC = DatabaseDescriptor.getLocalDataCenter();
+ if (config.getIgnoreDCs(repairType).contains(localDC))
+ {
+ logger.info("Not running repair as this node belongs to datacenter {}", localDC);
+ return;
+ }
+
+ // refresh the longest unrepaired node
+ repairState.setLongestUnrepairedNode(AutoRepairUtils.getHostWithLongestUnrepairTime(repairType));
+
+ //consistency level to use for local query
+ UUID myId = Gossiper.instance.getHostId(FBUtilities.getBroadcastAddressAndPort());
+ RepairTurn turn = AutoRepairUtils.myTurnToRunRepair(repairType, myId);
+ if (turn == MY_TURN || turn == MY_TURN_DUE_TO_PRIORITY || turn == MY_TURN_FORCE_REPAIR)
+ {
+ repairState.recordTurn(turn);
+ // For normal auto repair, we will use primary range only repairs (Repair with -pr option).
+ // For some cases, we may set the auto_repair_primary_token_range_only flag to false then we will do repair
+ // without -pr. We may also do force repair for certain node that we want to repair all the data on one node
+ // When doing force repair, we want to repair without -pr.
+ boolean primaryRangeOnly = config.getRepairPrimaryTokenRangeOnly(repairType)
+ && turn != MY_TURN_FORCE_REPAIR;
+ if (tooSoonToRunRepair(repairType, repairState, config, myId))
+ {
+ return;
+ }
+
+ long startTime = timeFunc.get();
+ logger.info("My host id: {}, my turn to run repair...repair primary-ranges only? {}", myId,
+ config.getRepairPrimaryTokenRangeOnly(repairType));
+ AutoRepairUtils.updateStartAutoRepairHistory(repairType, myId, timeFunc.get(), turn);
+
+ repairState.setRepairKeyspaceCount(0);
+ repairState.setRepairInProgress(true);
+ repairState.setTotalTablesConsideredForRepair(0);
+ repairState.setTotalMVTablesConsideredForRepair(0);
+
+ int failedTokenRanges = 0;
+ int succeededTokenRanges = 0;
+ int skippedTokenRanges = 0;
+
+ List<Keyspace> keyspaces = new ArrayList<>();
+ Keyspace.all().forEach(keyspaces::add);
+ // Auto-repair is likely to be run on multiple nodes independently, we want to avoid running multiple repair
+ // sessions on overlapping datasets at the same time. Shuffling keyspaces reduces the likelihood of this happening.
+ shuffleFunc.accept(keyspaces);
+
+ for (Keyspace keyspace : keyspaces)
+ {
+ if (!AutoRepairUtils.checkNodeContainsKeyspaceReplica(keyspace))
+ {
+ continue;
+ }
+
+ repairState.setRepairKeyspaceCount(repairState.getRepairKeyspaceCount() + 1);
+ List<String> tablesToBeRepaired = retrieveTablesToBeRepaired(keyspace, repairType, repairState);
+ shuffleFunc.accept(tablesToBeRepaired);
+ for (String tableName : tablesToBeRepaired)
+ {
+ String keyspaceName = keyspace.getName();
+ try
+ {
+ List<Pair<Token, Token>> subRangesToBeRepaired = tokenRangeSplitters.get(repairType).getRange(repairType, primaryRangeOnly, keyspaceName, tableName);
+ int totalSubRanges = subRangesToBeRepaired.size();
+
+ ColumnFamilyStore columnFamilyStore = keyspace.getColumnFamilyStore(tableName);
+ if (!columnFamilyStore.metadata().params.automatedRepair.get(repairType).repairEnabled())
+ {
+ logger.info("Repair is disabled for keyspace {} for tables: {}", keyspaceName, tableName);
+ repairState.setTotalDisabledTablesRepairCount(repairState.getTotalDisabledTablesRepairCount() + 1);
+ continue;
+ }
+ // this is done to make autorepair safe as running repair on table with more sstables
+ // may have its own challenges
+ int size = columnFamilyStore.getLiveSSTables().size();
+ if (size > config.getRepairSSTableCountHigherThreshold(repairType))
+ {
+ logger.info("Too many SSTables for repair, not doing repair on table {}.{} " +
+ "totalSSTables {}", keyspaceName, tableName, columnFamilyStore.getLiveSSTables().size());
+ skippedTokenRanges += totalSubRanges;
+ continue;
+ }
+
+ if (config.getRepairByKeyspace(repairType))
+ {
+ logger.info("Repair keyspace {} for tables: {}", keyspaceName, tablesToBeRepaired);
+ }
+ else
+ {
+ logger.info("Repair table {}.{}", keyspaceName, tableName);
+ }
+ long tableStartTime = timeFunc.get();
+ Set<Range<Token>> ranges = new HashSet<>();
+ int totalProcessedSubRanges = 0;
+ for (Pair<Token, Token> token : subRangesToBeRepaired)
+ {
+ if (!config.isAutoRepairEnabled(repairType))
+ {
+ logger.error("Auto-repair for type {} is disabled hence not running repair", repairType);
+ repairState.setRepairInProgress(false);
+ return;
+ }
+
+ if (config.getRepairByKeyspace(repairType))
+ {
+ if (AutoRepairUtils.keyspaceMaxRepairTimeExceeded(repairType, tableStartTime, tablesToBeRepaired.size()))
+ {
+ skippedTokenRanges += totalSubRanges - totalProcessedSubRanges;
+ logger.info("Keyspace took too much time to repair hence skipping it {}",
+ keyspaceName);
+ break;
+ }
+ }
+ else
+ {
+ if (AutoRepairUtils.tableMaxRepairTimeExceeded(repairType, tableStartTime))
+ {
+ skippedTokenRanges += totalSubRanges - totalProcessedSubRanges;
+ logger.info("Table took too much time to repair hence skipping it {}.{}",
+ keyspaceName, tableName);
+ break;
+ }
+ }
+ Token childStartToken = token.left;
+ Token childEndToken = token.right;
+ logger.debug("Current Token Left side {}, right side {}", childStartToken
+ .toString(), childEndToken.toString());
+
+ ranges.add(new Range<>(childStartToken, childEndToken));
+ totalProcessedSubRanges++;
+ if ((totalProcessedSubRanges % config.getRepairThreads(repairType) == 0) ||
+ (totalProcessedSubRanges == totalSubRanges))
+ {
+ int retryCount = 0;
+ Future<?> f = null;
+ while (retryCount <= config.getRepairMaxRetries())
+ {
+ RepairRunnable task = repairState.getRepairRunnable(keyspaceName,
+ config.getRepairByKeyspace(repairType) ? tablesToBeRepaired : ImmutableList.of(tableName),
Review Comment:
Since this is being evaluated for each table, won't this mean that for each Table in a keyspace, it will issue a full keyspace repair?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] autorepair v2 framework (against 4.1.6) [cassandra]
Posted by "jaydeepkumar1984 (via GitHub)" <gi...@apache.org>.
jaydeepkumar1984 commented on code in PR #3367:
URL: https://github.com/apache/cassandra/pull/3367#discussion_r1794436900
##########
src/java/org/apache/cassandra/tools/nodetool/GetAutoRepairConfig.java:
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.tools.nodetool;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Joiner;
+import io.airlift.airline.Command;
+import org.apache.cassandra.repair.autorepair.AutoRepairConfig;
+import org.apache.cassandra.repair.autorepair.AutoRepairConfig.RepairType;
+import org.apache.cassandra.tools.NodeProbe;
+import org.apache.cassandra.tools.NodeTool.NodeToolCmd;
+
+import java.io.PrintStream;
+
+@Command(name = "getautorepairconfig", description = "Print autorepair configurations")
+public class GetAutoRepairConfig extends NodeToolCmd
+{
+ @VisibleForTesting
+ protected static PrintStream out = System.out;
+
+ @Override
+ public void execute(NodeProbe probe)
+ {
+ AutoRepairConfig config = probe.getAutoRepairConfig();
+ if (config == null || !config.isAutoRepairSchedulingEnabled())
+ {
+ out.println("Auto-repair is not enabled");
+ return;
+ }
+
+ StringBuilder sb = new StringBuilder();
+ sb.append("repair scheduler configuration:");
+ sb.append("\n\trepair eligibility check interval: " + config.getRepairCheckInterval());
+ sb.append("\n\tTTL for repair history for dead nodes: " + config.getAutoRepairHistoryClearDeleteHostsBufferInterval());
+ sb.append("\n\tmax retries for repair: " + config.getRepairMaxRetries());
+ sb.append("\n\tretry backoff: " + config.getRepairRetryBackoff());
+ for (RepairType repairType : RepairType.values())
+ {
+ sb.append(formatRepairTypeConfig(probe, repairType, config));
+ }
+
+ out.println(sb);
+ }
+
+ private String formatRepairTypeConfig(NodeProbe probe, RepairType repairType, AutoRepairConfig config)
+ {
+ StringBuilder sb = new StringBuilder();
+ sb.append("\nconfiguration for repair type: " + repairType);
+ sb.append("\n\tenabled: " + config.isAutoRepairEnabled(repairType));
+ sb.append("\n\tminimum repair interval: " + config.getRepairMinInterval(repairType));
+ sb.append("\n\trepair threads: " + config.getRepairThreads(repairType));
+ sb.append("\n\tnumber of repair subranges: " + config.getRepairSubRangeNum(repairType));
+ sb.append("\n\tpriority hosts: " + Joiner.on(',').skipNulls().join(probe.getRepairPriorityForHosts(repairType)));
+ sb.append("\n\tsstable count higher threshold: " + config.getRepairSSTableCountHigherThreshold(repairType));
+ sb.append("\n\ttable max repair time in sec: " + config.getAutoRepairTableMaxRepairTime(repairType));
+ sb.append("\n\tignore datacenters: " + Joiner.on(',').skipNulls().join(config.getIgnoreDCs(repairType)));
+ sb.append("\n\trepair primary token-range: " + config.getRepairPrimaryTokenRangeOnly(repairType));
+ sb.append("\n\tnumber of parallel repairs within group: " + config.getParallelRepairCount(repairType));
+ sb.append("\n\tpercentage of parallel repairs within group: " + config.getParallelRepairPercentage(repairType));
+ sb.append("\n\tmv repair enabled: " + config.getMVRepairEnabled(repairType));
+ sb.append("\n\tinitial scheduler delay: " + config.getInitialSchedulerDelay(repairType));
+ sb.append("\n\trepair setssion timeout: " + config.getRepairSessionTimeout(repairType));
Review Comment:
Incorporated
##########
test/unit/org/apache/cassandra/cql3/statements/DescribeStatementTest.java:
##########
@@ -1004,7 +1004,9 @@ private static String tableParametersCql()
" AND memtable_flush_period_in_ms = 0\n" +
" AND min_index_interval = 128\n" +
" AND read_repair = 'BLOCKING'\n" +
- " AND speculative_retry = '99p';";
+ " AND speculative_retry = '99p'\n" +
+ " AND automated_repair_full = {'enabled': 'true'}\n" +
Review Comment:
Incorporated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] autorepair v2 framework (against 4.1.6) [cassandra]
Posted by "tolbertam (via GitHub)" <gi...@apache.org>.
tolbertam commented on code in PR #3367:
URL: https://github.com/apache/cassandra/pull/3367#discussion_r1827044723
##########
src/java/org/apache/cassandra/config/CassandraRelevantProperties.java:
##########
@@ -346,6 +346,17 @@ public enum CassandraRelevantProperties
*/
GOSSIP_SETTLE_POLL_SUCCESSES_REQUIRED("cassandra.gossip_settle_poll_success_required", "3"),
+ /**
+ * If set to true, mutations streamed during anti-entropy repair will be replayed via the regular write path for associated views.
+ */
+ STREAMING_REQUIRES_VIEW_BUILD_DURING_REPAIR("cassandra.streaming.requires_view_build_during_repair", "true"),
Review Comment:
This currently prevents startup if trying to configure an incremental repair schedule:
```
ERROR [main] 2024-11-03 12:48:38,892 CassandraDaemon.java:900 - Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: Cannot run incremental repair while materialized view replay is enabled.
at org.apache.cassandra.service.AutoRepairService.checkCanRun(AutoRepairService.java:69)
at org.apache.cassandra.repair.autorepair.AutoRepair.setup(AutoRepair.java:128)
at org.apache.cassandra.service.StorageService.doAutoRepairSetup(StorageService.java:1355)
at org.apache.cassandra.service.StorageService.finishJoiningRing(StorageService.java:1326)
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1214)
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1150)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:941)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:859)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:421)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:744)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:878)
```
It seems a bit unnecessary to have to set some system properties to run an incremental repair schedule if we aren't using mvs or cdc, was wondering if we could make this smarter. Currently both materialized views and cdc have guardrail properties, `materialized_views_enabled` and `cdc_enabled` which both default to false. Can we involve these properties somehow? If they are not set, then we don't need to consider either during streaming.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] autorepair v2 framework (against 4.1.6) [cassandra]
Posted by "clohfink (via GitHub)" <gi...@apache.org>.
clohfink commented on code in PR #3367:
URL: https://github.com/apache/cassandra/pull/3367#discussion_r1683348623
##########
src/java/org/apache/cassandra/schema/TableParams.java:
##########
@@ -105,6 +112,13 @@ private TableParams(Builder builder)
extensions = builder.extensions;
cdc = builder.cdc;
readRepair = builder.readRepair;
+ automatedRepair = new HashMap<>()
Review Comment:
```
automatedRepair = new EnumMap<>(AutoRepairConfig.RepairType.class);
automatedRepair.put(AutoRepairConfig.RepairType.full, builder.automatedRepairFull);
automatedRepair.put(AutoRepairConfig.RepairType.incremental, builder.automatedRepairIncremental);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] autorepair v2 framework (against 4.1.6) [cassandra]
Posted by "clohfink (via GitHub)" <gi...@apache.org>.
clohfink commented on code in PR #3367:
URL: https://github.com/apache/cassandra/pull/3367#discussion_r1794096745
##########
test/unit/org/apache/cassandra/cql3/statements/DescribeStatementTest.java:
##########
@@ -1004,7 +1004,9 @@ private static String tableParametersCql()
" AND memtable_flush_period_in_ms = 0\n" +
" AND min_index_interval = 128\n" +
" AND read_repair = 'BLOCKING'\n" +
- " AND speculative_retry = '99p';";
+ " AND speculative_retry = '99p'\n" +
+ " AND automated_repair_full = {'enabled': 'true'}\n" +
Review Comment:
np: should we maybe nix the "automated" prefix?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] autorepair v2 framework (against 4.1.6) [cassandra]
Posted by "clohfink (via GitHub)" <gi...@apache.org>.
clohfink commented on code in PR #3367:
URL: https://github.com/apache/cassandra/pull/3367#discussion_r1771956176
##########
test/unit/org/apache/cassandra/repair/autorepair/AutoRepairParameterizedTest.java:
##########
@@ -0,0 +1,681 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.repair.autorepair;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Sets;
+
+import org.apache.cassandra.config.DurationSpec;
Review Comment:
unused import breaks build with checkstyles
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] autorepair v2 framework (against 4.1.6) [cassandra]
Posted by "jaydeepkumar1984 (via GitHub)" <gi...@apache.org>.
jaydeepkumar1984 commented on code in PR #3367:
URL: https://github.com/apache/cassandra/pull/3367#discussion_r1695735953
##########
src/java/org/apache/cassandra/repair/autorepair/AutoRepair.java:
##########
@@ -0,0 +1,401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.repair.autorepair;
+
+import java.util.ArrayList;
+import java.util.EnumMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.FutureTask;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableList;
+
+import org.apache.cassandra.repair.RepairRunnable;
+import org.apache.cassandra.utils.Pair;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.concurrent.NamedThreadFactory;
+import org.apache.cassandra.concurrent.ScheduledExecutorPlus;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.exceptions.ConfigurationException;
+import org.apache.cassandra.gms.Gossiper;
+import org.apache.cassandra.schema.TableMetadata;
+import org.apache.cassandra.schema.Tables;
+import org.apache.cassandra.service.AutoRepairService;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.repair.autorepair.AutoRepairUtils.RepairTurn;
+
+import static org.apache.cassandra.concurrent.ExecutorFactory.Global.executorFactory;
+import static org.apache.cassandra.repair.autorepair.AutoRepairUtils.RepairTurn.MY_TURN;
+import static org.apache.cassandra.repair.autorepair.AutoRepairUtils.RepairTurn.MY_TURN_DUE_TO_PRIORITY;
+import static org.apache.cassandra.repair.autorepair.AutoRepairUtils.RepairTurn.MY_TURN_FORCE_REPAIR;
+
+public class AutoRepair
+{
+ // Initial delay for repair session to start after setup
+ final static long INITIAL_REPAIR_DELAY_SEC = 30;
+
+ private static final Logger logger = LoggerFactory.getLogger(AutoRepair.class);
+
+ @VisibleForTesting
+ protected static Supplier<Long> timeFunc = System::currentTimeMillis;
+
+ public static AutoRepair instance = new AutoRepair();
+
+ @VisibleForTesting
+ protected final Map<AutoRepairConfig.RepairType, ScheduledExecutorPlus> repairExecutors;
+ @VisibleForTesting
+ protected final Map<AutoRepairConfig.RepairType, AutoRepairState> repairStates;
+
+ protected final Map<AutoRepairConfig.RepairType, IAutoRepairTokenRangeSplitter> tokenRangeSplitters = new EnumMap<>(AutoRepairConfig.RepairType.class);
+
+
+ @VisibleForTesting
+ protected AutoRepair()
+ {
+ AutoRepairConfig config = DatabaseDescriptor.getAutoRepairConfig();
+ repairExecutors = new EnumMap<>(AutoRepairConfig.RepairType.class);
+ repairStates = new EnumMap<>(AutoRepairConfig.RepairType.class);
+ for (AutoRepairConfig.RepairType repairType : AutoRepairConfig.RepairType.values())
+ {
+ repairExecutors.put(repairType, executorFactory().scheduled(false, "AutoRepair-Repair-" + repairType, Thread.NORM_PRIORITY));
+ repairStates.put(repairType, AutoRepairConfig.RepairType.getAutoRepairState(repairType));
+ tokenRangeSplitters.put(repairType, FBUtilities.newAutoRepairTokenRangeSplitter(config.getTokenRangeSplitter(repairType)));
+ }
+ }
+
+ public void setup()
+ {
+ verifyIsSafeToEnable();
+
+ AutoRepairConfig config = DatabaseDescriptor.getAutoRepairConfig();
+ AutoRepairService.setup();
+ AutoRepairUtils.setup();
+
+ for (AutoRepairConfig.RepairType repairType : AutoRepairConfig.RepairType.values())
+ {
+ repairExecutors.get(repairType).scheduleWithFixedDelay(
+ () -> repair(repairType, 60000),
+ INITIAL_REPAIR_DELAY_SEC,
+ config.getRepairCheckInterval().toSeconds(),
+ TimeUnit.SECONDS);
+ }
+ }
+
+ @VisibleForTesting
+ protected void verifyIsSafeToEnable()
+ {
+ AutoRepairConfig config = DatabaseDescriptor.getAutoRepairConfig();
+ if (config.isAutoRepairEnabled(AutoRepairConfig.RepairType.incremental) &&
+ (DatabaseDescriptor.getMaterializedViewsEnabled() || DatabaseDescriptor.isCDCEnabled()))
+ throw new ConfigurationException("Cannot enable incremental repair with materialized views or CDC enabled");
+ }
+
+ // repairAsync runs a repair session of the given type asynchronously.
+ public void repairAsync(AutoRepairConfig.RepairType repairType, long millisToWait)
+ {
+ repairExecutors.get(repairType).submit(() -> repair(repairType, millisToWait));
+ }
+
+ // repair runs a repair session of the given type synchronously.
+ public void repair(AutoRepairConfig.RepairType repairType, long millisToWait)
Review Comment:
Chopped it a bit - see if this looks better now
##########
test/unit/org/apache/cassandra/repair/autorepair/AutoRepairParameterizedTest.java:
##########
@@ -0,0 +1,581 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.repair.autorepair;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Sets;
+
+import org.apache.cassandra.cql3.statements.schema.TableAttributes;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.repair.RepairRunnable;
+import org.apache.cassandra.schema.AutoRepairParams;
+import org.apache.cassandra.schema.TableParams;
+import org.apache.cassandra.service.StorageService;
+import org.apache.cassandra.utils.Pair;
+
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import junit.framework.Assert;
+import org.apache.cassandra.SchemaLoader;
+import org.apache.cassandra.concurrent.ScheduledExecutorPlus;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.cql3.CQLTester;
+import org.apache.cassandra.cql3.QueryProcessor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.marshal.IntegerType;
+import org.apache.cassandra.db.marshal.UTF8Type;
+import org.apache.cassandra.gms.Gossiper;
+import org.apache.cassandra.io.sstable.format.SSTableReader;
+import org.apache.cassandra.metrics.AutoRepairMetricsManager;
+import org.apache.cassandra.metrics.AutoRepairMetrics;
+import org.apache.cassandra.schema.KeyspaceParams;
+import org.apache.cassandra.schema.Schema;
+import org.apache.cassandra.schema.SchemaConstants;
+import org.apache.cassandra.schema.TableMetadata;
+import org.apache.cassandra.service.AutoRepairService;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.utils.progress.ProgressEvent;
+import org.mockito.Mock;
+import org.mockito.Mockito;
+import org.mockito.MockitoAnnotations;
+
+import static org.apache.cassandra.Util.setAutoRepairEnabled;
+import static org.apache.cassandra.Util.token;
Review Comment:
Incorporated
##########
test/unit/org/apache/cassandra/repair/autorepair/AutoRepairParameterizedTest.java:
##########
@@ -0,0 +1,581 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.repair.autorepair;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Sets;
+
+import org.apache.cassandra.cql3.statements.schema.TableAttributes;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.repair.RepairRunnable;
+import org.apache.cassandra.schema.AutoRepairParams;
+import org.apache.cassandra.schema.TableParams;
+import org.apache.cassandra.service.StorageService;
+import org.apache.cassandra.utils.Pair;
+
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import junit.framework.Assert;
Review Comment:
Incorporated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] autorepair v2 framework (against 4.1.6) [cassandra]
Posted by "tolbertam (via GitHub)" <gi...@apache.org>.
tolbertam commented on code in PR #3367:
URL: https://github.com/apache/cassandra/pull/3367#discussion_r1827044723
##########
src/java/org/apache/cassandra/config/CassandraRelevantProperties.java:
##########
@@ -346,6 +346,17 @@ public enum CassandraRelevantProperties
*/
GOSSIP_SETTLE_POLL_SUCCESSES_REQUIRED("cassandra.gossip_settle_poll_success_required", "3"),
+ /**
+ * If set to true, mutations streamed during anti-entropy repair will be replayed via the regular write path for associated views.
+ */
+ STREAMING_REQUIRES_VIEW_BUILD_DURING_REPAIR("cassandra.streaming.requires_view_build_during_repair", "true"),
Review Comment:
This currently prevents startup if trying to configure an incremental repair schedule:
```
ERROR [main] 2024-11-03 12:48:38,892 CassandraDaemon.java:900 - Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: Cannot run incremental repair while materialized view replay is enabled.
at org.apache.cassandra.service.AutoRepairService.checkCanRun(AutoRepairService.java:69)
at org.apache.cassandra.repair.autorepair.AutoRepair.setup(AutoRepair.java:128)
at org.apache.cassandra.service.StorageService.doAutoRepairSetup(StorageService.java:1355)
at org.apache.cassandra.service.StorageService.finishJoiningRing(StorageService.java:1326)
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1214)
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1150)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:941)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:859)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:421)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:744)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:878)
```
It seems a bit rigid to have to set some system properties to run an incremental repair schedule, was wondering if we could make this smarter. Currently both materialized views and cdc have guardrail properties, `materialized_views_enabled` and `cdc_enabled` which both default to false. Can we involve these properties somehow? If they are not set, then we don't need to consider either during streaming.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] autorepair v2 framework (against 4.1.6) [cassandra]
Posted by "tolbertam (via GitHub)" <gi...@apache.org>.
tolbertam commented on code in PR #3367:
URL: https://github.com/apache/cassandra/pull/3367#discussion_r1827044723
##########
src/java/org/apache/cassandra/config/CassandraRelevantProperties.java:
##########
@@ -346,6 +346,17 @@ public enum CassandraRelevantProperties
*/
GOSSIP_SETTLE_POLL_SUCCESSES_REQUIRED("cassandra.gossip_settle_poll_success_required", "3"),
+ /**
+ * If set to true, mutations streamed during anti-entropy repair will be replayed via the regular write path for associated views.
+ */
+ STREAMING_REQUIRES_VIEW_BUILD_DURING_REPAIR("cassandra.streaming.requires_view_build_during_repair", "true"),
Review Comment:
This currently prevents startup if trying to configure an incremental repair schedule:
```
ERROR [main] 2024-11-03 12:48:38,892 CassandraDaemon.java:900 - Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: Cannot run incremental repair while materialized view replay is enabled.
at org.apache.cassandra.service.AutoRepairService.checkCanRun(AutoRepairService.java:69)
at org.apache.cassandra.repair.autorepair.AutoRepair.setup(AutoRepair.java:128)
at org.apache.cassandra.service.StorageService.doAutoRepairSetup(StorageService.java:1355)
at org.apache.cassandra.service.StorageService.finishJoiningRing(StorageService.java:1326)
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1214)
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1150)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:941)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:859)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:421)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:744)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:878)
```
It seems a bit terse to have to set some system properties to run an incremental repair schedule, was wondering if we could make this smarter. Currently both materialized views and cdc have guardrail properties, `materialized_views_enabled` and `cdc_enabled` which both default to false. Can we involve these properties somehow? If they are not set, then we don't need to consider either during streaming.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] autorepair v2 framework (against 4.1.6) [cassandra]
Posted by "tolbertam (via GitHub)" <gi...@apache.org>.
tolbertam commented on code in PR #3367:
URL: https://github.com/apache/cassandra/pull/3367#discussion_r1827044723
##########
src/java/org/apache/cassandra/config/CassandraRelevantProperties.java:
##########
@@ -346,6 +346,17 @@ public enum CassandraRelevantProperties
*/
GOSSIP_SETTLE_POLL_SUCCESSES_REQUIRED("cassandra.gossip_settle_poll_success_required", "3"),
+ /**
+ * If set to true, mutations streamed during anti-entropy repair will be replayed via the regular write path for associated views.
+ */
+ STREAMING_REQUIRES_VIEW_BUILD_DURING_REPAIR("cassandra.streaming.requires_view_build_during_repair", "true"),
Review Comment:
This currently prevents startup if trying to configure an incremental repair schedule:
```
ERROR [main] 2024-11-03 12:48:38,892 CassandraDaemon.java:900 - Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: Cannot run incremental repair while materialized view replay is enabled.
at org.apache.cassandra.service.AutoRepairService.checkCanRun(AutoRepairService.java:69)
at org.apache.cassandra.repair.autorepair.AutoRepair.setup(AutoRepair.java:128)
at org.apache.cassandra.service.StorageService.doAutoRepairSetup(StorageService.java:1355)
at org.apache.cassandra.service.StorageService.finishJoiningRing(StorageService.java:1326)
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1214)
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1150)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:941)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:859)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:421)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:744)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:878)
```
It seems a bit complex to have to set some system properties to run an incremental repair schedule, was wondering if we could make this smarter. Currently both materialized views and cdc have guardrail properties, `materialized_views_enabled` and `cdc_enabled` which both default to false. Can we involve these properties somehow? If they are not set, then we don't need to consider either during streaming.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] autorepair v2 framework (against 4.1.6) [cassandra]
Posted by "clohfink (via GitHub)" <gi...@apache.org>.
clohfink commented on code in PR #3367:
URL: https://github.com/apache/cassandra/pull/3367#discussion_r1794018195
##########
src/java/org/apache/cassandra/repair/autorepair/AutoRepairKeyspace.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.repair.autorepair;
+
+import java.util.concurrent.TimeUnit;
+
+import org.apache.cassandra.cql3.statements.schema.CreateTableStatement;
+import org.apache.cassandra.schema.KeyspaceMetadata;
+import org.apache.cassandra.schema.KeyspaceParams;
+import org.apache.cassandra.schema.SchemaConstants;
+import org.apache.cassandra.schema.TableId;
+import org.apache.cassandra.schema.TableMetadata;
+import org.apache.cassandra.schema.Tables;
+
+public class AutoRepairKeyspace
+{
+ private AutoRepairKeyspace()
+ {
+ }
+
+ /**
+ * Generation is used as a timestamp for automatic table creation on startup.
+ * If you make any changes to the tables below, make sure to increment the
+ * generation and document your change here.
+ */
+ public static final long GENERATION = 0;
Review Comment:
I think this should just be folded into SystemDistributedKeyspace, since the metadata() returns that but missing the correct defaults from the system distributed it might cause some weirdness, also having the GENERATION number 0 when being merged into the system_distributed. I think would be better to hve these tables in the keyspace that owns them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] autorepair v2 framework (against 4.1.6) [cassandra]
Posted by "clohfink (via GitHub)" <gi...@apache.org>.
clohfink commented on code in PR #3367:
URL: https://github.com/apache/cassandra/pull/3367#discussion_r1683344614
##########
src/java/org/apache/cassandra/schema/TableParams.java:
##########
@@ -105,6 +112,13 @@ private TableParams(Builder builder)
extensions = builder.extensions;
cdc = builder.cdc;
readRepair = builder.readRepair;
+ automatedRepair = new HashMap<>()
Review Comment:
this breaks java8 builds, cant use <> on anonymous inner classes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] autorepair v2 framework (against 4.1.6) [cassandra]
Posted by "clohfink (via GitHub)" <gi...@apache.org>.
clohfink commented on code in PR #3367:
URL: https://github.com/apache/cassandra/pull/3367#discussion_r1683345806
##########
src/java/org/apache/cassandra/schema/TableParams.java:
##########
@@ -84,12 +89,14 @@ public String toString()
public final boolean cdc;
public final ReadRepairStrategy readRepair;
+ public final Map<AutoRepairConfig.RepairType, AutoRepairParams> automatedRepair;
Review Comment:
Can change to EnumMap
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] autorepair v2 framework (against 4.1.6) [cassandra]
Posted by "clohfink (via GitHub)" <gi...@apache.org>.
clohfink commented on code in PR #3367:
URL: https://github.com/apache/cassandra/pull/3367#discussion_r1683382819
##########
test/unit/org/apache/cassandra/repair/autorepair/AutoRepairParameterizedTest.java:
##########
@@ -0,0 +1,581 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.repair.autorepair;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Sets;
+
+import org.apache.cassandra.cql3.statements.schema.TableAttributes;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.repair.RepairRunnable;
+import org.apache.cassandra.schema.AutoRepairParams;
+import org.apache.cassandra.schema.TableParams;
+import org.apache.cassandra.service.StorageService;
+import org.apache.cassandra.utils.Pair;
+
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import junit.framework.Assert;
Review Comment:
also in AutoRepairTest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] autorepair v2 framework (against 4.1.6) [cassandra]
Posted by "jaydeepkumar1984 (via GitHub)" <gi...@apache.org>.
jaydeepkumar1984 commented on code in PR #3367:
URL: https://github.com/apache/cassandra/pull/3367#discussion_r1794436808
##########
src/java/org/apache/cassandra/repair/autorepair/AutoRepairKeyspace.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.repair.autorepair;
+
+import java.util.concurrent.TimeUnit;
+
+import org.apache.cassandra.cql3.statements.schema.CreateTableStatement;
+import org.apache.cassandra.schema.KeyspaceMetadata;
+import org.apache.cassandra.schema.KeyspaceParams;
+import org.apache.cassandra.schema.SchemaConstants;
+import org.apache.cassandra.schema.TableId;
+import org.apache.cassandra.schema.TableMetadata;
+import org.apache.cassandra.schema.Tables;
+
+public class AutoRepairKeyspace
+{
+ private AutoRepairKeyspace()
+ {
+ }
+
+ /**
+ * Generation is used as a timestamp for automatic table creation on startup.
+ * If you make any changes to the tables below, make sure to increment the
+ * generation and document your change here.
+ */
+ public static final long GENERATION = 0;
Review Comment:
Incorporated
##########
src/java/org/apache/cassandra/repair/autorepair/AutoRepairState.java:
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.repair.autorepair;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import org.apache.cassandra.config.DurationSpec;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.view.TableViews;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.metrics.AutoRepairMetricsManager;
+import org.apache.cassandra.metrics.AutoRepairMetrics;
+import org.apache.cassandra.repair.autorepair.AutoRepairConfig.RepairType;
+import org.apache.cassandra.repair.autorepair.AutoRepairUtils.AutoRepairHistory;
+import org.apache.cassandra.repair.RepairParallelism;
+import org.apache.cassandra.repair.RepairRunnable;
+import org.apache.cassandra.repair.messages.RepairOption;
+import org.apache.cassandra.service.AutoRepairService;
+import org.apache.cassandra.service.StorageService;
+import org.apache.cassandra.streaming.PreviewKind;
+import org.apache.cassandra.utils.concurrent.Condition;
+import org.apache.cassandra.utils.progress.ProgressEvent;
+import org.apache.cassandra.utils.progress.ProgressEventType;
+import org.apache.cassandra.utils.progress.ProgressListener;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.text.SimpleDateFormat;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+import static org.apache.cassandra.utils.Clock.Global.currentTimeMillis;
+import static org.apache.cassandra.utils.concurrent.Condition.newOneTimeCondition;
+
+// AutoRepairState represents the state of automated repair for a given repair type.
+public abstract class AutoRepairState implements ProgressListener
+{
+ protected static final Logger logger = LoggerFactory.getLogger(AutoRepairState.class);
+ private final SimpleDateFormat format = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss,SSS");
+ @VisibleForTesting
+ protected static Supplier<Long> timeFunc = System::currentTimeMillis;
Review Comment:
Incorporated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] autorepair v2 framework (against 4.1.6) [cassandra]
Posted by "clohfink (via GitHub)" <gi...@apache.org>.
clohfink commented on code in PR #3367:
URL: https://github.com/apache/cassandra/pull/3367#discussion_r1794094083
##########
src/java/org/apache/cassandra/tools/nodetool/GetAutoRepairConfig.java:
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.tools.nodetool;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Joiner;
+import io.airlift.airline.Command;
+import org.apache.cassandra.repair.autorepair.AutoRepairConfig;
+import org.apache.cassandra.repair.autorepair.AutoRepairConfig.RepairType;
+import org.apache.cassandra.tools.NodeProbe;
+import org.apache.cassandra.tools.NodeTool.NodeToolCmd;
+
+import java.io.PrintStream;
+
+@Command(name = "getautorepairconfig", description = "Print autorepair configurations")
+public class GetAutoRepairConfig extends NodeToolCmd
+{
+ @VisibleForTesting
+ protected static PrintStream out = System.out;
+
+ @Override
+ public void execute(NodeProbe probe)
+ {
+ AutoRepairConfig config = probe.getAutoRepairConfig();
+ if (config == null || !config.isAutoRepairSchedulingEnabled())
+ {
+ out.println("Auto-repair is not enabled");
+ return;
+ }
+
+ StringBuilder sb = new StringBuilder();
+ sb.append("repair scheduler configuration:");
+ sb.append("\n\trepair eligibility check interval: " + config.getRepairCheckInterval());
+ sb.append("\n\tTTL for repair history for dead nodes: " + config.getAutoRepairHistoryClearDeleteHostsBufferInterval());
+ sb.append("\n\tmax retries for repair: " + config.getRepairMaxRetries());
+ sb.append("\n\tretry backoff: " + config.getRepairRetryBackoff());
+ for (RepairType repairType : RepairType.values())
+ {
+ sb.append(formatRepairTypeConfig(probe, repairType, config));
+ }
+
+ out.println(sb);
+ }
+
+ private String formatRepairTypeConfig(NodeProbe probe, RepairType repairType, AutoRepairConfig config)
+ {
+ StringBuilder sb = new StringBuilder();
+ sb.append("\nconfiguration for repair type: " + repairType);
+ sb.append("\n\tenabled: " + config.isAutoRepairEnabled(repairType));
+ sb.append("\n\tminimum repair interval: " + config.getRepairMinInterval(repairType));
+ sb.append("\n\trepair threads: " + config.getRepairThreads(repairType));
+ sb.append("\n\tnumber of repair subranges: " + config.getRepairSubRangeNum(repairType));
+ sb.append("\n\tpriority hosts: " + Joiner.on(',').skipNulls().join(probe.getRepairPriorityForHosts(repairType)));
+ sb.append("\n\tsstable count higher threshold: " + config.getRepairSSTableCountHigherThreshold(repairType));
+ sb.append("\n\ttable max repair time in sec: " + config.getAutoRepairTableMaxRepairTime(repairType));
+ sb.append("\n\tignore datacenters: " + Joiner.on(',').skipNulls().join(config.getIgnoreDCs(repairType)));
+ sb.append("\n\trepair primary token-range: " + config.getRepairPrimaryTokenRangeOnly(repairType));
+ sb.append("\n\tnumber of parallel repairs within group: " + config.getParallelRepairCount(repairType));
+ sb.append("\n\tpercentage of parallel repairs within group: " + config.getParallelRepairPercentage(repairType));
+ sb.append("\n\tmv repair enabled: " + config.getMVRepairEnabled(repairType));
+ sb.append("\n\tinitial scheduler delay: " + config.getInitialSchedulerDelay(repairType));
+ sb.append("\n\trepair setssion timeout: " + config.getRepairSessionTimeout(repairType));
Review Comment:
spelling setssion -> session
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] autorepair v2 framework (against 4.1.6) [cassandra]
Posted by "jaydeepkumar1984 (via GitHub)" <gi...@apache.org>.
jaydeepkumar1984 commented on code in PR #3367:
URL: https://github.com/apache/cassandra/pull/3367#discussion_r1772071357
##########
test/unit/org/apache/cassandra/repair/autorepair/AutoRepairParameterizedTest.java:
##########
@@ -0,0 +1,681 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.repair.autorepair;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Sets;
+
+import org.apache.cassandra.config.DurationSpec;
Review Comment:
Sorry, I missed this again. I forgot to run this against Java8. I've just fixed everything and pushed the changes - it now works with Java8. Could you please retry?
Next time, I will make sure to compile it with Java8 before pushing the newer changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] autorepair v2 framework (against 4.1.6) [cassandra]
Posted by "clohfink (via GitHub)" <gi...@apache.org>.
clohfink commented on code in PR #3367:
URL: https://github.com/apache/cassandra/pull/3367#discussion_r1683363500
##########
src/java/org/apache/cassandra/repair/autorepair/AutoRepair.java:
##########
@@ -0,0 +1,401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.repair.autorepair;
+
+import java.util.ArrayList;
+import java.util.EnumMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.FutureTask;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableList;
+
+import org.apache.cassandra.repair.RepairRunnable;
+import org.apache.cassandra.utils.Pair;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.concurrent.NamedThreadFactory;
+import org.apache.cassandra.concurrent.ScheduledExecutorPlus;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.exceptions.ConfigurationException;
+import org.apache.cassandra.gms.Gossiper;
+import org.apache.cassandra.schema.TableMetadata;
+import org.apache.cassandra.schema.Tables;
+import org.apache.cassandra.service.AutoRepairService;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.repair.autorepair.AutoRepairUtils.RepairTurn;
+
+import static org.apache.cassandra.concurrent.ExecutorFactory.Global.executorFactory;
+import static org.apache.cassandra.repair.autorepair.AutoRepairUtils.RepairTurn.MY_TURN;
+import static org.apache.cassandra.repair.autorepair.AutoRepairUtils.RepairTurn.MY_TURN_DUE_TO_PRIORITY;
+import static org.apache.cassandra.repair.autorepair.AutoRepairUtils.RepairTurn.MY_TURN_FORCE_REPAIR;
+
+public class AutoRepair
+{
+ // Initial delay for repair session to start after setup
+ final static long INITIAL_REPAIR_DELAY_SEC = 30;
+
+ private static final Logger logger = LoggerFactory.getLogger(AutoRepair.class);
+
+ @VisibleForTesting
+ protected static Supplier<Long> timeFunc = System::currentTimeMillis;
+
+ public static AutoRepair instance = new AutoRepair();
+
+ @VisibleForTesting
+ protected final Map<AutoRepairConfig.RepairType, ScheduledExecutorPlus> repairExecutors;
+ @VisibleForTesting
+ protected final Map<AutoRepairConfig.RepairType, AutoRepairState> repairStates;
+
+ protected final Map<AutoRepairConfig.RepairType, IAutoRepairTokenRangeSplitter> tokenRangeSplitters = new EnumMap<>(AutoRepairConfig.RepairType.class);
+
+
+ @VisibleForTesting
+ protected AutoRepair()
+ {
+ AutoRepairConfig config = DatabaseDescriptor.getAutoRepairConfig();
+ repairExecutors = new EnumMap<>(AutoRepairConfig.RepairType.class);
+ repairStates = new EnumMap<>(AutoRepairConfig.RepairType.class);
+ for (AutoRepairConfig.RepairType repairType : AutoRepairConfig.RepairType.values())
+ {
+ repairExecutors.put(repairType, executorFactory().scheduled(false, "AutoRepair-Repair-" + repairType, Thread.NORM_PRIORITY));
+ repairStates.put(repairType, AutoRepairConfig.RepairType.getAutoRepairState(repairType));
+ tokenRangeSplitters.put(repairType, FBUtilities.newAutoRepairTokenRangeSplitter(config.getTokenRangeSplitter(repairType)));
+ }
+ }
+
+ public void setup()
+ {
+ verifyIsSafeToEnable();
+
+ AutoRepairConfig config = DatabaseDescriptor.getAutoRepairConfig();
+ AutoRepairService.setup();
+ AutoRepairUtils.setup();
+
+ for (AutoRepairConfig.RepairType repairType : AutoRepairConfig.RepairType.values())
+ {
+ repairExecutors.get(repairType).scheduleWithFixedDelay(
+ () -> repair(repairType, 60000),
+ INITIAL_REPAIR_DELAY_SEC,
+ config.getRepairCheckInterval().toSeconds(),
+ TimeUnit.SECONDS);
+ }
+ }
+
+ @VisibleForTesting
+ protected void verifyIsSafeToEnable()
+ {
+ AutoRepairConfig config = DatabaseDescriptor.getAutoRepairConfig();
+ if (config.isAutoRepairEnabled(AutoRepairConfig.RepairType.incremental) &&
+ (DatabaseDescriptor.getMaterializedViewsEnabled() || DatabaseDescriptor.isCDCEnabled()))
+ throw new ConfigurationException("Cannot enable incremental repair with materialized views or CDC enabled");
+ }
+
+ // repairAsync runs a repair session of the given type asynchronously.
+ public void repairAsync(AutoRepairConfig.RepairType repairType, long millisToWait)
+ {
+ repairExecutors.get(repairType).submit(() -> repair(repairType, millisToWait));
+ }
+
+ // repair runs a repair session of the given type synchronously.
+ public void repair(AutoRepairConfig.RepairType repairType, long millisToWait)
Review Comment:
NP: maybe simplify this method a little breaking it up, its very long and deep
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] autorepair v2 framework (against 4.1.6) [cassandra]
Posted by "clohfink (via GitHub)" <gi...@apache.org>.
clohfink commented on code in PR #3367:
URL: https://github.com/apache/cassandra/pull/3367#discussion_r1683351680
##########
src/java/org/apache/cassandra/repair/autorepair/AutoRepairUtils.java:
##########
@@ -0,0 +1,806 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.repair.autorepair;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.MoreObjects;
+import com.google.common.collect.Lists;
+
+import org.apache.cassandra.locator.LocalStrategy;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.cql3.QueryOptions;
+import org.apache.cassandra.cql3.QueryProcessor;
+import org.apache.cassandra.cql3.UntypedResultSet;
+import org.apache.cassandra.cql3.statements.ModificationStatement;
+import org.apache.cassandra.cql3.statements.SelectStatement;
+import org.apache.cassandra.db.ConsistencyLevel;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.marshal.UTF8Type;
+import org.apache.cassandra.db.marshal.UUIDType;
+import org.apache.cassandra.gms.Gossiper;
+import org.apache.cassandra.locator.AbstractReplicationStrategy;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.locator.NetworkTopologyStrategy;
+import org.apache.cassandra.schema.Schema;
+import org.apache.cassandra.schema.SchemaConstants;
+import org.apache.cassandra.schema.TableMetadata;
+import org.apache.cassandra.schema.ViewMetadata;
+import org.apache.cassandra.serializers.SetSerializer;
+import org.apache.cassandra.serializers.UUIDSerializer;
+import org.apache.cassandra.service.AutoRepairService;
+import org.apache.cassandra.service.ClientState;
+import org.apache.cassandra.service.QueryState;
+import org.apache.cassandra.service.StorageService;
+import org.apache.cassandra.transport.Dispatcher;
+import org.apache.cassandra.transport.ProtocolVersion;
+import org.apache.cassandra.transport.messages.ResultMessage;
+import org.apache.cassandra.utils.ByteBufferUtil;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.repair.autorepair.AutoRepairConfig.RepairType;
+
+import static org.apache.cassandra.repair.autorepair.AutoRepairUtils.RepairTurn.MY_TURN;
+import static org.apache.cassandra.repair.autorepair.AutoRepairUtils.RepairTurn.MY_TURN_DUE_TO_PRIORITY;
+import static org.apache.cassandra.repair.autorepair.AutoRepairUtils.RepairTurn.NOT_MY_TURN;
+import static org.apache.cassandra.repair.autorepair.AutoRepairUtils.RepairTurn.MY_TURN_FORCE_REPAIR;
+
+/**
+ * This class serves as a utility class for AutoRepair. It contains various helper APIs
+ * to store/retrieve repair status, decide whose turn is next, etc.
+ */
+public class AutoRepairUtils
+{
+ private static final Logger logger = LoggerFactory.getLogger(AutoRepairUtils.class);
+ static final String COL_REPAIR_TYPE = "repair_type";
+ static final String COL_HOST_ID = "host_id";
+ static final String COL_REPAIR_START_TS = "repair_start_ts";
+ static final String COL_REPAIR_FINISH_TS = "repair_finish_ts";
+ static final String COL_REPAIR_PRIORITY = "repair_priority";
+ static final String COL_DELETE_HOSTS = "delete_hosts"; // this set stores the host ids which think the row should be deleted
+ static final String COL_REPAIR_TURN = "repair_turn"; // this record the last repair turn. Normal turn or turn due to priority
+ static final String COL_DELETE_HOSTS_UPDATE_TIME = "delete_hosts_update_time"; // the time when delete hosts are upated
+ static final String COL_FORCE_REPAIR = "force_repair"; // if set to true, the node will do non-primary range rapair
+
+ final static String SELECT_REPAIR_HISTORY = String.format(
+ "SELECT * FROM %s.%s WHERE %s = ?", SchemaConstants.DISTRIBUTED_KEYSPACE_NAME,
+ AutoRepairKeyspace.AUTO_REPAIR_HISTORY, COL_REPAIR_TYPE);
+ final static String SELECT_REPAIR_PRIORITY = String.format(
+ "SELECT * FROM %s.%s WHERE %s = ?", SchemaConstants.DISTRIBUTED_KEYSPACE_NAME,
+ AutoRepairKeyspace.AUTO_REPAIR_PRIORITY, COL_REPAIR_TYPE);
+ final static String DEL_REPAIR_PRIORITY = String.format(
+ "DELETE %s[?] FROM %s.%s WHERE %s = ?", COL_REPAIR_PRIORITY, SchemaConstants.DISTRIBUTED_KEYSPACE_NAME,
+ AutoRepairKeyspace.AUTO_REPAIR_PRIORITY, COL_REPAIR_TYPE);
+ final static String ADD_PRIORITY_HOST = String.format(
+ "UPDATE %s.%s SET %s = %s + ? WHERE %s = ?", SchemaConstants.DISTRIBUTED_KEYSPACE_NAME,
+ AutoRepairKeyspace.AUTO_REPAIR_PRIORITY, COL_REPAIR_PRIORITY, COL_REPAIR_PRIORITY, COL_REPAIR_TYPE);
+
+ final static String INSERT_NEW_REPAIR_HISTORY = String.format(
+ "INSERT INTO %s.%s (%s, %s, %s, %s, %s, %s) values (?, ? ,?, ?, {}, ?) IF NOT EXISTS",
+ SchemaConstants.DISTRIBUTED_KEYSPACE_NAME, AutoRepairKeyspace.AUTO_REPAIR_HISTORY, COL_REPAIR_TYPE,
+ COL_HOST_ID, COL_REPAIR_START_TS, COL_REPAIR_FINISH_TS, COL_DELETE_HOSTS, COL_DELETE_HOSTS_UPDATE_TIME);
+
+ final static String ADD_HOST_ID_TO_DELETE_HOSTS = String.format(
+ "UPDATE %s.%s SET %s = %s + ?, %s = ? WHERE %s = ? AND %s = ? IF EXISTS"
+ , SchemaConstants.DISTRIBUTED_KEYSPACE_NAME, AutoRepairKeyspace.AUTO_REPAIR_HISTORY, COL_DELETE_HOSTS,
+ COL_DELETE_HOSTS, COL_DELETE_HOSTS_UPDATE_TIME, COL_REPAIR_TYPE, COL_HOST_ID);
+
+ final static String DEL_AUTO_REPAIR_HISTORY = String.format(
+ "DELETE FROM %s.%s WHERE %s = ? AND %s = ?"
+ , SchemaConstants.DISTRIBUTED_KEYSPACE_NAME, AutoRepairKeyspace.AUTO_REPAIR_HISTORY, COL_REPAIR_TYPE,
+ COL_HOST_ID);
+
+ final static String RECORD_START_REPAIR_HISTORY = String.format(
+ "UPDATE %s.%s SET %s= ?, repair_turn = ? WHERE %s = ? AND %s = ?"
+ , SchemaConstants.DISTRIBUTED_KEYSPACE_NAME, AutoRepairKeyspace.AUTO_REPAIR_HISTORY, COL_REPAIR_START_TS,
+ COL_REPAIR_TYPE, COL_HOST_ID);
+
+ final static String RECORD_FINISH_REPAIR_HISTORY = String.format(
+
+ "UPDATE %s.%s SET %s= ?, %s=false WHERE %s = ? AND %s = ?"
+ , SchemaConstants.DISTRIBUTED_KEYSPACE_NAME, AutoRepairKeyspace.AUTO_REPAIR_HISTORY, COL_REPAIR_FINISH_TS,
+ COL_FORCE_REPAIR, COL_REPAIR_TYPE, COL_HOST_ID);
+
+ final static String CLEAR_DELETE_HOSTS = String.format(
+ "UPDATE %s.%s SET %s= {} WHERE %s = ? AND %s = ?"
+ , SchemaConstants.DISTRIBUTED_KEYSPACE_NAME, AutoRepairKeyspace.AUTO_REPAIR_HISTORY, COL_DELETE_HOSTS,
+ COL_REPAIR_TYPE, COL_HOST_ID);
+
+ final static String SET_FORCE_REPAIR = String.format(
+ "UPDATE %s.%s SET %s=true WHERE %s = ? AND %s = ?"
+ , SchemaConstants.DISTRIBUTED_KEYSPACE_NAME, AutoRepairKeyspace.AUTO_REPAIR_HISTORY, COL_FORCE_REPAIR,
+ COL_REPAIR_TYPE, COL_HOST_ID);
+
+ static ModificationStatement delStatementRepairHistory;
+ static SelectStatement selectStatementRepairHistory;
+ static ModificationStatement delStatementPriorityStatus;
+ static SelectStatement selectStatementRepairPriority;
+ static ModificationStatement addPriorityHost;
+ static ModificationStatement insertNewRepairHistoryStatement;
+ static ModificationStatement recordStartRepairHistoryStatement;
+ static ModificationStatement recordFinishRepairHistoryStatement;
+ static ModificationStatement addHostIDToDeleteHostsStatement;
+ static ModificationStatement clearDeleteHostsStatement;
+ static ModificationStatement setForceRepairStatement;
+ static ConsistencyLevel internalQueryCL;
+
+ public enum RepairTurn
+ {
+ MY_TURN,
+ NOT_MY_TURN,
+ MY_TURN_DUE_TO_PRIORITY,
+ MY_TURN_FORCE_REPAIR
+ }
+
+ public static void setup()
+ {
+ selectStatementRepairHistory = (SelectStatement) QueryProcessor.getStatement(SELECT_REPAIR_HISTORY, ClientState
+ .forInternalCalls());
+ selectStatementRepairPriority = (SelectStatement) QueryProcessor.getStatement(SELECT_REPAIR_PRIORITY, ClientState
+ .forInternalCalls());
+ delStatementPriorityStatus = (ModificationStatement) QueryProcessor.getStatement(DEL_REPAIR_PRIORITY, ClientState
+ .forInternalCalls());
+ addPriorityHost = (ModificationStatement) QueryProcessor.getStatement(ADD_PRIORITY_HOST, ClientState
+ .forInternalCalls());
+ insertNewRepairHistoryStatement = (ModificationStatement) QueryProcessor.getStatement(INSERT_NEW_REPAIR_HISTORY, ClientState
+ .forInternalCalls());
+ recordStartRepairHistoryStatement = (ModificationStatement) QueryProcessor.getStatement(RECORD_START_REPAIR_HISTORY, ClientState
+ .forInternalCalls());
+ recordFinishRepairHistoryStatement = (ModificationStatement) QueryProcessor.getStatement(RECORD_FINISH_REPAIR_HISTORY, ClientState
+ .forInternalCalls());
+ addHostIDToDeleteHostsStatement = (ModificationStatement) QueryProcessor.getStatement(ADD_HOST_ID_TO_DELETE_HOSTS, ClientState
+ .forInternalCalls());
+ setForceRepairStatement = (ModificationStatement) QueryProcessor.getStatement(SET_FORCE_REPAIR, ClientState
+ .forInternalCalls());
+ clearDeleteHostsStatement = (ModificationStatement) QueryProcessor.getStatement(CLEAR_DELETE_HOSTS, ClientState
+ .forInternalCalls());
+ delStatementRepairHistory = (ModificationStatement) QueryProcessor.getStatement(DEL_AUTO_REPAIR_HISTORY, ClientState
+ .forInternalCalls());
+ Keyspace autoRepairKS = Schema.instance.getKeyspaceInstance(SchemaConstants.DISTRIBUTED_KEYSPACE_NAME);
+ internalQueryCL = autoRepairKS.getReplicationStrategy().getClass() == NetworkTopologyStrategy.class ?
+ ConsistencyLevel.LOCAL_QUORUM : ConsistencyLevel.ONE;
+ }
+
+ public static class AutoRepairHistory
+ {
+ UUID hostId;
+ String repairTurn;
+ long lastRepairStartTime;
+ long lastRepairFinishTime;
+ Set<UUID> deleteHosts;
+ long deleteHostsUpdateTime;
+ boolean forceRepair;
+
+ public AutoRepairHistory(UUID hostId, String repairTurn, long lastRepairStartTime, long lastRepairFinishTime,
+ Set<UUID> deleteHosts, long deleteHostsUpateTime, boolean forceRepair)
+ {
+ this.hostId = hostId;
+ this.repairTurn = repairTurn;
+ this.lastRepairStartTime = lastRepairStartTime;
+ this.lastRepairFinishTime = lastRepairFinishTime;
+ this.deleteHosts = deleteHosts;
+ if (this.deleteHosts == null)
+ {
+ this.deleteHosts = new HashSet<>();
+ }
+ this.deleteHostsUpdateTime = deleteHostsUpateTime;
+ this.forceRepair = forceRepair;
+ }
+
+ public String toString()
+ {
+ return MoreObjects.toStringHelper(this).
+ add("hostId", hostId).
+ add("repairTurn", repairTurn).
+ add("lastRepairStartTime", lastRepairStartTime).
+ add("lastRepairFinishTime", lastRepairFinishTime).
+ add("deleteHosts", deleteHosts).
+ toString();
+ }
+
+ public boolean isRepairRunning()
+ {
+ // if a repair history record has start time laster than finish time, it means the repair is running
+ return lastRepairStartTime > lastRepairFinishTime;
+ }
+
+ public long getLastRepairFinishTime()
+ {
+ return lastRepairFinishTime;
+ }
+ }
+
+ public static class CurrentRepairStatus
+ {
+ public Set<UUID> hostIdsWithOnGoingRepair; // hosts that is running repair
+ public Set<UUID> hostIdsWithOnGoingForceRepair; // hosts that is running repair because of force repair
+ Set<UUID> priority;
+ List<AutoRepairHistory> historiesWithoutOnGoingRepair; // hosts that is NOT running repair
+
+ public CurrentRepairStatus(List<AutoRepairHistory> repairHistories, Set<UUID> priority)
+ {
+ hostIdsWithOnGoingRepair = new HashSet<>();
+ hostIdsWithOnGoingForceRepair = new HashSet<>();
+ historiesWithoutOnGoingRepair = new ArrayList<>();
+
+ for (AutoRepairHistory history : repairHistories)
+ {
+ if (history.isRepairRunning())
+ {
+ if (history.forceRepair)
+ {
+ hostIdsWithOnGoingForceRepair.add(history.hostId);
+ }
+ else
+ {
+ hostIdsWithOnGoingRepair.add(history.hostId);
+ }
+ }
+ else
+ {
+ historiesWithoutOnGoingRepair.add(history);
+ }
+ }
+ this.priority = priority;
+ }
+
+ public String toString()
+ {
+ return MoreObjects.toStringHelper(this).
+ add("hostIdsWithOnGoingRepair", hostIdsWithOnGoingRepair).
+ add("hostIdsWithOnGoingForceRepair", hostIdsWithOnGoingForceRepair).
+ add("historiesWithoutOnGoingRepair", historiesWithoutOnGoingRepair).
+ add("priority", priority).
+ toString();
+ }
+ }
+
+ @VisibleForTesting
+ public static List<AutoRepairHistory> getAutoRepairHistoryByGroupID(RepairType repairType)
+ {
+ UntypedResultSet repairHistoryResult;
+
+ ResultMessage.Rows repairStatusRows = selectStatementRepairHistory.execute(QueryState.forInternalCalls(),
+ QueryOptions.forInternalCalls(internalQueryCL, Lists.newArrayList(ByteBufferUtil.bytes(repairType.toString()))), Dispatcher.RequestTime.forImmediateExecution());
+ repairHistoryResult = UntypedResultSet.create(repairStatusRows.result);
+
+ List<AutoRepairHistory> repairHistories = new ArrayList<>();
+ if (repairHistoryResult.size() > 0)
+ {
+ for (UntypedResultSet.Row row : repairHistoryResult)
+ {
+ UUID hostId = row.getUUID(COL_HOST_ID);
+ String repairTurn = null;
+ if (row.has(COL_REPAIR_TURN))
+ repairTurn = row.getString(COL_REPAIR_TURN);
+ long lastRepairStartTime = row.getLong(COL_REPAIR_START_TS, 0);
+ long lastRepairFinishTime = row.getLong(COL_REPAIR_FINISH_TS, 0);
+ Set<UUID> deleteHosts = row.getSet(COL_DELETE_HOSTS, UUIDType.instance);
+ long deleteHostsUpdateTime = row.getLong(COL_DELETE_HOSTS_UPDATE_TIME, 0);
+ Boolean forceRepair = row.has(COL_FORCE_REPAIR) ? row.getBoolean(COL_FORCE_REPAIR) : false;
+ repairHistories.add(new AutoRepairHistory(hostId, repairTurn, lastRepairStartTime, lastRepairFinishTime,
+ deleteHosts, deleteHostsUpdateTime, forceRepair));
+ }
+ return repairHistories;
+ }
+ logger.info("No repair history found");
+ return null;
+ }
+
+ public static List<AutoRepairHistory> getAutoRepairHistoryForLocalGroup(RepairType repairType)
+ {
+ return getAutoRepairHistoryByGroupID(repairType);
+ }
+
+ // A host may add itself in delete hosts for some other hosts due to restart or some temp gossip issue. If a node's record
+ // delete_hosts is not growing for more than 2 hours, we consider it as a normal node so we clear the delete_hosts for that node
+ public static void clearDeleteHosts(RepairType repairType, UUID hostId)
+ {
+ clearDeleteHostsStatement.execute(QueryState.forInternalCalls(),
+ QueryOptions.forInternalCalls(internalQueryCL,
+ Lists.newArrayList(ByteBufferUtil.bytes(repairType.toString()),
+ ByteBufferUtil.bytes(hostId))), Dispatcher.RequestTime.forImmediateExecution());
+ }
+
+ public static void setForceRepairNewNode(RepairType repairType)
+ {
+ // this function will be called when a node bootstrap finished
+ UUID hostId = Gossiper.instance.getHostId(FBUtilities.getBroadcastAddressAndPort());
+ // insert the data first
+ insertNewRepairHistory(repairType, System.currentTimeMillis(), System.currentTimeMillis());
Review Comment:
fails checkstyles, use org.apache.cassandra.utils.Clock.Global or org.apache.cassandra.utils.Clock interface
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] autorepair v2 framework (against 4.1.6) [cassandra]
Posted by "jaydeepkumar1984 (via GitHub)" <gi...@apache.org>.
jaydeepkumar1984 commented on code in PR #3367:
URL: https://github.com/apache/cassandra/pull/3367#discussion_r1695735524
##########
src/java/org/apache/cassandra/repair/autorepair/AutoRepair.java:
##########
@@ -0,0 +1,401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.repair.autorepair;
+
+import java.util.ArrayList;
+import java.util.EnumMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.FutureTask;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableList;
+
+import org.apache.cassandra.repair.RepairRunnable;
+import org.apache.cassandra.utils.Pair;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.concurrent.NamedThreadFactory;
+import org.apache.cassandra.concurrent.ScheduledExecutorPlus;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.exceptions.ConfigurationException;
+import org.apache.cassandra.gms.Gossiper;
+import org.apache.cassandra.schema.TableMetadata;
+import org.apache.cassandra.schema.Tables;
+import org.apache.cassandra.service.AutoRepairService;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.repair.autorepair.AutoRepairUtils.RepairTurn;
+
+import static org.apache.cassandra.concurrent.ExecutorFactory.Global.executorFactory;
+import static org.apache.cassandra.repair.autorepair.AutoRepairUtils.RepairTurn.MY_TURN;
+import static org.apache.cassandra.repair.autorepair.AutoRepairUtils.RepairTurn.MY_TURN_DUE_TO_PRIORITY;
+import static org.apache.cassandra.repair.autorepair.AutoRepairUtils.RepairTurn.MY_TURN_FORCE_REPAIR;
+
+public class AutoRepair
+{
+ // Initial delay for repair session to start after setup
+ final static long INITIAL_REPAIR_DELAY_SEC = 30;
+
+ private static final Logger logger = LoggerFactory.getLogger(AutoRepair.class);
+
+ @VisibleForTesting
+ protected static Supplier<Long> timeFunc = System::currentTimeMillis;
+
+ public static AutoRepair instance = new AutoRepair();
+
+ @VisibleForTesting
+ protected final Map<AutoRepairConfig.RepairType, ScheduledExecutorPlus> repairExecutors;
+ @VisibleForTesting
+ protected final Map<AutoRepairConfig.RepairType, AutoRepairState> repairStates;
+
+ protected final Map<AutoRepairConfig.RepairType, IAutoRepairTokenRangeSplitter> tokenRangeSplitters = new EnumMap<>(AutoRepairConfig.RepairType.class);
+
+
+ @VisibleForTesting
+ protected AutoRepair()
+ {
+ AutoRepairConfig config = DatabaseDescriptor.getAutoRepairConfig();
+ repairExecutors = new EnumMap<>(AutoRepairConfig.RepairType.class);
+ repairStates = new EnumMap<>(AutoRepairConfig.RepairType.class);
+ for (AutoRepairConfig.RepairType repairType : AutoRepairConfig.RepairType.values())
+ {
+ repairExecutors.put(repairType, executorFactory().scheduled(false, "AutoRepair-Repair-" + repairType, Thread.NORM_PRIORITY));
+ repairStates.put(repairType, AutoRepairConfig.RepairType.getAutoRepairState(repairType));
+ tokenRangeSplitters.put(repairType, FBUtilities.newAutoRepairTokenRangeSplitter(config.getTokenRangeSplitter(repairType)));
+ }
+ }
+
+ public void setup()
+ {
+ verifyIsSafeToEnable();
+
+ AutoRepairConfig config = DatabaseDescriptor.getAutoRepairConfig();
+ AutoRepairService.setup();
+ AutoRepairUtils.setup();
+
+ for (AutoRepairConfig.RepairType repairType : AutoRepairConfig.RepairType.values())
+ {
+ repairExecutors.get(repairType).scheduleWithFixedDelay(
+ () -> repair(repairType, 60000),
+ INITIAL_REPAIR_DELAY_SEC,
+ config.getRepairCheckInterval().toSeconds(),
+ TimeUnit.SECONDS);
+ }
+ }
+
+ @VisibleForTesting
+ protected void verifyIsSafeToEnable()
+ {
+ AutoRepairConfig config = DatabaseDescriptor.getAutoRepairConfig();
+ if (config.isAutoRepairEnabled(AutoRepairConfig.RepairType.incremental) &&
+ (DatabaseDescriptor.getMaterializedViewsEnabled() || DatabaseDescriptor.isCDCEnabled()))
+ throw new ConfigurationException("Cannot enable incremental repair with materialized views or CDC enabled");
+ }
+
+ // repairAsync runs a repair session of the given type asynchronously.
+ public void repairAsync(AutoRepairConfig.RepairType repairType, long millisToWait)
+ {
+ repairExecutors.get(repairType).submit(() -> repair(repairType, millisToWait));
+ }
+
+ // repair runs a repair session of the given type synchronously.
+ public void repair(AutoRepairConfig.RepairType repairType, long millisToWait)
+ {
+ AutoRepairConfig config = AutoRepairService.instance.getAutoRepairConfig();
+ if (!config.isAutoRepairEnabled(repairType))
+ {
+ logger.debug("Auto-repair is disabled for repair type {}", repairType);
+ return;
+ }
+
+
+ AutoRepairState repairState = repairStates.get(repairType);
+
+ try
+ {
+ String localDC = DatabaseDescriptor.getLocalDataCenter();
+ if (config.getIgnoreDCs(repairType).contains(localDC))
+ {
+ logger.info("Not running repair as this node belongs to datacenter {}", localDC);
+ return;
+ }
+
+ // refresh the longest unrepaired node
+ repairState.setLongestUnrepairedNode(AutoRepairUtils.getHostWithLongestUnrepairTime(repairType));
+
+ //consistency level to use for local query
+ UUID myId = Gossiper.instance.getHostId(FBUtilities.getBroadcastAddressAndPort());
+ RepairTurn turn = AutoRepairUtils.myTurnToRunRepair(repairType, myId);
+ if (turn == MY_TURN || turn == MY_TURN_DUE_TO_PRIORITY || turn == MY_TURN_FORCE_REPAIR)
+ {
+ repairState.recordTurn(turn);
+ // For normal auto repair, we will use primary range only repairs (Repair with -pr option).
+ // For some cases, we may set the auto_repair_primary_token_range_only flag to false then we will do repair
+ // without -pr. We may also do force repair for certain node that we want to repair all the data on one node
+ // When doing force repair, we want to repair without -pr.
+ boolean primaryRangeOnly = config.getRepairPrimaryTokenRangeOnly(repairType)
+ && turn != MY_TURN_FORCE_REPAIR;
+ repairState.setTotalTablesConsideredForRepair(0);
+ if (repairState.getLastRepairTime() != 0)
+ {
+ /** check if it is too soon to run repair. one of the reason we
+ * should not run frequent repair is because repair triggers
+ * memtable flush
+ */
+ long timeElapsedSinceLastRepair = TimeUnit.MILLISECONDS.toSeconds(timeFunc.get() - repairState.getLastRepairTime());
+ if (timeElapsedSinceLastRepair < config.getRepairMinInterval(repairType).toSeconds())
+ {
+ logger.info("Too soon to run repair, last repair was done {} seconds ago",
+ timeElapsedSinceLastRepair);
+ return;
+ }
+ }
+
+ long startTime = timeFunc.get();
+ logger.info("My host id: {}, my turn to run repair...repair primary-ranges only? {}", myId,
+ config.getRepairPrimaryTokenRangeOnly(repairType));
+ AutoRepairUtils.updateStartAutoRepairHistory(repairType, myId, timeFunc.get(), turn);
+
+ repairState.setRepairKeyspaceCount(0);
+ repairState.setRepairTableSuccessCount(0);
+ repairState.setRepairFailedTablesCount(0);
+ repairState.setRepairSkippedTablesCount(0);
+ repairState.setRepairInProgress(true);
+ repairState.setTotalMVTablesConsideredForRepair(0);
+ for (Keyspace keyspace : Keyspace.all())
+ {
+ Tables tables = keyspace.getMetadata().tables;
+ Iterator<TableMetadata> iter = tables.iterator();
+ String keyspaceName = keyspace.getName();
+ if (!AutoRepairUtils.checkNodeContainsKeyspaceReplica(keyspace))
+ {
+ continue;
+ }
+
+ repairState.setRepairKeyspaceCount(repairState.getRepairKeyspaceCount() + 1);
+ List<String> tablesToBeRepaired = new ArrayList<>();
+ while (iter.hasNext())
+ {
+ repairState.setTotalTablesConsideredForRepair(repairState.getTotalTablesConsideredForRepair() + 1);
+ TableMetadata tableMetadata = iter.next();
+ String tableName = tableMetadata.name;
+ tablesToBeRepaired.add(tableName);
+
+ // See if we should repair MVs as well that are associated with this given table
+ List<String> mvs = AutoRepairUtils.getAllMVs(repairType, keyspace, tableMetadata);
+ if (!mvs.isEmpty())
+ {
+ tablesToBeRepaired.addAll(mvs);
+ repairState.setTotalMVTablesConsideredForRepair(repairState.getTotalMVTablesConsideredForRepair() + mvs.size());
+ }
+ }
+
+ for (String tableName : tablesToBeRepaired)
+ {
+ try
+ {
+ ColumnFamilyStore columnFamilyStore = keyspace.getColumnFamilyStore(tableName);
+ if (!columnFamilyStore.metadata().params.automatedRepair.get(repairType).repairEnabled())
+ {
+ logger.info("Repair is disabled for keyspace {} for tables: {}", keyspaceName, tableName);
+ repairState.setTotalDisabledTablesRepairCount(repairState.getTotalDisabledTablesRepairCount() + 1);
+ continue;
+ }
+ // this is done to make autorepair safe as running repair on table with more sstables
+ // may have its own challenges
+ int size = columnFamilyStore.getLiveSSTables().size();
+ if (size > config.getRepairSSTableCountHigherThreshold(repairType))
+ {
+ logger.info("Too many SSTables for repair, not doing repair on table {}.{} " +
+ "totalSSTables {}", keyspaceName, tableName, columnFamilyStore.getLiveSSTables().size());
+ repairState.setRepairSkippedTablesCount(repairState.getRepairSkippedTablesCount() + 1);
+ continue;
+ }
+
+ if (config.getRepairByKeyspace(repairType))
+ {
+ logger.info("Repair keyspace {} for tables: {}", keyspaceName, tablesToBeRepaired);
+ }
+ else
+ {
+ logger.info("Repair table {}.{}", keyspaceName, tableName);
+ }
+ long tableStartTime = timeFunc.get();
+ boolean repairSuccess = true;
+ Set<Range<Token>> ranges = new HashSet<>();
+ List<Pair<Token, Token>> subRangesToBeRepaired = tokenRangeSplitters.get(repairType).getRange(repairType, primaryRangeOnly, keyspaceName, tableName);
+ int totalSubRanges = subRangesToBeRepaired.size();
+ int totalProcessedSubRanges = 0;
+ for (Pair<Token, Token> token : subRangesToBeRepaired)
+ {
+ if (!config.isAutoRepairEnabled(repairType))
+ {
+ logger.error("Auto-repair for type {} is disabled hence not running repair", repairType);
+ repairState.setRepairInProgress(false);
+ return;
+ }
+
+ if (config.getRepairByKeyspace(repairType))
+ {
+ if (AutoRepairUtils.keyspaceMaxRepairTimeExceeded(repairType, tableStartTime, tablesToBeRepaired.size()))
+ {
+ repairState.setRepairSkippedTablesCount(repairState.getRepairSkippedTablesCount() + tablesToBeRepaired.size());
+ logger.info("Keyspace took too much time to repair hence skipping it {}",
+ keyspaceName);
+ break;
+ }
+ }
+ else
+ {
+ if (AutoRepairUtils.tableMaxRepairTimeExceeded(repairType, tableStartTime))
+ {
+ repairState.setRepairSkippedTablesCount(repairState.getRepairSkippedTablesCount() + 1);
+ logger.info("Table took too much time to repair hence skipping it {}.{}",
+ keyspaceName, tableName);
+ break;
+ }
+ }
+ Token childStartToken = token.left;
+ Token childEndToken = token.right;
+ logger.debug("Current Token Left side {}, right side {}", childStartToken
+ .toString(), childEndToken.toString());
+
+ ranges.add(new Range<>(childStartToken, childEndToken));
+ totalProcessedSubRanges++;
+ if ((totalProcessedSubRanges % config.getRepairThreads(repairType) == 0) ||
+ (totalProcessedSubRanges == totalSubRanges))
+ {
+ RepairRunnable task = repairState.getRepairRunnable(keyspaceName,
+ config.getRepairByKeyspace(repairType) ? tablesToBeRepaired : ImmutableList.of(tableName),
+ ranges, primaryRangeOnly);
+ repairState.resetWaitCondition();
+ new Thread(NamedThreadFactory.createAnonymousThread(new FutureTask<>(task, null))).start();
Review Comment:
Incorporated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] autorepair v2 framework (against 4.1.6) [cassandra]
Posted by "jaydeepkumar1984 (via GitHub)" <gi...@apache.org>.
jaydeepkumar1984 commented on code in PR #3367:
URL: https://github.com/apache/cassandra/pull/3367#discussion_r1828310099
##########
src/java/org/apache/cassandra/config/CassandraRelevantProperties.java:
##########
@@ -346,6 +346,17 @@ public enum CassandraRelevantProperties
*/
GOSSIP_SETTLE_POLL_SUCCESSES_REQUIRED("cassandra.gossip_settle_poll_success_required", "3"),
+ /**
+ * If set to true, mutations streamed during anti-entropy repair will be replayed via the regular write path for associated views.
+ */
+ STREAMING_REQUIRES_VIEW_BUILD_DURING_REPAIR("cassandra.streaming.requires_view_build_during_repair", "true"),
Review Comment:
Ok, I have made the following two changes:
1. Backported https://issues.apache.org/jira/browse/CASSANDRA-17666
2. Implemented a similar flag to https://issues.apache.org/jira/browse/CASSANDRA-17666 for MV
tl;dr - I've removed these properties instead it is now controlled by two yaml flags
```
cdc_on_repair_enabled: false
materialized_views_on_repair_enabled: false
```
The above two flags are `true` by default, so one needs to disable them. The changes have been pushed to the same PR; please take a look at it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org