You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pr@cassandra.apache.org by "nivykani (via GitHub)" <gi...@apache.org> on 2026/07/31 00:08:10 UTC

[PR] CASSANDRA-21527: Make runWithCompactionsDisabled return non-null on success (4.0) [cassandra]

nivykani opened a new pull request, #4985:
URL: https://github.com/apache/cassandra/pull/4985

   Currently, runWithCompactionsDisabled can’t tell the difference between a successful truncate vs a truncate that wasn’t successful due to being unable to stop compaction, because both return null. In either case, it logs “Truncate is complete” and returns success to the user. TruncateBlockingTest reproduces this bug.
   
   The proposed fix is adding a return value to check for success on truncateBlocking, as well as adding a new exception type for Truncate.
   
   The [Cassandra Jira](https://issues.apache.org/jira/browse/CASSANDRA-21527?filter=-2)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] CASSANDRA-21527: Make runWithCompactionsDisabled return non-null on success (4.0) [cassandra]

Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz closed pull request #4985: CASSANDRA-21527: Make runWithCompactionsDisabled return non-null on success (4.0)
URL: https://github.com/apache/cassandra/pull/4985


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] CASSANDRA-21527: Make runWithCompactionsDisabled return non-null on success (4.0) [cassandra]

Posted by "nivykani (via GitHub)" <gi...@apache.org>.
nivykani commented on code in PR #4985:
URL: https://github.com/apache/cassandra/pull/4985#discussion_r3761627791


##########
test/unit/org/apache/cassandra/db/TruncateBlockingTest.java:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.db;
+
+import net.bytebuddy.ByteBuddy;
+import net.bytebuddy.agent.ByteBuddyAgent;
+import net.bytebuddy.dynamic.loading.ClassReloadingStrategy;
+import net.bytebuddy.implementation.StubMethod;
+
+import org.assertj.core.api.Assertions;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import org.apache.cassandra.cql3.CQLTester;
+import org.apache.cassandra.db.compaction.CompactionManager;
+import org.apache.cassandra.db.compaction.OperationType;
+import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
+import org.apache.cassandra.exceptions.TruncateException;
+import org.apache.cassandra.io.sstable.format.SSTableReader;
+
+import static net.bytebuddy.matcher.ElementMatchers.named;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+
+public class TruncateBlockingTest extends CQLTester
+{
+    @BeforeClass
+    public static void setUp()
+    {
+        ByteBuddyAgent.install();
+        // Stub out waitForCessation so the test doesn't wait 60s
+        new ByteBuddy().redefine(CompactionManager.class)
+                .method(named("waitForCessation"))
+                .intercept(StubMethod.INSTANCE)
+                .make()
+                .load(CompactionManager.class.getClassLoader(), ClassReloadingStrategy.fromInstalledAgent());
+    }
+
+    @Test

Review Comment:
   Good to know, fixed!



##########
src/java/org/apache/cassandra/exceptions/RequestFailureReason.java:
##########


Review Comment:
   Great catch, thank you for pointing this out. Based on the comment, it sounds like we'd never want to return null here, so I changed the return statement to return `UNKNOWN` if the map lookup was `null`. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] CASSANDRA-21527: Make runWithCompactionsDisabled return non-null on success (4.0) [cassandra]

Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on code in PR #4985:
URL: https://github.com/apache/cassandra/pull/4985#discussion_r3754396892


##########
test/unit/org/apache/cassandra/db/TruncateBlockingTest.java:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.db;
+
+import net.bytebuddy.ByteBuddy;
+import net.bytebuddy.agent.ByteBuddyAgent;
+import net.bytebuddy.dynamic.loading.ClassReloadingStrategy;
+import net.bytebuddy.implementation.StubMethod;
+
+import org.assertj.core.api.Assertions;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import org.apache.cassandra.cql3.CQLTester;
+import org.apache.cassandra.db.compaction.CompactionManager;
+import org.apache.cassandra.db.compaction.OperationType;
+import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
+import org.apache.cassandra.exceptions.TruncateException;
+import org.apache.cassandra.io.sstable.format.SSTableReader;
+
+import static net.bytebuddy.matcher.ElementMatchers.named;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+
+public class TruncateBlockingTest extends CQLTester
+{
+    @BeforeClass
+    public static void setUp()
+    {
+        ByteBuddyAgent.install();
+        // Stub out waitForCessation so the test doesn't wait 60s
+        new ByteBuddy().redefine(CompactionManager.class)
+                .method(named("waitForCessation"))
+                .intercept(StubMethod.INSTANCE)
+                .make()
+                .load(CompactionManager.class.getClassLoader(), ClassReloadingStrategy.fromInstalledAgent());
+    }
+
+    @Test

Review Comment:
   nit: There's a purely declarative approach you could use here that also handles cleanup and possible test contamination:
   
   ```
   @Test
     @BMRule(name = "no-op waitForCessation",
             targetClass = "org.apache.cassandra.db.compaction.CompactionManager",
             targetMethod = "waitForCessation",
             action = "return;")
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] CASSANDRA-21527: Make runWithCompactionsDisabled return non-null on success (4.0) [cassandra]

Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on code in PR #4985:
URL: https://github.com/apache/cassandra/pull/4985#discussion_r3786123022


##########
src/java/org/apache/cassandra/net/InboundSink.java:
##########
@@ -22,6 +22,8 @@
 import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
 import java.util.function.Predicate;
 
+import org.apache.cassandra.exceptions.TruncateException;

Review Comment:
   nit: move down to where the other C* imports are?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] CASSANDRA-21527: Make runWithCompactionsDisabled return non-null on success (4.0) [cassandra]

Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on PR #4985:
URL: https://github.com/apache/cassandra/pull/4985#issuecomment-5319771641

   Committed as https://github.com/apache/cassandra/commit/7b4170491d124cd817278e01f0ff3f09012db889


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] CASSANDRA-21527: Make runWithCompactionsDisabled return non-null on success (4.0) [cassandra]

Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on code in PR #4985:
URL: https://github.com/apache/cassandra/pull/4985#discussion_r3754363161


##########
src/java/org/apache/cassandra/db/ColumnFamilyStore.java:
##########
@@ -2328,39 +2344,56 @@ private void truncateBlocking(boolean noSnapshot)
             for (SSTableReader sstable : cfs.getLiveSSTables())
                 now = Math.max(now, sstable.maxDataAge);
         truncatedAt = now;
-
-        Runnable truncateRunnable = new Runnable()
+        Throwable failure = null;
+        try
         {
-            public void run()
-            {
-                logger.info("Truncating {}.{} with truncatedAt={}", keyspace.getName(), getTableName(), truncatedAt);
-                // since truncation can happen at different times on different nodes, we need to make sure
-                // that any repairs are aborted, otherwise we might clear the data on one node and then
-                // stream in data that is actually supposed to have been deleted
-                ActiveRepairService.instance.abort((prs) -> prs.getTableIds().contains(metadata.id),
-                                                   "Stopping parent sessions {} due to truncation of tableId="+metadata.id);
-                data.notifyTruncated(truncatedAt);
+            Boolean succeeded = runWithCompactionsDisabled(() -> runTruncate(truncatedAt, noSnapshot, replayAfter),
+                                                             true, true);
+            // null means compactions couldn't be disabled, not failure of the truncate work itself
+            if (succeeded == null)
+                failure = new TruncateException("Unable to stop compaction. Usually retrying truncate will work.");

Review Comment:
   nit: I'd consider something a little more prescriptive. Operators can always check the logs, and they'd find...
   
   ```
   logger.warn("Unable to cancel in-progress compactions for {}.  Perhaps there is an unusually large row in progress somewhere, or the system is simply overloaded.", metadata.name);
   ```
   
   If they don't check the logs, perhaps it would be helpful to indicate that retrying when overall system load is lower might be a good idea. Something like...
   
   ```
   failure = new TruncateException("Unable to stop in-progress compactions. Please retry when current compaction tasks have completed or overall system load has decreased.");
   ```
   
   Not a blocker, and language is up to you.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] CASSANDRA-21527: Make runWithCompactionsDisabled return non-null on success (4.0) [cassandra]

Posted by "nivykani (via GitHub)" <gi...@apache.org>.
nivykani commented on code in PR #4985:
URL: https://github.com/apache/cassandra/pull/4985#discussion_r3761624646


##########
src/java/org/apache/cassandra/db/ColumnFamilyStore.java:
##########
@@ -2328,39 +2344,56 @@ private void truncateBlocking(boolean noSnapshot)
             for (SSTableReader sstable : cfs.getLiveSSTables())
                 now = Math.max(now, sstable.maxDataAge);
         truncatedAt = now;
-
-        Runnable truncateRunnable = new Runnable()
+        Throwable failure = null;
+        try
         {
-            public void run()
-            {
-                logger.info("Truncating {}.{} with truncatedAt={}", keyspace.getName(), getTableName(), truncatedAt);
-                // since truncation can happen at different times on different nodes, we need to make sure
-                // that any repairs are aborted, otherwise we might clear the data on one node and then
-                // stream in data that is actually supposed to have been deleted
-                ActiveRepairService.instance.abort((prs) -> prs.getTableIds().contains(metadata.id),
-                                                   "Stopping parent sessions {} due to truncation of tableId="+metadata.id);
-                data.notifyTruncated(truncatedAt);
+            Boolean succeeded = runWithCompactionsDisabled(() -> runTruncate(truncatedAt, noSnapshot, replayAfter),
+                                                             true, true);
+            // null means compactions couldn't be disabled, not failure of the truncate work itself
+            if (succeeded == null)
+                failure = new TruncateException("Unable to stop compaction. Usually retrying truncate will work.");

Review Comment:
   Fixed. Also added the boolean check from https://github.com/apache/cassandra/pull/4983#discussion_r3748516344, but kept the error message intentionally brief because nothing currently reaches that code path (only for future-proofing).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] CASSANDRA-21527: Make runWithCompactionsDisabled return non-null on success (4.0) [cassandra]

Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on code in PR #4985:
URL: https://github.com/apache/cassandra/pull/4985#discussion_r3754217474


##########
src/java/org/apache/cassandra/exceptions/RequestFailureReason.java:
##########


Review Comment:
   So we have a gap here, and in the 4.1 and 5.0 patches. If a 4.1 node tries to send us a `NODE_DOWN` code, we'll return `null` instead of `UNKNOWN`. We can either pre-populate `UNKNOWN` for the nulls in `codeToReasonMap` or apprehend the nulls here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] CASSANDRA-21527: Make runWithCompactionsDisabled return non-null on success (4.0) [cassandra]

Posted by "nivykani (via GitHub)" <gi...@apache.org>.
nivykani commented on code in PR #4985:
URL: https://github.com/apache/cassandra/pull/4985#discussion_r3797349783


##########
src/java/org/apache/cassandra/net/InboundSink.java:
##########
@@ -22,6 +22,8 @@
 import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
 import java.util.function.Predicate;
 
+import org.apache.cassandra.exceptions.TruncateException;

Review Comment:
   Fixed!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org