You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pr@cassandra.apache.org by "dcapwell (via GitHub)" <gi...@apache.org> on 2026/05/29 22:34:00 UTC

[PR] CASSANDRA-21412: Journal compaction did not get a reference to segments to segfault due to use-after-free [cassandra]

dcapwell opened a new pull request, #4852:
URL: https://github.com/apache/cassandra/pull/4852

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] CASSANDRA-21412: Journal compaction did not get a reference to segments to segfault due to use-after-free [cassandra]

Posted by "iamaleksey (via GitHub)" <gi...@apache.org>.
iamaleksey commented on PR #4852:
URL: https://github.com/apache/cassandra/pull/4852#issuecomment-5252143434

   > 4. If we add ref grabbing and address nothing else, the race with Journal shutdown still remains, but manifests itself differently. `segmentCompactor.compact(toCompact)` call will end up publishing an SSTable, but `journal.replaceCompactedSegments(toCompact, newSegments)` is going to hit `Invariants.require(isNotStopped())` _and_ leave the segments in place as well, which is not ideal.
   > 
   > I think that we should at the very least as part of the fix make sure that Journal shutdown really waits for compaction to complete before force-closing all the segments.
   
   It might even be sufficient to just address the shutdown-not-waiting-for-compaction-to-complete issue without changing any ref logic at all.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] CASSANDRA-21412: Journal compaction did not get a reference to segments to segfault due to use-after-free [cassandra]

Posted by "dcapwell (via GitHub)" <gi...@apache.org>.
dcapwell commented on code in PR #4852:
URL: https://github.com/apache/cassandra/pull/4852#discussion_r3808052491


##########
src/java/org/apache/cassandra/journal/Compactor.java:
##########
@@ -61,6 +76,7 @@ private synchronized void schedule(long period, TimeUnit units)
         scheduled = executor.scheduleWithFixedDelay(this, period, period, units);
     }
 
+    @VisibleForTesting

Review Comment:
   was walking all code paths that called
   
   ```
   if (scheduled != null)
     scheduled.cancel(false);
   ```
   
   to understand the locking and this test is only used for tests, so documenting it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] CASSANDRA-21412: Journal compaction did not get a reference to segments to segfault due to use-after-free [cassandra]

Posted by "dcapwell (via GitHub)" <gi...@apache.org>.
dcapwell commented on code in PR #4852:
URL: https://github.com/apache/cassandra/pull/4852#discussion_r3808086756


##########
src/java/org/apache/cassandra/journal/Compactor.java:
##########
@@ -106,16 +137,57 @@ public void run()
         }
     }
 
+    void withoutCompaction(Runnable task)
+    {
+        lockUninterruptibly();
+        try
+        {
+            task.run();
+        }
+        finally
+        {
+            compactionLock.unlock();
+        }
+    }
+
+    void awaitQuiescence()
+    {
+        withoutCompaction(() -> {});
+    }
+
+    private void lockUninterruptibly()
+    {
+        boolean interrupted = false;
+        long startNanos = nanoTime();
+        while (true)

Review Comment:
   debated bounded vs unbounded... im open to bounding.
   
   What i see is the only non-production path is the `ShutdownAccord` thread which is a background daemon so blocking here won't cause the JVM to stay up.  
   
   The other code paths were mostly test related so bounded will be timed out and hopefully we have the right stack trace to see this...
   
   At least for jvm-dtest we fail `close` after 1m so we would see this in the stack trace



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] CASSANDRA-21412: Journal compaction did not get a reference to segments to segfault due to use-after-free [cassandra]

Posted by "dcapwell (via GitHub)" <gi...@apache.org>.
dcapwell commented on code in PR #4852:
URL: https://github.com/apache/cassandra/pull/4852#discussion_r3808058595


##########
src/java/org/apache/cassandra/journal/Compactor.java:
##########
@@ -106,16 +137,57 @@ public void run()
         }
     }
 
+    void withoutCompaction(Runnable task)
+    {
+        lockUninterruptibly();
+        try
+        {
+            task.run();
+        }
+        finally
+        {
+            compactionLock.unlock();
+        }
+    }
+
+    void awaitQuiescence()
+    {
+        withoutCompaction(() -> {});
+    }
+
+    private void lockUninterruptibly()
+    {
+        boolean interrupted = false;
+        long startNanos = nanoTime();
+        while (true)
+        {
+            try
+            {
+                if (compactionLock.tryLock(1, TimeUnit.MINUTES))
+                    break;
+                logger.warn("Waited {}s for an in-flight {} compaction to complete",
+                            TimeUnit.NANOSECONDS.toSeconds(nanoTime() - startNanos), journal.name);
+            }
+            catch (InterruptedException e)
+            {
+                interrupted = true;
+            }
+        }
+        if (interrupted)
+            Thread.currentThread().interrupt();
+    }
+
     @Override
     public boolean isTerminated()
     {
         return executor.isTerminated();
     }
 
     @Override
-    public void shutdown()
+    public synchronized void shutdown()

Review Comment:
   sync wasn't added for `isShutdown`, it was added for
   
   ```
   if (scheduled != null)
     scheduled.cancel(false);
   ```
   
   start and shutdown are different threads so we don't have perfect happens-before so shutdown might not see `scheduled` so is unable to cancel. 
   
   To keep the usage in-sync i added the sync claude



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org