[jira] [Created] (CASSANDRA-21609) A failed flush can strand every thread that waits for it

["Michael Semb Wever (Jira)" <ji...@apache.org>]

Michael Semb Wever created CASSANDRA-21609:
----------------------------------------------

             Summary: A failed flush can strand every thread that waits for it
                 Key: CASSANDRA-21609
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-21609
             Project: Apache Cassandra
          Issue Type: Bug
          Components: Local/Memtable
            Reporter: Michael Semb Wever
            Assignee: Michael Semb Wever


In {{ColumnFamilyStore.Flush.run}} the flush failure path is:

{code:java}
catch (Throwable t)
{
    JVMStabilityInspector.inspectThrowable(t);
    postFlush.flushFailure = t;
}

// signal the post-flush we've done our work
postFlush.latch.decrement();
{code}

{{JVMStabilityInspector.inspectThrowable}} can itself throw. When it does, control never reaches {{postFlush.latch.decrement()}}, so every thread that waits for the flush blocks forever and the failure never propagates up the call chain. Nothing is logged, the node reports a pending flush, and no flush runs. The lockup was seen in tests that reached the open file limit.

The patch moves the latch decrement and the trace logging into a {{finally}} block, and records the failure on {{postFlush}} before it inspects the throwable.

Patch: [mck/upstream/flush-failure-latch/5.0|https://github.com/thelastpickle/cassandra/tree/mck/upstream/flush-failure-latch/5.0]
Provenance: [ee8426a8db|https://github.com/datastax/cassandra/commit/ee8426a8dbcf11ab17510f401d67c514f5e8fd21] by Piotr Kołaczkowski



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org