[jira] [Updated] (CASSANDRA-21727) SAI vector graph is broken after an interrupt during the wait for a permit

["Jeremiah Jordan (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CASSANDRA-21727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jeremiah Jordan updated CASSANDRA-21727:
----------------------------------------
     Bug Category: Parent values: Correctness(12982)Level 1 values: Unrecoverable Corruption / Loss(13161)
       Complexity: Normal
    Discovered By: Adhoc Test
    Fix Version/s: 5.0.x
                   6.0
                   7.0
         Severity: Normal
           Status: Open  (was: Triage Needed)

> SAI vector graph is broken after an interrupt during the wait for a permit
> --------------------------------------------------------------------------
>
>                 Key: CASSANDRA-21727
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-21727
>             Project: Apache Cassandra
>          Issue Type: Bug
>          Components: Feature/SAI
>            Reporter: Jeremiah Jordan
>            Assignee: Jeremiah Jordan
>            Priority: Normal
>             Fix For: 5.0.x, 6.0, 7.0
>
>
> h2. Problem
> CASSANDRA-21644 added a semaphore to {{OnHeapGraph.add}}. The writer gets the semaphore too late. Before the wait, the writer already added the new vector to {{postingsMap}}, {{vectorValues}} and {{postingsByOrdinal}}. It also already wrote the vector to {{vectorsByKey}}.
> If the writer receives an interrupt during the wait, the wait throws {{UncheckedInterruptedException}}. The insert stops before {{addGraphNode}}. The result is as follows:
> * The vector is in {{postingsMap}}, {{vectorValues}} and {{postingsByOrdinal}}, but it is not in the graph.
> * A retry of the same row finds the vector in {{postingsMap}}. The retry adds only the key. It does not add the vector to the graph.
> * A vector search does not find the row.
> * At flush, the assertion in {{OnHeapGraph.writeData}} fails. This assertion checks that each ordinal is in the graph. If the JVM runs without assertions, no error occurs and the index is incorrect.
> * For an update, {{vectorsByKey}} contains the new vector, but the row keeps the old vector. The brute-force search for {{ORDER BY}} uses the new vector to score the row.
> h2. Steps to reproduce
> # Create an {{OnHeapGraph}} with a memtable for which {{limitsConcurrentWritesTo}} returns false.
> # Set the interrupt flag of the thread.
> # Call {{add}} with a new vector. The call throws {{UncheckedInterruptedException}}.
> # Clear the interrupt flag. Call {{add}} again for the same row and vector.
> # Search the graph for the vector.
> Expected result: the search finds the vector. Actual result: {{size()}} returns 1, but the search finds no vector.
> h2. Proposed fix
> Change {{OnHeapGraph.add}} as follows:
> # Read the entry for the vector from {{postingsMap}}.
> # If the vector is new, get the permit before any change to the data. This includes {{vectorsByKey}}.
> # Release the permit in one {{finally}} block.
> With this order, an interrupt during the wait does not change the graph or {{vectorsByKey}}. A retry then adds the vector to the graph.
> Do not try to roll back the changes after the interrupt. Before the rollback, other writers can add keys for the same vector and send acknowledgements to clients. Also, the graph cannot reuse an ordinal after it assigns that ordinal.
> Add tests for these conditions:
> * After an interrupt during the wait for a permit, the graph, {{vectorsByKey}} and the number of permits do not change. A retry adds the vector to the graph.
> * If two writers add the same new vector, the writer that does not add the vector returns its permit.
> * If more writers than permits add the same vectors at the same time, all inserts complete.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org