[jira] [Commented] (CASSANDRA-21649) Ensure SSTables are deleted in order of last-updated timestamp

["Caleb Rackliffe (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-21649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18114312#comment-18114312 ] 

Caleb Rackliffe commented on CASSANDRA-21649:
---------------------------------------------

Left one comment in the PR that's worth looking at, but otherwise the {{trunk}} patch LGTM. I imagine this will go back a few versions...

> Ensure SSTables are deleted in order of last-updated timestamp
> --------------------------------------------------------------
>
>                 Key: CASSANDRA-21649
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-21649
>             Project: Apache Cassandra
>          Issue Type: Bug
>          Components: Local/SSTable
>            Reporter: Nivy Kani
>            Assignee: Nivy Kani
>            Priority: Normal
>         Attachments: ci_summary.html, result_details.tar.gz
>
>          Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> When a node restarts after crashing, we need to clean up any partially-deleted SSTables, but we do a safety check on REMOVE log records to verify that the last-modified timestamps match the actual files on disk. If the timestamps don’t match, we throw an error that the log is corrupted, and the node doesn’t start.
>  
> The comment in LogFile.verifyRecord() states: "Because we delete files from oldest to newest, the latest update time should always match.” But this isn’t always true; regular SSTable deletion after compaction always deletes DATA files first, and doesn’t explicitly set the deletion order of other files.
>  
> So we can have this scenario:
> 1. SSTable files are written, and there’s some tiny delay between the written-timestamp of each file.
> 2. Compaction succeeds. SSTable files can be deleted now, and we write a REMOVE record with the timestamp of the most-recently-updated file of each SSTable.
> 3. SSTableTidier starts deleting the SSTable files. Problem: it can delete the files in non-timestamp order.
> 4. Halfway through, the node crashes. On restart, LogFile.verifyRecord() goes through each REMOVE record to compare timestamps. It fails because the last-written file was deleted, so the most-recent-timestamp is different now.
> 5. Now the node won’t start up, and some SSTable files are leaked.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org