[jira] [Commented] (IGNITE-17254) Storage API for RAFT snapshot streaming

["Roman Puchkovskiy (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IGNITE-17254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17617111#comment-17617111 ] 

Roman Puchkovskiy commented on IGNITE-17254:
--------------------------------------------

Thank you!

> Storage API for RAFT snapshot streaming
> ---------------------------------------
>
>                 Key: IGNITE-17254
>                 URL: https://issues.apache.org/jira/browse/IGNITE-17254
>             Project: Ignite
>          Issue Type: Improvement
>            Reporter: Roman Puchkovskiy
>            Assignee: Roman Puchkovskiy
>            Priority: Major
>              Labels: ignite-3
>             Fix For: 3.0.0-beta1
>
>          Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Current PartitionStorage has a method for taking snapshots (and a method for restoring from them). The  methods work with files. In IGNITE-17253, we are going to switch to streaming protocol (which does not use files at all). So we need streaming snapshot-related methods on MvPartitionStorage.
> h3. UPDATE
> From my understanding, we basically need two more methods in the MV partition storage to manipulate data:
> {code:java}
> // This one represents a combination of "addWrite" and "commitWrite", but a faster one.
> // It will be used in incoming snapshot copier to write data.
> void addWriteCommitted(RowId rowId, BinaryRow row, HybridTimestamp commitTimestamp);
> // This one will be used to get all versions of a particular row and form a SnapshotMvDataResponse message.
> Cursor<ReadResult> scanVersionsEx(RowId rowId);{code}
> Both of them are very simple and, for the most part, match their counterparts in every storage.
> What's not as simple is the fail-over scenario. We shouldn't treat partially-downloaded partition as a valid set of data. So, the easiest was (in My opinion) to achieve this is to:
>  * set "lastAppliedIndex" value to "-1" before writing any the data
>  * drop old data (these two steps may be swapped, it's not that important)
>  * download and write everything
>  * set valid value for "lastAppliedIndex"
> This is a crude solution, but should work well, especially if properly documented.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)