[jira] [Created] (IGNITE-26279) Implement segment index files and a checkpointer thread

["Ivan Bessonov (Jira)" <ji...@apache.org>]

Ivan Bessonov created IGNITE-26279:
--------------------------------------

             Summary: Implement segment index files and a checkpointer thread
                 Key: IGNITE-26279
                 URL: https://issues.apache.org/jira/browse/IGNITE-26279
             Project: Ignite
          Issue Type: Improvement
            Reporter: Ivan Bessonov


Index file is a persistent analogue to https://issues.apache.org/jira/browse/IGNITE-26272's mem-tables.
h2. Files

First of all, the general data format:

 
{code:java}
+--------+---------------------+-----+---------------------+
| Header | Payload for group 1 | ... | Payload for group N |
+--------+---------------------+-----+---------------------+ {code}
Header should look like this:

 

 
{noformat}
+---------------------+-----------------------------------------+-----+
|     Common meta     |            Raft group 1 meta            | ... |
+---------------------+-----------------------------------------+-----+
| Magic | Version | N | Group ID | Flags | Offset | Start | End | ... |
+---------------------+-----------------------------------------+-----+{noformat}
 
 * {{Magic number}} is for checking that this is the right file. 4 bytes.
 * {{Version}} is for format extensibility. 4 bytes.
 * {{N}} is a number of raft groups in this file. 4 bytes.
 * {{GroupID}} is a raft group identifier. 8 bytes.
 * {{Flags}} means additional information about the group, for example “this is the last index for this group” flag. 4 bytes.
 * {{Offset}} is an offset within this index file that points to "Payload for group X" block. 4 bytes.
 * {{Start}} and {{End}} here mean the range of raft indexes in this index file. 8 bytes each.

Payload should look like this:

 
{code:java}
+----------+-----+----------+
| Offset 1 | ... | Offset M |
+----------+-----+----------+ {code}
 * {{Offset}} is an offset in a corresponding segment file that points to a corresponding log entry.
 * There are exactly {{End-Start}} offsets there.

h2. Checkpointer

We can chose other name if this one is confusing.

The checkpointer thread has to be awaken when log storage switches to the next segment. It does the following:
 * Collect mem-tables from all stripes and make them immutable. New mem-tables will be allocated automatically.
 * {{sync}} the segment file.
 * Write an index file that matches the mem-table.
 * [Out of scope for now] Create an in-memory metadata structures for the index.
 * Remove the mem-table from memory.
 * {{sync}} the index file.
 * Rename {{.bin.tmp}} file to the {{{}.bin{}}}.
 * Wait for a next {{{}mem-table{}}}.

There might be a queue of unwritten mem-tables. We should limit its size. Throttling is out of scope.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)