You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Aleksandr Polovtsev (Jira)" <ji...@apache.org> on 2025/09/19 08:19:00 UTC
[jira] [Commented] (IGNITE-26279) Implement segment index files and a checkpointer thread
[ https://issues.apache.org/jira/browse/IGNITE-26279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18021355#comment-18021355 ]
Aleksandr Polovtsev commented on IGNITE-26279:
----------------------------------------------
Only checkpoint thread will be implemented under this ticket, index files are going to be implemented in IGNITE-26473
> Implement segment index files and a checkpointer thread
> -------------------------------------------------------
>
> Key: IGNITE-26279
> URL: https://issues.apache.org/jira/browse/IGNITE-26279
> Project: Ignite
> Issue Type: Improvement
> Reporter: Ivan Bessonov
> Assignee: Aleksandr Polovtsev
> Priority: Major
> Labels: ignite-3
>
> Index file is a persistent analogue to https://issues.apache.org/jira/browse/IGNITE-26272's mem-tables.
> h2. Files
> First of all, the general data format:
> {code:java}
> +--------+---------------------+-----+---------------------+
> | Header | Payload for group 1 | ... | Payload for group N |
> +--------+---------------------+-----+---------------------+ {code}
> Header should look like this:
> {noformat}
> +---------------------+-----------------------------------------+-----+
> | Common meta | Raft group 1 meta | ... |
> +---------------------+-----------------------------------------+-----+
> | Magic | Version | N | Group ID | Flags | Offset | Start | End | ... |
> +---------------------+-----------------------------------------+-----+{noformat}
> * {{Magic number}} is for checking that this is the right file. 4 bytes.
> * {{Version}} is for format extensibility. 4 bytes.
> * {{N}} is a number of raft groups in this file. 4 bytes.
> * {{GroupID}} is a raft group identifier. 8 bytes.
> * {{Flags}} means additional information about the group, for example “this is the last index for this group” flag. 4 bytes.
> * {{Offset}} is an offset within this index file that points to "Payload for group X" block. 4 bytes.
> * {{Start}} and {{End}} here mean the range of raft indexes in this index file. 8 bytes each.
> Payload should look like this:
> {code:java}
> +----------+-----+----------+
> | Offset 1 | ... | Offset M |
> +----------+-----+----------+ {code}
> * {{Offset}} is an offset in a corresponding segment file that points to a corresponding log entry.
> * There are exactly {{End-Start}} offsets there.
> h2. Checkpointer
> We can chose other name if this one is confusing.
> The checkpointer thread has to be awaken when log storage switches to the next segment. It does the following:
> * Collect mem-tables from all stripes and make them immutable. New mem-tables will be allocated automatically.
> * {{sync}} the segment file.
> * Write an index file that matches the mem-table.
> * [Out of scope for now] Create an in-memory metadata structures for the index.
> * Remove the mem-table from memory.
> * {{sync}} the index file.
> * Rename {{.bin.tmp}} file to the {{{}.bin{}}}.
> * Wait for a next {{{}mem-table{}}}.
> There might be a queue of unwritten mem-tables. We should limit its size. Throttling is out of scope.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)