You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Ivan Bessonov (Jira)" <ji...@apache.org> on 2025/08/22 12:27:00 UTC
[jira] [Updated] (IGNITE-26270) Implement append and segment switching
[ https://issues.apache.org/jira/browse/IGNITE-26270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Bessonov updated IGNITE-26270:
-----------------------------------
Description:
In this ticket we need to implement the basic log storage class with {{append}} function. The log will initially consist of a number of files with the following naming convention:
{code:java}
segment-N-G.bin
{code}
* N is a sequence number of the segment. Zero-based, fixed-width, encoded as decimal.
* G is a generation number assigned to particular segment number. Also zero-based, fixed-width, encoded as decimal.
The first ever file when we start fresh log storage will be called {{{}segment-00000000-00000000.bin{}}}. This way the lexicographic order will match the natural {{
{N,G}
}} ascending order.
The file structure is the following:
{code:java}
+--------+---------+-----+---------+
| Header | Payload | ... | Payload |
+--------+---------+-----+---------+
Header (8 bytes):
+--------------+---------+
| Magic number | Version |
+--------------+---------+
Payload (16+ bytes):
+---------------+--------+-------+------+
| Raft Group ID | Length | Bytes | Hash |
+---------------+--------+-------+------+{code}
* {{Magic number}} is for checking that this is the right file.
* {{Version}} is for format extensibility.
* {{Raft Group ID}} is an 8-bytes identifier, unique for each raft group.
* {{Length}} is 4 bytes.
* 32-bit hash covers everything starting with {{Raft Group ID}} and ending with the last byte of {{Bytes}} section.
These files will have a fixed size in order to use {{{}mmap{}}}. Default size should be 1Gb. Max size is 4 Gb. We will write to the file using {{SegmentRingByteBuffer}} as {{IO}} (maybe with necessary refactorings).
Hash algorithm could be anything, but it should be fast.
Switch segment record will either be an 8-bytes payload filled with zeroes (raft group ID can't be 0), or any tail of the file if there's less than 8 bytes left (no need for padding with zeroes then).
While reaching the end of file, {{append}} operation should optionally write a switch segment record and atomically switch to next file. Thread-safe implementation is presumed.
{{Append}} should only return execution to the caller when its bytes and all previous bytes have been written into segments, like in AI2.
{{V1Encoder}} can already be utilized in API, but we may want to extend it or even duplicate.
Unit tests (including concurrency tests) must be included in the implementation.
was:
In this ticket we need to implement the basic log storage class with {{append}} function. The log will initially consist of a number of files with the following naming convention:
{code:java}
segment-N-G.bin
{code}
* N is a sequence number of the segment. Zero-based, fixed-width, encoded as decimal.
* G is a generation number assigned to particular segment number. Also zero-based, fixed-width, encoded as decimal.
The first ever file when we start fresh log storage will be called {{{}segment-00000000-00000000.bin{}}}. This way the lexicographic order will match the natural {{{N,G}}} ascending order.
The file structure is the following:
{code:java}
+--------+---------+-----+---------+
| Header | Payload | ... | Payload |
+--------+---------+-----+---------+
Header (8 bytes):
+--------------+---------+
| Magic number | Version |
+--------------+---------+
Payload (16+ bytes):
+---------------+--------+-------+------+
| Raft Group ID | Length | Bytes | Hash |
+---------------+--------+-------+------+{code}
* {{Magic number}} is for checking that this is the right file.
* {{Version}} is for format extensibility.
* {{Raft Group ID}} is an 8-bytes identifier, unique for each raft group.
* {{Length}} is 4 bytes.
* 32-bit hash covers everything starting with {{Raft Group ID}} and ending with the last byte of {{Bytes}} section.
These files will have a fixed size in order to use {{{}mmap{}}}. Default size should be 1Gb. Max size is 4 Gb. We will write to the file using {{SegmentRingByteBuffer}} as {{IO}} (maybe with necessary refactorings).
Hash algorithm could be anything, but it should be fast.
Switch segment record will either be an 8-bytes payload filled with zeroes (raft group ID can't be 0), or any tail of the file if there's less than 8 bytes left (no need for padding with zeroes then).
While reaching the end of file, {{append}} operation should optionally write a switch segment record and atomically switch to next file. Thread-safe implementation is presumed.
{{Append}} should only return execution to the caller when its bytes and all previous bytes have been written into segments, like in AI2.
{{V1Encoder}} can already be utilized in API, but we may want to extend it or even duplicate.
Unit tests (including concurrency tests) must be included in the implementation.
> Implement append and segment switching
> --------------------------------------
>
> Key: IGNITE-26270
> URL: https://issues.apache.org/jira/browse/IGNITE-26270
> Project: Ignite
> Issue Type: Improvement
> Reporter: Ivan Bessonov
> Priority: Major
> Labels: ignite-3
>
> In this ticket we need to implement the basic log storage class with {{append}} function. The log will initially consist of a number of files with the following naming convention:
> {code:java}
> segment-N-G.bin
> {code}
> * N is a sequence number of the segment. Zero-based, fixed-width, encoded as decimal.
> * G is a generation number assigned to particular segment number. Also zero-based, fixed-width, encoded as decimal.
> The first ever file when we start fresh log storage will be called {{{}segment-00000000-00000000.bin{}}}. This way the lexicographic order will match the natural {{
> {N,G}
> }} ascending order.
> The file structure is the following:
> {code:java}
> +--------+---------+-----+---------+
> | Header | Payload | ... | Payload |
> +--------+---------+-----+---------+
> Header (8 bytes):
> +--------------+---------+
> | Magic number | Version |
> +--------------+---------+
> Payload (16+ bytes):
> +---------------+--------+-------+------+
> | Raft Group ID | Length | Bytes | Hash |
> +---------------+--------+-------+------+{code}
> * {{Magic number}} is for checking that this is the right file.
> * {{Version}} is for format extensibility.
> * {{Raft Group ID}} is an 8-bytes identifier, unique for each raft group.
> * {{Length}} is 4 bytes.
> * 32-bit hash covers everything starting with {{Raft Group ID}} and ending with the last byte of {{Bytes}} section.
> These files will have a fixed size in order to use {{{}mmap{}}}. Default size should be 1Gb. Max size is 4 Gb. We will write to the file using {{SegmentRingByteBuffer}} as {{IO}} (maybe with necessary refactorings).
> Hash algorithm could be anything, but it should be fast.
> Switch segment record will either be an 8-bytes payload filled with zeroes (raft group ID can't be 0), or any tail of the file if there's less than 8 bytes left (no need for padding with zeroes then).
> While reaching the end of file, {{append}} operation should optionally write a switch segment record and atomically switch to next file. Thread-safe implementation is presumed.
> {{Append}} should only return execution to the caller when its bytes and all previous bytes have been written into segments, like in AI2.
> {{V1Encoder}} can already be utilized in API, but we may want to extend it or even duplicate.
> Unit tests (including concurrency tests) must be included in the implementation.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)