[jira] [Created] (IGNITE-26270) Implement append and segment switching

["Ivan Bessonov (Jira)" <ji...@apache.org>]

Ivan Bessonov created IGNITE-26270:
--------------------------------------

             Summary: Implement append and segment switching
                 Key: IGNITE-26270
                 URL: https://issues.apache.org/jira/browse/IGNITE-26270
             Project: Ignite
          Issue Type: Improvement
            Reporter: Ivan Bessonov


In this ticket we need to implement the basic log storage class with {{append}} function. The log will initially consist of a number of files with the following naming convention:

 
{code:java}
segment-N-G.bin
{code}
 * N is a sequence number of the segment. Zero-based, fixed-width, encoded as decimal.
 * G is a generation number assigned to particular segment number. Also zero-based, fixed-width, encoded as decimal.

The first ever file when we start fresh log storage will be called {{{}segment-00000000-00000000.bin{}}}.  This way the lexicographic order will match the natural {{{N,G}}} ascending order.

The file structure is the following:
{code:java}
+--------+---------+-----+---------+
| Header | Payload | ... | Payload |
+--------+---------+-----+---------+ 

Header (8 bytes):
+--------------+---------+
| Magic number | Version |
+--------------+---------+

Payload (16+ bytes):
+---------------+--------+-------+------+
| Raft Group ID | Length | Bytes | Hash |
+---------------+--------+-------+------+{code}
 * {{Magic number}} is for checking that this is the right file.
 * {{Version}} is for format extensibility.
 * {{Raft Group ID}} is an 8-bytes identifier, unique for each raft group.
 * {{Length}} is 4 bytes.
 * 32-bit hash covers everything starting with {{Raft Group ID}} and ending with the last byte of {{Bytes}} section.

These files will have a fixed size in order to use {{{}mmap{}}}. Default size should be 1Gb. Max size is 4 Gb. We will write to the file using {{SegmentRingByteBuffer}} as {{IO}} (maybe with necessary refactorings).

Hash algorithm could be anything, but it should be fast.

Switch segment record will either be an 8-bytes payload filled with zeroes (raft group ID can't be 0), or any tail of the file if there's less than 8 bytes left (no need for padding with zeroes then).

While reaching the end of file, {{append}} operation should optionally write a switch segment record and atomically switch to next file. Thread-safe implementation is presumed.

{{Append}} should only return execution to the caller when its bytes and all previous bytes have been written into segments, like in AI2.

{{V1Encoder}} can already be utilized in API, but we may want to extend it or even duplicate.

Unit tests (including concurrency tests) must be included in the implementation.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)