[jira] [Created] (IGNITE-26272) Implement mem-table for incomplete segment files

["Ivan Bessonov (Jira)" <ji...@apache.org>]

Ivan Bessonov created IGNITE-26272:
--------------------------------------

             Summary: Implement mem-table for incomplete segment files
                 Key: IGNITE-26272
                 URL: https://issues.apache.org/jira/browse/IGNITE-26272
             Project: Ignite
          Issue Type: Improvement
            Reporter: Ivan Bessonov


Here we need to have an in-memory structure for offset table of incomplete segment files. In essence it should look like this:
{code:java}
MemTable = Map<
  ReplicationGroupId,
  IndexSegmentInfo
>

IndexSegmentInfo = {
  flags
  startIndex
  offsets[]
} {code}
In code it might look something like this:
{code:java}
class IndexSegmentInfo {
  volatile int flags;
  final int startOffset;
  volatile int[] offsets;
  volatile int nextOffsetIndex;
}{code}
This structure has to be partitioned into stripes, because it's going to be used in a disruptor pool and we want as little contention as possible. Concurrent reads from it should still be possible, that's why there are all these volatiles.

Append to log storage should be followed with an append to this mem-table.

During the segment switch there must be a way to wait for all stripes in other thread, in order to collect all mem-table for further asynchronous processing.

It can be expressed with a current pseudo-code:
{code:java}
void append(logEntry) {
  while (true) {
    segment = currentSegment()

    if (!segment.semaphore.acquire())
      continue; // Retry, segment had switched in background.

    try {
      offset = segment.append(logEntry)

      if (offset == SEGMENT_SWITCHED)
        continue; // Retry.

       segment.memTable.append(this, logEntry, offset)
    } finally {
      segment.semaphore.release()
    }
  }
}{code}
Of course, "{{{}memTable{}}}" doesn't have to be a field of "{{{}segment{}}}", this is pseudo-code.

We can use a counter instead of a semaphore, or any other type of primitive that won't cost much to acquire and release.

Doing the "{{{}segment.semaphore.acquireAll(){}}}" after segment switch guarantees that all "{{{}segment.memTable.append(...){}}}" have finished their execution, giving us a fast solution.

Of course, we should have concurrency tests for these all algorithms.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)