[jira] [Created] (FLINK-12886) Support container memory segment

["Liya Fan (JIRA)" <ji...@apache.org>]

Liya Fan created FLINK-12886:
--------------------------------

             Summary: Support container memory segment
                 Key: FLINK-12886
                 URL: https://issues.apache.org/jira/browse/FLINK-12886
             Project: Flink
          Issue Type: New Feature
          Components: Table SQL / Runtime
            Reporter: Liya Fan
            Assignee: Liya Fan
         Attachments: image-2019-06-18-17-59-42-136.png

We observe that in many scenarios, the operations/algorithms are based on an array of MemorySegment. These memory segments form a large, combined, and continuous memory space.

For example, suppose we have an array of n memory segments. Memory addresses from 0 to segment_size - 1 are served by the first memory segment; memory addresses from segment_size to 2 * segment_size - 1 are served by the second memory segment, and so on.

Specific algorithms decide the actual MemorySegment to serve the operation requests. For some rare cases, two or more memory segments serve the requests. There are many operations based on such a paradigm, for example, {{BinaryString#matchAt}}, {{SegmentsUtil#copyToBytes}}, {{LongHashPartition#MatchIterator#get}}, etc.

The problem is that, for memory segment array based operations, large amounts of code is devoted to

1. Computing the memory segment index & offset within the memory segment.
 2. Processing boundary cases. For example, to write an integer, there are only 2 bytes left in the first memory segment, and the remaining 2 bytes must be written to the next memory segment.
 3. Differentiate processing for short/long data. For example, when copying memory data to a byte array. Different methods are implemented for cases when 1) the data fits in a single segment; 2) the data spans multiple segments.

Therefore, there are much duplicated code to achieve above purposes. What is worse, this paradigm significantly increases the amount of code, making the code more difficult to read and maintain. Furthermore, it easily gives rise to bugs which difficult to find and debug.

To address these problems, we propose a new type of memory segment: {{ContainerMemorySegment}}. It is based on an array of underlying memory segments with the same size. It extends from the {{MemorySegment}} base class, so it provides all the functionalities provided by {{MemorySegment}}. In addition, it hides all the details for dealing with specific memory segments, and acts as if it were a big continuous memory region.

A prototype implementation is given below:

 !image-2019-06-18-17-59-42-136.png|thumbnail! 

With this new type of memory segment, many operations/algorithms can be greatly simplified, without affecting performance. This is because,

1. Many checks, boundary processing are already there. We just move them to the new class.
 2. We optimize the implementation of the new class, so the special optimizations (e.g. optimizations for short data) are still preserved.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)