You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by "Yingjie Cao (Jira)" <ji...@apache.org> on 2020/09/21 03:32:00 UTC

[jira] [Created] (FLINK-19297) Make ResultPartitionWriter record-oriented

Yingjie Cao created FLINK-19297:
-----------------------------------

             Summary: Make ResultPartitionWriter record-oriented
                 Key: FLINK-19297
                 URL: https://issues.apache.org/jira/browse/FLINK-19297
             Project: Flink
          Issue Type: Improvement
          Components: Runtime / Network
    Affects Versions: 1.12.0
            Reporter: Yingjie Cao
             Fix For: 1.12.0


Currently, ResultPartitionWriter is buffer-oriented, that is, RecordWriter will add buffer of different channels to ResultPartitionWriter and the buffer boundary serves as a nature boundary of data belonging to different channels. However, this abstraction is not flexible enough to handle some cases where records are appended a joint-structure shared by all channels and sorting is used to cluster data belonging to different channels.

In this ticket, we propose to make ResultPartitionWriter record oriented which offers more flexibility to the implementation of ResultPartitionWriter. And based on the new record-oriented Interface, we will introduce the sort-merge based blocking shuffle to Flink in the future.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)