[jira] [Created] (FLINK-29053) Hybrid shuffle has concurrent modification of buffer when compression is enabled

["Weijie Guo (Jira)" <ji...@apache.org>]

Weijie Guo created FLINK-29053:
----------------------------------

             Summary: Hybrid shuffle has concurrent modification of buffer when compression is enabled
                 Key: FLINK-29053
                 URL: https://issues.apache.org/jira/browse/FLINK-29053
             Project: Flink
          Issue Type: Bug
          Components: Runtime / Network
    Affects Versions: 1.16.0
            Reporter: Weijie Guo
             Fix For: 1.16.0


When the downstream thread obtains the buffer and consuming it, if the data is compressed in the spilling thread and copied to the original buffer in the same time, since the two threads share the same memory data, the consuming thread will consume incorrect data, causing problems such as deserialize the data disorder.
Considering that the downstream consumption is prohibited during compression, or block spilling thread when the downstream consumption is not completed will have a great impact on performance. I think we should move the compression operation to the write thread and store the compressed buffer directly in memory.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)