[jira] [Created] (FLINK-30189) HsSubpartitionFileReader may load data that has been consumed from memory

["Weijie Guo (Jira)" <ji...@apache.org>]

Weijie Guo created FLINK-30189:
----------------------------------

             Summary: HsSubpartitionFileReader may load data that has been consumed from memory
                 Key: FLINK-30189
                 URL: https://issues.apache.org/jira/browse/FLINK-30189
             Project: Flink
          Issue Type: Bug
          Components: Runtime / Network
    Affects Versions: 1.16.0, 1.17.0
            Reporter: Weijie Guo
            Assignee: Weijie Guo


In order to solve the problem that data cannot be read from the disk correctly after failover, we changed the calculation logical of the buffer's readable state in FLINK-29238.  Buffers that are greater than consumingOffset and have been released can be pre-load from file. However, the update of consumingOffset is asynchronous, If it lags behind the actual consumption progress, the buffer will have a chance to be load from the disk again. 

IMO, we can record the consumed status of buffer by each consumer in the InternalRegion. Only the buffers that have not been consumed and have been released will be considered as readable. In the case of failover, a new consumerId will be generated, so all buffers will be considered as unconsumed and can be correctly read from the disk too.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)