[jira] [Resolved] (FLINK-32298) The outputQueueSize is negative

["Rui Fan (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-32298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rui Fan resolved FLINK-32298.
-----------------------------
    Fix Version/s: 1.18.0
       Resolution: Fixed

> The outputQueueSize is negative 
> --------------------------------
>
>                 Key: FLINK-32298
>                 URL: https://issues.apache.org/jira/browse/FLINK-32298
>             Project: Flink
>          Issue Type: Bug
>          Components: Runtime / Network
>    Affects Versions: 1.18.0
>            Reporter: Rui Fan
>            Assignee: Rui Fan
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.18.0
>
>         Attachments: image-2023-06-09-17-27-46-429.png
>
>
> h1. Backgraound
> The outputQueueSize indicates `The real size of queued output buffers in bytes.`, so it shouldn't be negative. However, it may be negative in some cases.
> h2. How outputQueueSize is generated?
> TotalWrittenBytes: *_BufferWritingResultPartition#totalWrittenBytes_* records how many data is written to ResultPartition.
> TotalSentNumberOfBytes: *_PipelinedSubpartition#totalNumberOfBytes_* records how many data is sent to downstream.
> The outputQueueSize = TotalWrittenBytes - TotalSentNumberOfBytes.
> h1. Bug
> The TotalSentNumberOfBytes may be larger than TotalWrittenBytes due to some data are written to the PipelinedSubpartition without the BufferWritingResultPartition, such as : 
>  # PipelinedSubpartition#finishReadRecoveredState writes the `EndOfChannelStateEvent` even if the unaligned checkpoint is disable
>  # PipelinedSubpartition#addRecovered writes channel state(if the job recovered from unaligned checkpoint, the outputQueueSize is totally wrong)
>  # PipelinedSubpartition#finish writes the `EndOfPartitionEvent`
> !image-2023-06-09-17-27-46-429.png|width=1033,height=296!
>  
> h1. Solution
> PipelinedSubpartition should is written through BufferWritingResultPartition, and all writes should be counted.
>  
> By the way, outputQueueSize doesn't matter because it's just a metric, it doesn't affect data processing. I found this bug because some of our flink scenarios need to use adaptive rebalance (FLINK-31655), I'm developing it in our internal version, which relies on the correct outputQueueSize to select the low pressure channel.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)