[jira] [Resolved] (BEAM-9439) KinesisReader does not report correct backlog statistics

["Alexey Romanenko (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexey Romanenko resolved BEAM-9439.
------------------------------------
    Fix Version/s: 2.22.0
       Resolution: Fixed

> KinesisReader does not report correct backlog statistics 
> ---------------------------------------------------------
>
>                 Key: BEAM-9439
>                 URL: https://issues.apache.org/jira/browse/BEAM-9439
>             Project: Beam
>          Issue Type: Bug
>          Components: io-java-kinesis
>            Reporter: Sam Whittle
>            Assignee: Sebastian Graca
>            Priority: Major
>             Fix For: 2.22.0
>
>          Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> The KinesisReader implementing KinesisIO reports backlog by implementing the
> UnboundedSource.getTotalBacklogBytes()
> method as opposed to the
> UnboundedSource.getSplitBacklogBytes()
> This value is supposed to represent the total backlog across all shards.  This function is implemented by calling SimplifiedKinesisClient.getBacklogBytes with the watermark of the kinesis shards managed within the UnboundedReader instance.  As this watermark may be further ahead than the watermark across all shards, this may miss backlog bytes.
> An additional concern is that the watermark is calculated using a WatermarkPolicy, which means that the watermark may be inconsistent to the kinesis timestamp for querying backlog.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)