[jira] [Updated] (FLINK-27399) Pulsar connector didn't set start consuming position correctly

["Martijn Visser (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-27399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Martijn Visser updated FLINK-27399:
-----------------------------------
    Priority: Blocker  (was: Critical)

> Pulsar connector didn't set start consuming position correctly
> --------------------------------------------------------------
>
>                 Key: FLINK-27399
>                 URL: https://issues.apache.org/jira/browse/FLINK-27399
>             Project: Flink
>          Issue Type: Bug
>          Components: Connectors / Pulsar
>    Affects Versions: 1.14.4, 1.15.0, 1.16.0
>            Reporter: Yufan Sheng
>            Assignee: Yufan Sheng
>            Priority: Blocker
>              Labels: pull-request-available
>             Fix For: 1.16.0
>
>
> The Pulsar connector didn't use the consuming position from the checkpoint. They just commit the position to Pulsar after the checkpoint is complete. And the connector start to consume message from Pulsar directly by the offset stored on the Pulsar subscription.
> This causes the test could be failed in some situations. The start cursor (position on Pulsar) would be reset to the wrong position which causes the results didn't match the desired records.
> This issue fixes
> # FLINK-23944
> # FLINK-25884
> # FLINK-26177
> # FLINK-26237
> # FLINK-26721
> Although the test failure message could be various. They are truly sharing the same cause.
> h2. How to fix this issue:
> SourceEvent protocol for limiting the {{Consumer.seek}} operation.
> The Pulsar source needs to seek the desired consuming position when bootstrapping. The seeking action couldn’t be executed concurrently. We have designed a [new mechanism|https://github.com/apache/flink/pull/17119#pullrequestreview-746035072].



--
This message was sent by Atlassian Jira
(v8.20.10#820010)