[jira] [Assigned] (FLINK-41009) Use a single Pulsar transaction per checkpoint in the Pulsar sink

["Ferenc Csaky (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-41009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ferenc Csaky reassigned FLINK-41009:
------------------------------------

    Assignee: Yubiao Feng

> Use a single Pulsar transaction per checkpoint in the Pulsar sink
> -----------------------------------------------------------------
>
>                 Key: FLINK-41009
>                 URL: https://issues.apache.org/jira/browse/FLINK-41009
>             Project: Flink
>          Issue Type: Improvement
>          Components: Connectors / Pulsar
>    Affects Versions: 2.0.0
>            Reporter: Yubiao Feng
>            Assignee: Yubiao Feng
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: pulsar-5.0.0
>
>
>  
> {{Currently, the Pulsar sink creates a separate Pulsar transaction for each target topic within the same Flink checkpoint.}}
> {{This weakens checkpoint-level atomicity: when committing a checkpoint, some topic transactions may be committed successfully while others fail. A later recovery may then encounter inconsistent transaction states, including transaction timeout or transaction-not-found errors.}}
> Pulsar transactions can span multiple topics and namespaces. The sink should therefore create and reuse one transaction for all records written during a single checkpoint, regardless of their target topics.
> At checkpoint completion, the sink should emit one committable and commit that single transaction. A new transaction should only be created after the previous checkpoint's transaction has been handed over for commit.
> {{Benefits:}}
>  * Provide atomic commit semantics across all Pulsar topics written in one checkpoint. 
>  * Avoid partial transaction commits within a checkpoint. 
>  * Reduce transaction creation and commit overhead.
>  * {{Avoid recovery failures caused by inconsistent per-topic transaction states. }}
>  
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)