You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Xintong Song (Jira)" <ji...@apache.org> on 2021/08/20 09:09:00 UTC

[jira] [Assigned] (FLINK-23896) The new KafkaSink drops data if job fails between checkpoint and transaction commit.

     [ https://issues.apache.org/jira/browse/FLINK-23896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xintong Song reassigned FLINK-23896:
------------------------------------

    Assignee: David Morávek

> The new KafkaSink drops data if job fails between checkpoint and transaction commit.
> ------------------------------------------------------------------------------------
>
>                 Key: FLINK-23896
>                 URL: https://issues.apache.org/jira/browse/FLINK-23896
>             Project: Flink
>          Issue Type: Bug
>          Components: Connectors / Kafka
>            Reporter: David Morávek
>            Assignee: David Morávek
>            Priority: Blocker
>              Labels: pull-request-available
>             Fix For: 1.14.0
>
>
> * Any time a new *transactional producer* is started, "[KafkaProducer#initTransactions()|https://kafka.apache.org/24/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html#initTransactions--]" needs to be called in order to obtain new *ProducerId* from *TransactionCoordinator* (Kafka Broker component).
>  ** *ProducerId* is increased every time a new producer with the same *TransactionalId* is registered.
>  ** Publication of new ProducerId *FENCES* all prior ProducerIds and *ABORTS* all of uncommitted transactions assigned with them.
>  * *KafkaCommitter* uses a separate producer, that hacks into Kafka internals and resumes transaction using epoch and producer, without actually assigning a new ProducerId for itself. This committer uses *ProducerId* that is stored in *KafkaComittable* state to commit transaction.
>  * If a *new producer is started before committing the transaction* (this can happen in some failover scenarios), ProducerId stored in the state is already *FENCED* and commit fails with *ProducerFencedException*. Because we currently ignore this exception (we just log a warning), we effectively *DROP* all uncommitted data from the previous checkpoint.
> Basically any job failure that happens between successfully taking a checkpoint and committing transactions, will trigger this behavior.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)