You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Fabian Paul (Jira)" <ji...@apache.org> on 2021/10/13 13:15:00 UTC
[jira] [Updated] (FLINK-24530) GlobalCommitter might not commit all
records on drain
[ https://issues.apache.org/jira/browse/FLINK-24530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Fabian Paul updated FLINK-24530:
--------------------------------
Description:
If a pipeline has a unified Sink that implements a {{Committer}} and {{GlobalCommitter}} committables are forwarded from the {{Committer}} on either receiving {{notifyCheckpointCompleted}} or {{endInput}}.
Since both methods are triggered by RPC calls we currently do not guarantee an order when either method is called on the {{Committer}} or {{GlobalCommitter}}. This can lead to that the {{GlobalCommitter}} receives the RPC call at first and tries to commit although the {{Committer}} has nothing forwarded yet. For the normal streaming execution, it means that the visibility of the final commit could be delayed by one checkpoint.
Unfortunately, if the pipeline is stopped with a savepoint and drained the {{endInput}} might be called at the {{GlobalCommitter}} before the {{Committer}} leading to potential data loss.
was:
If a pipeline has a unified Sink that implements a {{Committer}} and {{GlobalCommitter}} committables are forwarded from the {{Committer}} on either receiving {{notifyCheckpointCompleted}} or {{endInput}}.
Since both methods are triggered by RPC calls we currently do not guarantee an order when either method is called on the {{Committer}} or {{GlobalCommitter}}. This can lead to that the {{GlobalCommitter}} receives the RPC call at first and tries to commit although the {{Committer}} has nothing forwarded yet. For the normal streaming execution, it means that the visibility of the final commit could be delayed by one checkpoint.
Unfortunately, if the pipeline is stopped with a savepoint and drained the {{endInput}} might be called at the {{GlobalCommitter}} before the {{Committer}} leading to potential data loss.
With FLIP-147 (checkpoints after task finish) fully implemented the problem is mitigated because it guarantees the order of the final {{notifyCheckpointCompleted}} call although it still suffers from the potentially delayed visibility.
> GlobalCommitter might not commit all records on drain
> -----------------------------------------------------
>
> Key: FLINK-24530
> URL: https://issues.apache.org/jira/browse/FLINK-24530
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream, Connectors / Common
> Affects Versions: 1.14.0, 1.13.2, 1.15.0
> Reporter: Fabian Paul
> Priority: Major
>
> If a pipeline has a unified Sink that implements a {{Committer}} and {{GlobalCommitter}} committables are forwarded from the {{Committer}} on either receiving {{notifyCheckpointCompleted}} or {{endInput}}.
> Since both methods are triggered by RPC calls we currently do not guarantee an order when either method is called on the {{Committer}} or {{GlobalCommitter}}. This can lead to that the {{GlobalCommitter}} receives the RPC call at first and tries to commit although the {{Committer}} has nothing forwarded yet. For the normal streaming execution, it means that the visibility of the final commit could be delayed by one checkpoint.
> Unfortunately, if the pipeline is stopped with a savepoint and drained the {{endInput}} might be called at the {{GlobalCommitter}} before the {{Committer}} leading to potential data loss.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)