[jira] [Created] (FLINK-24530) GlobalCommitter might not commit all records on drain

["Fabian Paul (Jira)" <ji...@apache.org>]

Fabian Paul created FLINK-24530:
-----------------------------------

             Summary: GlobalCommitter might not commit all records on drain
                 Key: FLINK-24530
                 URL: https://issues.apache.org/jira/browse/FLINK-24530
             Project: Flink
          Issue Type: Bug
          Components: API / DataStream, Connectors / Common
    Affects Versions: 1.13.2, 1.14.0, 1.15.0
            Reporter: Fabian Paul


If a pipeline has a unified Sink that implements a {{Committer}} and {{GlobalCommitter}} committables are forwarded from the {{Committer}} on either receiving {{notifyCheckpointCompleted}} or {{endInput}}. 

Since both methods are triggered by RPC calls we currently do not guarantee an order when either method is called on the {{Committer}} or {{GlobalCommitter}}. This can lead to that the {{GlobalCommitter}} receives the RPC call at first and tries to commit although the {{Committer}} has nothing forwarded yet. For the normal streaming execution, it means that the visibility of the final commit could be delayed by one checkpoint.

Unfortunately, if the pipeline is stopped with a savepoint and drained the {{endInput}} might be called at the {{GlobalCommitter}} before the {{Committer}} leading to potential data loss.

With FLIP-147 (checkpoints after task finish) fully implemented the problem is mitigated because it guarantees the order of the final {{notifyCheckpointCompleted}} call although it still suffers from the potentially delayed visibility.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)