You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@beam.apache.org by "Maximilian Michels (Jira)" <ji...@apache.org> on 2019/11/04 10:48:00 UTC

[jira] [Created] (BEAM-8549) Buffered state during checkpointing may be keyed incorrectly

Maximilian Michels created BEAM-8549:
----------------------------------------

             Summary: Buffered state during checkpointing may be keyed incorrectly
                 Key: BEAM-8549
                 URL: https://issues.apache.org/jira/browse/BEAM-8549
             Project: Beam
          Issue Type: Bug
          Components: runner-flink
            Reporter: Maximilian Michels
            Assignee: Maximilian Michels
             Fix For: 2.17.0


The current buffer logic for items emitted during checkpointing is faulty in the sense that the buffer is partitioned on the output keys of the operator. The key may be changed or even be dropped. Thus, the original key partitioning will not be maintained which will cause checkpointing to fail.

An alternative solution would be BEAM-6733, but we may want to keep the current buffering logic in place. The proposed solution here is to remove the keyed buffering. The output buffer may then always be redistributed round-robin upon restoring from a checkpoint. Note that this is fine because no assumption can be made about the distribution of output elements of an operation.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)