You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/02/27 16:04:45 UTC
[jira] [Commented] (FLINK-5928) Externalized checkpoints
overwritting each other
[ https://issues.apache.org/jira/browse/FLINK-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15886022#comment-15886022 ]
ASF GitHub Bot commented on FLINK-5928:
---------------------------------------
GitHub user uce opened a pull request:
https://github.com/apache/flink/pull/3424
[FLINK-5928] [checkpoints] Use custom metadata file for externalized checkpoints
- Adds a checkpoint coordinator test for externalized checkpoints. This was covered only in unit tests for the involved checkpoint components like PendingCheckpoint etc. This would have caught the issue.
- The fix is to not use a `_metadata` but a random `checkpoint_metadata-:randomSuffix` file for externalized checkpoints, because they are not unique per configured directory. Hopefully, we can get rid of this soon with @StephanEwen's refactorings.
This is based on top of #3411, which is already good to merge.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/uce/flink 5928-ext_chk_metadata
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3424.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3424
----
commit a3d2405f690e08d3de4f641428887ab04ba2ca2d
Author: Stephan Ewen <se...@apache.org>
Date: 2017-02-17T16:51:00Z
[FLINK-5822] [state backends] Make JobManager / Checkpoint Coordinator aware of the root state backend
commit b2f3bc41bc991e8deb22fb89822f28c75d94c8f7
Author: Stephan Ewen <se...@apache.org>
Date: 2017-02-22T21:18:50Z
[FLINK-5897] [checkpoints] Make checkpoint externalization not depend strictly on FileSystems
That is the first step towards checkpoints that can be externalized to other stores as well,
like k/v stores and databases, if supported by the state backend.
commit 537be203dab0614383645e859bbafb6ebfeb3161
Author: Ufuk Celebi <uc...@apache.org>
Date: 2017-02-27T15:12:37Z
[FLINK-5928] [checkpoints] Add CheckpointCoordinatorExternalizedCheckpointsTest
Problem: there were only unit tests for the checkpoint instances available
that don't test the behaviour of the checkpoint coordinator with respect
to externalized checkpoints.
commit 88e4700cce630f8ae869abff22acfd46ab999aa0
Author: Ufuk Celebi <uc...@apache.org>
Date: 2017-02-27T15:58:14Z
[FLINK-5928] [checkpoints] Use custom metadata file for externalized checkpoints
----
> Externalized checkpoints overwritting each other
> ------------------------------------------------
>
> Key: FLINK-5928
> URL: https://issues.apache.org/jira/browse/FLINK-5928
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
> Reporter: Ufuk Celebi
> Assignee: Ufuk Celebi
> Priority: Critical
>
> I noticed that PR #3346 accidentally broke externalized checkpoints by using a fixed meta data file name. We should restore the old behaviour with creating random files and double check why no test caught this.
> This will likely superseded by upcoming changes from [~StephanEwen] to use metadata streams on the JobManager.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)