You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by uce <gi...@git.apache.org> on 2016/11/03 14:54:51 UTC
[GitHub] flink pull request #2750: [FLINK-5007] [checkpointing] Retain externalized c...
GitHub user uce opened a pull request:
https://github.com/apache/flink/pull/2750
[FLINK-5007] [checkpointing] Retain externalized checkpoint on suspension
Handles graceful cluster shut down (non-HA) like cancellation and respects the configured clean up behaviour.
```
ExternalizedCheckpointCleanup.DELETE_ON_CANCELLATION => delete on suspension
ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION => retain on suspension
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/uce/flink 5007-suspend_external
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2750.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2750
----
commit 9f422c23c0e99259f39c84c292d034d70d8078cb
Author: Ufuk Celebi <uc...@apache.org>
Date: 2016-11-03T14:52:24Z
[FLINK-5007] [checkpointing] Retain externalized checkpoint on suspension
Handles graceful cluster shut down (non-HA) like cancellation.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2750: [FLINK-5007] [checkpointing] Retain externalized checkpoi...
Posted by uce <gi...@git.apache.org>.
Github user uce commented on the issue:
https://github.com/apache/flink/pull/2750
In HA mode, checkpoints are not deleted on suspension. This PR won't change that behaviour. It only affects non-HA behaviour.
Currently, the behaviour is to remove checkpoints on suspension, which is definitely a problem. But in non-HA mode suspension happens also for graceful shut down (for example when terminating a YARN session). Never deleting on suspend means that users who have `DELETE_ON_CANCELLATION` configured, will have externalized checkpoints lingering around when they shut down their non-HA cluster. That's why I thought it might be better to treat this the same as the retain/delete on cancellation configuration.
Does this make sense? For HA, these setting do not apply during suspension.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2750: [FLINK-5007] [checkpointing] Retain externalized checkpoi...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/2750
A suspension is usually a master failure or loss of leadership. Suspending a job does not delete/remove any HA checkpoints.
The same should be the case for externalized checkpoints, in my opinion. Why should an externalized checkpoint be deleted in suspend, when a regular one is not?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2750: [FLINK-5007] [checkpointing] Retain externalized checkpoi...
Posted by uce <gi...@git.apache.org>.
Github user uce commented on the issue:
https://github.com/apache/flink/pull/2750
@StephanEwen Do you have time to look at this? Currently, when externalized checkpoints are configured and the cluster shuts down via suspending all jobs, the externalized checkpoints are cleaned up. This PR proposes to handle suspension like a cancellation and respect the corresponding cleanup configuration, e.g. retain if `RETAIN_ON_CANCELLATION` and delete if `DELETE_ON_CANCELLATION`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2750: [FLINK-5007] [checkpointing] Retain externalized c...
Posted by uce <gi...@git.apache.org>.
Github user uce closed the pull request at:
https://github.com/apache/flink/pull/2750
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2750: [FLINK-5007] [checkpointing] Retain externalized checkpoi...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/2750
Okay, understood, that makes sense.
+1 from my side
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---