You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by StephanEwen <gi...@git.apache.org> on 2015/11/25 20:51:14 UTC
[GitHub] flink pull request: [FLINK-3051] Control the maximum number of con...
GitHub user StephanEwen opened a pull request:
https://github.com/apache/flink/pull/1408
[FLINK-3051] Control the maximum number of concurrent checkpoints
This change introduces a parameter that lets users control at most how many checkpoints
should be in progress at any given point in time. This is very useful for
cases where the best checkpoint interval is not easy to determine, or where once in
a while checkpoints may take long. Previously, such situations lead to checkpoint
queue-up, where the system ended up being more busy with checkpoints than with processing.
After this change, the checkpoint rate will by default sort of self-regulate: The checkpoint interval is the most frequent time at which checkpoints will occur, but they will occur slower if they take longer than that interval.
### Checkpoint Config
This also introduces the CheckpointConfig class that holds all checkpointing related parameters to more simply pass them between environment, streamgraph, etc...
### Default number of concurrent checkpoints
Previously, the maximum number of concurrent checkpoints was implicitly infinite.
I would suspect that most people will want to run the system such that only one checkpoint is ever concurrently active, so I set the default value for this to 1.
In some corner cases, it may be interesting to set that value higher.
### WIP for delay between checkpoints
This also contains some WIP to define a minimum time between checkpoint attempts. That flag would tell the system not only to not do more than one checkpoint concurrently, but to leave at least a certain time between completion of one checkpoint, and the triggering of the next. It basically defines a guaranteed time that is "work only" between checkpoints.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/StephanEwen/incubator-flink checkpoints_configure
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1408.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1408
----
commit e4d063867ac09cbec6666ecbca92816cfb6faf6c
Author: Stephan Ewen <se...@apache.org>
Date: 2015-11-19T17:03:55Z
[hotfix] Java-7-ify the ExecutionConfig class
commit 79def44a2e973c8b6817821d5d7342731f7e7aa2
Author: Stephan Ewen <se...@apache.org>
Date: 2015-11-19T18:05:47Z
[FLINK-3051] [streaming] Add mechanisms to control the maximum number of concurrent checkpoints
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3051] Control the maximum number of con...
Posted by gyfora <gi...@git.apache.org>.
Github user gyfora commented on the pull request:
https://github.com/apache/flink/pull/1408#issuecomment-159910663
Please give me an hour or so to look at this :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3051] Control the maximum number of con...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/1408#issuecomment-159932475
Missing for that is a bit of code in the checkpoint coordinator that marks the time when checkpoints become possible again and adds that delay to the time when the next checkpoint is scheduled.
I'd suggest to add that as a followup...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3051] Control the maximum number of con...
Posted by gyfora <gi...@git.apache.org>.
Github user gyfora commented on the pull request:
https://github.com/apache/flink/pull/1408#issuecomment-159915515
Looks good and the minimum delay between checkpoints would be an extremely useful feature. What's missing for that?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3051] Control the maximum number of con...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/1408#issuecomment-159868163
I think it fits best into the "Fault Tolerance" section of the Streaming Guide: https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/streaming_guide.html#fault-tolerance
If you think that section will grow to big, we could move it to a new page and link it from the streaming guide.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3051] Control the maximum number of con...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/1408#issuecomment-159866582
Where is the docs would be the best place for that?
- A new entry in the "Programing Guides" menu?
- Or a section in the streaming guide?
I would vote for the first
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3051] Control the maximum number of con...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/1408#issuecomment-159876995
Will do this in a separate pull request as follow up.
Any concerns about merging this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3051] Control the maximum number of con...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/1408
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3051] Control the maximum number of con...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/1408#issuecomment-159851509
Good addition! More control over our checkpointing is something people were asking me at talks.
Could you add a few sentences to the documentation about this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3051] Control the maximum number of con...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/1408#issuecomment-159872462
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3051] Control the maximum number of con...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/1408#issuecomment-159872110
Should we pull that into a separate document? It becomes quite large...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---