You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by yanghua <gi...@git.apache.org> on 2018/05/29 08:04:11 UTC
[GitHub] flink pull request #6092: [FLINK-9352] In Standalone checkpoint recover mode...
GitHub user yanghua opened a pull request:
https://github.com/apache/flink/pull/6092
[FLINK-9352] In Standalone checkpoint recover mode many jobs with same checkpoint interval cause IO pressure
## What is the purpose of the change
*This pull request fixed a problem : In Standalone checkpoint recover mode many jobs with same checkpoint interval cause IO pressure*
## Brief change log
- *Replace the scheduler's initial delay time from baseInterval to a random num between min pause and base interval*
## Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / **no**)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / **not documented**)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yanghua/flink FLINK-9352
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/6092.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6092
----
commit 1eb432833bf2dd23187194500b6e1c6523f30605
Author: yanghua <ya...@...>
Date: 2018-05-29T07:59:48Z
[FLINK-9352] In Standalone checkpoint recover mode many jobs with same checkpoint interval cause IO pressure
----
---
[GitHub] flink pull request #6092: [FLINK-9352] In Standalone checkpoint recover mode...
Posted by yuqi1129 <gi...@git.apache.org>.
Github user yuqi1129 commented on a diff in the pull request:
https://github.com/apache/flink/pull/6092#discussion_r191340849
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java ---
@@ -1173,9 +1179,10 @@ public void startCheckpointScheduler() {
stopCheckpointScheduler();
periodicScheduling = true;
+ long initialDelay = schedulerInitialDelayGenerator.nextLong(
+ minPauseBetweenCheckpointsNanos / 1_000_000, baseInterval);
--- End diff --
Any place to check the value `minPauseBetweenCheckpointsNanos / 1_000_000` is less or equal than baseInterval?
---
[GitHub] flink issue #6092: [FLINK-9352] In Standalone checkpoint recover mode many j...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/6092
Can we do `ThreadLocalRandom.current().nextLong(minPauseBetweenCheckpoints, baseInterval + 1L)`?
---
[GitHub] flink pull request #6092: [FLINK-9352] In Standalone checkpoint recover mode...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/6092#discussion_r200065877
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java ---
@@ -1173,9 +1179,10 @@ public void startCheckpointScheduler() {
stopCheckpointScheduler();
periodicScheduling = true;
+ long initialDelay = schedulerInitialDelayGenerator.nextLong(
+ minPauseBetweenCheckpointsNanos / 1_000_000, baseInterval);
--- End diff --
Could we replace `schedulerInitialDelayGenerator` with `long initialDelay = ThreadLocalRandom.current().nextLong(minPauseBetweenCheckpointsNanos / 1_000_000, baseInterval);`? That way we would not have to use `RandomUtils`.
---
[GitHub] flink issue #6092: [FLINK-9352] In Standalone checkpoint recover mode many j...
Posted by yanghua <gi...@git.apache.org>.
Github user yanghua commented on the issue:
https://github.com/apache/flink/pull/6092
@pnowojski @dawidwys can you review this?
---
[GitHub] flink issue #6092: [FLINK-9352] In Standalone checkpoint recover mode many j...
Posted by yanghua <gi...@git.apache.org>.
Github user yanghua commented on the issue:
https://github.com/apache/flink/pull/6092
cc @tillrohrmann @StephanEwen
---
[GitHub] flink pull request #6092: [FLINK-9352] In Standalone checkpoint recover mode...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/6092
---
[GitHub] flink issue #6092: [FLINK-9352] In Standalone checkpoint recover mode many j...
Posted by yanghua <gi...@git.apache.org>.
Github user yanghua commented on the issue:
https://github.com/apache/flink/pull/6092
cc @zentol @kl0u
---
[GitHub] flink issue #6092: [FLINK-9352] In Standalone checkpoint recover mode many j...
Posted by yanghua <gi...@git.apache.org>.
Github user yanghua commented on the issue:
https://github.com/apache/flink/pull/6092
hi @tillrohrmann there is a little difference between `RandomUtils.nextLong ` and `ThreadLocalRandom.current().nextLong` , that is the former allows **startInclusive = endExclusive** but the latter does not allow. However, in CheckpointCoordinator exists :
```
if (baseInterval < minPauseBetweenCheckpoints) {
baseInterval = minPauseBetweenCheckpoints;
}
```
so Travis-CI test failed.
---
[GitHub] flink issue #6092: [FLINK-9352] In Standalone checkpoint recover mode many j...
Posted by yanghua <gi...@git.apache.org>.
Github user yanghua commented on the issue:
https://github.com/apache/flink/pull/6092
cc @tillrohrmann @zentol
---
[GitHub] flink issue #6092: [FLINK-9352] In Standalone checkpoint recover mode many j...
Posted by yanghua <gi...@git.apache.org>.
Github user yanghua commented on the issue:
https://github.com/apache/flink/pull/6092
hi @tillrohrmann can you review this PR? this issue happens in our inner standalone Flink cluster. When recover many jobs, will cause performance problem.
---
[GitHub] flink pull request #6092: [FLINK-9352] In Standalone checkpoint recover mode...
Posted by yanghua <gi...@git.apache.org>.
Github user yanghua commented on a diff in the pull request:
https://github.com/apache/flink/pull/6092#discussion_r191341746
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java ---
@@ -1173,9 +1179,10 @@ public void startCheckpointScheduler() {
stopCheckpointScheduler();
periodicScheduling = true;
+ long initialDelay = schedulerInitialDelayGenerator.nextLong(
+ minPauseBetweenCheckpointsNanos / 1_000_000, baseInterval);
--- End diff --
@yuqi1129 please see the constructor's [code segment](https://github.com/yanghua/flink/blob/1eb432833bf2dd23187194500b6e1c6523f30605/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java#L218)
---
[GitHub] flink issue #6092: [FLINK-9352] In Standalone checkpoint recover mode many j...
Posted by yanghua <gi...@git.apache.org>.
Github user yanghua commented on the issue:
https://github.com/apache/flink/pull/6092
cc @zentol if you have time, can you review this?
---
[GitHub] flink issue #6092: [FLINK-9352] In Standalone checkpoint recover mode many j...
Posted by yanghua <gi...@git.apache.org>.
Github user yanghua commented on the issue:
https://github.com/apache/flink/pull/6092
@tillrohrmann has refactored, please review again. the reason of the travis error is not because of this PR.
---