You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by tillrohrmann <gi...@git.apache.org> on 2018/02/08 17:25:10 UTC
[GitHub] flink pull request #5437: [FLINK-8614] [flip6] Activate Flip-6 mode per defa...
GitHub user tillrohrmann opened a pull request:
https://github.com/apache/flink/pull/5437
[FLINK-8614] [flip6] Activate Flip-6 mode per default
## What is the purpose of the change
This commit enables the Flip-6 mode per default. Additionally, it disables
some of the Yarn tests which no longer apply to Flip-6 (tests which wait for
a number of started TM container without a job submission).
This PR is based on #5436, #5432 and #5388.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tillrohrmann/flink flip6Default
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5437.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5437
----
commit 56ed240dd2c3e8f2775208ae349f137acb53bd34
Author: Till Rohrmann <tr...@...>
Date: 2018-02-07T16:00:40Z
[FLINK-8613] [flip6] [yarn] Return excess containers
Upon notification of newly allocated containers, the YarnResourceManager
will only accept as many containers as there are pending container requests.
All excess containers will be returned.
commit f9d9fd0d4c22defd4f839b7078596e6e440d26c1
Author: Till Rohrmann <tr...@...>
Date: 2018-01-30T14:15:49Z
[FLINK-8529] [flip6] Let Yarn entry points use APPLICATION_MASTER_PORT
Let all Yarn entry points use the YarnConfigOptions.APPLICATION_MASTER_PORT option
to specify the valid port range for the common RpcService.
commit acc942e5237bcf8cc482c564c061d741befbff53
Author: Till Rohrmann <tr...@...>
Date: 2018-02-06T15:47:28Z
[FLINK-8609] [flip6] Enable Flip-6 job mode in CliFrontend
This commit allows to deploy detached job mode clusters via the
CliFrontend. In order to do that, it first extracts the JobGraph
from the PackagedProgram and then uses the ClusterDescriptor to
deploy the job mode cluster.
commit 52e8e74a6c977c68be0bdb62017bcbac0bb5ed2f
Author: Till Rohrmann <tr...@...>
Date: 2018-02-08T13:34:54Z
[FLINK-8608] [flip6] Implement MiniDispatcher for job mode
The MiniDispatcher is responsible for submitting the single job with which
a job mode cluster is started. Once the job has completed and if the cluster
has been started in detached mode, the MiniDispatcher will terminate.
In order to reduce code duplication, the MiniDispatcher is a sub class of the
Dispatcher which is started with a single job submitted job graph store.
commit d0ab48e090f142bf5287528b6d0198b702e169f0
Author: Till Rohrmann <tr...@...>
Date: 2018-02-07T17:21:06Z
[hotfix] [yarn] Write number of slots to configuration
commit 1d38657ab86b6f7cff7b9b6c878ee4573aedff71
Author: Till Rohrmann <tr...@...>
Date: 2018-02-07T17:58:32Z
[hotfix] [yarn] Remove unnecessary TaskManager configuration generation
commit 76b6ce0ca587377b14fbd6eda48f52d4da13f09e
Author: Till Rohrmann <tr...@...>
Date: 2018-02-08T09:27:27Z
[hotfix] Only log retrying exception on debug in RetryingRegistration
commit 00de9bf451f5cb56e96938a7cf0dd9c5e38ea595
Author: Till Rohrmann <tr...@...>
Date: 2018-01-30T08:22:03Z
[FLINK-8614] [flip6] Activate Flip-6 mode per default
This commit enables the Flip-6 mode per default. Additionally, it disables
some of the Yarn tests which no longer apply to Flip-6 (tests which wait for
a number of started TM container without a job submission).
----
---
[GitHub] flink issue #5437: [FLINK-8614] [flip6] Activate Flip-6 mode per default
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/5437
Thanks for the review @GJL. Merging once Travis gives green light.
---
[GitHub] flink pull request #5437: [FLINK-8614] [flip6] Activate Flip-6 mode per defa...
Posted by GJL <gi...@git.apache.org>.
Github user GJL commented on a diff in the pull request:
https://github.com/apache/flink/pull/5437#discussion_r168459950
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java ---
@@ -402,6 +403,24 @@ public void start() throws Exception {
}
}
+ @Override
+ public CompletableFuture<JobResult> requestJobResult(JobID jobId, Time timeout) {
+ final JobManagerRunner jobManagerRunner = jobManagerRunners.get(jobId);
+
+ if (jobManagerRunner == null) {
+ final ArchivedExecutionGraph archivedExecutionGraph = archivedExecutionGraphStore.get(jobId);
+
+ if (archivedExecutionGraph == null) {
+ return FutureUtils.completedExceptionally(new FlinkJobNotFoundException(jobId));
+ } else {
+ return CompletableFuture.completedFuture(JobResult.createFrom(archivedExecutionGraph));
+ }
+ } else {
+ return jobManagerRunner.getResultFuture().thenApply(
--- End diff --
nit: can be rewritten as `return jobManagerRunner.getResultFuture().thenApply(JobResult::createFrom);`
---
[GitHub] flink pull request #5437: [FLINK-8614] [flip6] Activate Flip-6 mode per defa...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/5437#discussion_r168525466
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/MiniDispatcher.java ---
@@ -100,6 +101,18 @@ public MiniDispatcher(
return acknowledgeCompletableFuture;
}
+ @Override
+ public CompletableFuture<JobResult> requestJobResult(JobID jobId, Time timeout) {
+ final CompletableFuture<JobResult> jobResultFuture = super.requestJobResult(jobId, timeout);
+
+ if (executionMode == ClusterEntrypoint.ExecutionMode.NORMAL) {
+ // terminate the MiniDispatcher once we served the first JobResult successfully
+ jobResultFuture.thenRun(this::shutDown);
--- End diff --
I think you're right. We should also shut down the `MiniDispatcher` in case of a failure.
---
[GitHub] flink pull request #5437: [FLINK-8614] [flip6] Activate Flip-6 mode per defa...
Posted by GJL <gi...@git.apache.org>.
Github user GJL commented on a diff in the pull request:
https://github.com/apache/flink/pull/5437#discussion_r168461190
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/MiniDispatcher.java ---
@@ -100,6 +101,18 @@ public MiniDispatcher(
return acknowledgeCompletableFuture;
}
+ @Override
+ public CompletableFuture<JobResult> requestJobResult(JobID jobId, Time timeout) {
+ final CompletableFuture<JobResult> jobResultFuture = super.requestJobResult(jobId, timeout);
+
+ if (executionMode == ClusterEntrypoint.ExecutionMode.NORMAL) {
+ // terminate the MiniDispatcher once we served the first JobResult successfully
+ jobResultFuture.thenRun(this::shutDown);
--- End diff --
Don't know if it is a problem but if the future is completed exceptionally, `shutDown` will not be invoked.
public static void main(String[] args) {
FutureUtils.completedExceptionally(new RuntimeException()).thenRun(() -> System.out.println("not printed"));
}
---
[GitHub] flink pull request #5437: [FLINK-8614] [flip6] Activate Flip-6 mode per defa...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/5437#discussion_r168525199
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java ---
@@ -402,6 +403,24 @@ public void start() throws Exception {
}
}
+ @Override
+ public CompletableFuture<JobResult> requestJobResult(JobID jobId, Time timeout) {
+ final JobManagerRunner jobManagerRunner = jobManagerRunners.get(jobId);
+
+ if (jobManagerRunner == null) {
+ final ArchivedExecutionGraph archivedExecutionGraph = archivedExecutionGraphStore.get(jobId);
+
+ if (archivedExecutionGraph == null) {
+ return FutureUtils.completedExceptionally(new FlinkJobNotFoundException(jobId));
+ } else {
+ return CompletableFuture.completedFuture(JobResult.createFrom(archivedExecutionGraph));
+ }
+ } else {
+ return jobManagerRunner.getResultFuture().thenApply(
--- End diff --
Good catch. Will change it.
---
[GitHub] flink pull request #5437: [FLINK-8614] [flip6] Activate Flip-6 mode per defa...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/5437
---