You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/03/05 08:55:08 UTC
[GitHub] [flink] tillrohrmann opened a new pull request #15093: [FLINK-21401] Consolidate JobGraph generation to properly instantiate streaming and batch JobGraphs
tillrohrmann opened a new pull request #15093:
URL: https://github.com/apache/flink/pull/15093
This PR consolidates the `JobGraph` generation into as few places as possible. Moreover it introduces explicit factory methods for streaming and batch `JobGraphs` (`JobType.STREAMING` and `JobType.BATCH`) in order to better classify which tests can be run with the `AdaptiveScheduler` and which strictly require the `DefaultScheduler`.
This PR also introduces a `JobGraphBuilder` which can be used to make the `JobGraph` an immutable data structure. This builder is then used for the more complex `JobGraph` creations for tests.
Last but not least, this PR adapts some failing tests (mainly `DispatcherTest`) to also work with the `AdaptiveScheduler`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15093: [FLINK-21401] Consolidate JobGraph generation to properly instantiate streaming and batch JobGraphs
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #15093:
URL: https://github.com/apache/flink/pull/15093#issuecomment-791289763
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ff8af80ef146dd30a890eaab9cab9bc8d416d953",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ff8af80ef146dd30a890eaab9cab9bc8d416d953",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ff8af80ef146dd30a890eaab9cab9bc8d416d953 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15093: [FLINK-21401] Consolidate JobGraph generation to properly instantiate streaming and batch JobGraphs
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15093:
URL: https://github.com/apache/flink/pull/15093#issuecomment-791289763
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ff8af80ef146dd30a890eaab9cab9bc8d416d953",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14182",
"triggerID" : "ff8af80ef146dd30a890eaab9cab9bc8d416d953",
"triggerType" : "PUSH"
}, {
"hash" : "80919f42cf3909b73b4a9f3f4259fe1c8bb2d598",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14203",
"triggerID" : "80919f42cf3909b73b4a9f3f4259fe1c8bb2d598",
"triggerType" : "PUSH"
}, {
"hash" : "c1139e525265f8887c1fa0e05631a0345676b120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14208",
"triggerID" : "c1139e525265f8887c1fa0e05631a0345676b120",
"triggerType" : "PUSH"
}, {
"hash" : "364a8efe51a80007a6609bf43b890a7287b7f247",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14210",
"triggerID" : "364a8efe51a80007a6609bf43b890a7287b7f247",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 364a8efe51a80007a6609bf43b890a7287b7f247 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14210)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #15093: [FLINK-21401] Consolidate JobGraph generation to properly instantiate streaming and batch JobGraphs
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #15093:
URL: https://github.com/apache/flink/pull/15093#discussion_r588350564
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/dispatcher/DispatcherTest.java
##########
@@ -991,13 +1027,11 @@ public TestingJobManagerRunner createJobManagerRunner(
final BlockingJobVertex blockingJobVertex = new BlockingJobVertex("testVertex");
blockingJobVertex.setInvokableClass(NoOpInvokable.class);
return Tuple2.of(
- new JobGraph(TEST_JOB_ID, "blockingTestJob", blockingJobVertex), blockingJobVertex);
- }
-
- private JobGraph createFailingJobGraph(Exception failureCause) {
- final FailingJobVertex jobVertex = new FailingJobVertex("Failing JobVertex", failureCause);
- jobVertex.setInvokableClass(NoOpInvokable.class);
- return new JobGraph(jobGraph.getJobID(), "Failing JobGraph", jobVertex);
+ JobGraphBuilder.newStreamingJobGraphBuilder()
+ .setJobId(jobId)
+ .addJobVertex(blockingJobVertex)
+ .build(),
+ blockingJobVertex);
}
private static class FailingJobVertex extends JobVertex {
Review comment:
I also added a couple of tests which relied on scheduler details. I think before we did not have the necessity to separate tests thoroughly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #15093: [FLINK-21401] Consolidate JobGraph generation to properly instantiate streaming and batch JobGraphs
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #15093:
URL: https://github.com/apache/flink/pull/15093#discussion_r588229663
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/entrypoint/component/FileJobGraphRetrieverTest.java
##########
@@ -67,7 +68,11 @@ public static void init() throws IOException {
final JobVertex source = new JobVertex("source");
final JobVertex target = new JobVertex("target");
- final JobGraph jobGraph = new JobGraph(new JobID(), "test", source, target);
+ final JobGraph jobGraph =
+ JobGraphBuilder.newStreamingJobGraphBuilder()
+ .addJobVertices(Arrays.asList(source, target))
+ .addClasspaths(Collections.singletonList(jarFileInJobGraph.toUri().toURL()))
+ .build();
jobGraph.setClasspaths(Collections.singletonList(jarFileInJobGraph.toUri().toURL()));
Review comment:
```suggestion
```
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/dispatcher/DispatcherTest.java
##########
@@ -392,46 +418,47 @@ public void testNonBlockingJobSubmission() throws Exception {
assertEquals(jobID, multiDetails.getJobs().iterator().next().getJobId());
// submission has succeeded, let the initialization finish.
- blockingJobGraph.f1.unblock();
+ latch.trigger();
// ensure job is running
CommonTestUtils.waitUntilCondition(
- () ->
- dispatcherGateway.requestJobStatus(jobGraph.getJobID(), TIMEOUT).get()
- == JobStatus.RUNNING,
+ () -> dispatcherGateway.requestJobStatus(jobID, TIMEOUT).get() == JobStatus.RUNNING,
Deadline.fromNow(Duration.of(10, ChronoUnit.SECONDS)),
5L);
}
@Test(timeout = 5_000L)
public void testInvalidCallDuringInitialization() throws Exception {
+ final OneShotLatch latch = new OneShotLatch();
dispatcher =
createAndStartDispatcher(
heartbeatServices,
haServices,
- new ExpectedJobIdJobManagerRunnerFactory(
- TEST_JOB_ID, createdJobManagerRunnerLatch));
- jobMasterLeaderElectionService.isLeader(UUID.randomUUID());
+ new BlockingJobManagerRunnerFactory(latch::await));
+
DispatcherGateway dispatcherGateway = dispatcher.getSelfGateway(DispatcherGateway.class);
- Tuple2<JobGraph, BlockingJobVertex> blockingJobGraph = getBlockingJobGraphAndVertex();
- JobID jid = blockingJobGraph.f0.getJobID();
+ final JobGraph emptyJobGraph =
+ JobGraphBuilder.newStreamingJobGraphBuilder().setJobId(jobId).build();
- dispatcherGateway.submitJob(blockingJobGraph.f0, TIMEOUT).get();
+ dispatcherGateway.submitJob(emptyJobGraph, TIMEOUT).get();
assertThat(
- dispatcherGateway.requestJobStatus(jid, TIMEOUT).get(), is(JobStatus.INITIALIZING));
+ dispatcherGateway.requestJobStatus(jobId, TIMEOUT).get(),
+ is(JobStatus.INITIALIZING));
// this call is supposed to fail
try {
- dispatcherGateway.triggerSavepoint(jid, "file:///tmp/savepoint", false, TIMEOUT).get();
+ dispatcherGateway
+ .triggerSavepoint(jobId, "file:///tmp/savepoint", false, TIMEOUT)
+ .get();
fail("Previous statement should have failed");
} catch (ExecutionException t) {
assertTrue(t.getCause() instanceof UnavailableDispatcherOperationException);
}
// submission has succeeded, let the initialization finish.
- blockingJobGraph.f1.unblock();
+ latch.trigger();
Review comment:
This is much nicer than the Tuple2 with the vertex! Thanks for the cleanup.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/JobMasterTest.java
##########
@@ -260,100 +245,6 @@ public static void teardownClass() {
}
}
- @Test
- public void testDeclineCheckpointInvocationWithUserException() throws Exception {
Review comment:
found https://github.com/apache/flink/pull/14662/files#r585721398 ;)
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/dispatcher/DispatcherTest.java
##########
@@ -991,13 +1027,11 @@ public TestingJobManagerRunner createJobManagerRunner(
final BlockingJobVertex blockingJobVertex = new BlockingJobVertex("testVertex");
blockingJobVertex.setInvokableClass(NoOpInvokable.class);
return Tuple2.of(
- new JobGraph(TEST_JOB_ID, "blockingTestJob", blockingJobVertex), blockingJobVertex);
- }
-
- private JobGraph createFailingJobGraph(Exception failureCause) {
- final FailingJobVertex jobVertex = new FailingJobVertex("Failing JobVertex", failureCause);
- jobVertex.setInvokableClass(NoOpInvokable.class);
- return new JobGraph(jobGraph.getJobID(), "Failing JobGraph", jobVertex);
+ JobGraphBuilder.newStreamingJobGraphBuilder()
+ .setJobId(jobId)
+ .addJobVertex(blockingJobVertex)
+ .build(),
+ blockingJobVertex);
}
private static class FailingJobVertex extends JobVertex {
Review comment:
Thanks a lot for the various cleanups! I took some notes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann closed pull request #15093: [FLINK-21401] Consolidate JobGraph generation to properly instantiate streaming and batch JobGraphs
Posted by GitBox <gi...@apache.org>.
tillrohrmann closed pull request #15093:
URL: https://github.com/apache/flink/pull/15093
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #15093: [FLINK-21401] Consolidate JobGraph generation to properly instantiate streaming and batch JobGraphs
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #15093:
URL: https://github.com/apache/flink/pull/15093#discussion_r588347940
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/entrypoint/component/FileJobGraphRetrieverTest.java
##########
@@ -67,7 +68,11 @@ public static void init() throws IOException {
final JobVertex source = new JobVertex("source");
final JobVertex target = new JobVertex("target");
- final JobGraph jobGraph = new JobGraph(new JobID(), "test", source, target);
+ final JobGraph jobGraph =
+ JobGraphBuilder.newStreamingJobGraphBuilder()
+ .addJobVertices(Arrays.asList(source, target))
+ .addClasspaths(Collections.singletonList(jarFileInJobGraph.toUri().toURL()))
+ .build();
jobGraph.setClasspaths(Collections.singletonList(jarFileInJobGraph.toUri().toURL()));
Review comment:
Good catch. Will remove it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15093: [FLINK-21401] Consolidate JobGraph generation to properly instantiate streaming and batch JobGraphs
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #15093:
URL: https://github.com/apache/flink/pull/15093#issuecomment-791277178
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit ff8af80ef146dd30a890eaab9cab9bc8d416d953 (Fri Mar 05 09:05:42 UTC 2021)
**Warnings:**
* **1 pom.xml files were touched**: Check for build and licensing issues.
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15093: [FLINK-21401] Consolidate JobGraph generation to properly instantiate streaming and batch JobGraphs
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15093:
URL: https://github.com/apache/flink/pull/15093#issuecomment-791289763
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ff8af80ef146dd30a890eaab9cab9bc8d416d953",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14182",
"triggerID" : "ff8af80ef146dd30a890eaab9cab9bc8d416d953",
"triggerType" : "PUSH"
}, {
"hash" : "80919f42cf3909b73b4a9f3f4259fe1c8bb2d598",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14203",
"triggerID" : "80919f42cf3909b73b4a9f3f4259fe1c8bb2d598",
"triggerType" : "PUSH"
}, {
"hash" : "c1139e525265f8887c1fa0e05631a0345676b120",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14208",
"triggerID" : "c1139e525265f8887c1fa0e05631a0345676b120",
"triggerType" : "PUSH"
}, {
"hash" : "364a8efe51a80007a6609bf43b890a7287b7f247",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "364a8efe51a80007a6609bf43b890a7287b7f247",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c1139e525265f8887c1fa0e05631a0345676b120 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14208)
* 364a8efe51a80007a6609bf43b890a7287b7f247 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15093: [FLINK-21401] Consolidate JobGraph generation to properly instantiate streaming and batch JobGraphs
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15093:
URL: https://github.com/apache/flink/pull/15093#issuecomment-791289763
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ff8af80ef146dd30a890eaab9cab9bc8d416d953",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14182",
"triggerID" : "ff8af80ef146dd30a890eaab9cab9bc8d416d953",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ff8af80ef146dd30a890eaab9cab9bc8d416d953 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14182)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15093: [FLINK-21401] Consolidate JobGraph generation to properly instantiate streaming and batch JobGraphs
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15093:
URL: https://github.com/apache/flink/pull/15093#issuecomment-791289763
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ff8af80ef146dd30a890eaab9cab9bc8d416d953",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14182",
"triggerID" : "ff8af80ef146dd30a890eaab9cab9bc8d416d953",
"triggerType" : "PUSH"
}, {
"hash" : "80919f42cf3909b73b4a9f3f4259fe1c8bb2d598",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14203",
"triggerID" : "80919f42cf3909b73b4a9f3f4259fe1c8bb2d598",
"triggerType" : "PUSH"
}, {
"hash" : "c1139e525265f8887c1fa0e05631a0345676b120",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14208",
"triggerID" : "c1139e525265f8887c1fa0e05631a0345676b120",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 80919f42cf3909b73b4a9f3f4259fe1c8bb2d598 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14203)
* c1139e525265f8887c1fa0e05631a0345676b120 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14208)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #15093: [FLINK-21401] Consolidate JobGraph generation to properly instantiate streaming and batch JobGraphs
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #15093:
URL: https://github.com/apache/flink/pull/15093#discussion_r588350992
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/dispatcher/DispatcherTest.java
##########
@@ -392,46 +418,47 @@ public void testNonBlockingJobSubmission() throws Exception {
assertEquals(jobID, multiDetails.getJobs().iterator().next().getJobId());
// submission has succeeded, let the initialization finish.
- blockingJobGraph.f1.unblock();
+ latch.trigger();
// ensure job is running
CommonTestUtils.waitUntilCondition(
- () ->
- dispatcherGateway.requestJobStatus(jobGraph.getJobID(), TIMEOUT).get()
- == JobStatus.RUNNING,
+ () -> dispatcherGateway.requestJobStatus(jobID, TIMEOUT).get() == JobStatus.RUNNING,
Deadline.fromNow(Duration.of(10, ChronoUnit.SECONDS)),
5L);
}
@Test(timeout = 5_000L)
public void testInvalidCallDuringInitialization() throws Exception {
+ final OneShotLatch latch = new OneShotLatch();
dispatcher =
createAndStartDispatcher(
heartbeatServices,
haServices,
- new ExpectedJobIdJobManagerRunnerFactory(
- TEST_JOB_ID, createdJobManagerRunnerLatch));
- jobMasterLeaderElectionService.isLeader(UUID.randomUUID());
+ new BlockingJobManagerRunnerFactory(latch::await));
+
DispatcherGateway dispatcherGateway = dispatcher.getSelfGateway(DispatcherGateway.class);
- Tuple2<JobGraph, BlockingJobVertex> blockingJobGraph = getBlockingJobGraphAndVertex();
- JobID jid = blockingJobGraph.f0.getJobID();
+ final JobGraph emptyJobGraph =
+ JobGraphBuilder.newStreamingJobGraphBuilder().setJobId(jobId).build();
- dispatcherGateway.submitJob(blockingJobGraph.f0, TIMEOUT).get();
+ dispatcherGateway.submitJob(emptyJobGraph, TIMEOUT).get();
assertThat(
- dispatcherGateway.requestJobStatus(jid, TIMEOUT).get(), is(JobStatus.INITIALIZING));
+ dispatcherGateway.requestJobStatus(jobId, TIMEOUT).get(),
+ is(JobStatus.INITIALIZING));
// this call is supposed to fail
try {
- dispatcherGateway.triggerSavepoint(jid, "file:///tmp/savepoint", false, TIMEOUT).get();
+ dispatcherGateway
+ .triggerSavepoint(jobId, "file:///tmp/savepoint", false, TIMEOUT)
+ .get();
fail("Previous statement should have failed");
} catch (ExecutionException t) {
assertTrue(t.getCause() instanceof UnavailableDispatcherOperationException);
}
// submission has succeeded, let the initialization finish.
- blockingJobGraph.f1.unblock();
+ latch.trigger();
Review comment:
Yeah, `Tuple2` is actually quite ugly. We should try to never use it if possible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15093: [FLINK-21401] Consolidate JobGraph generation to properly instantiate streaming and batch JobGraphs
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15093:
URL: https://github.com/apache/flink/pull/15093#issuecomment-791289763
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ff8af80ef146dd30a890eaab9cab9bc8d416d953",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14182",
"triggerID" : "ff8af80ef146dd30a890eaab9cab9bc8d416d953",
"triggerType" : "PUSH"
}, {
"hash" : "80919f42cf3909b73b4a9f3f4259fe1c8bb2d598",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14203",
"triggerID" : "80919f42cf3909b73b4a9f3f4259fe1c8bb2d598",
"triggerType" : "PUSH"
}, {
"hash" : "c1139e525265f8887c1fa0e05631a0345676b120",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c1139e525265f8887c1fa0e05631a0345676b120",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ff8af80ef146dd30a890eaab9cab9bc8d416d953 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14182)
* 80919f42cf3909b73b4a9f3f4259fe1c8bb2d598 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14203)
* c1139e525265f8887c1fa0e05631a0345676b120 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15093: [FLINK-21401] Consolidate JobGraph generation to properly instantiate streaming and batch JobGraphs
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15093:
URL: https://github.com/apache/flink/pull/15093#issuecomment-791289763
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ff8af80ef146dd30a890eaab9cab9bc8d416d953",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14182",
"triggerID" : "ff8af80ef146dd30a890eaab9cab9bc8d416d953",
"triggerType" : "PUSH"
}, {
"hash" : "80919f42cf3909b73b4a9f3f4259fe1c8bb2d598",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14203",
"triggerID" : "80919f42cf3909b73b4a9f3f4259fe1c8bb2d598",
"triggerType" : "PUSH"
}, {
"hash" : "c1139e525265f8887c1fa0e05631a0345676b120",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14208",
"triggerID" : "c1139e525265f8887c1fa0e05631a0345676b120",
"triggerType" : "PUSH"
}, {
"hash" : "364a8efe51a80007a6609bf43b890a7287b7f247",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14210",
"triggerID" : "364a8efe51a80007a6609bf43b890a7287b7f247",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c1139e525265f8887c1fa0e05631a0345676b120 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14208)
* 364a8efe51a80007a6609bf43b890a7287b7f247 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14210)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #15093: [FLINK-21401] Consolidate JobGraph generation to properly instantiate streaming and batch JobGraphs
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #15093:
URL: https://github.com/apache/flink/pull/15093#discussion_r588347385
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/JobMasterTest.java
##########
@@ -260,100 +245,6 @@ public static void teardownClass() {
}
}
- @Test
- public void testDeclineCheckpointInvocationWithUserException() throws Exception {
Review comment:
I should have added a reference to this discussion. Thanks for digging it up.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15093: [FLINK-21401] Consolidate JobGraph generation to properly instantiate streaming and batch JobGraphs
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15093:
URL: https://github.com/apache/flink/pull/15093#issuecomment-791289763
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ff8af80ef146dd30a890eaab9cab9bc8d416d953",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14182",
"triggerID" : "ff8af80ef146dd30a890eaab9cab9bc8d416d953",
"triggerType" : "PUSH"
}, {
"hash" : "80919f42cf3909b73b4a9f3f4259fe1c8bb2d598",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "80919f42cf3909b73b4a9f3f4259fe1c8bb2d598",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ff8af80ef146dd30a890eaab9cab9bc8d416d953 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14182)
* 80919f42cf3909b73b4a9f3f4259fe1c8bb2d598 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15093: [FLINK-21401] Consolidate JobGraph generation to properly instantiate streaming and batch JobGraphs
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15093:
URL: https://github.com/apache/flink/pull/15093#issuecomment-791289763
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ff8af80ef146dd30a890eaab9cab9bc8d416d953",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14182",
"triggerID" : "ff8af80ef146dd30a890eaab9cab9bc8d416d953",
"triggerType" : "PUSH"
}, {
"hash" : "80919f42cf3909b73b4a9f3f4259fe1c8bb2d598",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14203",
"triggerID" : "80919f42cf3909b73b4a9f3f4259fe1c8bb2d598",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ff8af80ef146dd30a890eaab9cab9bc8d416d953 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14182)
* 80919f42cf3909b73b4a9f3f4259fe1c8bb2d598 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14203)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15093: [FLINK-21401] Consolidate JobGraph generation to properly instantiate streaming and batch JobGraphs
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15093:
URL: https://github.com/apache/flink/pull/15093#issuecomment-791289763
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ff8af80ef146dd30a890eaab9cab9bc8d416d953",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14182",
"triggerID" : "ff8af80ef146dd30a890eaab9cab9bc8d416d953",
"triggerType" : "PUSH"
}, {
"hash" : "80919f42cf3909b73b4a9f3f4259fe1c8bb2d598",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14203",
"triggerID" : "80919f42cf3909b73b4a9f3f4259fe1c8bb2d598",
"triggerType" : "PUSH"
}, {
"hash" : "c1139e525265f8887c1fa0e05631a0345676b120",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14208",
"triggerID" : "c1139e525265f8887c1fa0e05631a0345676b120",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ff8af80ef146dd30a890eaab9cab9bc8d416d953 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14182)
* 80919f42cf3909b73b4a9f3f4259fe1c8bb2d598 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14203)
* c1139e525265f8887c1fa0e05631a0345676b120 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14208)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on pull request #15093: [FLINK-21401] Consolidate JobGraph generation to properly instantiate streaming and batch JobGraphs
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on pull request #15093:
URL: https://github.com/apache/flink/pull/15093#issuecomment-791467318
Thanks for the review @rmetzger. I've addressed your comments. I will merge this PR once AZP gives green light.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org