You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/08/19 09:49:31 UTC
[GitHub] [flink] Aitozi opened a new pull request #16894: [FLINK-23871][Runtime/Coordination]Dispatcher should handle finishing…
Aitozi opened a new pull request #16894:
URL: https://github.com/apache/flink/pull/16894
… job exception when recover
## What is the purpose of the change
This pr is to harden the dispatcher to solve the case that master crashed at job finishing phase.
## Brief change log
Add a new exception to represent job have reached Done status, and handle this exception in jobManagerResultFuture.
## Verifying this change
This change added tests and can be verified as follows:
org.apache.flink.runtime.dispatcher.DispatcherTest#testNonFatalErrorWhenRecoveredJobFinishingException
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16894: [FLINK-23871][Runtime/Coordination]Dispatcher should handle finishing…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16894:
URL: https://github.com/apache/flink/pull/16894#issuecomment-901800802
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22489",
"triggerID" : "a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a",
"triggerType" : "PUSH"
}, {
"hash" : "9c0242e54be44f5e9aeb3596a3df73c62d126518",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22559",
"triggerID" : "9c0242e54be44f5e9aeb3596a3df73c62d126518",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c0242e54be44f5e9aeb3596a3df73c62d126518 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22559)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Aitozi commented on a change in pull request #16894: [FLINK-23871][Runtime/Coordination]Dispatcher should handle finishing…
Posted by GitBox <gi...@apache.org>.
Aitozi commented on a change in pull request #16894:
URL: https://github.com/apache/flink/pull/16894#discussion_r692873178
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobFinishingException.java
##########
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.JobException;
+
+/** Exception indicating that a Flink job failed in finishing phase. */
+public class JobFinishingException extends JobException {
+ private static final long serialVersionUID = 5906282500322681417L;
+
+ public JobFinishingException(JobID jobId) {
+ super("The job (" + jobId + ") is in finishing phase.");
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16894: [FLINK-23871][Runtime/Coordination]Dispatcher should handle finishing…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16894:
URL: https://github.com/apache/flink/pull/16894#issuecomment-901800802
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22489",
"triggerID" : "a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a",
"triggerType" : "PUSH"
}, {
"hash" : "9c0242e54be44f5e9aeb3596a3df73c62d126518",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22559",
"triggerID" : "9c0242e54be44f5e9aeb3596a3df73c62d126518",
"triggerType" : "PUSH"
}, {
"hash" : "86dddf2aa5a5783f9ef2e4e17b6b1489fed50560",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22565",
"triggerID" : "86dddf2aa5a5783f9ef2e4e17b6b1489fed50560",
"triggerType" : "PUSH"
}, {
"hash" : "17f59b1f33ca1c7613c2c204580fadf42b57e8a2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22583",
"triggerID" : "17f59b1f33ca1c7613c2c204580fadf42b57e8a2",
"triggerType" : "PUSH"
}, {
"hash" : "1c80d779378fc92ccc8667b5e271caea5d5312b3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1c80d779378fc92ccc8667b5e271caea5d5312b3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 17f59b1f33ca1c7613c2c204580fadf42b57e8a2 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22583)
* 1c80d779378fc92ccc8667b5e271caea5d5312b3 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16894: [FLINK-23871][Runtime/Coordination]Dispatcher should handle finishing…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16894:
URL: https://github.com/apache/flink/pull/16894#issuecomment-901800802
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22489",
"triggerID" : "a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a",
"triggerType" : "PUSH"
}, {
"hash" : "9c0242e54be44f5e9aeb3596a3df73c62d126518",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c0242e54be44f5e9aeb3596a3df73c62d126518",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22489)
* 9c0242e54be44f5e9aeb3596a3df73c62d126518 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #16894: [FLINK-23871][Runtime/Coordination]Dispatcher should handle finishing…
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #16894:
URL: https://github.com/apache/flink/pull/16894#discussion_r692917816
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java
##########
@@ -446,9 +446,13 @@ private CleanupJobState handleJobManagerRunnerResult(
JobManagerRunnerResult jobManagerRunnerResult, ExecutionType executionType) {
if (jobManagerRunnerResult.isInitializationFailure()) {
if (executionType == ExecutionType.RECOVERY) {
- return jobManagerRunnerFailed(
- jobManagerRunnerResult.getExecutionGraphInfo().getJobId(),
- jobManagerRunnerResult.getInitializationFailure());
+ if (jobManagerRunnerResult.isJobFinishing()) {
+ return CleanupJobState.GLOBAL;
Review comment:
I would refrain from adding a new `JobStatus.UNKNOWN` value because `JobStatus` is public evolving and exposed at many places. I agree that `FAILED` is not optimal but we should state in the exception message that we could not properly deduce the final state. Moreover, I hope that this can be fixed with FLINK-11813 properly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16894: [FLINK-23871][Runtime/Coordination]Dispatcher should handle finishing…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16894:
URL: https://github.com/apache/flink/pull/16894#issuecomment-901800802
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22489",
"triggerID" : "a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22489)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16894: [FLINK-23871][Runtime/Coordination]Dispatcher should handle finishing…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16894:
URL: https://github.com/apache/flink/pull/16894#issuecomment-901800802
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22489",
"triggerID" : "a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a",
"triggerType" : "PUSH"
}, {
"hash" : "9c0242e54be44f5e9aeb3596a3df73c62d126518",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22559",
"triggerID" : "9c0242e54be44f5e9aeb3596a3df73c62d126518",
"triggerType" : "PUSH"
}, {
"hash" : "86dddf2aa5a5783f9ef2e4e17b6b1489fed50560",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22565",
"triggerID" : "86dddf2aa5a5783f9ef2e4e17b6b1489fed50560",
"triggerType" : "PUSH"
}, {
"hash" : "17f59b1f33ca1c7613c2c204580fadf42b57e8a2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22583",
"triggerID" : "17f59b1f33ca1c7613c2c204580fadf42b57e8a2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 86dddf2aa5a5783f9ef2e4e17b6b1489fed50560 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22565)
* 17f59b1f33ca1c7613c2c204580fadf42b57e8a2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22583)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16894: [FLINK-23871][Runtime/Coordination]Dispatcher should handle finishing…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16894:
URL: https://github.com/apache/flink/pull/16894#issuecomment-901800802
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22489",
"triggerID" : "a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a",
"triggerType" : "PUSH"
}, {
"hash" : "9c0242e54be44f5e9aeb3596a3df73c62d126518",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22559",
"triggerID" : "9c0242e54be44f5e9aeb3596a3df73c62d126518",
"triggerType" : "PUSH"
}, {
"hash" : "86dddf2aa5a5783f9ef2e4e17b6b1489fed50560",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22565",
"triggerID" : "86dddf2aa5a5783f9ef2e4e17b6b1489fed50560",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 86dddf2aa5a5783f9ef2e4e17b6b1489fed50560 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22565)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16894: [FLINK-23871][Runtime/Coordination]Dispatcher should handle finishing…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16894:
URL: https://github.com/apache/flink/pull/16894#issuecomment-901800802
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22489",
"triggerID" : "a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a",
"triggerType" : "PUSH"
}, {
"hash" : "9c0242e54be44f5e9aeb3596a3df73c62d126518",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22559",
"triggerID" : "9c0242e54be44f5e9aeb3596a3df73c62d126518",
"triggerType" : "PUSH"
}, {
"hash" : "86dddf2aa5a5783f9ef2e4e17b6b1489fed50560",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22565",
"triggerID" : "86dddf2aa5a5783f9ef2e4e17b6b1489fed50560",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c0242e54be44f5e9aeb3596a3df73c62d126518 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22559)
* 86dddf2aa5a5783f9ef2e4e17b6b1489fed50560 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22565)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16894: [FLINK-23871][Runtime/Coordination]Dispatcher should handle finishing…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16894:
URL: https://github.com/apache/flink/pull/16894#issuecomment-901800802
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22489",
"triggerID" : "a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a",
"triggerType" : "PUSH"
}, {
"hash" : "9c0242e54be44f5e9aeb3596a3df73c62d126518",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22559",
"triggerID" : "9c0242e54be44f5e9aeb3596a3df73c62d126518",
"triggerType" : "PUSH"
}, {
"hash" : "86dddf2aa5a5783f9ef2e4e17b6b1489fed50560",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22565",
"triggerID" : "86dddf2aa5a5783f9ef2e4e17b6b1489fed50560",
"triggerType" : "PUSH"
}, {
"hash" : "17f59b1f33ca1c7613c2c204580fadf42b57e8a2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22583",
"triggerID" : "17f59b1f33ca1c7613c2c204580fadf42b57e8a2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 17f59b1f33ca1c7613c2c204580fadf42b57e8a2 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22583)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16894: [FLINK-23871][Runtime/Coordination]Dispatcher should handle finishing…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16894:
URL: https://github.com/apache/flink/pull/16894#issuecomment-901800802
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22489",
"triggerID" : "a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a",
"triggerType" : "PUSH"
}, {
"hash" : "9c0242e54be44f5e9aeb3596a3df73c62d126518",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22559",
"triggerID" : "9c0242e54be44f5e9aeb3596a3df73c62d126518",
"triggerType" : "PUSH"
}, {
"hash" : "86dddf2aa5a5783f9ef2e4e17b6b1489fed50560",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22565",
"triggerID" : "86dddf2aa5a5783f9ef2e4e17b6b1489fed50560",
"triggerType" : "PUSH"
}, {
"hash" : "17f59b1f33ca1c7613c2c204580fadf42b57e8a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22583",
"triggerID" : "17f59b1f33ca1c7613c2c204580fadf42b57e8a2",
"triggerType" : "PUSH"
}, {
"hash" : "1c80d779378fc92ccc8667b5e271caea5d5312b3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22588",
"triggerID" : "1c80d779378fc92ccc8667b5e271caea5d5312b3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1c80d779378fc92ccc8667b5e271caea5d5312b3 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22588)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Aitozi commented on a change in pull request #16894: [FLINK-23871][Runtime/Coordination]Dispatcher should handle finishing…
Posted by GitBox <gi...@apache.org>.
Aitozi commented on a change in pull request #16894:
URL: https://github.com/apache/flink/pull/16894#discussion_r692876011
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/DefaultJobMasterServiceProcess.java
##########
@@ -214,15 +214,6 @@ public void jobReachedGloballyTerminalState(ExecutionGraphInfo executionGraphInf
resultFuture.complete(JobManagerRunnerResult.forSuccess(executionGraphInfo));
}
- @Override
Review comment:
Attention: I didn't see any product code use this method, so i delete it in this pr. Please help make sure for safe. @tillrohrmann
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16894: [FLINK-23871][Runtime/Coordination]Dispatcher should handle finishing…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16894:
URL: https://github.com/apache/flink/pull/16894#issuecomment-901800802
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22489",
"triggerID" : "a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a",
"triggerType" : "PUSH"
}, {
"hash" : "9c0242e54be44f5e9aeb3596a3df73c62d126518",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22559",
"triggerID" : "9c0242e54be44f5e9aeb3596a3df73c62d126518",
"triggerType" : "PUSH"
}, {
"hash" : "86dddf2aa5a5783f9ef2e4e17b6b1489fed50560",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22565",
"triggerID" : "86dddf2aa5a5783f9ef2e4e17b6b1489fed50560",
"triggerType" : "PUSH"
}, {
"hash" : "17f59b1f33ca1c7613c2c204580fadf42b57e8a2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "17f59b1f33ca1c7613c2c204580fadf42b57e8a2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 86dddf2aa5a5783f9ef2e4e17b6b1489fed50560 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22565)
* 17f59b1f33ca1c7613c2c204580fadf42b57e8a2 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #16894: [FLINK-23871][Runtime/Coordination]Dispatcher should handle finishing…
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #16894:
URL: https://github.com/apache/flink/pull/16894#discussion_r693040621
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobAlreadyDoneException.java
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.JobException;
+
+/** Exception indicating that a Flink job failed in finishing phase. */
+class JobAlreadyDoneException extends JobException {
+ private static final long serialVersionUID = 5906282500322681417L;
+
+ JobAlreadyDoneException(JobID jobId) {
+ super(
+ "The job ("
+ + jobId
+ + ") has already terminated, but we cannot tell the final "
+ + "state and also not serve the jobResult for some unexpected reason.");
Review comment:
```suggestion
+ "state and also not serve the job result. This can happen if Flink fails over shortly after finishing a job and before reporting the result. Please check the output of your job to see whether the execution was successful or not.");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16894: [FLINK-23871][Runtime/Coordination]Dispatcher should handle finishing…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16894:
URL: https://github.com/apache/flink/pull/16894#issuecomment-901800802
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22489",
"triggerID" : "a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a",
"triggerType" : "PUSH"
}, {
"hash" : "9c0242e54be44f5e9aeb3596a3df73c62d126518",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22559",
"triggerID" : "9c0242e54be44f5e9aeb3596a3df73c62d126518",
"triggerType" : "PUSH"
}, {
"hash" : "86dddf2aa5a5783f9ef2e4e17b6b1489fed50560",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "86dddf2aa5a5783f9ef2e4e17b6b1489fed50560",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c0242e54be44f5e9aeb3596a3df73c62d126518 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22559)
* 86dddf2aa5a5783f9ef2e4e17b6b1489fed50560 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #16894: [FLINK-23871][Runtime/Coordination]Dispatcher should handle finishing…
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #16894:
URL: https://github.com/apache/flink/pull/16894#discussion_r692914530
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobAlreadyDoneException.java
##########
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.JobException;
+
+/** Exception indicating that a Flink job failed in finishing phase. */
+public class JobAlreadyDoneException extends JobException {
Review comment:
Can this be package private?
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/JobMasterServiceLeadershipRunnerTest.java
##########
@@ -659,6 +660,31 @@ public void testJobMasterServiceProcessCreationFailureIsForwardedToResultFuture(
"Result future should be completed exceptionally."));
}
+ @Test
+ public void testJobAlreadyDone() throws Exception {
+ JobID jobID = new JobID();
+ try (JobManagerRunner jobManagerRunner =
+ newJobMasterServiceLeadershipRunnerBuilder()
+ .setJobMasterServiceProcessFactory(
+ TestingJobMasterServiceProcessFactory.newBuilder()
+ .setJobId(jobID)
+ .build())
+ .build()) {
+ runningJobsRegistry.setJobFinished(jobID);
+ jobManagerRunner.start();
+ leaderElectionService.isLeader(UUID.randomUUID());
+
+ final CompletableFuture<JobManagerRunnerResult> resultFuture =
+ jobManagerRunner.getResultFuture();
+
+ JobManagerRunnerResult result =
+ resultFuture.get(TESTING_TIMEOUT.toMilliseconds(), TimeUnit.MILLISECONDS);
Review comment:
```suggestion
resultFuture.get();
```
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobMasterServiceLeadershipRunner.java
##########
@@ -285,7 +285,12 @@ private ExecutionGraphInfo createExecutionGraphInfoWithJobStatus(JobStatus jobSt
}
private void jobAlreadyDone() {
- resultFuture.completeExceptionally(new JobNotFinishedException(getJobID()));
+ resultFuture.complete(
+ JobManagerRunnerResult.forSuccess(
+ new ExecutionGraphInfo(
+ jobMasterServiceProcessFactory.createArchivedExecutionGraph(
+ JobStatus.FAILED,
+ new JobAlreadyDoneException(getJobID())))));
Review comment:
I think if we communicate that way to the `Dispatcher` that a job has been already terminated, then we don't need `JobManagerRunnerResult.isJobAlreadyDone()`.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/DefaultJobMasterServiceProcess.java
##########
@@ -214,15 +214,6 @@ public void jobReachedGloballyTerminalState(ExecutionGraphInfo executionGraphInf
resultFuture.complete(JobManagerRunnerResult.forSuccess(executionGraphInfo));
}
- @Override
Review comment:
I also couldn't find another user of this method. Hence, I believe it is ok to remove it.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java
##########
@@ -446,9 +446,13 @@ private CleanupJobState handleJobManagerRunnerResult(
JobManagerRunnerResult jobManagerRunnerResult, ExecutionType executionType) {
if (jobManagerRunnerResult.isInitializationFailure()) {
if (executionType == ExecutionType.RECOVERY) {
- return jobManagerRunnerFailed(
- jobManagerRunnerResult.getExecutionGraphInfo().getJobId(),
- jobManagerRunnerResult.getInitializationFailure());
+ if (jobManagerRunnerResult.isJobAlreadyDone()) {
+ return jobReachedTerminalState(jobManagerRunnerResult.getExecutionGraphInfo());
Review comment:
With the current state of this PR, we probably don't need this branch.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobAlreadyDoneException.java
##########
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.JobException;
+
+/** Exception indicating that a Flink job failed in finishing phase. */
+public class JobAlreadyDoneException extends JobException {
+ private static final long serialVersionUID = 5906282500322681417L;
+
+ public JobAlreadyDoneException(JobID jobId) {
+ super("The job (" + jobId + ") has already terminated.");
Review comment:
I think we should add that we cannot tell the final state and also not serve the `JobResult`. Consequently, the user would have to check the result of the job manually (e.g. whether output files are there).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #16894: [FLINK-23871][Runtime/Coordination]Dispatcher should handle finishing…
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #16894:
URL: https://github.com/apache/flink/pull/16894#issuecomment-901800802
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Aitozi commented on a change in pull request #16894: [FLINK-23871][Runtime/Coordination]Dispatcher should handle finishing…
Posted by GitBox <gi...@apache.org>.
Aitozi commented on a change in pull request #16894:
URL: https://github.com/apache/flink/pull/16894#discussion_r692873064
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java
##########
@@ -446,9 +446,13 @@ private CleanupJobState handleJobManagerRunnerResult(
JobManagerRunnerResult jobManagerRunnerResult, ExecutionType executionType) {
if (jobManagerRunnerResult.isInitializationFailure()) {
if (executionType == ExecutionType.RECOVERY) {
- return jobManagerRunnerFailed(
- jobManagerRunnerResult.getExecutionGraphInfo().getJobId(),
- jobManagerRunnerResult.getInitializationFailure());
+ if (jobManagerRunnerResult.isJobFinishing()) {
+ return CleanupJobState.GLOBAL;
Review comment:
I have implemented this by complete the jobManagerRunner resultFuture with
```java
resultFuture.complete(
JobManagerRunnerResult.forSuccess(
new ExecutionGraphInfo(
jobMasterServiceProcessFactory.createArchivedExecutionGraph(
JobStatus.FAILED,
new JobAlreadyDoneException(getJobID())))));
```
But, I'm not sure is it right to set state to `JobStatus.FAILED`. We do not know the status of job at the time of recovery of an already done job. I think we can do these things like :
- Add a special `JobStatus.UNKNOWN` to handle this case
- Or when the FLIP for [FLINK-11813](https://issues.apache.org/jira/browse/FLINK-11813) have been introduced , we can extract the job result directly from `JobResultStore`.
Looking forward to your opinion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #16894: [FLINK-23871][Runtime/Coordination]Dispatcher should handle finishing…
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #16894:
URL: https://github.com/apache/flink/pull/16894#issuecomment-901775127
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a (Thu Aug 19 09:52:22 UTC 2021)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
* **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-23871).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work.
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16894: [FLINK-23871][Runtime/Coordination]Dispatcher should handle finishing…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16894:
URL: https://github.com/apache/flink/pull/16894#issuecomment-901800802
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22489",
"triggerID" : "a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a",
"triggerType" : "PUSH"
}, {
"hash" : "9c0242e54be44f5e9aeb3596a3df73c62d126518",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22559",
"triggerID" : "9c0242e54be44f5e9aeb3596a3df73c62d126518",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22489)
* 9c0242e54be44f5e9aeb3596a3df73c62d126518 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22559)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Aitozi commented on pull request #16894: [FLINK-23871][Runtime/Coordination]Dispatcher should handle finishing…
Posted by GitBox <gi...@apache.org>.
Aitozi commented on pull request #16894:
URL: https://github.com/apache/flink/pull/16894#issuecomment-902756472
Thanks for your review @tillrohrmann , I have addressed your comments , please take a look again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16894: [FLINK-23871][Runtime/Coordination]Dispatcher should handle finishing…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16894:
URL: https://github.com/apache/flink/pull/16894#issuecomment-901800802
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22489",
"triggerID" : "a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22489)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann closed pull request #16894: [FLINK-23871][Runtime/Coordination]Dispatcher should handle finishing…
Posted by GitBox <gi...@apache.org>.
tillrohrmann closed pull request #16894:
URL: https://github.com/apache/flink/pull/16894
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16894: [FLINK-23871][Runtime/Coordination]Dispatcher should handle finishing…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16894:
URL: https://github.com/apache/flink/pull/16894#issuecomment-901800802
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22489",
"triggerID" : "a7ac6694734a9fbc7e2b436b472dd9f7cb8f831a",
"triggerType" : "PUSH"
}, {
"hash" : "9c0242e54be44f5e9aeb3596a3df73c62d126518",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22559",
"triggerID" : "9c0242e54be44f5e9aeb3596a3df73c62d126518",
"triggerType" : "PUSH"
}, {
"hash" : "86dddf2aa5a5783f9ef2e4e17b6b1489fed50560",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22565",
"triggerID" : "86dddf2aa5a5783f9ef2e4e17b6b1489fed50560",
"triggerType" : "PUSH"
}, {
"hash" : "17f59b1f33ca1c7613c2c204580fadf42b57e8a2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22583",
"triggerID" : "17f59b1f33ca1c7613c2c204580fadf42b57e8a2",
"triggerType" : "PUSH"
}, {
"hash" : "1c80d779378fc92ccc8667b5e271caea5d5312b3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22588",
"triggerID" : "1c80d779378fc92ccc8667b5e271caea5d5312b3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 17f59b1f33ca1c7613c2c204580fadf42b57e8a2 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22583)
* 1c80d779378fc92ccc8667b5e271caea5d5312b3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22588)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Aitozi commented on a change in pull request #16894: [FLINK-23871][Runtime/Coordination]Dispatcher should handle finishing…
Posted by GitBox <gi...@apache.org>.
Aitozi commented on a change in pull request #16894:
URL: https://github.com/apache/flink/pull/16894#discussion_r692873731
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/dispatcher/DispatcherTest.java
##########
@@ -578,6 +579,45 @@ public void testFatalErrorIfRecoveredJobsCannotBeStarted() throws Exception {
fatalErrorHandler.clearError();
}
+ /** Tests that the {@link Dispatcher} do not fail by {@link JobFinishingException}. */
+ @Test
+ public void testNonFatalErrorWhenRecoveredJobFinishingException() throws Exception {
Review comment:
I add another test case in `org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunnerTest#testJobAlreadyDone` I think this can cover the current case, what do you think ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #16894: [FLINK-23871][Runtime/Coordination]Dispatcher should handle finishing…
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #16894:
URL: https://github.com/apache/flink/pull/16894#discussion_r692176396
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/dispatcher/DispatcherTest.java
##########
@@ -578,6 +579,45 @@ public void testFatalErrorIfRecoveredJobsCannotBeStarted() throws Exception {
fatalErrorHandler.clearError();
}
+ /** Tests that the {@link Dispatcher} do not fail by {@link JobFinishingException}. */
Review comment:
```suggestion
/** Tests that the {@link Dispatcher} does not fail by {@link JobFinishingException}. */
```
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobFinishingException.java
##########
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.JobException;
+
+/** Exception indicating that a Flink job failed in finishing phase. */
+public class JobFinishingException extends JobException {
+ private static final long serialVersionUID = 5906282500322681417L;
+
+ public JobFinishingException(JobID jobId) {
+ super("The job (" + jobId + ") is in finishing phase.");
Review comment:
Maybe we should say that the job has already terminated and we cannot say what its final state was and also cannot return its `JobResult`.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobFinishingException.java
##########
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.JobException;
+
+/** Exception indicating that a Flink job failed in finishing phase. */
+public class JobFinishingException extends JobException {
Review comment:
Maybe
```suggestion
public class JobAlreadyDoneException extends JobException {
```
is a slightly better name.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java
##########
@@ -446,9 +446,13 @@ private CleanupJobState handleJobManagerRunnerResult(
JobManagerRunnerResult jobManagerRunnerResult, ExecutionType executionType) {
if (jobManagerRunnerResult.isInitializationFailure()) {
if (executionType == ExecutionType.RECOVERY) {
- return jobManagerRunnerFailed(
- jobManagerRunnerResult.getExecutionGraphInfo().getJobId(),
- jobManagerRunnerResult.getInitializationFailure());
+ if (jobManagerRunnerResult.isJobFinishing()) {
+ return CleanupJobState.GLOBAL;
Review comment:
I think we should call `jobReachedTerminalState` so that we create an `ArchivedExecutionGraph`.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/dispatcher/DispatcherTest.java
##########
@@ -578,6 +579,45 @@ public void testFatalErrorIfRecoveredJobsCannotBeStarted() throws Exception {
fatalErrorHandler.clearError();
}
+ /** Tests that the {@link Dispatcher} do not fail by {@link JobFinishingException}. */
+ @Test
+ public void testNonFatalErrorWhenRecoveredJobFinishingException() throws Exception {
Review comment:
Maybe we should add another test similar to `DispatcherTest.testJobDataAreCleanedUpInCorrectOrder` that ensures that we clean up things properly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Aitozi commented on a change in pull request #16894: [FLINK-23871][Runtime/Coordination]Dispatcher should handle finishing…
Posted by GitBox <gi...@apache.org>.
Aitozi commented on a change in pull request #16894:
URL: https://github.com/apache/flink/pull/16894#discussion_r693277592
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobAlreadyDoneException.java
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.JobException;
+
+/** Exception indicating that a Flink job failed in finishing phase. */
+class JobAlreadyDoneException extends JobException {
+ private static final long serialVersionUID = 5906282500322681417L;
+
+ JobAlreadyDoneException(JobID jobId) {
+ super(
+ "The job ("
+ + jobId
+ + ") has already terminated, but we cannot tell the final "
+ + "state and also not serve the jobResult for some unexpected reason.");
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on pull request #16894: [FLINK-23871][Runtime/Coordination]Dispatcher should handle finishing…
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on pull request #16894:
URL: https://github.com/apache/flink/pull/16894#issuecomment-903087614
Great, thanks @Aitozi. Merging this PR now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org