You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/07/29 11:51:38 UTC
[GitHub] [flink] zlzhang0122 opened a new pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
zlzhang0122 opened a new pull request #16637:
URL: https://github.com/apache/flink/pull/16637
## What is the purpose of the change
This pull request make some kinds of failures be checked against CheckpointFailureManager and be deciding whether these errors should be just logged or checked against the number of tolerable failures and maybe fail the job.
That way we can let users perceived some unrecoverable error faster.
## Brief change log
- *add IO_EXCEPTION to identify the disk error and increase the number of failure count*
- *add some other failure reason to increase the failure count*
- *add a unit test case for IO_EXCEPTION scene*
## Verifying this change
- *Added IO_EXCEPTION scene to CheckpointCoordinatorTest and modify the number of CheckpointFailureManagerTest *
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-889063245
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit d7928fd1f51e6cc4296df21a9bf8521988b599f2 (Thu Jul 29 12:08:57 UTC 2021)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zlzhang0122 commented on a change in pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
zlzhang0122 commented on a change in pull request #16637:
URL: https://github.com/apache/flink/pull/16637#discussion_r684122805
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java
##########
@@ -862,11 +862,19 @@ private void onTriggerSuccess() {
*/
private void onTriggerFailure(
CheckpointTriggerRequest onCompletionPromise, Throwable throwable) {
- final CheckpointException checkpointException =
- getCheckpointException(
- CheckpointFailureReason.TRIGGER_CHECKPOINT_FAILURE, throwable);
- onCompletionPromise.completeExceptionally(checkpointException);
- onTriggerFailure((PendingCheckpoint) null, checkpointException);
+ Throwable rootThrowable = ExceptionUtils.stripCompletionException(throwable);
+ if (rootThrowable instanceof IOException) {
+ final CheckpointException checkpointException =
+ getCheckpointException(CheckpointFailureReason.IO_EXCEPTION, rootThrowable);
+ onCompletionPromise.completeExceptionally(checkpointException);
+ onTriggerFailure((PendingCheckpoint) null, throwable);
+ } else {
+ final CheckpointException checkpointException =
+ getCheckpointException(
+ CheckpointFailureReason.TRIGGER_CHECKPOINT_FAILURE, throwable);
+ onCompletionPromise.completeExceptionally(checkpointException);
+ onTriggerFailure((PendingCheckpoint) null, checkpointException);
Review comment:
u r right and I have also made a change here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #16637:
URL: https://github.com/apache/flink/pull/16637#discussion_r685896989
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointFailureManager.java
##########
@@ -136,6 +135,8 @@ public void checkFailureCounter(CheckpointException exception, long checkpointId
// ignore
break;
+ case EXCEPTION:
Review comment:
Renaming should be completely fine as its not exposed or does it appear in the logs?
Do you plan to adjust the message? That could cause some issues if folks are parsing the logs, so maybe we can skip that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] akalash commented on a change in pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
akalash commented on a change in pull request #16637:
URL: https://github.com/apache/flink/pull/16637#discussion_r683558429
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java
##########
@@ -862,11 +862,19 @@ private void onTriggerSuccess() {
*/
private void onTriggerFailure(
CheckpointTriggerRequest onCompletionPromise, Throwable throwable) {
- final CheckpointException checkpointException =
- getCheckpointException(
- CheckpointFailureReason.TRIGGER_CHECKPOINT_FAILURE, throwable);
- onCompletionPromise.completeExceptionally(checkpointException);
- onTriggerFailure((PendingCheckpoint) null, checkpointException);
+ Throwable rootThrowable = ExceptionUtils.stripCompletionException(throwable);
Review comment:
It is actually not a root throwable. stripCompletionException only unwraps CompletionException. But what if IOException would be hidden deeper than one level? According to the task description, I suppose we interesting in any IOException at any cause level. So if I understand everything right `ExceptionUtils.findThrowable(IOException).isPresent()` is more correct construction here.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointFailureManager.java
##########
@@ -136,6 +135,8 @@ public void checkFailureCounter(CheckpointException exception, long checkpointId
// ignore
break;
+ case EXCEPTION:
Review comment:
What is the reason for moving EXCEPTION out of ignore? I don't see any clue of this either in the ticket or in the tests.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java
##########
@@ -862,11 +862,19 @@ private void onTriggerSuccess() {
*/
private void onTriggerFailure(
CheckpointTriggerRequest onCompletionPromise, Throwable throwable) {
- final CheckpointException checkpointException =
- getCheckpointException(
- CheckpointFailureReason.TRIGGER_CHECKPOINT_FAILURE, throwable);
- onCompletionPromise.completeExceptionally(checkpointException);
- onTriggerFailure((PendingCheckpoint) null, checkpointException);
+ Throwable rootThrowable = ExceptionUtils.stripCompletionException(throwable);
Review comment:
Also, why do you use unwrapped throwable only if it is IOException but if it is not you use the initial one? I think logic should be consistent here. If we know that this throwable can be wrapped by CompletionException we should unwrap it in both cases(Perhaps, it makes sense to move unwrapping into `getCheckpointException`).
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/CheckpointFailureManagerTest.java
##########
@@ -96,7 +96,7 @@ public void testTotalCountValue() {
}
// CHECKPOINT_DECLINED, CHECKPOINT_EXPIRED and CHECKPOINT_ASYNC_EXCEPTION
- assertEquals(3, callback.getInvokeCounter());
+ assertEquals(5, callback.getInvokeCounter());
Review comment:
Inconsistency between number and comment.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java
##########
@@ -862,11 +862,19 @@ private void onTriggerSuccess() {
*/
private void onTriggerFailure(
CheckpointTriggerRequest onCompletionPromise, Throwable throwable) {
- final CheckpointException checkpointException =
- getCheckpointException(
- CheckpointFailureReason.TRIGGER_CHECKPOINT_FAILURE, throwable);
- onCompletionPromise.completeExceptionally(checkpointException);
- onTriggerFailure((PendingCheckpoint) null, checkpointException);
+ Throwable rootThrowable = ExceptionUtils.stripCompletionException(throwable);
+ if (rootThrowable instanceof IOException) {
+ final CheckpointException checkpointException =
+ getCheckpointException(CheckpointFailureReason.IO_EXCEPTION, rootThrowable);
+ onCompletionPromise.completeExceptionally(checkpointException);
+ onTriggerFailure((PendingCheckpoint) null, throwable);
+ } else {
+ final CheckpointException checkpointException =
+ getCheckpointException(
+ CheckpointFailureReason.TRIGGER_CHECKPOINT_FAILURE, throwable);
+ onCompletionPromise.completeExceptionally(checkpointException);
+ onTriggerFailure((PendingCheckpoint) null, checkpointException);
Review comment:
I didn't get why `if` and `else` blocks are different(one passes throwable to onTriggerFailure while another passes checkpointException). As I understand you should only choose CheckpointFailureReason but another code should be the same. Or Did I understand something wrong?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointFailureReason.java
##########
@@ -81,6 +81,8 @@
FINALIZE_CHECKPOINT_FAILURE(false, "Failure to finalize checkpoint."),
+ IO_EXCEPTION(false, "Trigger checkpoint failure."),
Review comment:
Please fix the description. right now it is just copy-paste of `TRIGGER_CHECKPOINT_FAILURE`.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinatorTest.java
##########
@@ -624,6 +625,36 @@ public void testTriggerAndDeclineCheckpointThenFailureManagerThrowsException()
}
}
+ @Test
+ public void testIOExceptionCheckpointExceedsTolerableFailureNumber() throws Exception {
Review comment:
This test checks that the task is failed when it receives the new reason type but I don't see tests that check that your new code inside of `onTriggerFailure` and `onTriggerFailure` work.
As I can see, at least `testPeriodicSchedulingWithInactiveTasks`, `testTriggerCheckpointAfterCancel` and `testCheckpointAbortsIfTriggerTasksAreFinished`(and I sure many more) trigger somehow the exception. Perhaps, you can take a look how these tests works and you can write something similar.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java
##########
@@ -892,9 +900,12 @@ private void onTriggerFailure(@Nullable PendingCheckpoint checkpoint, Throwable
job,
numUnsuccessful,
throwable);
- final CheckpointException cause =
- getCheckpointException(
- CheckpointFailureReason.TRIGGER_CHECKPOINT_FAILURE, throwable);
+ CheckpointFailureReason defaultReason =
+ CheckpointFailureReason.TRIGGER_CHECKPOINT_FAILURE;
+ if (throwable instanceof IOException) {
Review comment:
The same here. Why are you so sure that IOException would not be wrapped by other exceptions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zlzhang0122 commented on a change in pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
zlzhang0122 commented on a change in pull request #16637:
URL: https://github.com/apache/flink/pull/16637#discussion_r684115825
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java
##########
@@ -862,11 +862,19 @@ private void onTriggerSuccess() {
*/
private void onTriggerFailure(
CheckpointTriggerRequest onCompletionPromise, Throwable throwable) {
- final CheckpointException checkpointException =
- getCheckpointException(
- CheckpointFailureReason.TRIGGER_CHECKPOINT_FAILURE, throwable);
- onCompletionPromise.completeExceptionally(checkpointException);
- onTriggerFailure((PendingCheckpoint) null, checkpointException);
+ Throwable rootThrowable = ExceptionUtils.stripCompletionException(throwable);
Review comment:
Sure, I have push a new commit and make the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] akalash commented on a change in pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
akalash commented on a change in pull request #16637:
URL: https://github.com/apache/flink/pull/16637#discussion_r685136777
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorCoordinatorCheckpoints.java
##########
@@ -121,14 +123,17 @@ private static void acknowledgeAllCoordinators(
final Throwable error =
checkpoint.isDisposed() ? checkpoint.getFailureCause() : null;
+ CheckpointFailureReason reason = CheckpointFailureReason.TRIGGER_CHECKPOINT_FAILURE;
Review comment:
I do not really follow the idea of this method(the original one). As I understand method `acknowledgeAllCoordinators` can be called only after all taskManagers finish the checkpoint it means that we can check `checkpoint.getFailureCause()` before the loop and throw the exception if needed. I also don't understand why if we already have the `CheckpointException`(from checkpoint.getFailureCause()) with specific `ReasonType` we always wrap it by one more `CheckpointException` with the constant type `TRIGGER_CHECKPOINT_FAILURE`.
Perhaps, @zlzhang0122 or @pnowojski can explain it?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointFailureManager.java
##########
@@ -136,6 +135,8 @@ public void checkFailureCounter(CheckpointException exception, long checkpointId
// ignore
break;
+ case EXCEPTION:
Review comment:
Hm, yes, I see. Actually, I don't see the difference between EXCEPTION and IO_EXCEPTION right now. I personally don't think that it makes sense to keep both of them, perhaps it is better to use IO_EXCEPTION in both places. But perhaps, it was some specific scenario why the EXCEPTION was introduced.
@pnowojski what do you think. Do we really need to keep both IO_EXCEPTION(new one) and EXCEPTION? Why is EXCEPTION ignored in FailureManager right now?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorCoordinatorCheckpoints.java
##########
@@ -121,14 +123,17 @@ private static void acknowledgeAllCoordinators(
final Throwable error =
checkpoint.isDisposed() ? checkpoint.getFailureCause() : null;
+ CheckpointFailureReason reason = CheckpointFailureReason.TRIGGER_CHECKPOINT_FAILURE;
if (error != null) {
- throw new CheckpointException(
- errorMessage,
- CheckpointFailureReason.TRIGGER_CHECKPOINT_FAILURE,
- error);
+ Throwable rootThrowable = ExceptionUtils.stripCompletionException(error);
Review comment:
Again, I don't sure that we should have `stripCompletionException` here, perhaps, it should be `findThrowable`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] akalash commented on a change in pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
akalash commented on a change in pull request #16637:
URL: https://github.com/apache/flink/pull/16637#discussion_r686592632
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointFailureManager.java
##########
@@ -136,6 +135,8 @@ public void checkFailureCounter(CheckpointException exception, long checkpointId
// ignore
break;
+ case EXCEPTION:
Review comment:
Yes, we kind of want to replace only for IO-related failures but the thing is EXCEPTION is using only at one place and only for IO exception. So when we replace EXCEPTION with IO_EXCEPTION, there will be no other usages for EXCEPTION.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-889099584
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"triggerType" : "PUSH"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21186",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21188",
"triggerID" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"triggerType" : "PUSH"
}, {
"hash" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21680",
"triggerID" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"triggerType" : "PUSH"
}, {
"hash" : "eb8c1b59b1fe328cfefc51c1204ced5202ad4448",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21702",
"triggerID" : "eb8c1b59b1fe328cfefc51c1204ced5202ad4448",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21680)
* eb8c1b59b1fe328cfefc51c1204ced5202ad4448 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21702)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] akalash commented on a change in pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
akalash commented on a change in pull request #16637:
URL: https://github.com/apache/flink/pull/16637#discussion_r686157796
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointFailureManager.java
##########
@@ -136,6 +135,8 @@ public void checkFailureCounter(CheckpointException exception, long checkpointId
// ignore
break;
+ case EXCEPTION:
Review comment:
It, of course, appears in the logs somewhere(at least in stack trace but I don't see explicit printing of `CheckpointFailureReason`) but if we take into account that this PR split existing one `CheckpointFailureReason` into two which in any case changes the previous behavior I still think that it will be ok to rename it. And yes, the message will be slightly different.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zlzhang0122 commented on a change in pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
zlzhang0122 commented on a change in pull request #16637:
URL: https://github.com/apache/flink/pull/16637#discussion_r684066191
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointFailureManager.java
##########
@@ -136,6 +135,8 @@ public void checkFailureCounter(CheckpointException exception, long checkpointId
// ignore
break;
+ case EXCEPTION:
Review comment:
CheckpointFailureReason.EXCEPTION is also a kind of IOException may produced by StreamTaskNetworkInput.prepareSnapshot(), to keep the consistency, I think maybe we should moved it out of ignore. What do u think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-889099584
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"triggerType" : "PUSH"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21186",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21188",
"triggerID" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"triggerType" : "PUSH"
}, {
"hash" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21680",
"triggerID" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"triggerType" : "PUSH"
}, {
"hash" : "eb8c1b59b1fe328cfefc51c1204ced5202ad4448",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21702",
"triggerID" : "eb8c1b59b1fe328cfefc51c1204ced5202ad4448",
"triggerType" : "PUSH"
}, {
"hash" : "afe05192b1bd26789bd0b7da3a773e63bc2215a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21817",
"triggerID" : "afe05192b1bd26789bd0b7da3a773e63bc2215a1",
"triggerType" : "PUSH"
}, {
"hash" : "60352a88cfe01d3f9ad9d297753c3f2f941fcc9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21871",
"triggerID" : "60352a88cfe01d3f9ad9d297753c3f2f941fcc9a",
"triggerType" : "PUSH"
}, {
"hash" : "a9af148bb5760c640fe768b0476a53202b7e6792",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21900",
"triggerID" : "a9af148bb5760c640fe768b0476a53202b7e6792",
"triggerType" : "PUSH"
}, {
"hash" : "59d43e6a1bfc216cbfa8ca43224576312adb57d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "59d43e6a1bfc216cbfa8ca43224576312adb57d2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a9af148bb5760c640fe768b0476a53202b7e6792 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21900)
* 59d43e6a1bfc216cbfa8ca43224576312adb57d2 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] akalash commented on a change in pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
akalash commented on a change in pull request #16637:
URL: https://github.com/apache/flink/pull/16637#discussion_r687554740
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointFailureManager.java
##########
@@ -127,7 +127,6 @@ public void checkFailureCounter(CheckpointException exception, long checkpointId
case CHECKPOINT_DECLINED_SUBSUMED:
case CHECKPOINT_DECLINED_INPUT_END_OF_STREAM:
- case EXCEPTION:
Review comment:
It is ok to have two separate commits under one jira ticket. You can check the git log and see that it is the pretty usual stuff. It makes sense to do so when you have tightly connected changes but they are still logically different and for example, they require different commit names. So it is exactly your case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-889099584
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"triggerType" : "PUSH"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21186",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21188",
"triggerID" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"triggerType" : "PUSH"
}, {
"hash" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21680",
"triggerID" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"triggerType" : "PUSH"
}, {
"hash" : "eb8c1b59b1fe328cfefc51c1204ced5202ad4448",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21702",
"triggerID" : "eb8c1b59b1fe328cfefc51c1204ced5202ad4448",
"triggerType" : "PUSH"
}, {
"hash" : "afe05192b1bd26789bd0b7da3a773e63bc2215a1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21817",
"triggerID" : "afe05192b1bd26789bd0b7da3a773e63bc2215a1",
"triggerType" : "PUSH"
}, {
"hash" : "60352a88cfe01d3f9ad9d297753c3f2f941fcc9a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "60352a88cfe01d3f9ad9d297753c3f2f941fcc9a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afe05192b1bd26789bd0b7da3a773e63bc2215a1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21817)
* 60352a88cfe01d3f9ad9d297753c3f2f941fcc9a UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zlzhang0122 commented on a change in pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
zlzhang0122 commented on a change in pull request #16637:
URL: https://github.com/apache/flink/pull/16637#discussion_r684122805
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java
##########
@@ -862,11 +862,19 @@ private void onTriggerSuccess() {
*/
private void onTriggerFailure(
CheckpointTriggerRequest onCompletionPromise, Throwable throwable) {
- final CheckpointException checkpointException =
- getCheckpointException(
- CheckpointFailureReason.TRIGGER_CHECKPOINT_FAILURE, throwable);
- onCompletionPromise.completeExceptionally(checkpointException);
- onTriggerFailure((PendingCheckpoint) null, checkpointException);
+ Throwable rootThrowable = ExceptionUtils.stripCompletionException(throwable);
+ if (rootThrowable instanceof IOException) {
+ final CheckpointException checkpointException =
+ getCheckpointException(CheckpointFailureReason.IO_EXCEPTION, rootThrowable);
+ onCompletionPromise.completeExceptionally(checkpointException);
+ onTriggerFailure((PendingCheckpoint) null, throwable);
+ } else {
+ final CheckpointException checkpointException =
+ getCheckpointException(
+ CheckpointFailureReason.TRIGGER_CHECKPOINT_FAILURE, throwable);
+ onCompletionPromise.completeExceptionally(checkpointException);
+ onTriggerFailure((PendingCheckpoint) null, checkpointException);
Review comment:
You are right and I have also made a change here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-889099584
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"triggerType" : "PUSH"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21186",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21188",
"triggerID" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"triggerType" : "PUSH"
}, {
"hash" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21680",
"triggerID" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"triggerType" : "PUSH"
}, {
"hash" : "eb8c1b59b1fe328cfefc51c1204ced5202ad4448",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21702",
"triggerID" : "eb8c1b59b1fe328cfefc51c1204ced5202ad4448",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eb8c1b59b1fe328cfefc51c1204ced5202ad4448 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21702)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zlzhang0122 commented on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
zlzhang0122 commented on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-898121984
@akalash yes, I have fixed it. The unit test has passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-889099584
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"triggerType" : "PUSH"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21186",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21188",
"triggerID" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d7928fd1f51e6cc4296df21a9bf8521988b599f2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21186) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163)
* 08da26fefae5d23c49518fcb6716813410c8d5fb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21188)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] pnowojski commented on a change in pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
pnowojski commented on a change in pull request #16637:
URL: https://github.com/apache/flink/pull/16637#discussion_r688922304
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinatorTest.java
##########
@@ -626,6 +627,194 @@ public void testTriggerAndDeclineCheckpointThenFailureManagerThrowsException()
}
}
+ @Test
+ public void testIOExceptionCheckpointExceedsTolerableFailureNumber() throws Exception {
+ // create some mock Execution vertices that receive the checkpoint trigger messages
+ ExecutionGraph graph =
+ new CheckpointCoordinatorTestingUtils.CheckpointExecutionGraphBuilder()
+ .addJobVertex(new JobVertexID())
+ .addJobVertex(new JobVertexID())
+ .build();
+
+ final String errorMsg = "Exceeded checkpoint failure tolerance number!";
+ CheckpointFailureManager checkpointFailureManager = getCheckpointFailureManager(errorMsg);
+ CheckpointCoordinator checkpointCoordinator =
+ getCheckpointCoordinator(graph, checkpointFailureManager);
+
+ try {
+ checkpointCoordinator.triggerCheckpoint(false);
+ manuallyTriggeredScheduledExecutor.triggerAll();
+
+ checkpointCoordinator.abortPendingCheckpoints(new CheckpointException(IO_EXCEPTION));
+
+ fail("Test failed.");
+ } catch (Exception e) {
+ // expected
+ assertTrue(e instanceof RuntimeException);
+ assertEquals(errorMsg, e.getMessage());
+ } finally {
+ checkpointCoordinator.shutdown();
+ }
+ }
+
+ @Test
+ public void testIOExceptionForPeriodicSchedulingWithInactiveTasks() {
+ TestingIOExceptionCheckpointIDCounter idCounter =
+ new TestingIOExceptionCheckpointIDCounter();
+
+ try {
+ JobVertexID jobVertexID1 = new JobVertexID();
+
+ ExecutionGraph graph =
+ new CheckpointCoordinatorTestingUtils.CheckpointExecutionGraphBuilder()
+ .addJobVertex(jobVertexID1)
+ .setTransitToRunning(false)
+ .build();
+
+ ExecutionVertex vertex1 = graph.getJobVertex(jobVertexID1).getTaskVertices()[0];
+
+ CheckpointCoordinatorConfiguration chkConfig =
+ new CheckpointCoordinatorConfiguration
+ .CheckpointCoordinatorConfigurationBuilder()
+ .setCheckpointInterval(10) // periodic interval is 10 ms
+ .setCheckpointTimeout(200000) // timeout is very long (200 s)
+ .setMinPauseBetweenCheckpoints(0) // no extra delay
+ .setMaxConcurrentCheckpoints(2) // max two concurrent checkpoints
+ .build();
+ CheckpointCoordinator checkpointCoordinator =
+ new CheckpointCoordinatorBuilder()
+ .setExecutionGraph(graph)
+ .setCheckpointCoordinatorConfiguration(chkConfig)
+ .setCompletedCheckpointStore(new StandaloneCompletedCheckpointStore(2))
+ .setTimer(manuallyTriggeredScheduledExecutor)
+ .setCheckpointIDCounter(idCounter)
+ .build();
+ idCounter.setOwner(checkpointCoordinator);
+
+ checkpointCoordinator.startCheckpointScheduler();
+
+ manuallyTriggeredScheduledExecutor.triggerPeriodicScheduledTasks();
+ manuallyTriggeredScheduledExecutor.triggerAll();
+ // no checkpoint should have started so far
+ assertEquals(0, checkpointCoordinator.getNumberOfPendingCheckpoints());
+
+ // now move the state to RUNNING
+ vertex1.getCurrentExecutionAttempt().transitionState(ExecutionState.RUNNING);
+
+ final CompletableFuture<CompletedCheckpoint> onCompletionPromise =
+ checkpointCoordinator.triggerCheckpoint(
+ CheckpointProperties.forCheckpoint(
+ CheckpointRetentionPolicy.NEVER_RETAIN_AFTER_TERMINATION),
+ null,
+ true);
+ manuallyTriggeredScheduledExecutor.triggerAll();
+
+ try {
+ onCompletionPromise.get();
+ fail("should not trigger periodic checkpoint after IOException occurred.");
+ } catch (Exception e) {
+ final Optional<CheckpointException> checkpointExceptionOptional =
+ ExceptionUtils.findThrowable(e, CheckpointException.class);
+ assertTrue(checkpointExceptionOptional.isPresent());
+ assertEquals(
+ IO_EXCEPTION,
+ checkpointExceptionOptional.get().getCheckpointFailureReason());
+ }
+ } catch (Exception e) {
+ e.printStackTrace();
+ fail(e.getMessage());
+ }
+ }
+
+ /** Tests that do not trigger checkpoint when IOException occurred. */
+ @Test
+ public void testTriggerCheckpointAfterIOException() throws Exception {
Review comment:
This test duplicates code with `testTriggerCheckpointAfterCancel`
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinatorTest.java
##########
@@ -626,6 +627,194 @@ public void testTriggerAndDeclineCheckpointThenFailureManagerThrowsException()
}
}
+ @Test
+ public void testIOExceptionCheckpointExceedsTolerableFailureNumber() throws Exception {
+ // create some mock Execution vertices that receive the checkpoint trigger messages
+ ExecutionGraph graph =
+ new CheckpointCoordinatorTestingUtils.CheckpointExecutionGraphBuilder()
+ .addJobVertex(new JobVertexID())
+ .addJobVertex(new JobVertexID())
+ .build();
+
+ final String errorMsg = "Exceeded checkpoint failure tolerance number!";
+ CheckpointFailureManager checkpointFailureManager = getCheckpointFailureManager(errorMsg);
+ CheckpointCoordinator checkpointCoordinator =
+ getCheckpointCoordinator(graph, checkpointFailureManager);
+
+ try {
+ checkpointCoordinator.triggerCheckpoint(false);
+ manuallyTriggeredScheduledExecutor.triggerAll();
+
+ checkpointCoordinator.abortPendingCheckpoints(new CheckpointException(IO_EXCEPTION));
+
+ fail("Test failed.");
+ } catch (Exception e) {
+ // expected
+ assertTrue(e instanceof RuntimeException);
+ assertEquals(errorMsg, e.getMessage());
+ } finally {
+ checkpointCoordinator.shutdown();
+ }
+ }
+
+ @Test
+ public void testIOExceptionForPeriodicSchedulingWithInactiveTasks() {
Review comment:
This test duplicates a lot of code from `testPeriodicSchedulingWithInactiveTasks()`.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinatorTest.java
##########
@@ -626,6 +627,194 @@ public void testTriggerAndDeclineCheckpointThenFailureManagerThrowsException()
}
}
+ @Test
+ public void testIOExceptionCheckpointExceedsTolerableFailureNumber() throws Exception {
+ // create some mock Execution vertices that receive the checkpoint trigger messages
+ ExecutionGraph graph =
+ new CheckpointCoordinatorTestingUtils.CheckpointExecutionGraphBuilder()
+ .addJobVertex(new JobVertexID())
+ .addJobVertex(new JobVertexID())
+ .build();
+
+ final String errorMsg = "Exceeded checkpoint failure tolerance number!";
+ CheckpointFailureManager checkpointFailureManager = getCheckpointFailureManager(errorMsg);
+ CheckpointCoordinator checkpointCoordinator =
+ getCheckpointCoordinator(graph, checkpointFailureManager);
+
+ try {
+ checkpointCoordinator.triggerCheckpoint(false);
+ manuallyTriggeredScheduledExecutor.triggerAll();
+
+ checkpointCoordinator.abortPendingCheckpoints(new CheckpointException(IO_EXCEPTION));
+
+ fail("Test failed.");
+ } catch (Exception e) {
+ // expected
+ assertTrue(e instanceof RuntimeException);
+ assertEquals(errorMsg, e.getMessage());
Review comment:
please never hide the original exception. This code should look like this:
```
if (!ExceptionUtils.findThrowable(e, errorMsg).isPresent()) {
throw e;
}
```
This way if an unexpected error happens we can immediately see what was the actual exception and it's stack trace.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-889099584
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"triggerType" : "PUSH"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21186",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21188",
"triggerID" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 08da26fefae5d23c49518fcb6716813410c8d5fb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21188)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-889099584
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d7928fd1f51e6cc4296df21a9bf8521988b599f2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-889099584
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"triggerType" : "PUSH"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21186",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21188",
"triggerID" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d7928fd1f51e6cc4296df21a9bf8521988b599f2 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21186) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163)
* 08da26fefae5d23c49518fcb6716813410c8d5fb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21188)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-889099584
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"triggerType" : "PUSH"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21186",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21188",
"triggerID" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"triggerType" : "PUSH"
}, {
"hash" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21680",
"triggerID" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"triggerType" : "PUSH"
}, {
"hash" : "eb8c1b59b1fe328cfefc51c1204ced5202ad4448",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21702",
"triggerID" : "eb8c1b59b1fe328cfefc51c1204ced5202ad4448",
"triggerType" : "PUSH"
}, {
"hash" : "afe05192b1bd26789bd0b7da3a773e63bc2215a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21817",
"triggerID" : "afe05192b1bd26789bd0b7da3a773e63bc2215a1",
"triggerType" : "PUSH"
}, {
"hash" : "60352a88cfe01d3f9ad9d297753c3f2f941fcc9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21871",
"triggerID" : "60352a88cfe01d3f9ad9d297753c3f2f941fcc9a",
"triggerType" : "PUSH"
}, {
"hash" : "a9af148bb5760c640fe768b0476a53202b7e6792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21900",
"triggerID" : "a9af148bb5760c640fe768b0476a53202b7e6792",
"triggerType" : "PUSH"
}, {
"hash" : "59d43e6a1bfc216cbfa8ca43224576312adb57d2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21992",
"triggerID" : "59d43e6a1bfc216cbfa8ca43224576312adb57d2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 59d43e6a1bfc216cbfa8ca43224576312adb57d2 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21992)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] akalash commented on a change in pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
akalash commented on a change in pull request #16637:
URL: https://github.com/apache/flink/pull/16637#discussion_r686854033
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointFailureManager.java
##########
@@ -127,7 +127,6 @@ public void checkFailureCounter(CheckpointException exception, long checkpointId
case CHECKPOINT_DECLINED_SUBSUMED:
case CHECKPOINT_DECLINED_INPUT_END_OF_STREAM:
- case EXCEPTION:
Review comment:
@zlzhang0122 You don't need to touch it in this commit because this looks strange - you move it to another section and then you remove it at the next commit. It is better to remove it from the original place in your next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zlzhang0122 commented on a change in pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
zlzhang0122 commented on a change in pull request #16637:
URL: https://github.com/apache/flink/pull/16637#discussion_r687423896
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointFailureManager.java
##########
@@ -127,7 +127,6 @@ public void checkFailureCounter(CheckpointException exception, long checkpointId
case CHECKPOINT_DECLINED_SUBSUMED:
case CHECKPOINT_DECLINED_INPUT_END_OF_STREAM:
- case EXCEPTION:
Review comment:
I'm not so sure about this, am I need to create a new jira ticket? Because I have separated this in two commits under one ticket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] akalash commented on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
akalash commented on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-897641105
@zlzhang0122, thanks. Did you fix the test which failed highly likely because of your changes? (testRestartCheckpointCoordinatorIfStopWithSavepointFails). If so then let's wait for another test result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zlzhang0122 commented on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
zlzhang0122 commented on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-889593787
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zlzhang0122 commented on a change in pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
zlzhang0122 commented on a change in pull request #16637:
URL: https://github.com/apache/flink/pull/16637#discussion_r684151372
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/CheckpointFailureManagerTest.java
##########
@@ -96,7 +96,7 @@ public void testTotalCountValue() {
}
// CHECKPOINT_DECLINED, CHECKPOINT_EXPIRED and CHECKPOINT_ASYNC_EXCEPTION
- assertEquals(3, callback.getInvokeCounter());
+ assertEquals(5, callback.getInvokeCounter());
Review comment:
I have fixed this too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zlzhang0122 commented on a change in pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
zlzhang0122 commented on a change in pull request #16637:
URL: https://github.com/apache/flink/pull/16637#discussion_r684151641
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java
##########
@@ -892,9 +900,12 @@ private void onTriggerFailure(@Nullable PendingCheckpoint checkpoint, Throwable
job,
numUnsuccessful,
throwable);
- final CheckpointException cause =
- getCheckpointException(
- CheckpointFailureReason.TRIGGER_CHECKPOINT_FAILURE, throwable);
+ CheckpointFailureReason defaultReason =
+ CheckpointFailureReason.TRIGGER_CHECKPOINT_FAILURE;
+ if (throwable instanceof IOException) {
Review comment:
You are right and I have also made a change here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-889099584
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"triggerType" : "PUSH"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21186",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21188",
"triggerID" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"triggerType" : "PUSH"
}, {
"hash" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21680",
"triggerID" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21680)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-889099584
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"triggerType" : "PUSH"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21186",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21188",
"triggerID" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"triggerType" : "PUSH"
}, {
"hash" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21680",
"triggerID" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"triggerType" : "PUSH"
}, {
"hash" : "eb8c1b59b1fe328cfefc51c1204ced5202ad4448",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21702",
"triggerID" : "eb8c1b59b1fe328cfefc51c1204ced5202ad4448",
"triggerType" : "PUSH"
}, {
"hash" : "afe05192b1bd26789bd0b7da3a773e63bc2215a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21817",
"triggerID" : "afe05192b1bd26789bd0b7da3a773e63bc2215a1",
"triggerType" : "PUSH"
}, {
"hash" : "60352a88cfe01d3f9ad9d297753c3f2f941fcc9a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21871",
"triggerID" : "60352a88cfe01d3f9ad9d297753c3f2f941fcc9a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 60352a88cfe01d3f9ad9d297753c3f2f941fcc9a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21871)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] akalash commented on a change in pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
akalash commented on a change in pull request #16637:
URL: https://github.com/apache/flink/pull/16637#discussion_r686849015
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointFailureManager.java
##########
@@ -136,6 +135,8 @@ public void checkFailureCounter(CheckpointException exception, long checkpointId
// ignore
break;
+ case EXCEPTION:
Review comment:
Ok, then. We can migrate from EXCEPTION enum to IO_EXCEPTION but we can keep the original message from EXCEPTION and we can add the second sentence with an explanation about IO-exception.
So right now we have the message `An Exception occurred while triggering the checkpoint.` and we can transform it to something like `An Exception occurred while triggering the checkpoint. IO-problem detected.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] pnowojski commented on a change in pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
pnowojski commented on a change in pull request #16637:
URL: https://github.com/apache/flink/pull/16637#discussion_r685314374
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointFailureManager.java
##########
@@ -136,6 +135,8 @@ public void checkFailureCounter(CheckpointException exception, long checkpointId
// ignore
break;
+ case EXCEPTION:
Review comment:
I also doesn't see a reason why should we have both `EXCEPTION` and `IO_EXCEPTION`? Unless we want to differentiate checkpoint failures on task vs job manager, but in that case names are miss-leading.
> Why is EXCEPTION ignored in FailureManager right now?
I don't know. Maybe you can reach out to Arvid to ask about this, but I presume it's a simple omission.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zlzhang0122 commented on a change in pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
zlzhang0122 commented on a change in pull request #16637:
URL: https://github.com/apache/flink/pull/16637#discussion_r687423896
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointFailureManager.java
##########
@@ -127,7 +127,6 @@ public void checkFailureCounter(CheckpointException exception, long checkpointId
case CHECKPOINT_DECLINED_SUBSUMED:
case CHECKPOINT_DECLINED_INPUT_END_OF_STREAM:
- case EXCEPTION:
Review comment:
I'm not so sure about this, am I need to create a new jira ticket? Because I have separated this in two commits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-889099584
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"triggerType" : "PUSH"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21186",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21188",
"triggerID" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"triggerType" : "PUSH"
}, {
"hash" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21680",
"triggerID" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"triggerType" : "PUSH"
}, {
"hash" : "eb8c1b59b1fe328cfefc51c1204ced5202ad4448",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "eb8c1b59b1fe328cfefc51c1204ced5202ad4448",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21680)
* eb8c1b59b1fe328cfefc51c1204ced5202ad4448 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-889099584
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"triggerType" : "PUSH"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21186",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d7928fd1f51e6cc4296df21a9bf8521988b599f2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21186) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163)
* 08da26fefae5d23c49518fcb6716813410c8d5fb UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-889099584
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"triggerType" : "PUSH"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21186",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21188",
"triggerID" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"triggerType" : "PUSH"
}, {
"hash" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21680",
"triggerID" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"triggerType" : "PUSH"
}, {
"hash" : "eb8c1b59b1fe328cfefc51c1204ced5202ad4448",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21702",
"triggerID" : "eb8c1b59b1fe328cfefc51c1204ced5202ad4448",
"triggerType" : "PUSH"
}, {
"hash" : "afe05192b1bd26789bd0b7da3a773e63bc2215a1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21817",
"triggerID" : "afe05192b1bd26789bd0b7da3a773e63bc2215a1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eb8c1b59b1fe328cfefc51c1204ced5202ad4448 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21702)
* afe05192b1bd26789bd0b7da3a773e63bc2215a1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21817)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zlzhang0122 commented on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
zlzhang0122 commented on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-897632704
sure @akalash I have already finished it and please take a look, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zlzhang0122 removed a comment on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
zlzhang0122 removed a comment on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-889592975
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zlzhang0122 commented on a change in pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
zlzhang0122 commented on a change in pull request #16637:
URL: https://github.com/apache/flink/pull/16637#discussion_r684124071
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointFailureReason.java
##########
@@ -81,6 +81,8 @@
FINALIZE_CHECKPOINT_FAILURE(false, "Failure to finalize checkpoint."),
+ IO_EXCEPTION(false, "Trigger checkpoint failure."),
Review comment:
Sorry for that and I have fixed this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-889099584
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"triggerType" : "PUSH"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21186",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* d7928fd1f51e6cc4296df21a9bf8521988b599f2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21186)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-889099584
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"triggerType" : "PUSH"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21186",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21188",
"triggerID" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"triggerType" : "PUSH"
}, {
"hash" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21680",
"triggerID" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"triggerType" : "PUSH"
}, {
"hash" : "eb8c1b59b1fe328cfefc51c1204ced5202ad4448",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21702",
"triggerID" : "eb8c1b59b1fe328cfefc51c1204ced5202ad4448",
"triggerType" : "PUSH"
}, {
"hash" : "afe05192b1bd26789bd0b7da3a773e63bc2215a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21817",
"triggerID" : "afe05192b1bd26789bd0b7da3a773e63bc2215a1",
"triggerType" : "PUSH"
}, {
"hash" : "60352a88cfe01d3f9ad9d297753c3f2f941fcc9a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21871",
"triggerID" : "60352a88cfe01d3f9ad9d297753c3f2f941fcc9a",
"triggerType" : "PUSH"
}, {
"hash" : "a9af148bb5760c640fe768b0476a53202b7e6792",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21900",
"triggerID" : "a9af148bb5760c640fe768b0476a53202b7e6792",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 60352a88cfe01d3f9ad9d297753c3f2f941fcc9a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21871)
* a9af148bb5760c640fe768b0476a53202b7e6792 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21900)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] akalash commented on a change in pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
akalash commented on a change in pull request #16637:
URL: https://github.com/apache/flink/pull/16637#discussion_r685828704
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointFailureManager.java
##########
@@ -136,6 +135,8 @@ public void checkFailureCounter(CheckpointException exception, long checkpointId
// ignore
break;
+ case EXCEPTION:
Review comment:
@AHeise In this PR new `CheckpointFailureReason` is introduced(`IO_EXCEPTION`). Now we want to replace the existing `EXCEPTION` with `IO_EXCEPTION`(because according to the code it is exactly IO_EXCEPTION). Also, `IO_EXCEPTION` would not be ignored by `FailureManager` as it was with `EXCEPTION`. Do you have any objections to this?
@zlzhang0122 If nobody will mind, I suggest getting rid of the EXCEPTION and replace it with IO_EXCEPTION. But it is better to do so in two separated commits(under one ticket). As a result of this ticket, you will have two commits - one of them adding the IO_EXCEPTION but don't touch the EXCEPTION and the second one just replace EXCEPTION with IO_EXCEPTION.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-889099584
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d7928fd1f51e6cc4296df21a9bf8521988b599f2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-889099584
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"triggerType" : "PUSH"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21186",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21188",
"triggerID" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"triggerType" : "PUSH"
}, {
"hash" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21680",
"triggerID" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"triggerType" : "PUSH"
}, {
"hash" : "eb8c1b59b1fe328cfefc51c1204ced5202ad4448",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21702",
"triggerID" : "eb8c1b59b1fe328cfefc51c1204ced5202ad4448",
"triggerType" : "PUSH"
}, {
"hash" : "afe05192b1bd26789bd0b7da3a773e63bc2215a1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21817",
"triggerID" : "afe05192b1bd26789bd0b7da3a773e63bc2215a1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afe05192b1bd26789bd0b7da3a773e63bc2215a1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21817)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-889099584
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"triggerType" : "PUSH"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21186",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21188",
"triggerID" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"triggerType" : "PUSH"
}, {
"hash" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 08da26fefae5d23c49518fcb6716813410c8d5fb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21188)
* 0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #16637:
URL: https://github.com/apache/flink/pull/16637#discussion_r686213928
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointFailureManager.java
##########
@@ -136,6 +135,8 @@ public void checkFailureCounter(CheckpointException exception, long checkpointId
// ignore
break;
+ case EXCEPTION:
Review comment:
So you are not entirely replacing it, but only replacing it for IO related failures? Well then, there is nothing that you can do to maintain compability except making sure that the existing EXCEPTION is displaying as it was before. Make sure to point out the change in the release notes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] pnowojski closed pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
pnowojski closed pull request #16637:
URL: https://github.com/apache/flink/pull/16637
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #16637:
URL: https://github.com/apache/flink/pull/16637#discussion_r686653773
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointFailureManager.java
##########
@@ -136,6 +135,8 @@ public void checkFailureCounter(CheckpointException exception, long checkpointId
// ignore
break;
+ case EXCEPTION:
Review comment:
Okay thanks for clarifying. In that case, do we really need to change the exception message? Could we maybe leave the original message and just add a small sentence that this most likely caused by a disk issue? Then old log monitoring will still work and you get even additional advise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-889099584
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"triggerType" : "PUSH"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21186",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21188",
"triggerID" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"triggerType" : "PUSH"
}, {
"hash" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21680",
"triggerID" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"triggerType" : "PUSH"
}, {
"hash" : "eb8c1b59b1fe328cfefc51c1204ced5202ad4448",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21702",
"triggerID" : "eb8c1b59b1fe328cfefc51c1204ced5202ad4448",
"triggerType" : "PUSH"
}, {
"hash" : "afe05192b1bd26789bd0b7da3a773e63bc2215a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21817",
"triggerID" : "afe05192b1bd26789bd0b7da3a773e63bc2215a1",
"triggerType" : "PUSH"
}, {
"hash" : "60352a88cfe01d3f9ad9d297753c3f2f941fcc9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21871",
"triggerID" : "60352a88cfe01d3f9ad9d297753c3f2f941fcc9a",
"triggerType" : "PUSH"
}, {
"hash" : "a9af148bb5760c640fe768b0476a53202b7e6792",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21900",
"triggerID" : "a9af148bb5760c640fe768b0476a53202b7e6792",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a9af148bb5760c640fe768b0476a53202b7e6792 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21900)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-889099584
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"triggerType" : "PUSH"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21186",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21188",
"triggerID" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"triggerType" : "PUSH"
}, {
"hash" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21680",
"triggerID" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"triggerType" : "PUSH"
}, {
"hash" : "eb8c1b59b1fe328cfefc51c1204ced5202ad4448",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21702",
"triggerID" : "eb8c1b59b1fe328cfefc51c1204ced5202ad4448",
"triggerType" : "PUSH"
}, {
"hash" : "afe05192b1bd26789bd0b7da3a773e63bc2215a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21817",
"triggerID" : "afe05192b1bd26789bd0b7da3a773e63bc2215a1",
"triggerType" : "PUSH"
}, {
"hash" : "60352a88cfe01d3f9ad9d297753c3f2f941fcc9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21871",
"triggerID" : "60352a88cfe01d3f9ad9d297753c3f2f941fcc9a",
"triggerType" : "PUSH"
}, {
"hash" : "a9af148bb5760c640fe768b0476a53202b7e6792",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21900",
"triggerID" : "a9af148bb5760c640fe768b0476a53202b7e6792",
"triggerType" : "PUSH"
}, {
"hash" : "59d43e6a1bfc216cbfa8ca43224576312adb57d2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21992",
"triggerID" : "59d43e6a1bfc216cbfa8ca43224576312adb57d2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a9af148bb5760c640fe768b0476a53202b7e6792 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21900)
* 59d43e6a1bfc216cbfa8ca43224576312adb57d2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21992)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zlzhang0122 commented on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
zlzhang0122 commented on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-889592975
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-889099584
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d7928fd1f51e6cc4296df21a9bf8521988b599f2 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] pnowojski commented on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
pnowojski commented on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-898886723
Superseded by https://github.com/apache/flink/pull/16829
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-889099584
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"triggerType" : "PUSH"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21186",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21188",
"triggerID" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"triggerType" : "PUSH"
}, {
"hash" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21680",
"triggerID" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"triggerType" : "PUSH"
}, {
"hash" : "eb8c1b59b1fe328cfefc51c1204ced5202ad4448",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21702",
"triggerID" : "eb8c1b59b1fe328cfefc51c1204ced5202ad4448",
"triggerType" : "PUSH"
}, {
"hash" : "afe05192b1bd26789bd0b7da3a773e63bc2215a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21817",
"triggerID" : "afe05192b1bd26789bd0b7da3a773e63bc2215a1",
"triggerType" : "PUSH"
}, {
"hash" : "60352a88cfe01d3f9ad9d297753c3f2f941fcc9a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21871",
"triggerID" : "60352a88cfe01d3f9ad9d297753c3f2f941fcc9a",
"triggerType" : "PUSH"
}, {
"hash" : "a9af148bb5760c640fe768b0476a53202b7e6792",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a9af148bb5760c640fe768b0476a53202b7e6792",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 60352a88cfe01d3f9ad9d297753c3f2f941fcc9a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21871)
* a9af148bb5760c640fe768b0476a53202b7e6792 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zlzhang0122 commented on a change in pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
zlzhang0122 commented on a change in pull request #16637:
URL: https://github.com/apache/flink/pull/16637#discussion_r684228554
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinatorTest.java
##########
@@ -624,6 +625,36 @@ public void testTriggerAndDeclineCheckpointThenFailureManagerThrowsException()
}
}
+ @Test
+ public void testIOExceptionCheckpointExceedsTolerableFailureNumber() throws Exception {
Review comment:
Sure, I have already add some unit tests that check getCheckpointException function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zlzhang0122 commented on a change in pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
zlzhang0122 commented on a change in pull request #16637:
URL: https://github.com/apache/flink/pull/16637#discussion_r685807958
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorCoordinatorCheckpoints.java
##########
@@ -121,14 +123,17 @@ private static void acknowledgeAllCoordinators(
final Throwable error =
checkpoint.isDisposed() ? checkpoint.getFailureCause() : null;
+ CheckpointFailureReason reason = CheckpointFailureReason.TRIGGER_CHECKPOINT_FAILURE;
Review comment:
I agree with @pnowojski that `acknowledgeAllCoordinators` is called very early, and as far as I can see, `acknowledgeAllCoordinators` is used to take checkpoint for `OperatorCoordinator` which is runing on JobMaster. So maybe we can simply consider it as another kind of masterStates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-889099584
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"triggerType" : "PUSH"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21186",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21188",
"triggerID" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"triggerType" : "PUSH"
}, {
"hash" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21680",
"triggerID" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 08da26fefae5d23c49518fcb6716813410c8d5fb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21188)
* 0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21680)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-889099584
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"triggerType" : "PUSH"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21186",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21188",
"triggerID" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"triggerType" : "PUSH"
}, {
"hash" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21680",
"triggerID" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"triggerType" : "PUSH"
}, {
"hash" : "eb8c1b59b1fe328cfefc51c1204ced5202ad4448",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21702",
"triggerID" : "eb8c1b59b1fe328cfefc51c1204ced5202ad4448",
"triggerType" : "PUSH"
}, {
"hash" : "afe05192b1bd26789bd0b7da3a773e63bc2215a1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21817",
"triggerID" : "afe05192b1bd26789bd0b7da3a773e63bc2215a1",
"triggerType" : "PUSH"
}, {
"hash" : "60352a88cfe01d3f9ad9d297753c3f2f941fcc9a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21871",
"triggerID" : "60352a88cfe01d3f9ad9d297753c3f2f941fcc9a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afe05192b1bd26789bd0b7da3a773e63bc2215a1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21817)
* 60352a88cfe01d3f9ad9d297753c3f2f941fcc9a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21871)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16637:
URL: https://github.com/apache/flink/pull/16637#issuecomment-889099584
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"triggerType" : "PUSH"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21186",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "d7928fd1f51e6cc4296df21a9bf8521988b599f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21163",
"triggerID" : "889593787",
"triggerType" : "MANUAL"
}, {
"hash" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21188",
"triggerID" : "08da26fefae5d23c49518fcb6716813410c8d5fb",
"triggerType" : "PUSH"
}, {
"hash" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21680",
"triggerID" : "0fb4be42643b0d2803477ee6e2b3bf4fc4b6ce17",
"triggerType" : "PUSH"
}, {
"hash" : "eb8c1b59b1fe328cfefc51c1204ced5202ad4448",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21702",
"triggerID" : "eb8c1b59b1fe328cfefc51c1204ced5202ad4448",
"triggerType" : "PUSH"
}, {
"hash" : "afe05192b1bd26789bd0b7da3a773e63bc2215a1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "afe05192b1bd26789bd0b7da3a773e63bc2215a1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eb8c1b59b1fe328cfefc51c1204ced5202ad4448 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21702)
* afe05192b1bd26789bd0b7da3a773e63bc2215a1 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] pnowojski commented on a change in pull request #16637: [FLINK-23189][checkpoint]Count and fail the task when the disk is error on JobManager
Posted by GitBox <gi...@apache.org>.
pnowojski commented on a change in pull request #16637:
URL: https://github.com/apache/flink/pull/16637#discussion_r685319318
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorCoordinatorCheckpoints.java
##########
@@ -121,14 +123,17 @@ private static void acknowledgeAllCoordinators(
final Throwable error =
checkpoint.isDisposed() ? checkpoint.getFailureCause() : null;
+ CheckpointFailureReason reason = CheckpointFailureReason.TRIGGER_CHECKPOINT_FAILURE;
Review comment:
From what I can see `acknowledgeAllCoordinators` is called before (CheckpointCoordinator:L578) triggerCheckpoint RPC is called on Task Managers (CheckpointCoordinator:L623). So this happens very early in the in the checkpoint triggering process and is consistent with out `TRIGGER_CHECKPOINT_FAILURE` usages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org