You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/01/10 08:14:06 UTC
[GitHub] [flink] klion26 opened a new pull request #10824:
[FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if
StopWithSavepoint failed
klion26 opened a new pull request #10824: [FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if StopWithSavepoint failed
URL: https://github.com/apache/flink/pull/10824
## What is the purpose of the change
Restart `CheckpointCoordinator` if `StopWithSavepoint` failed.
## Verifying this change
This change added tests and can be verified as follows:
- `JobMasterStopWithSavepointIT#testRestartCheckpointCoordinatorIfStopWithSavepointFails`
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (yes / no / don't know)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes)
- The S3 file system connector: ( no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] klion26 commented on a change in pull request #10824:
[FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if
StopWithSavepoint failed
Posted by GitBox <gi...@apache.org>.
klion26 commented on a change in pull request #10824: [FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if StopWithSavepoint failed
URL: https://github.com/apache/flink/pull/10824#discussion_r367226527
##########
File path: flink-tests/src/test/java/org/apache/flink/runtime/jobmaster/JobMasterStopWithSavepointIT.java
##########
@@ -357,4 +391,36 @@ public void finishTask() throws Exception {
finishLatch.trigger();
}
}
+
+ /**
+ * A {@link StreamTask} that simply calls {@link CountDownLatch#countDown()} when
+ * invoking {@link #triggerCheckpointAsync}.
+ */
+ public static class CheckpointCountingTask extends NoOpStreamTask {
Review comment:
Ok, will update it. previously did not change to this, because I though in `CheckpointCountingTask` we do not need to distinguish whether this task is in subtask 0 (and can tolerate some code dup in test), but distinguish whether this task is subtask 0 seems has no side effect.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] pnowojski commented on a change in pull request #10824:
[FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if
StopWithSavepoint failed
Posted by GitBox <gi...@apache.org>.
pnowojski commented on a change in pull request #10824: [FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if StopWithSavepoint failed
URL: https://github.com/apache/flink/pull/10824#discussion_r366875446
##########
File path: flink-tests/src/test/java/org/apache/flink/runtime/jobmaster/JobMasterStopWithSavepointIT.java
##########
@@ -357,4 +391,36 @@ public void finishTask() throws Exception {
finishLatch.trigger();
}
}
+
+ /**
+ * A {@link StreamTask} that simply calls {@link CountDownLatch#countDown()} when
+ * invoking {@link #triggerCheckpointAsync}.
+ */
+ public static class CheckpointCountingTask extends NoOpStreamTask {
Review comment:
By extracting I meant that `ExceptionOnCallbackStreamTask` could also use this `CheckpointCountingTask` class to avoid some (minor) code duplication. Could you re-use this code in `ExceptionOnCallbackStreamTask`? Make it extend `CheckpointCountingTask` and delegate methods/constructors to `super` calls?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10824:
[FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if
StopWithSavepoint failed
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10824: [FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if StopWithSavepoint failed
URL: https://github.com/apache/flink/pull/10824#issuecomment-572926550
<!--
Meta data
Hash:7a61978007a5a02723a561ee145d82e2413e51ae Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4250 TriggerType:PUSH TriggerID:7a61978007a5a02723a561ee145d82e2413e51ae
Hash:7a61978007a5a02723a561ee145d82e2413e51ae Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143855396 TriggerType:PUSH TriggerID:7a61978007a5a02723a561ee145d82e2413e51ae
Hash:e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648
-->
## CI report:
* 7a61978007a5a02723a561ee145d82e2413e51ae Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143855396) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4250)
* e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10824:
[FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if
StopWithSavepoint failed
Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #10824: [FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if StopWithSavepoint failed
URL: https://github.com/apache/flink/pull/10824#issuecomment-572926550
<!--
Meta data
Hash:7a61978007a5a02723a561ee145d82e2413e51ae Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:7a61978007a5a02723a561ee145d82e2413e51ae
-->
## CI report:
* 7a61978007a5a02723a561ee145d82e2413e51ae UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] pnowojski merged pull request #10824:
[FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if
StopWithSavepoint failed
Posted by GitBox <gi...@apache.org>.
pnowojski merged pull request #10824: [FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if StopWithSavepoint failed
URL: https://github.com/apache/flink/pull/10824
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10824:
[FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if
StopWithSavepoint failed
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10824: [FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if StopWithSavepoint failed
URL: https://github.com/apache/flink/pull/10824#issuecomment-572926550
<!--
Meta data
Hash:7a61978007a5a02723a561ee145d82e2413e51ae Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4250 TriggerType:PUSH TriggerID:7a61978007a5a02723a561ee145d82e2413e51ae
Hash:7a61978007a5a02723a561ee145d82e2413e51ae Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143855396 TriggerType:PUSH TriggerID:7a61978007a5a02723a561ee145d82e2413e51ae
Hash:e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4351 TriggerType:PUSH TriggerID:e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648
Hash:e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144450734 TriggerType:PUSH TriggerID:e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648
Hash:3b07862f15aaddd2b6d2a0d21cceaba4a1f55f6e Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4385 TriggerType:PUSH TriggerID:3b07862f15aaddd2b6d2a0d21cceaba4a1f55f6e
Hash:3b07862f15aaddd2b6d2a0d21cceaba4a1f55f6e Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/144670673 TriggerType:PUSH TriggerID:3b07862f15aaddd2b6d2a0d21cceaba4a1f55f6e
-->
## CI report:
* 7a61978007a5a02723a561ee145d82e2413e51ae Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143855396) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4250)
* e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144450734) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4351)
* 3b07862f15aaddd2b6d2a0d21cceaba4a1f55f6e Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/144670673) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4385)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] klion26 commented on issue #10824:
[FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if
StopWithSavepoint failed
Posted by GitBox <gi...@apache.org>.
klion26 commented on issue #10824: [FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if StopWithSavepoint failed
URL: https://github.com/apache/flink/pull/10824#issuecomment-574478632
@pnowojski thanks for the review, updated the test. please have another look.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10824:
[FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if
StopWithSavepoint failed
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10824: [FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if StopWithSavepoint failed
URL: https://github.com/apache/flink/pull/10824#issuecomment-572926550
<!--
Meta data
Hash:7a61978007a5a02723a561ee145d82e2413e51ae Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4250 TriggerType:PUSH TriggerID:7a61978007a5a02723a561ee145d82e2413e51ae
Hash:7a61978007a5a02723a561ee145d82e2413e51ae Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143855396 TriggerType:PUSH TriggerID:7a61978007a5a02723a561ee145d82e2413e51ae
-->
## CI report:
* 7a61978007a5a02723a561ee145d82e2413e51ae Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143855396) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4250)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10824:
[FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if
StopWithSavepoint failed
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10824: [FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if StopWithSavepoint failed
URL: https://github.com/apache/flink/pull/10824#issuecomment-572926550
<!--
Meta data
Hash:7a61978007a5a02723a561ee145d82e2413e51ae Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4250 TriggerType:PUSH TriggerID:7a61978007a5a02723a561ee145d82e2413e51ae
Hash:7a61978007a5a02723a561ee145d82e2413e51ae Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143855396 TriggerType:PUSH TriggerID:7a61978007a5a02723a561ee145d82e2413e51ae
Hash:e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4351 TriggerType:PUSH TriggerID:e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648
Hash:e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144450734 TriggerType:PUSH TriggerID:e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648
Hash:3b07862f15aaddd2b6d2a0d21cceaba4a1f55f6e Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:3b07862f15aaddd2b6d2a0d21cceaba4a1f55f6e
-->
## CI report:
* 7a61978007a5a02723a561ee145d82e2413e51ae Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143855396) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4250)
* e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144450734) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4351)
* 3b07862f15aaddd2b6d2a0d21cceaba4a1f55f6e UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] pnowojski commented on a change in pull request #10824:
[FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if
StopWithSavepoint failed
Posted by GitBox <gi...@apache.org>.
pnowojski commented on a change in pull request #10824: [FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if StopWithSavepoint failed
URL: https://github.com/apache/flink/pull/10824#discussion_r365774289
##########
File path: flink-tests/src/test/java/org/apache/flink/runtime/jobmaster/JobMasterStopWithSavepointIT.java
##########
@@ -194,6 +197,30 @@ private void throwingExceptionOnCallbackWithRestartsHelper(final boolean termina
assertThat(getJobStatus(), either(equalTo(JobStatus.CANCELLING)).or(equalTo(JobStatus.CANCELED)));
}
+ @Test
+ public void testRestartCheckpointCoordinatorIfStopWithSavepointFails() throws Exception {
+ setUpJobGraph(ExceptionOnCallbackStreamTask.class, RestartStrategies.noRestart());
+
+ try {
+ Files.setPosixFilePermissions(savepointDirectory, Collections.emptySet());
+ } catch (IOException e) {
+ Assume.assumeNoException(e);
+ }
+
+ try {
+ stopWithSavepoint(true).get();
+ fail();
+ } catch (Exception e) {
+ assertThat(ExceptionUtils.findThrowable(e, CheckpointException.class).isPresent(), equalTo(true));
+ }
+
+ final JobStatus jobStatus = clusterClient.getJobStatus(jobGraph.getJobID()).get(60, TimeUnit.SECONDS);
+ assertThat(jobStatus, equalTo(JobStatus.RUNNING));
+ // assert that checkpoints are continued to be triggered
+ checkpointsToWaitFor = new CountDownLatch(1);
Review comment:
I'm not sure if this is correct and whether the newly created `checkpointsToWaitFor` is guaranteed to be visible by the `ExceptionOnCallbackStreamTask` instance, as `ExceptionOnCallbackStreamTask`'s thread is instantiated before this line.
Probably the easiest fix would be to make `checkpointsToWaitFor` `volatile`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10824:
[FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if
StopWithSavepoint failed
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10824: [FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if StopWithSavepoint failed
URL: https://github.com/apache/flink/pull/10824#issuecomment-572926550
<!--
Meta data
Hash:7a61978007a5a02723a561ee145d82e2413e51ae Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4250 TriggerType:PUSH TriggerID:7a61978007a5a02723a561ee145d82e2413e51ae
Hash:7a61978007a5a02723a561ee145d82e2413e51ae Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143855396 TriggerType:PUSH TriggerID:7a61978007a5a02723a561ee145d82e2413e51ae
Hash:e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4351 TriggerType:PUSH TriggerID:e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648
Hash:e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144450734 TriggerType:PUSH TriggerID:e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648
Hash:3b07862f15aaddd2b6d2a0d21cceaba4a1f55f6e Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4385 TriggerType:PUSH TriggerID:3b07862f15aaddd2b6d2a0d21cceaba4a1f55f6e
Hash:3b07862f15aaddd2b6d2a0d21cceaba4a1f55f6e Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144670673 TriggerType:PUSH TriggerID:3b07862f15aaddd2b6d2a0d21cceaba4a1f55f6e
-->
## CI report:
* 7a61978007a5a02723a561ee145d82e2413e51ae Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143855396) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4250)
* e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144450734) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4351)
* 3b07862f15aaddd2b6d2a0d21cceaba4a1f55f6e Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144670673) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4385)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10824:
[FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if
StopWithSavepoint failed
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10824: [FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if StopWithSavepoint failed
URL: https://github.com/apache/flink/pull/10824#issuecomment-572926550
<!--
Meta data
Hash:7a61978007a5a02723a561ee145d82e2413e51ae Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4250 TriggerType:PUSH TriggerID:7a61978007a5a02723a561ee145d82e2413e51ae
Hash:7a61978007a5a02723a561ee145d82e2413e51ae Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143855396 TriggerType:PUSH TriggerID:7a61978007a5a02723a561ee145d82e2413e51ae
Hash:e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4351 TriggerType:PUSH TriggerID:e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648
Hash:e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144450734 TriggerType:PUSH TriggerID:e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648
-->
## CI report:
* 7a61978007a5a02723a561ee145d82e2413e51ae Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143855396) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4250)
* e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144450734) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4351)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] klion26 commented on issue #10824:
[FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if
StopWithSavepoint failed
Posted by GitBox <gi...@apache.org>.
klion26 commented on issue #10824: [FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if StopWithSavepoint failed
URL: https://github.com/apache/flink/pull/10824#issuecomment-575162505
@pnowojski thanks for the review and merging.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10824:
[FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if
StopWithSavepoint failed
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10824: [FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if StopWithSavepoint failed
URL: https://github.com/apache/flink/pull/10824#issuecomment-572926550
<!--
Meta data
Hash:7a61978007a5a02723a561ee145d82e2413e51ae Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4250 TriggerType:PUSH TriggerID:7a61978007a5a02723a561ee145d82e2413e51ae
Hash:7a61978007a5a02723a561ee145d82e2413e51ae Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143855396 TriggerType:PUSH TriggerID:7a61978007a5a02723a561ee145d82e2413e51ae
Hash:e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648 Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4351 TriggerType:PUSH TriggerID:e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648
Hash:e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/144450734 TriggerType:PUSH TriggerID:e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648
-->
## CI report:
* 7a61978007a5a02723a561ee145d82e2413e51ae Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143855396) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4250)
* e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/144450734) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4351)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10824:
[FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if
StopWithSavepoint failed
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10824: [FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if StopWithSavepoint failed
URL: https://github.com/apache/flink/pull/10824#issuecomment-572926550
<!--
Meta data
Hash:7a61978007a5a02723a561ee145d82e2413e51ae Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4250 TriggerType:PUSH TriggerID:7a61978007a5a02723a561ee145d82e2413e51ae
Hash:7a61978007a5a02723a561ee145d82e2413e51ae Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143855396 TriggerType:PUSH TriggerID:7a61978007a5a02723a561ee145d82e2413e51ae
Hash:e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648 Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4351 TriggerType:PUSH TriggerID:e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648
Hash:e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144450734 TriggerType:PUSH TriggerID:e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648
-->
## CI report:
* 7a61978007a5a02723a561ee145d82e2413e51ae Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143855396) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4250)
* e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144450734) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4351)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] pnowojski commented on a change in pull request #10824:
[FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if
StopWithSavepoint failed
Posted by GitBox <gi...@apache.org>.
pnowojski commented on a change in pull request #10824: [FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if StopWithSavepoint failed
URL: https://github.com/apache/flink/pull/10824#discussion_r365769560
##########
File path: flink-tests/src/test/java/org/apache/flink/runtime/jobmaster/JobMasterStopWithSavepointIT.java
##########
@@ -194,6 +197,30 @@ private void throwingExceptionOnCallbackWithRestartsHelper(final boolean termina
assertThat(getJobStatus(), either(equalTo(JobStatus.CANCELLING)).or(equalTo(JobStatus.CANCELED)));
}
+ @Test
+ public void testRestartCheckpointCoordinatorIfStopWithSavepointFails() throws Exception {
+ setUpJobGraph(ExceptionOnCallbackStreamTask.class, RestartStrategies.noRestart());
+
+ try {
+ Files.setPosixFilePermissions(savepointDirectory, Collections.emptySet());
+ } catch (IOException e) {
+ Assume.assumeNoException(e);
+ }
+
+ try {
+ stopWithSavepoint(true).get();
+ fail();
+ } catch (Exception e) {
+ assertThat(ExceptionUtils.findThrowable(e, CheckpointException.class).isPresent(), equalTo(true));
Review comment:
1. I think you should be more specific and verify for example exact `CheckpointFailureReason`.
2. If the exception is not what you expected, you should just `throw e` it, instead of using `assertThat`, to get an even nicer exception message with a stack trace.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] pnowojski commented on a change in pull request #10824:
[FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if
StopWithSavepoint failed
Posted by GitBox <gi...@apache.org>.
pnowojski commented on a change in pull request #10824: [FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if StopWithSavepoint failed
URL: https://github.com/apache/flink/pull/10824#discussion_r365769560
##########
File path: flink-tests/src/test/java/org/apache/flink/runtime/jobmaster/JobMasterStopWithSavepointIT.java
##########
@@ -194,6 +197,30 @@ private void throwingExceptionOnCallbackWithRestartsHelper(final boolean termina
assertThat(getJobStatus(), either(equalTo(JobStatus.CANCELLING)).or(equalTo(JobStatus.CANCELED)));
}
+ @Test
+ public void testRestartCheckpointCoordinatorIfStopWithSavepointFails() throws Exception {
+ setUpJobGraph(ExceptionOnCallbackStreamTask.class, RestartStrategies.noRestart());
+
+ try {
+ Files.setPosixFilePermissions(savepointDirectory, Collections.emptySet());
+ } catch (IOException e) {
+ Assume.assumeNoException(e);
+ }
+
+ try {
+ stopWithSavepoint(true).get();
+ fail();
+ } catch (Exception e) {
+ assertThat(ExceptionUtils.findThrowable(e, CheckpointException.class).isPresent(), equalTo(true));
Review comment:
1. I think you should be more specific and verify for example exact `CheckpointFailureReason` if this is easy thing to do (otherwise there are higher chances for the test incorrectly passing because of different bug/exception).
2. If the exception is not what you expected, you should just `throw e` it, instead of using `assertThat`, to get an even nicer exception message with a stack trace.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] pnowojski commented on a change in pull request #10824:
[FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if
StopWithSavepoint failed
Posted by GitBox <gi...@apache.org>.
pnowojski commented on a change in pull request #10824: [FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if StopWithSavepoint failed
URL: https://github.com/apache/flink/pull/10824#discussion_r365769560
##########
File path: flink-tests/src/test/java/org/apache/flink/runtime/jobmaster/JobMasterStopWithSavepointIT.java
##########
@@ -194,6 +197,30 @@ private void throwingExceptionOnCallbackWithRestartsHelper(final boolean termina
assertThat(getJobStatus(), either(equalTo(JobStatus.CANCELLING)).or(equalTo(JobStatus.CANCELED)));
}
+ @Test
+ public void testRestartCheckpointCoordinatorIfStopWithSavepointFails() throws Exception {
+ setUpJobGraph(ExceptionOnCallbackStreamTask.class, RestartStrategies.noRestart());
+
+ try {
+ Files.setPosixFilePermissions(savepointDirectory, Collections.emptySet());
+ } catch (IOException e) {
+ Assume.assumeNoException(e);
+ }
+
+ try {
+ stopWithSavepoint(true).get();
+ fail();
+ } catch (Exception e) {
+ assertThat(ExceptionUtils.findThrowable(e, CheckpointException.class).isPresent(), equalTo(true));
Review comment:
nit:
1. I think you should be more specific and verify for example exact `CheckpointFailureReason` if this is easy thing to do (otherwise there are higher chances for the test incorrectly passing because of different bug/exception).
2. If the exception is not what you expected, you should just `throw e` it, instead of using `assertThat`, to get an even nicer exception message with a stack trace.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10824:
[FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if
StopWithSavepoint failed
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10824: [FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if StopWithSavepoint failed
URL: https://github.com/apache/flink/pull/10824#issuecomment-572926550
<!--
Meta data
Hash:7a61978007a5a02723a561ee145d82e2413e51ae Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4250 TriggerType:PUSH TriggerID:7a61978007a5a02723a561ee145d82e2413e51ae
Hash:7a61978007a5a02723a561ee145d82e2413e51ae Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/143855396 TriggerType:PUSH TriggerID:7a61978007a5a02723a561ee145d82e2413e51ae
-->
## CI report:
* 7a61978007a5a02723a561ee145d82e2413e51ae Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/143855396) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4250)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10824:
[FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if
StopWithSavepoint failed
Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #10824: [FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if StopWithSavepoint failed
URL: https://github.com/apache/flink/pull/10824#issuecomment-572923526
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 7a61978007a5a02723a561ee145d82e2413e51ae (Fri Jan 10 08:17:18 UTC 2020)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] pnowojski commented on a change in pull request #10824:
[FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if
StopWithSavepoint failed
Posted by GitBox <gi...@apache.org>.
pnowojski commented on a change in pull request #10824: [FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if StopWithSavepoint failed
URL: https://github.com/apache/flink/pull/10824#discussion_r365775963
##########
File path: flink-tests/src/test/java/org/apache/flink/runtime/jobmaster/JobMasterStopWithSavepointIT.java
##########
@@ -194,6 +197,30 @@ private void throwingExceptionOnCallbackWithRestartsHelper(final boolean termina
assertThat(getJobStatus(), either(equalTo(JobStatus.CANCELLING)).or(equalTo(JobStatus.CANCELED)));
}
+ @Test
+ public void testRestartCheckpointCoordinatorIfStopWithSavepointFails() throws Exception {
+ setUpJobGraph(ExceptionOnCallbackStreamTask.class, RestartStrategies.noRestart());
Review comment:
I think the fact that you are using `ExceptionOnCallbackStreamTask` might be a bit misleading, as as far as I understand, you are doing it only for the `checkpointsToWaitFor` right? And you are ignoring the whole "exception on notifyCheckpointComplete callback" part of the `ExceptionOnCallbackStreamTask`?
If that's the case, I think it would be better to move/extract the count down latch to a new
```
public static class CheckpointCountingTask extends NoOpStreamTask {
}
```
which would be just a `NoOpStreamTask` with the the `CountDownLatch`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10824:
[FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if
StopWithSavepoint failed
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10824: [FLINK-15152][checkpointing] Restatrt CheckpointCoordinator if StopWithSavepoint failed
URL: https://github.com/apache/flink/pull/10824#issuecomment-572926550
<!--
Meta data
Hash:7a61978007a5a02723a561ee145d82e2413e51ae Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4250 TriggerType:PUSH TriggerID:7a61978007a5a02723a561ee145d82e2413e51ae
Hash:7a61978007a5a02723a561ee145d82e2413e51ae Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/143855396 TriggerType:PUSH TriggerID:7a61978007a5a02723a561ee145d82e2413e51ae
Hash:e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4351 TriggerType:PUSH TriggerID:e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648
Hash:e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144450734 TriggerType:PUSH TriggerID:e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648
Hash:3b07862f15aaddd2b6d2a0d21cceaba4a1f55f6e Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4385 TriggerType:PUSH TriggerID:3b07862f15aaddd2b6d2a0d21cceaba4a1f55f6e
Hash:3b07862f15aaddd2b6d2a0d21cceaba4a1f55f6e Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144670673 TriggerType:PUSH TriggerID:3b07862f15aaddd2b6d2a0d21cceaba4a1f55f6e
-->
## CI report:
* 7a61978007a5a02723a561ee145d82e2413e51ae Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/143855396) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4250)
* e718b2d4d8b3f4c6f846a72ee0cfe28b7a0ce648 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144450734) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4351)
* 3b07862f15aaddd2b6d2a0d21cceaba4a1f55f6e Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144670673) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4385)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services