You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/05/14 04:10:00 UTC
[jira] [Commented] (FLINK-8977) End-to-end test: Manually resume
job after terminal failure
[ https://issues.apache.org/jira/browse/FLINK-8977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16473749#comment-16473749 ]
ASF GitHub Bot commented on FLINK-8977:
---------------------------------------
GitHub user tzulitai opened a pull request:
https://github.com/apache/flink/pull/6004
[FLINK-8977] [e2e] End-to-end test for manual job resume after terminal failure
## What is the purpose of the change
This PR is based on new e2e features introduced by #5941, #5990, and #5969.
Only the last two commits are relevant to FLINK-8977.
This PR adds e2e test coverage for the case that after a terminal failure caused by the user job code, manually resuming from a retained checkpoint works correctly.
This is achieved by extending the `test_resume_externalized_checkpoints.sh` test script to accept a `SIMULATE_FAILURE` flag.
## Brief change log
- 9360ea9 Extend the general purpose DataStream job to allow configuring restart strategies.
- b5d713c Extend `test_resume_externalized_checkpoints.sh` to allow simulating the job failure + manual resume case.
## Verifying this change
Verifiable by running locally the following e2e test script:
`SIMULATE_FAILURE=true flink-end-to-end-tests/test-scripts/test_resume_externalized_checkpoints.sh`
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / **no**)
- If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tzulitai/flink FLINK-8977
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/6004.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6004
----
commit 8db7f894b67b00f94148e0314a1c10d76266a350
Author: Tzu-Li (Gordon) Tai <tz...@...>
Date: 2018-04-30T10:04:43Z
[hotfix] [e2e-tests] Make SequenceGeneratorSource usable for 0-size key ranges
commit c8e14673e58aed0f9625e38875ec85a776282ad4
Author: Tzu-Li (Gordon) Tai <tz...@...>
Date: 2018-04-30T10:05:46Z
[FLINK-8971] [e2e-tests] Include broadcast / union state in general purpose DataStream job
commit 78354b295832fa2ec5d829ec4ac21150ecac1231
Author: Tzu-Li (Gordon) Tai <tz...@...>
Date: 2018-05-08T03:44:13Z
PR review - refactor source run function
commit f346fd0958e7c3361886680912630fe22761a63d
Author: Tzu-Li (Gordon) Tai <tz...@...>
Date: 2018-05-08T04:39:40Z
PR review - simplify broadcast / union state verification
commit b01cfda7d77723e8ded2ce99ee12f17352a3ca1f
Author: Tzu-Li (Gordon) Tai <tz...@...>
Date: 2018-05-11T03:51:12Z
[FLINK-9322] [e2e] Add failure simulation to the general purpose DataStream job
commit 0931f6ed48523ca46e2c99adc24950777d843ac8
Author: Tzu-Li (Gordon) Tai <tz...@...>
Date: 2018-05-11T07:09:00Z
[FLINK-9320] [e2e] Update test_ha e2e to use general purpose DataStream job
commit a819e56e0998e09bb6461b6c76be0807d83a1ef5
Author: Tzu-Li (Gordon) Tai <tz...@...>
Date: 2018-05-09T03:40:24Z
[FLINK-9074] [e2e] Allow configuring externalized checkpoints for the general purpose DataStream job
commit 3d0c83a991ab78d03c3cc1c9ff2abb61e0329d9d
Author: Tzu-Li (Gordon) Tai <tz...@...>
Date: 2018-05-09T04:17:25Z
[FLINK-9074] [e2e] Add e2e test for resuming jobs from retained checkpoints
commit 9360ea9ad0db858e7fdeecb54b1918e6b84cae1d
Author: Tzu-Li (Gordon) Tai <tz...@...>
Date: 2018-05-14T03:56:07Z
[FLINK-8977] [e2e] Allow configuring restart strategy for general purpose DataStream job
commit b5d713cf19290be437286e152d921b23ff532c7d
Author: Tzu-Li (Gordon) Tai <tz...@...>
Date: 2018-05-14T03:56:41Z
[FLINK-8977] [e2e] Extend externalized checkpoint e2e to simulate job failures
----
> End-to-end test: Manually resume job after terminal failure
> -----------------------------------------------------------
>
> Key: FLINK-8977
> URL: https://issues.apache.org/jira/browse/FLINK-8977
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
> Affects Versions: 1.5.0
> Reporter: Till Rohrmann
> Assignee: Tzu-Li (Gordon) Tai
> Priority: Blocker
> Fix For: 1.6.0, 1.5.1
>
>
> We should add an end-to-end test which verifies that a job can be resumed manually after a terminal job failure if there is a checkpoint. In order to do that we should
> # run the general purpose testing job FLINK-8971
> # wait for the completion of a checkpoint
> # Trigger a failure which leads to a terminal failure
> # Resume the job from the retained checkpoint
> This end-to-end test should run with all state backend combinations: RocksDB (incremental/full, async/sync), FsStateBackend (async/sync).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)