You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/06/09 15:50:12 UTC
[GitHub] [flink] rkhachatryan opened a new pull request #12550: [FLINK-17869][task][checkpointing] Fix race condition when caling ChannelStateWriter.abort (backport to 1.11)
rkhachatryan opened a new pull request #12550:
URL: https://github.com/apache/flink/pull/12550
__backport of # 12478 to 1.11__
## What is the purpose of the change
Before FLINK-17218 there were two problems:
1. Exceeding `ChannelStateWriterImpl.maxCheckpoints`
1. Race condition with out of order barriers (see be8fbcf506b8dd38e5425cf772a55f033f0962b0 of this PR)
FLINK-17218 (24ff415f1b76392f75dea7c3538558d24fcb7058) fixed the first one but introduced a new race condition when netty thread modifies a ChannelStateWriter map used by the task thread.
This PR reverts FLINK-17218 and addresses the issues above.
## Brief change log
1. Revert 24ff415f1b76392f75dea7c3538558d24fcb7058 `"[FLINK-17218][checkpointing] Ensuring that ChannelStateWriter aborts previous checkpoints before a new checkpoint`
1. Abort channel state write if checkpoint is subsumed - using future callback and task thread - fix race condition
1. Increase `ChannelStateWriterImpl.DEFAULT_MAX_CHECKPOINTS` - fix the original problem of `FLINK-17218`
1. Ignore out of order checkpoints in `SubtaskCheckpointCoordinator` - fix a more subtle race condition
1. Unignore `UnalignedCheckpointITCase`
## Verifying this change
Unignore `UnalignedCheckpointITCase`.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on pull request #12550: [FLINK-17869][task][checkpointing] Fix race condition when calling ChannelStateWriter.abort (backport to 1.11)
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on pull request #12550:
URL: https://github.com/apache/flink/pull/12550#issuecomment-641893471
CI build failure is unrelated: FLINK-17159 (other tests succeeded).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhijiangW merged pull request #12550: [FLINK-17869][task][checkpointing] Fix race condition when calling ChannelStateWriter.abort (backport to 1.11)
Posted by GitBox <gi...@apache.org>.
zhijiangW merged pull request #12550:
URL: https://github.com/apache/flink/pull/12550
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12550: [FLINK-17869][task][checkpointing] Fix race condition when caling ChannelStateWriter.abort (backport to 1.11)
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12550:
URL: https://github.com/apache/flink/pull/12550#issuecomment-641276762
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7a7da939c802e59ea78a4cc77a0209facb34b92a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3062",
"triggerID" : "7a7da939c802e59ea78a4cc77a0209facb34b92a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7a7da939c802e59ea78a4cc77a0209facb34b92a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3062)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12550: [FLINK-17869][task][checkpointing] Fix race condition when calling ChannelStateWriter.abort (backport to 1.11)
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12550:
URL: https://github.com/apache/flink/pull/12550#issuecomment-641276762
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7a7da939c802e59ea78a4cc77a0209facb34b92a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3062",
"triggerID" : "7a7da939c802e59ea78a4cc77a0209facb34b92a",
"triggerType" : "PUSH"
}, {
"hash" : "0a032d4b2f7409a8b7343256c97906999b33cccd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3157",
"triggerID" : "0a032d4b2f7409a8b7343256c97906999b33cccd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7a7da939c802e59ea78a4cc77a0209facb34b92a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3062)
* 0a032d4b2f7409a8b7343256c97906999b33cccd Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3157)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12550: [FLINK-17869][task][checkpointing] Fix race condition when calling ChannelStateWriter.abort (backport to 1.11)
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12550:
URL: https://github.com/apache/flink/pull/12550#issuecomment-641276762
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7a7da939c802e59ea78a4cc77a0209facb34b92a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3062",
"triggerID" : "7a7da939c802e59ea78a4cc77a0209facb34b92a",
"triggerType" : "PUSH"
}, {
"hash" : "0a032d4b2f7409a8b7343256c97906999b33cccd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0a032d4b2f7409a8b7343256c97906999b33cccd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7a7da939c802e59ea78a4cc77a0209facb34b92a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3062)
* 0a032d4b2f7409a8b7343256c97906999b33cccd UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12550: [FLINK-17869][task][checkpointing] Fix race condition when caling ChannelStateWriter.abort (backport to 1.11)
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #12550:
URL: https://github.com/apache/flink/pull/12550#issuecomment-641264368
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on pull request #12550: [FLINK-17869][task][checkpointing] Fix race condition when calling ChannelStateWriter.abort (backport to 1.11)
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on pull request #12550:
URL: https://github.com/apache/flink/pull/12550#issuecomment-641958438
The latest CI build failure is also unrelated: FLINK-18239.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12550: [FLINK-17869][task][checkpointing] Fix race condition when calling ChannelStateWriter.abort (backport to 1.11)
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12550:
URL: https://github.com/apache/flink/pull/12550#issuecomment-641276762
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7a7da939c802e59ea78a4cc77a0209facb34b92a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3062",
"triggerID" : "7a7da939c802e59ea78a4cc77a0209facb34b92a",
"triggerType" : "PUSH"
}, {
"hash" : "0a032d4b2f7409a8b7343256c97906999b33cccd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3157",
"triggerID" : "0a032d4b2f7409a8b7343256c97906999b33cccd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0a032d4b2f7409a8b7343256c97906999b33cccd Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3157)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12550: [FLINK-17869][task][checkpointing] Fix race condition when caling ChannelStateWriter.abort (backport to 1.11)
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12550:
URL: https://github.com/apache/flink/pull/12550#issuecomment-641276762
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7a7da939c802e59ea78a4cc77a0209facb34b92a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3062",
"triggerID" : "7a7da939c802e59ea78a4cc77a0209facb34b92a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7a7da939c802e59ea78a4cc77a0209facb34b92a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3062)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhijiangW commented on pull request #12550: [FLINK-17869][task][checkpointing] Fix race condition when calling ChannelStateWriter.abort (backport to 1.11)
Posted by GitBox <gi...@apache.org>.
zhijiangW commented on pull request #12550:
URL: https://github.com/apache/flink/pull/12550#issuecomment-642108719
The azure e2e failure seems unrelated and report in [FLINK-16821](https://issues.apache.org/jira/browse/FLINK-16821), merging
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org