You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/01/25 16:50:01 UTC
[GitHub] [flink] gaoyunhaii opened a new pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
gaoyunhaii opened a new pull request #14754:
URL: https://github.com/apache/flink/pull/14754
## What is the purpose of the change
Stores the information of whether an operator is fully finished into the checkpoint metadata.
To not have to introduce a new version of checkpoint metadata and due to a fully finished operator state always has an empty states, we could stores a special content for the operator state to indicate that the operator is fully finished.
## Brief change log
- 37484933592e8b5a4f5cf83425ac0b7b612c7480 allows set / get fully finished flag for operator state, and marked fully finished operators according to the checkpoint brief computed.
## Verifying this change
This change added unit tests about set and get fully finished flag, and serialization / deserialization of the operator states marked as fully finished.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): **no**
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **no**
- The serializers: **no**
- The runtime per-record code paths (performance sensitive): **no**
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: **no**
- The S3 file system connector: **no**
## Documentation
- Does this pull request introduce a new feature? **no**
- If yes, how is the feature documented? **not applicable**
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-797303630
Hi Roman @rkhachatryan , very thanks for the review! I updated the PR to address the comments. And very sorry for not pay attention to the result... Now I fixed the issue via adding the `serialVersionUID`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-766965862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12479",
"triggerID" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12695",
"triggerID" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"triggerType" : "PUSH"
}, {
"hash" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12858",
"triggerID" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"triggerType" : "PUSH"
}, {
"hash" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12905",
"triggerID" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"triggerType" : "PUSH"
}, {
"hash" : "0f8924698701b1d6768f8757414d69836b83fa3a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0f8924698701b1d6768f8757414d69836b83fa3a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 13548fbb8d09727b172ebdf26a90c56d409fa1f2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12905)
* 0f8924698701b1d6768f8757414d69836b83fa3a UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #14754:
URL: https://github.com/apache/flink/pull/14754#discussion_r592232907
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/FullyFinishedOperatorState.java
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint;
+
+import org.apache.flink.runtime.jobgraph.OperatorID;
+
+/**
+ * A special operator state implementation representing the operators whose instances are all
+ * finished.
+ */
+public class FullyFinishedOperatorState extends OperatorState {
Review comment:
Should this class also override `equals`/`hashCode`?
And why do we allow to set coordinator state (i.e. don't override like `putState`)?
Can finished vertex have coordinator state?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/metadata/MetadataV3Serializer.java
##########
@@ -103,11 +105,16 @@ protected void serializeOperatorState(OperatorState operatorState, DataOutputStr
serializeStreamStateHandle(operatorState.getCoordinatorState(), dos);
// Sub task states
- final Map<Integer, OperatorSubtaskState> subtaskStateMap = operatorState.getSubtaskStates();
- dos.writeInt(subtaskStateMap.size());
- for (Map.Entry<Integer, OperatorSubtaskState> entry : subtaskStateMap.entrySet()) {
- dos.writeInt(entry.getKey());
- serializeSubtaskState(entry.getValue(), dos);
+ if (operatorState.isFullyFinished()) {
+ dos.writeInt(-1);
Review comment:
nit: to me, explicitly writing `isFullyFinished` would be more clear and less error-prone:
```
dos.writeBoolean(operatorState.isFullyFinished());
dos.writeInt(subtaskStateMap.size()); // zero
// empty loop
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-766965862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12479",
"triggerID" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12695",
"triggerID" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"triggerType" : "PUSH"
}, {
"hash" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12858",
"triggerID" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"triggerType" : "PUSH"
}, {
"hash" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12905",
"triggerID" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"triggerType" : "PUSH"
}, {
"hash" : "0f8924698701b1d6768f8757414d69836b83fa3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13084",
"triggerID" : "0f8924698701b1d6768f8757414d69836b83fa3a",
"triggerType" : "PUSH"
}, {
"hash" : "00bfeff9764880852aff536535e3fb8878bd2e74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13982",
"triggerID" : "00bfeff9764880852aff536535e3fb8878bd2e74",
"triggerType" : "PUSH"
}, {
"hash" : "574e29c9ba5c4a144b2ad6029c96c3fa2af43647",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14198",
"triggerID" : "574e29c9ba5c4a144b2ad6029c96c3fa2af43647",
"triggerType" : "PUSH"
}, {
"hash" : "5f3ace5f40d1ae73f6c4407d8a2e131f11829b50",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14528",
"triggerID" : "5f3ace5f40d1ae73f6c4407d8a2e131f11829b50",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f3ace5f40d1ae73f6c4407d8a2e131f11829b50 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14528)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #14754:
URL: https://github.com/apache/flink/pull/14754#discussion_r572102215
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorState.java
##########
@@ -81,10 +93,22 @@ public OperatorID getOperatorID() {
return operatorID;
}
+ public boolean isFullyFinished() {
+ return operatorSubtaskStates.equals(FULLY_FINISHED_STATE);
+ }
+
+ public void markedFullyFinished() {
+ operatorSubtaskStates.clear();
+ FULLY_FINISHED_STATE.forEach(operatorSubtaskStates::put);
+ }
+
public void putState(int subtaskIndex, OperatorSubtaskState subtaskState) {
Preconditions.checkNotNull(subtaskState);
+ checkState(!isFullyFinished());
- if (subtaskIndex < 0 || subtaskIndex >= parallelism) {
+ if (subtaskIndex == FULLY_FINISHED_PLACEHOLDER_INDEX) {
+ markedFullyFinished();
Review comment:
Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-766965862
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #14754:
URL: https://github.com/apache/flink/pull/14754#discussion_r592242971
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/PendingCheckpoint.java
##########
@@ -308,6 +316,25 @@ public CompletedCheckpoint finalizeCheckpoint(
// make sure we fulfill the promise with an exception if something fails
try {
+ // Completes the operator state for the fully finished operators
+ for (ExecutionJobVertex jobVertex : checkpointBrief.getFullyFinishedJobVertex()) {
+ for (OperatorIDPair operatorID : jobVertex.getOperatorIDs()) {
+ OperatorState operatorState =
+ operatorStates.get(operatorID.getGeneratedOperatorID());
+
+ if (operatorState == null) {
+ operatorState =
+ new OperatorState(
+ operatorID.getGeneratedOperatorID(),
+ jobVertex.getParallelism(),
+ jobVertex.getMaxParallelism());
+ operatorStates.put(operatorID.getGeneratedOperatorID(), operatorState);
+ }
+
+ operatorState.markedFullyFinished();
Review comment:
I think the check is still missing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-766965862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12479",
"triggerID" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12695",
"triggerID" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7a0a38b9960949cedae60f4574edc8382841c8ac Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12695)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-766965862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12479",
"triggerID" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12695",
"triggerID" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"triggerType" : "PUSH"
}, {
"hash" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12858",
"triggerID" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"triggerType" : "PUSH"
}, {
"hash" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12905",
"triggerID" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"triggerType" : "PUSH"
}, {
"hash" : "0f8924698701b1d6768f8757414d69836b83fa3a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13084",
"triggerID" : "0f8924698701b1d6768f8757414d69836b83fa3a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 13548fbb8d09727b172ebdf26a90c56d409fa1f2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12905)
* 0f8924698701b1d6768f8757414d69836b83fa3a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13084)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-766965862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12479",
"triggerID" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12695",
"triggerID" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"triggerType" : "PUSH"
}, {
"hash" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12858",
"triggerID" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"triggerType" : "PUSH"
}, {
"hash" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12905",
"triggerID" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fbe576aeefa4712cc9aaeb856cab8afab85668e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12858)
* 13548fbb8d09727b172ebdf26a90c56d409fa1f2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12905)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-766965862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12479",
"triggerID" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 37484933592e8b5a4f5cf83425ac0b7b612c7480 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12479)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on a change in pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #14754:
URL: https://github.com/apache/flink/pull/14754#discussion_r571815521
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/PendingCheckpoint.java
##########
@@ -308,6 +316,25 @@ public CompletedCheckpoint finalizeCheckpoint(
// make sure we fulfill the promise with an exception if something fails
try {
+ // Completes the operator state for the fully finished operators
+ for (ExecutionJobVertex jobVertex : checkpointBrief.getFullyFinishedJobVertex()) {
+ for (OperatorIDPair operatorID : jobVertex.getOperatorIDs()) {
+ OperatorState operatorState =
+ operatorStates.get(operatorID.getGeneratedOperatorID());
+
+ if (operatorState == null) {
+ operatorState =
+ new OperatorState(
+ operatorID.getGeneratedOperatorID(),
+ jobVertex.getParallelism(),
+ jobVertex.getMaxParallelism());
+ operatorStates.put(operatorID.getGeneratedOperatorID(), operatorState);
+ }
+
+ operatorState.markedFullyFinished();
Review comment:
It should be always null, I add the check for it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys closed pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
dawidwys closed pull request #14754:
URL: https://github.com/apache/flink/pull/14754
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-766954030
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aimiebell commented on a change in pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
aimiebell commented on a change in pull request #14754:
URL: https://github.com/apache/flink/pull/14754#discussion_r572622956
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ExecutionGraph.java
##########
@@ -1095,7 +1084,7 @@ public void initFailureCause(Throwable t) {
*/
void vertexFinished() {
assertRunningInJobMasterMainThread();
- final int numFinished = ++verticesFinished;
+ final int numFinished = ++numFinishedVertices;
Review comment:
I detect that this code is problematic. According to the [Correctness (CORRECTNESS)](https://spotbugs.readthedocs.io/en/stable/bugDescriptions.html#correctness-correctness), [DLS: Overwritten increment (DLS_OVERWRITTEN_INCREMENT)](https://spotbugs.readthedocs.io/en/stable/bugDescriptions.html#dls-overwritten-increment-dls-overwritten-increment).
The code performs an increment operation (e.g., i++) and then immediately overwrites it. For example, i = i++ immediately overwrites the incremented value with the original value.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-766965862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12479",
"triggerID" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12695",
"triggerID" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"triggerType" : "PUSH"
}, {
"hash" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12858",
"triggerID" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"triggerType" : "PUSH"
}, {
"hash" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12905",
"triggerID" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"triggerType" : "PUSH"
}, {
"hash" : "0f8924698701b1d6768f8757414d69836b83fa3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13084",
"triggerID" : "0f8924698701b1d6768f8757414d69836b83fa3a",
"triggerType" : "PUSH"
}, {
"hash" : "00bfeff9764880852aff536535e3fb8878bd2e74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13982",
"triggerID" : "00bfeff9764880852aff536535e3fb8878bd2e74",
"triggerType" : "PUSH"
}, {
"hash" : "574e29c9ba5c4a144b2ad6029c96c3fa2af43647",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14198",
"triggerID" : "574e29c9ba5c4a144b2ad6029c96c3fa2af43647",
"triggerType" : "PUSH"
}, {
"hash" : "5f3ace5f40d1ae73f6c4407d8a2e131f11829b50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14528",
"triggerID" : "5f3ace5f40d1ae73f6c4407d8a2e131f11829b50",
"triggerType" : "PUSH"
}, {
"hash" : "c11030faa9d70bd6b4cc3d1d8f15ea1db2641159",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18432",
"triggerID" : "c11030faa9d70bd6b4cc3d1d8f15ea1db2641159",
"triggerType" : "PUSH"
}, {
"hash" : "05bed4739dfdc1218bd039214e500d3d52a8c78a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "05bed4739dfdc1218bd039214e500d3d52a8c78a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c11030faa9d70bd6b4cc3d1d8f15ea1db2641159 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18432)
* 05bed4739dfdc1218bd039214e500d3d52a8c78a UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-791403576
Very thanks @rkhachatryan for the reviewing and very sorry for missing the commits again... I update the PR according to the comments and also include the previous ones.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-766965862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12479",
"triggerID" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12695",
"triggerID" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"triggerType" : "PUSH"
}, {
"hash" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12858",
"triggerID" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fbe576aeefa4712cc9aaeb856cab8afab85668e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12858)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-775200104
Hi @gaoyunhaii , thanks for clarifying and updating the PR.
It LGTM in general, I'd make a final pass after all conflicts resolved.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-788728013
Hi Roman @rkhachatryan , I rebased this PR to the latest master, and also have some changes in the `get*` methods of the `OperatorState` to avoid the finished flag affects the normal processes. Could you have another look~? Very thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on a change in pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #14754:
URL: https://github.com/apache/flink/pull/14754#discussion_r643141598
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/metadata/MetadataV2Serializer.java
##########
@@ -71,6 +73,10 @@ public CheckpointMetadata deserialize(
@Override
protected void serializeOperatorState(OperatorState operatorState, DataOutputStream dos)
throws IOException {
+ checkState(
+ !operatorState.isFullyFinished(),
+ "Could not support finished Operator state in stale serializers.");
Review comment:
```suggestion
"Could not support finished Operator state in state serializers.");
```
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/PendingCheckpointTest.java
##########
@@ -544,6 +546,69 @@ public void testDisposeDisposesCoordinatorStates() throws Exception {
assertTrue(handle2.isDisposed());
}
+ @Test
+ public void testFinalizeCheckpointWithFullyFinishedOperators() throws Exception {
+ JobVertexID finishedJobVertexID = new JobVertexID();
+ JobVertexID runningJobVertexID = new JobVertexID();
+ OperatorID finishedOperatorID = new OperatorID();
+ OperatorID runningOperatorID = new OperatorID();
+
+ ExecutionGraph executionGraph =
+ new CheckpointCoordinatorTestingUtils.CheckpointExecutionGraphBuilder()
+ .addJobVertex(
+ finishedJobVertexID,
+ 1,
+ 256,
+ Collections.singletonList(
+ OperatorIDPair.generatedIDOnly(finishedOperatorID)),
+ true)
+ .addJobVertex(
+ runningJobVertexID,
+ 1,
+ 256,
+ Collections.singletonList(
+ OperatorIDPair.generatedIDOnly(runningOperatorID)),
+ true)
+ .build();
+ executionGraph
+ .getJobVertex(finishedJobVertexID)
+ .getTaskVertices()[0]
+ .getCurrentExecutionAttempt()
+ .markFinished();
+ PendingCheckpoint pendingCheckpoint = createPendingCheckpoint(executionGraph);
+ assertEquals(1, pendingCheckpoint.getCheckpointPlan().getFullyFinishedJobVertex().size());
+ assertEquals(
+ finishedJobVertexID,
+ pendingCheckpoint
+ .getCheckpointPlan()
+ .getFullyFinishedJobVertex()
+ .get(0)
+ .getJobVertexId());
+
+ // Report the state for the running operator
+ ExecutionAttemptID runningTaskId =
+ executionGraph
+ .getJobVertex(runningJobVertexID)
+ .getTaskVertices()[0]
+ .getCurrentExecutionAttempt()
+ .getAttemptId();
+ TaskStateSnapshot taskStateSnapshot = new TaskStateSnapshot();
+ taskStateSnapshot.putSubtaskStateByOperatorID(
+ runningOperatorID, new OperatorSubtaskState());
+ TaskAcknowledgeResult result =
+ pendingCheckpoint.acknowledgeTask(
+ runningTaskId, taskStateSnapshot, new CheckpointMetrics(), null);
+ assertEquals(TaskAcknowledgeResult.SUCCESS, result);
Review comment:
nit: We prefer `assertThat` as it gives a nicer description in case of a failure.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/metadata/MetadataV3Serializer.java
##########
@@ -103,11 +105,16 @@ protected void serializeOperatorState(OperatorState operatorState, DataOutputStr
serializeStreamStateHandle(operatorState.getCoordinatorState(), dos);
// Sub task states
- final Map<Integer, OperatorSubtaskState> subtaskStateMap = operatorState.getSubtaskStates();
- dos.writeInt(subtaskStateMap.size());
- for (Map.Entry<Integer, OperatorSubtaskState> entry : subtaskStateMap.entrySet()) {
- dos.writeInt(entry.getKey());
- serializeSubtaskState(entry.getValue(), dos);
+ if (operatorState.isFullyFinished()) {
+ dos.writeInt(-1);
Review comment:
I see @gaoyunhaii concerns. I'd be fine with using `-1` here.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/FullyFinishedOperatorState.java
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint;
+
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.state.memory.ByteStreamStateHandle;
+
+import javax.annotation.Nullable;
+
+/**
+ * A special operator state implementation representing the operators whose instances are all
+ * finished.
+ */
+public class FullyFinishedOperatorState extends OperatorState {
+
+ private static final long serialVersionUID = -7094972830573632176L;
Review comment:
Please use `1L` for `serialVersionUID` as recommended in our coding guidelines: https://flink.apache.org/contributing/code-style-and-quality-java.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-766965862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12479",
"triggerID" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12695",
"triggerID" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"triggerType" : "PUSH"
}, {
"hash" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12858",
"triggerID" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"triggerType" : "PUSH"
}, {
"hash" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12905",
"triggerID" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"triggerType" : "PUSH"
}, {
"hash" : "0f8924698701b1d6768f8757414d69836b83fa3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13084",
"triggerID" : "0f8924698701b1d6768f8757414d69836b83fa3a",
"triggerType" : "PUSH"
}, {
"hash" : "00bfeff9764880852aff536535e3fb8878bd2e74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13982",
"triggerID" : "00bfeff9764880852aff536535e3fb8878bd2e74",
"triggerType" : "PUSH"
}, {
"hash" : "574e29c9ba5c4a144b2ad6029c96c3fa2af43647",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14198",
"triggerID" : "574e29c9ba5c4a144b2ad6029c96c3fa2af43647",
"triggerType" : "PUSH"
}, {
"hash" : "5f3ace5f40d1ae73f6c4407d8a2e131f11829b50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f3ace5f40d1ae73f6c4407d8a2e131f11829b50",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 574e29c9ba5c4a144b2ad6029c96c3fa2af43647 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14198)
* 5f3ace5f40d1ae73f6c4407d8a2e131f11829b50 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-766954030
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 37484933592e8b5a4f5cf83425ac0b7b612c7480 (Mon Jan 25 16:51:57 UTC 2021)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
* **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-21127).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work.
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
dawidwys commented on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-853767111
Thanks for the update. Merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-766965862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12479",
"triggerID" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12695",
"triggerID" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"triggerType" : "PUSH"
}, {
"hash" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12858",
"triggerID" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"triggerType" : "PUSH"
}, {
"hash" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12905",
"triggerID" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"triggerType" : "PUSH"
}, {
"hash" : "0f8924698701b1d6768f8757414d69836b83fa3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13084",
"triggerID" : "0f8924698701b1d6768f8757414d69836b83fa3a",
"triggerType" : "PUSH"
}, {
"hash" : "00bfeff9764880852aff536535e3fb8878bd2e74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13982",
"triggerID" : "00bfeff9764880852aff536535e3fb8878bd2e74",
"triggerType" : "PUSH"
}, {
"hash" : "574e29c9ba5c4a144b2ad6029c96c3fa2af43647",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14198",
"triggerID" : "574e29c9ba5c4a144b2ad6029c96c3fa2af43647",
"triggerType" : "PUSH"
}, {
"hash" : "5f3ace5f40d1ae73f6c4407d8a2e131f11829b50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14528",
"triggerID" : "5f3ace5f40d1ae73f6c4407d8a2e131f11829b50",
"triggerType" : "PUSH"
}, {
"hash" : "c11030faa9d70bd6b4cc3d1d8f15ea1db2641159",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18432",
"triggerID" : "c11030faa9d70bd6b4cc3d1d8f15ea1db2641159",
"triggerType" : "PUSH"
}, {
"hash" : "05bed4739dfdc1218bd039214e500d3d52a8c78a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18523",
"triggerID" : "05bed4739dfdc1218bd039214e500d3d52a8c78a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 05bed4739dfdc1218bd039214e500d3d52a8c78a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18523)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on a change in pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #14754:
URL: https://github.com/apache/flink/pull/14754#discussion_r592968789
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/metadata/MetadataV3Serializer.java
##########
@@ -103,11 +105,16 @@ protected void serializeOperatorState(OperatorState operatorState, DataOutputStr
serializeStreamStateHandle(operatorState.getCoordinatorState(), dos);
// Sub task states
- final Map<Integer, OperatorSubtaskState> subtaskStateMap = operatorState.getSubtaskStates();
- dos.writeInt(subtaskStateMap.size());
- for (Map.Entry<Integer, OperatorSubtaskState> entry : subtaskStateMap.entrySet()) {
- dos.writeInt(entry.getKey());
- serializeSubtaskState(entry.getValue(), dos);
+ if (operatorState.isFullyFinished()) {
+ dos.writeInt(-1);
Review comment:
I'm a bit hesitate here since we should not be able to add new field to the meta data since it would impact the compatibility. There would be the cases that user taking a savepoint or external checkpoint with an older version (like 1.12) and resume the savepoint/checkpoint with new version (like the 1.13) to do the upgrade. If we add new fields, we should not e able to parse the existing savepoint / checkpoint ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-766965862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12479",
"triggerID" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12695",
"triggerID" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"triggerType" : "PUSH"
}, {
"hash" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12858",
"triggerID" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"triggerType" : "PUSH"
}, {
"hash" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12905",
"triggerID" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"triggerType" : "PUSH"
}, {
"hash" : "0f8924698701b1d6768f8757414d69836b83fa3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13084",
"triggerID" : "0f8924698701b1d6768f8757414d69836b83fa3a",
"triggerType" : "PUSH"
}, {
"hash" : "00bfeff9764880852aff536535e3fb8878bd2e74",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13982",
"triggerID" : "00bfeff9764880852aff536535e3fb8878bd2e74",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 00bfeff9764880852aff536535e3fb8878bd2e74 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13982)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-766965862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12479",
"triggerID" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12695",
"triggerID" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"triggerType" : "PUSH"
}, {
"hash" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12858",
"triggerID" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"triggerType" : "PUSH"
}, {
"hash" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12905",
"triggerID" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"triggerType" : "PUSH"
}, {
"hash" : "0f8924698701b1d6768f8757414d69836b83fa3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13084",
"triggerID" : "0f8924698701b1d6768f8757414d69836b83fa3a",
"triggerType" : "PUSH"
}, {
"hash" : "00bfeff9764880852aff536535e3fb8878bd2e74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13982",
"triggerID" : "00bfeff9764880852aff536535e3fb8878bd2e74",
"triggerType" : "PUSH"
}, {
"hash" : "574e29c9ba5c4a144b2ad6029c96c3fa2af43647",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14198",
"triggerID" : "574e29c9ba5c4a144b2ad6029c96c3fa2af43647",
"triggerType" : "PUSH"
}, {
"hash" : "5f3ace5f40d1ae73f6c4407d8a2e131f11829b50",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14528",
"triggerID" : "5f3ace5f40d1ae73f6c4407d8a2e131f11829b50",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 574e29c9ba5c4a144b2ad6029c96c3fa2af43647 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14198)
* 5f3ace5f40d1ae73f6c4407d8a2e131f11829b50 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14528)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on a change in pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #14754:
URL: https://github.com/apache/flink/pull/14754#discussion_r588272831
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorState.java
##########
@@ -118,14 +142,30 @@ public ByteStreamStateHandle getCoordinatorState() {
}
public Map<Integer, OperatorSubtaskState> getSubtaskStates() {
+ return getSubtaskStates(false);
+ }
+
+ public Map<Integer, OperatorSubtaskState> getSubtaskStates(boolean includeFinishedFlag) {
+ if (!includeFinishedFlag && isFullyFinished()) {
+ return Collections.emptyMap();
+ }
+
return Collections.unmodifiableMap(operatorSubtaskStates);
}
public Collection<OperatorSubtaskState> getStates() {
+ if (isFullyFinished()) {
+ return Collections.emptySet();
+ }
Review comment:
I think it would indeed make it more clear and not worry about consistency between methods. I updated the PR with this method. Very thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-766965862
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #14754:
URL: https://github.com/apache/flink/pull/14754#discussion_r569721814
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorState.java
##########
@@ -81,10 +93,22 @@ public OperatorID getOperatorID() {
return operatorID;
}
+ public boolean isFullyFinished() {
+ return operatorSubtaskStates.equals(FULLY_FINISHED_STATE);
+ }
+
+ public void markedFullyFinished() {
+ operatorSubtaskStates.clear();
+ FULLY_FINISHED_STATE.forEach(operatorSubtaskStates::put);
+ }
+
public void putState(int subtaskIndex, OperatorSubtaskState subtaskState) {
Preconditions.checkNotNull(subtaskState);
+ checkState(!isFullyFinished());
- if (subtaskIndex < 0 || subtaskIndex >= parallelism) {
+ if (subtaskIndex == FULLY_FINISHED_PLACEHOLDER_INDEX) {
+ markedFullyFinished();
Review comment:
Just to clarify, this branch is for deserialization, right?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/PendingCheckpoint.java
##########
@@ -308,6 +316,25 @@ public CompletedCheckpoint finalizeCheckpoint(
// make sure we fulfill the promise with an exception if something fails
try {
+ // Completes the operator state for the fully finished operators
+ for (ExecutionJobVertex jobVertex : checkpointBrief.getFullyFinishedJobVertex()) {
+ for (OperatorIDPair operatorID : jobVertex.getOperatorIDs()) {
+ OperatorState operatorState =
+ operatorStates.get(operatorID.getGeneratedOperatorID());
+
+ if (operatorState == null) {
+ operatorState =
+ new OperatorState(
+ operatorID.getGeneratedOperatorID(),
+ jobVertex.getParallelism(),
+ jobVertex.getMaxParallelism());
+ operatorStates.put(operatorID.getGeneratedOperatorID(), operatorState);
+ }
+
+ operatorState.markedFullyFinished();
Review comment:
IIUC, the actual state should not be present for finished operators for this checkpoint, right?
Can we add a check for this?
Otherwise, we need to discard previous states.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorState.java
##########
@@ -41,6 +41,18 @@
*/
public class OperatorState implements CompositeStateHandle {
+ private static final int FULLY_FINISHED_PLACEHOLDER_INDEX = -1;
+
+ /** A special content that marks this operator as fully finished. */
+ private static final Map<Integer, OperatorSubtaskState> FULLY_FINISHED_STATE =
+ new HashMap<Integer, OperatorSubtaskState>() {
+ {
+ this.put(
+ FULLY_FINISHED_PLACEHOLDER_INDEX,
+ OperatorSubtaskState.builder().build());
+ }
+ };
Review comment:
Why not `Collections.singletonMap(FULLY_FINISHED_PLACEHOLDER_INDEX, OperatorSubtaskState.builder().build()));`?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/PendingCheckpoint.java
##########
@@ -308,6 +316,25 @@ public CompletedCheckpoint finalizeCheckpoint(
// make sure we fulfill the promise with an exception if something fails
try {
+ // Completes the operator state for the fully finished operators
+ for (ExecutionJobVertex jobVertex : checkpointBrief.getFullyFinishedJobVertex()) {
Review comment:
nit: extract method?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-766965862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12479",
"triggerID" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12695",
"triggerID" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"triggerType" : "PUSH"
}, {
"hash" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12858",
"triggerID" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"triggerType" : "PUSH"
}, {
"hash" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12905",
"triggerID" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 13548fbb8d09727b172ebdf26a90c56d409fa1f2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12905)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-766965862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12479",
"triggerID" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12695",
"triggerID" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"triggerType" : "PUSH"
}, {
"hash" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12858",
"triggerID" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"triggerType" : "PUSH"
}, {
"hash" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12905",
"triggerID" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"triggerType" : "PUSH"
}, {
"hash" : "0f8924698701b1d6768f8757414d69836b83fa3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13084",
"triggerID" : "0f8924698701b1d6768f8757414d69836b83fa3a",
"triggerType" : "PUSH"
}, {
"hash" : "00bfeff9764880852aff536535e3fb8878bd2e74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13982",
"triggerID" : "00bfeff9764880852aff536535e3fb8878bd2e74",
"triggerType" : "PUSH"
}, {
"hash" : "574e29c9ba5c4a144b2ad6029c96c3fa2af43647",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14198",
"triggerID" : "574e29c9ba5c4a144b2ad6029c96c3fa2af43647",
"triggerType" : "PUSH"
}, {
"hash" : "5f3ace5f40d1ae73f6c4407d8a2e131f11829b50",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14528",
"triggerID" : "5f3ace5f40d1ae73f6c4407d8a2e131f11829b50",
"triggerType" : "PUSH"
}, {
"hash" : "c11030faa9d70bd6b4cc3d1d8f15ea1db2641159",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18432",
"triggerID" : "c11030faa9d70bd6b4cc3d1d8f15ea1db2641159",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f3ace5f40d1ae73f6c4407d8a2e131f11829b50 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14528)
* c11030faa9d70bd6b4cc3d1d8f15ea1db2641159 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18432)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on a change in pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #14754:
URL: https://github.com/apache/flink/pull/14754#discussion_r592967766
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/FullyFinishedOperatorState.java
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint;
+
+import org.apache.flink.runtime.jobgraph.OperatorID;
+
+/**
+ * A special operator state implementation representing the operators whose instances are all
+ * finished.
+ */
+public class FullyFinishedOperatorState extends OperatorState {
Review comment:
We should indeed need to override equals / hashcode so we could distinguish the finished operator and the empty operator state. I added these methods.
Regarding coordinator state, currently I also do not have specific examples that keep coordinator state after all the operators are finished. For fully finished operators, we would directly skip their execution after failover, thus all its state should not take effect. Now I also disabled setting the coordinator state. Very thanks for pointing out these issues !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-766965862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12479",
"triggerID" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12695",
"triggerID" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"triggerType" : "PUSH"
}, {
"hash" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12858",
"triggerID" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"triggerType" : "PUSH"
}, {
"hash" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12905",
"triggerID" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"triggerType" : "PUSH"
}, {
"hash" : "0f8924698701b1d6768f8757414d69836b83fa3a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13084",
"triggerID" : "0f8924698701b1d6768f8757414d69836b83fa3a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0f8924698701b1d6768f8757414d69836b83fa3a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13084)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-766965862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12479",
"triggerID" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12695",
"triggerID" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"triggerType" : "PUSH"
}, {
"hash" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12858",
"triggerID" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"triggerType" : "PUSH"
}, {
"hash" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12905",
"triggerID" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"triggerType" : "PUSH"
}, {
"hash" : "0f8924698701b1d6768f8757414d69836b83fa3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13084",
"triggerID" : "0f8924698701b1d6768f8757414d69836b83fa3a",
"triggerType" : "PUSH"
}, {
"hash" : "00bfeff9764880852aff536535e3fb8878bd2e74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13982",
"triggerID" : "00bfeff9764880852aff536535e3fb8878bd2e74",
"triggerType" : "PUSH"
}, {
"hash" : "574e29c9ba5c4a144b2ad6029c96c3fa2af43647",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14198",
"triggerID" : "574e29c9ba5c4a144b2ad6029c96c3fa2af43647",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 574e29c9ba5c4a144b2ad6029c96c3fa2af43647 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14198)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #14754:
URL: https://github.com/apache/flink/pull/14754#discussion_r587599988
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorState.java
##########
@@ -118,14 +142,30 @@ public ByteStreamStateHandle getCoordinatorState() {
}
public Map<Integer, OperatorSubtaskState> getSubtaskStates() {
+ return getSubtaskStates(false);
+ }
+
+ public Map<Integer, OperatorSubtaskState> getSubtaskStates(boolean includeFinishedFlag) {
+ if (!includeFinishedFlag && isFullyFinished()) {
+ return Collections.emptyMap();
+ }
+
return Collections.unmodifiableMap(operatorSubtaskStates);
}
public Collection<OperatorSubtaskState> getStates() {
+ if (isFullyFinished()) {
+ return Collections.emptySet();
+ }
Review comment:
I'm not sure that this check is needed. If `isFullyFinished` then `subtaskStates` should be empty anyways. So continuing wouldn't hurt but the check makes me feel that it's **necessary** in **all** methods.
ditto: other methods
Probably a subclass `FinishedOperatorState` with always empty state would make it easier and less error-prone, WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on a change in pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #14754:
URL: https://github.com/apache/flink/pull/14754#discussion_r571816119
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorState.java
##########
@@ -81,10 +93,22 @@ public OperatorID getOperatorID() {
return operatorID;
}
+ public boolean isFullyFinished() {
+ return operatorSubtaskStates.equals(FULLY_FINISHED_STATE);
+ }
+
+ public void markedFullyFinished() {
+ operatorSubtaskStates.clear();
+ FULLY_FINISHED_STATE.forEach(operatorSubtaskStates::put);
+ }
+
public void putState(int subtaskIndex, OperatorSubtaskState subtaskState) {
Preconditions.checkNotNull(subtaskState);
+ checkState(!isFullyFinished());
- if (subtaskIndex < 0 || subtaskIndex >= parallelism) {
+ if (subtaskIndex == FULLY_FINISHED_PLACEHOLDER_INDEX) {
+ markedFullyFinished();
Review comment:
Yes, this branch is for deserialization. I added comments to this branch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-766954030
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 0f8924698701b1d6768f8757414d69836b83fa3a (Fri Feb 19 07:33:07 UTC 2021)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
* **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-21127).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work.
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-766965862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12479",
"triggerID" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 37484933592e8b5a4f5cf83425ac0b7b612c7480 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12479)
* 7a0a38b9960949cedae60f4574edc8382841c8ac UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-766965862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12479",
"triggerID" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12695",
"triggerID" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"triggerType" : "PUSH"
}, {
"hash" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12858",
"triggerID" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"triggerType" : "PUSH"
}, {
"hash" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fbe576aeefa4712cc9aaeb856cab8afab85668e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12858)
* 13548fbb8d09727b172ebdf26a90c56d409fa1f2 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-766965862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12479",
"triggerID" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12695",
"triggerID" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"triggerType" : "PUSH"
}, {
"hash" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12858",
"triggerID" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"triggerType" : "PUSH"
}, {
"hash" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12905",
"triggerID" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"triggerType" : "PUSH"
}, {
"hash" : "0f8924698701b1d6768f8757414d69836b83fa3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13084",
"triggerID" : "0f8924698701b1d6768f8757414d69836b83fa3a",
"triggerType" : "PUSH"
}, {
"hash" : "00bfeff9764880852aff536535e3fb8878bd2e74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13982",
"triggerID" : "00bfeff9764880852aff536535e3fb8878bd2e74",
"triggerType" : "PUSH"
}, {
"hash" : "574e29c9ba5c4a144b2ad6029c96c3fa2af43647",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14198",
"triggerID" : "574e29c9ba5c4a144b2ad6029c96c3fa2af43647",
"triggerType" : "PUSH"
}, {
"hash" : "5f3ace5f40d1ae73f6c4407d8a2e131f11829b50",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14528",
"triggerID" : "5f3ace5f40d1ae73f6c4407d8a2e131f11829b50",
"triggerType" : "PUSH"
}, {
"hash" : "c11030faa9d70bd6b4cc3d1d8f15ea1db2641159",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c11030faa9d70bd6b4cc3d1d8f15ea1db2641159",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f3ace5f40d1ae73f6c4407d8a2e131f11829b50 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14528)
* c11030faa9d70bd6b4cc3d1d8f15ea1db2641159 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys closed pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
dawidwys closed pull request #14754:
URL: https://github.com/apache/flink/pull/14754
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-766954030
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 05bed4739dfdc1218bd039214e500d3d52a8c78a (Thu Sep 23 17:21:35 UTC 2021)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-766965862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12479",
"triggerID" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 37484933592e8b5a4f5cf83425ac0b7b612c7480 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12479)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-766965862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12479",
"triggerID" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12695",
"triggerID" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"triggerType" : "PUSH"
}, {
"hash" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12858",
"triggerID" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"triggerType" : "PUSH"
}, {
"hash" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12905",
"triggerID" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"triggerType" : "PUSH"
}, {
"hash" : "0f8924698701b1d6768f8757414d69836b83fa3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13084",
"triggerID" : "0f8924698701b1d6768f8757414d69836b83fa3a",
"triggerType" : "PUSH"
}, {
"hash" : "00bfeff9764880852aff536535e3fb8878bd2e74",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13982",
"triggerID" : "00bfeff9764880852aff536535e3fb8878bd2e74",
"triggerType" : "PUSH"
}, {
"hash" : "574e29c9ba5c4a144b2ad6029c96c3fa2af43647",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14198",
"triggerID" : "574e29c9ba5c4a144b2ad6029c96c3fa2af43647",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 00bfeff9764880852aff536535e3fb8878bd2e74 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13982)
* 574e29c9ba5c4a144b2ad6029c96c3fa2af43647 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14198)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-766965862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12479",
"triggerID" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12695",
"triggerID" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"triggerType" : "PUSH"
}, {
"hash" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12858",
"triggerID" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"triggerType" : "PUSH"
}, {
"hash" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12905",
"triggerID" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"triggerType" : "PUSH"
}, {
"hash" : "0f8924698701b1d6768f8757414d69836b83fa3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13084",
"triggerID" : "0f8924698701b1d6768f8757414d69836b83fa3a",
"triggerType" : "PUSH"
}, {
"hash" : "00bfeff9764880852aff536535e3fb8878bd2e74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13982",
"triggerID" : "00bfeff9764880852aff536535e3fb8878bd2e74",
"triggerType" : "PUSH"
}, {
"hash" : "574e29c9ba5c4a144b2ad6029c96c3fa2af43647",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14198",
"triggerID" : "574e29c9ba5c4a144b2ad6029c96c3fa2af43647",
"triggerType" : "PUSH"
}, {
"hash" : "5f3ace5f40d1ae73f6c4407d8a2e131f11829b50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14528",
"triggerID" : "5f3ace5f40d1ae73f6c4407d8a2e131f11829b50",
"triggerType" : "PUSH"
}, {
"hash" : "c11030faa9d70bd6b4cc3d1d8f15ea1db2641159",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18432",
"triggerID" : "c11030faa9d70bd6b4cc3d1d8f15ea1db2641159",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c11030faa9d70bd6b4cc3d1d8f15ea1db2641159 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18432)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-766965862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12479",
"triggerID" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12695",
"triggerID" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"triggerType" : "PUSH"
}, {
"hash" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12858",
"triggerID" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"triggerType" : "PUSH"
}, {
"hash" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12905",
"triggerID" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"triggerType" : "PUSH"
}, {
"hash" : "0f8924698701b1d6768f8757414d69836b83fa3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13084",
"triggerID" : "0f8924698701b1d6768f8757414d69836b83fa3a",
"triggerType" : "PUSH"
}, {
"hash" : "00bfeff9764880852aff536535e3fb8878bd2e74",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13982",
"triggerID" : "00bfeff9764880852aff536535e3fb8878bd2e74",
"triggerType" : "PUSH"
}, {
"hash" : "574e29c9ba5c4a144b2ad6029c96c3fa2af43647",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "574e29c9ba5c4a144b2ad6029c96c3fa2af43647",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 00bfeff9764880852aff536535e3fb8878bd2e74 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13982)
* 574e29c9ba5c4a144b2ad6029c96c3fa2af43647 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-852222264
Hi @dawidwys very thanks for the review! I have updated the PR according to the comments~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
dawidwys commented on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-853767111
Thanks for the update. Merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on a change in pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #14754:
URL: https://github.com/apache/flink/pull/14754#discussion_r592964534
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/PendingCheckpoint.java
##########
@@ -308,6 +316,25 @@ public CompletedCheckpoint finalizeCheckpoint(
// make sure we fulfill the promise with an exception if something fails
try {
+ // Completes the operator state for the fully finished operators
+ for (ExecutionJobVertex jobVertex : checkpointBrief.getFullyFinishedJobVertex()) {
+ for (OperatorIDPair operatorID : jobVertex.getOperatorIDs()) {
+ OperatorState operatorState =
+ operatorStates.get(operatorID.getGeneratedOperatorID());
+
+ if (operatorState == null) {
+ operatorState =
+ new OperatorState(
+ operatorID.getGeneratedOperatorID(),
+ jobVertex.getParallelism(),
+ jobVertex.getMaxParallelism());
+ operatorStates.put(operatorID.getGeneratedOperatorID(), operatorState);
+ }
+
+ operatorState.markedFullyFinished();
Review comment:
Very sorry for miss the check when migrating the code... I re-added the check.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaoyunhaii edited a comment on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
gaoyunhaii edited a comment on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-797303630
Hi Roman @rkhachatryan , very thanks for the review! I updated the PR to address the comments. And very sorry for not pay attention to the test result... Now I fixed the issue via adding the `serialVersionUID`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-766965862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12479",
"triggerID" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12695",
"triggerID" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"triggerType" : "PUSH"
}, {
"hash" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12858",
"triggerID" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"triggerType" : "PUSH"
}, {
"hash" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12905",
"triggerID" : "13548fbb8d09727b172ebdf26a90c56d409fa1f2",
"triggerType" : "PUSH"
}, {
"hash" : "0f8924698701b1d6768f8757414d69836b83fa3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13084",
"triggerID" : "0f8924698701b1d6768f8757414d69836b83fa3a",
"triggerType" : "PUSH"
}, {
"hash" : "00bfeff9764880852aff536535e3fb8878bd2e74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13982",
"triggerID" : "00bfeff9764880852aff536535e3fb8878bd2e74",
"triggerType" : "PUSH"
}, {
"hash" : "574e29c9ba5c4a144b2ad6029c96c3fa2af43647",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14198",
"triggerID" : "574e29c9ba5c4a144b2ad6029c96c3fa2af43647",
"triggerType" : "PUSH"
}, {
"hash" : "5f3ace5f40d1ae73f6c4407d8a2e131f11829b50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14528",
"triggerID" : "5f3ace5f40d1ae73f6c4407d8a2e131f11829b50",
"triggerType" : "PUSH"
}, {
"hash" : "c11030faa9d70bd6b4cc3d1d8f15ea1db2641159",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18432",
"triggerID" : "c11030faa9d70bd6b4cc3d1d8f15ea1db2641159",
"triggerType" : "PUSH"
}, {
"hash" : "05bed4739dfdc1218bd039214e500d3d52a8c78a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18523",
"triggerID" : "05bed4739dfdc1218bd039214e500d3d52a8c78a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c11030faa9d70bd6b4cc3d1d8f15ea1db2641159 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18432)
* 05bed4739dfdc1218bd039214e500d3d52a8c78a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18523)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on a change in pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #14754:
URL: https://github.com/apache/flink/pull/14754#discussion_r571814278
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorState.java
##########
@@ -41,6 +41,18 @@
*/
public class OperatorState implements CompositeStateHandle {
+ private static final int FULLY_FINISHED_PLACEHOLDER_INDEX = -1;
+
+ /** A special content that marks this operator as fully finished. */
+ private static final Map<Integer, OperatorSubtaskState> FULLY_FINISHED_STATE =
+ new HashMap<Integer, OperatorSubtaskState>() {
+ {
+ this.put(
+ FULLY_FINISHED_PLACEHOLDER_INDEX,
+ OperatorSubtaskState.builder().build());
+ }
+ };
Review comment:
Very thanks, this is exactly I'm looking for at first, but I do not found it... :-)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-766965862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12479",
"triggerID" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12695",
"triggerID" : "7a0a38b9960949cedae60f4574edc8382841c8ac",
"triggerType" : "PUSH"
}, {
"hash" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fbe576aeefa4712cc9aaeb856cab8afab85668e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7a0a38b9960949cedae60f4574edc8382841c8ac Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12695)
* 0fbe576aeefa4712cc9aaeb856cab8afab85668e UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-766965862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "37484933592e8b5a4f5cf83425ac0b7b612c7480",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 37484933592e8b5a4f5cf83425ac0b7b612c7480 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on pull request #14754: [FLINK-21127][runtime][checkpoint] Stores finished status for fully finished operators in checkpoint
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on pull request #14754:
URL: https://github.com/apache/flink/pull/14754#issuecomment-774955426
Hi Roman @rkhachatryan, very thanks for the review! I addressed the comments and added the missing test in https://github.com/apache/flink/pull/14754/commits/0f8924698701b1d6768f8757414d69836b83fa3a.
The usage of the `fullyFinished` flag is indeed not include in this PR. The flag is expected to be passed to the task after recovery in [FLINK-21088](https://issues.apache.org/jira/browse/FLINK-21088), and used to skip the execution of the fully finished operator in [FLINK-21089](https://issues.apache.org/jira/browse/FLINK-21089). In detail, it would make the fully finished source directly send EndOfPartitionEvent to finish the channel, and skip lifecycle methods like `open`/`close` for the fully finished operators.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org