You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/06/22 08:19:48 UTC

[GitHub] [flink] gaoyunhaii opened a new pull request #16238: [FLINK-21084][runtime][checkpoint] Allows tasks to report operator subtask finished status

gaoyunhaii opened a new pull request #16238:
URL: https://github.com/apache/flink/pull/16238


   ## What is the purpose of the change
   
   This PR allows the task to report finished status if it has closed operators. The reported subtask status would be used to calculate if the whole operator is fully finished on the JM side. 
   
   ## Brief change log
   
   - df455f060b64611f06bc068094afcc48b98984af snapshot the finished status for the closed operator.
   
   ## Verifying this change
   
   This change added tests and can be verified by added UT.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): **no**
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **no**
     - The serializers: **no**
     - The runtime per-record code paths (performance sensitive): **no**
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: **no**
     - The S3 file system connector: **no**
   
   ## Documentation
   
     - Does this pull request introduce a new feature? **no**
     - If yes, how is the feature documented? **not applicable**
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #16238: [FLINK-21084][runtime][checkpoint] Allows tasks to report finished snapshot if finished on restore

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16238:
URL: https://github.com/apache/flink/pull/16238#issuecomment-865697511


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "df455f060b64611f06bc068094afcc48b98984af",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19298",
       "triggerID" : "df455f060b64611f06bc068094afcc48b98984af",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e38c1c707865aa3e3e6fe7e376819d6a599d2a15",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20371",
       "triggerID" : "e38c1c707865aa3e3e6fe7e376819d6a599d2a15",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7c5707b792ab4f0f91e567c462f29f485fc8184",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c7c5707b792ab4f0f91e567c462f29f485fc8184",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e38c1c707865aa3e3e6fe7e376819d6a599d2a15 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20371) 
   * c7c5707b792ab4f0f91e567c462f29f485fc8184 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #16238: [FLINK-21084][runtime][checkpoint] Allows tasks to report finished snapshot if finished on restore

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16238:
URL: https://github.com/apache/flink/pull/16238#issuecomment-865697511


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "df455f060b64611f06bc068094afcc48b98984af",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19298",
       "triggerID" : "df455f060b64611f06bc068094afcc48b98984af",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e38c1c707865aa3e3e6fe7e376819d6a599d2a15",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20371",
       "triggerID" : "e38c1c707865aa3e3e6fe7e376819d6a599d2a15",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7c5707b792ab4f0f91e567c462f29f485fc8184",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20497",
       "triggerID" : "c7c5707b792ab4f0f91e567c462f29f485fc8184",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c7c5707b792ab4f0f91e567c462f29f485fc8184 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20497) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] dawidwys commented on a change in pull request #16238: [FLINK-21084][runtime][checkpoint] Allows tasks to report finished snapshot if finished on restore

Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #16238:
URL: https://github.com/apache/flink/pull/16238#discussion_r671234850



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/AsyncCheckpointRunnable.java
##########
@@ -109,52 +112,25 @@ public void run() {
         FileSystemSafetyNet.initializeSafetyNetForThread();
         try {
 
-            TaskStateSnapshot jobManagerTaskOperatorSubtaskStates =
-                    new TaskStateSnapshot(operatorSnapshotsInProgress.size());
-            TaskStateSnapshot localTaskOperatorSubtaskStates =
-                    new TaskStateSnapshot(operatorSnapshotsInProgress.size());
-
-            long bytesPersistedDuringAlignment = 0;
-            for (Map.Entry<OperatorID, OperatorSnapshotFutures> entry :
-                    operatorSnapshotsInProgress.entrySet()) {
-
-                OperatorID operatorID = entry.getKey();
-                OperatorSnapshotFutures snapshotInProgress = entry.getValue();
-
-                // finalize the async part of all by executing all snapshot runnables
-                OperatorSnapshotFinalizer finalizedSnapshots =
-                        new OperatorSnapshotFinalizer(snapshotInProgress);
-
-                jobManagerTaskOperatorSubtaskStates.putSubtaskStateByOperatorID(
-                        operatorID, finalizedSnapshots.getJobManagerOwnedState());
-
-                localTaskOperatorSubtaskStates.putSubtaskStateByOperatorID(
-                        operatorID, finalizedSnapshots.getTaskLocalState());
-
-                bytesPersistedDuringAlignment +=
-                        finalizedSnapshots
-                                .getJobManagerOwnedState()
-                                .getResultSubpartitionState()
-                                .getStateSize();
-                bytesPersistedDuringAlignment +=
-                        finalizedSnapshots
-                                .getJobManagerOwnedState()
-                                .getInputChannelState()
-                                .getStateSize();
-            }
+            SnapshotsFinalizeResult snapshotsFinalizeResult =

Review comment:
       Nice! I really like extracting such parts of the code to separate methods. It makes it so much easier to read!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #16238: [FLINK-21084][runtime][checkpoint] Allows tasks to report finished snapshot if finished on restore

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16238:
URL: https://github.com/apache/flink/pull/16238#issuecomment-865697511


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "df455f060b64611f06bc068094afcc48b98984af",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19298",
       "triggerID" : "df455f060b64611f06bc068094afcc48b98984af",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e38c1c707865aa3e3e6fe7e376819d6a599d2a15",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20371",
       "triggerID" : "e38c1c707865aa3e3e6fe7e376819d6a599d2a15",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e38c1c707865aa3e3e6fe7e376819d6a599d2a15 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20371) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #16238: [FLINK-21084][runtime][checkpoint] Allows tasks to report finished snapshot if finished on restore

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16238:
URL: https://github.com/apache/flink/pull/16238#issuecomment-865697511


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "df455f060b64611f06bc068094afcc48b98984af",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19298",
       "triggerID" : "df455f060b64611f06bc068094afcc48b98984af",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e38c1c707865aa3e3e6fe7e376819d6a599d2a15",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e38c1c707865aa3e3e6fe7e376819d6a599d2a15",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * df455f060b64611f06bc068094afcc48b98984af Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19298) 
   * e38c1c707865aa3e3e6fe7e376819d6a599d2a15 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] gaoyunhaii commented on pull request #16238: [FLINK-21084][runtime][checkpoint] Allows tasks to report finished snapshot if finished on restore

Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on pull request #16238:
URL: https://github.com/apache/flink/pull/16238#issuecomment-880627752


   Hi @dawidwys very thanks for the review! Could you have a look at this PR next when convenient~?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] gaoyunhaii commented on pull request #16238: [FLINK-21084][runtime][checkpoint] Allows tasks to report finished snapshot if finished on restore

Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on pull request #16238:
URL: https://github.com/apache/flink/pull/16238#issuecomment-881637602


   Very thanks @dawidwys  for the review! :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] dawidwys closed pull request #16238: [FLINK-21084][runtime][checkpoint] Allows tasks to report finished snapshot if finished on restore

Posted by GitBox <gi...@apache.org>.
dawidwys closed pull request #16238:
URL: https://github.com/apache/flink/pull/16238


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #16238: [FLINK-21084][runtime][checkpoint] Allows tasks to report finished snapshot if finished on restore

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16238:
URL: https://github.com/apache/flink/pull/16238#issuecomment-865697511


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "df455f060b64611f06bc068094afcc48b98984af",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19298",
       "triggerID" : "df455f060b64611f06bc068094afcc48b98984af",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e38c1c707865aa3e3e6fe7e376819d6a599d2a15",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20371",
       "triggerID" : "e38c1c707865aa3e3e6fe7e376819d6a599d2a15",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * df455f060b64611f06bc068094afcc48b98984af Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19298) 
   * e38c1c707865aa3e3e6fe7e376819d6a599d2a15 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20371) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] gaoyunhaii commented on pull request #16238: [FLINK-21084][runtime][checkpoint] Allows tasks to report operator subtask finished status

Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on pull request #16238:
URL: https://github.com/apache/flink/pull/16238#issuecomment-866642683


   Hi @RocMarshal very thanks for the review~ This PR is a part of https://issues.apache.org/jira/browse/FLINK-2491 and is still blocked by some previous PRs, thus I opened it as a draft one first~


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #16238: [FLINK-21084][runtime][checkpoint] Allows tasks to report finished snapshot if finished on restore

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16238:
URL: https://github.com/apache/flink/pull/16238#issuecomment-865697511


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "df455f060b64611f06bc068094afcc48b98984af",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19298",
       "triggerID" : "df455f060b64611f06bc068094afcc48b98984af",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e38c1c707865aa3e3e6fe7e376819d6a599d2a15",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20371",
       "triggerID" : "e38c1c707865aa3e3e6fe7e376819d6a599d2a15",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7c5707b792ab4f0f91e567c462f29f485fc8184",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20497",
       "triggerID" : "c7c5707b792ab4f0f91e567c462f29f485fc8184",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e38c1c707865aa3e3e6fe7e376819d6a599d2a15 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20371) 
   * c7c5707b792ab4f0f91e567c462f29f485fc8184 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20497) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot commented on pull request #16238: [FLINK-21084][runtime][checkpoint] Allows tasks to report operator subtask finished status

Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #16238:
URL: https://github.com/apache/flink/pull/16238#issuecomment-865686419






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] RocMarshal commented on pull request #16238: [FLINK-21084][runtime][checkpoint] Allows tasks to report operator subtask finished status

Posted by GitBox <gi...@apache.org>.
RocMarshal commented on pull request #16238:
URL: https://github.com/apache/flink/pull/16238#issuecomment-865918075


   @gaoyunhaii Thank you for your contribution. Could you please resolve the conflicting files in order to active the CI robot  to run ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #16238: [FLINK-21084][runtime][checkpoint] Allows tasks to report operator subtask finished status

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16238:
URL: https://github.com/apache/flink/pull/16238#issuecomment-865697511


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "df455f060b64611f06bc068094afcc48b98984af",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19298",
       "triggerID" : "df455f060b64611f06bc068094afcc48b98984af",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * df455f060b64611f06bc068094afcc48b98984af Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=19298) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org