You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/11/10 21:45:55 UTC

[GitHub] [flink] rkhachatryan opened a new pull request #14024: [FLINK-20079][task] Fix StreamTask initialization order

rkhachatryan opened a new pull request #14024:
URL: https://github.com/apache/flink/pull/14024


   ## What is the purpose of the change
   
   In FLINK-19681, `UnalignedCheckpointITCase` was updated to put more backpressure. 
   This revealed a bug introduced in FLINK-19907: resultPartitions can be requested before the operator chain is initialized. 
   A proper fix would be to initialize the chain after the outputs but before the inputs.
   
   ## Brief change log
   
     - pull the change to `UnalignedCheckpointITCase` from FLINK-19681
     - fix initialization in `StreamTask`
   
   (the two commits can be squashed)
   
   ## Verifying this change
   
   Updated `UnalignedCheckpointITCase`, `OverWindowITCase`
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): no
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
     - The serializers: no
     - The runtime per-record code paths (performance sensitive): no
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: yes
     - The S3 file system connector: no
   
   ## Documentation
   
     - Does this pull request introduce a new feature? no
     - If yes, how is the feature documented? no
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #14024: [FLINK-20079][task] Fix StreamTask initialization order

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14024:
URL: https://github.com/apache/flink/pull/14024#issuecomment-724996806


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "10e38b5e449d42aa2d7563a9e44f6eb1c53c9509",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "10e38b5e449d42aa2d7563a9e44f6eb1c53c9509",
       "triggerType" : "PUSH"
     }, {
       "hash" : "26ec4e6904ae7ae4d307fc10c08bbbc264a820a5",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9427",
       "triggerID" : "26ec4e6904ae7ae4d307fc10c08bbbc264a820a5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1531ec1122918c4a30311af8fb3f2dcd9deaf774",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "1531ec1122918c4a30311af8fb3f2dcd9deaf774",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 10e38b5e449d42aa2d7563a9e44f6eb1c53c9509 UNKNOWN
   * 26ec4e6904ae7ae4d307fc10c08bbbc264a820a5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9427) 
   * 1531ec1122918c4a30311af8fb3f2dcd9deaf774 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #14024: [FLINK-20079][task] Fix StreamTask initialization order

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14024:
URL: https://github.com/apache/flink/pull/14024#issuecomment-724996806


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "10e38b5e449d42aa2d7563a9e44f6eb1c53c9509",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "10e38b5e449d42aa2d7563a9e44f6eb1c53c9509",
       "triggerType" : "PUSH"
     }, {
       "hash" : "26ec4e6904ae7ae4d307fc10c08bbbc264a820a5",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9427",
       "triggerID" : "26ec4e6904ae7ae4d307fc10c08bbbc264a820a5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 10e38b5e449d42aa2d7563a9e44f6eb1c53c9509 UNKNOWN
   * 26ec4e6904ae7ae4d307fc10c08bbbc264a820a5 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9427) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise merged pull request #14024: [FLINK-20079][task] Fix StreamTask initialization order

Posted by GitBox <gi...@apache.org>.
AHeise merged pull request #14024:
URL: https://github.com/apache/flink/pull/14024


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on pull request #14024: [FLINK-20079][task] Fix StreamTask initialization order

Posted by GitBox <gi...@apache.org>.
AHeise commented on pull request #14024:
URL: https://github.com/apache/flink/pull/14024#issuecomment-725534098


   Forgot to approve before merging. But LGTM anyways ;)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on a change in pull request #14024: [FLINK-20079][task] Fix StreamTask initialization order

Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #14024:
URL: https://github.com/apache/flink/pull/14024#discussion_r521147841



##########
File path: flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointITCase.java
##########
@@ -531,8 +534,10 @@ public void invoke(Long value, Context context) throws Exception {
 			state.numOutput++;
 
 			if (state.completedCheckpoints < minCheckpoints) {
-				// induce heavy backpressure until enough checkpoints have been written
-				Thread.sleep(0, 100_000);
+				// induce backpressure until enough checkpoints have been written
+				if (random.nextInt(1000) == 42) {

Review comment:
       Why do we need to reduce the backpressure here? I'm worried that any solution on top of it will only work with a certain minimal flow.
   
   Note that in https://github.com/apache/flink/pull/13845/commits/ef4f78d9b7bedb10a8c7be3e062032d2e7a0c77c#diff-c0d448d637f04cc203de3592d7fab0ba2412f2d75c59dcf6e44cc4a3c18e5851R568 I added a modification to the test that avoids backpressuring during recovery (by applying backpressure only after the first successful checkpoint.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #14024: [FLINK-20079][task] Fix StreamTask initialization order

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14024:
URL: https://github.com/apache/flink/pull/14024#issuecomment-724996806


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "10e38b5e449d42aa2d7563a9e44f6eb1c53c9509",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "10e38b5e449d42aa2d7563a9e44f6eb1c53c9509",
       "triggerType" : "PUSH"
     }, {
       "hash" : "26ec4e6904ae7ae4d307fc10c08bbbc264a820a5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "26ec4e6904ae7ae4d307fc10c08bbbc264a820a5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 10e38b5e449d42aa2d7563a9e44f6eb1c53c9509 UNKNOWN
   * 26ec4e6904ae7ae4d307fc10c08bbbc264a820a5 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #14024: [FLINK-20079][task] Fix StreamTask initialization order

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14024:
URL: https://github.com/apache/flink/pull/14024#issuecomment-724996806


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "10e38b5e449d42aa2d7563a9e44f6eb1c53c9509",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "10e38b5e449d42aa2d7563a9e44f6eb1c53c9509",
       "triggerType" : "PUSH"
     }, {
       "hash" : "26ec4e6904ae7ae4d307fc10c08bbbc264a820a5",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9427",
       "triggerID" : "26ec4e6904ae7ae4d307fc10c08bbbc264a820a5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 10e38b5e449d42aa2d7563a9e44f6eb1c53c9509 UNKNOWN
   * 26ec4e6904ae7ae4d307fc10c08bbbc264a820a5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9427) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot commented on pull request #14024: [FLINK-20079][task] Fix StreamTask initialization order

Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #14024:
URL: https://github.com/apache/flink/pull/14024#issuecomment-724986565


   Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress of the review.
   
   
   ## Automated Checks
   Last check on commit 10e38b5e449d42aa2d7563a9e44f6eb1c53c9509 (Tue Nov 10 21:47:45 UTC 2020)
   
   **Warnings:**
    * No documentation files were touched! Remember to keep the Flink docs up to date!
   
   
   <sub>Mention the bot in a comment to re-run the automated checks.</sub>
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
    The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
    - `@flinkbot approve all` to approve all aspects
    - `@flinkbot approve-until architecture` to approve everything until `architecture`
    - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
    - `@flinkbot disapprove architecture` to remove an approval you gave earlier
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] rkhachatryan commented on pull request #14024: [FLINK-20079][task] Fix StreamTask initialization order

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on pull request #14024:
URL: https://github.com/apache/flink/pull/14024#issuecomment-725585137


   Thanks!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #14024: [FLINK-20079][task] Fix StreamTask initialization order

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14024:
URL: https://github.com/apache/flink/pull/14024#issuecomment-724996806


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "10e38b5e449d42aa2d7563a9e44f6eb1c53c9509",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "10e38b5e449d42aa2d7563a9e44f6eb1c53c9509",
       "triggerType" : "PUSH"
     }, {
       "hash" : "26ec4e6904ae7ae4d307fc10c08bbbc264a820a5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9427",
       "triggerID" : "26ec4e6904ae7ae4d307fc10c08bbbc264a820a5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1531ec1122918c4a30311af8fb3f2dcd9deaf774",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9463",
       "triggerID" : "1531ec1122918c4a30311af8fb3f2dcd9deaf774",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 10e38b5e449d42aa2d7563a9e44f6eb1c53c9509 UNKNOWN
   * 1531ec1122918c4a30311af8fb3f2dcd9deaf774 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9463) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] rkhachatryan commented on pull request #14024: [FLINK-20079][task] Fix StreamTask initialization order

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on pull request #14024:
URL: https://github.com/apache/flink/pull/14024#issuecomment-725312076


   > why the previous fix was not sufficient. Could you please motivate it on the PR description or commit message?
   
   The previous fix was for a different problem: watermarks being emitted before the recovered state of result subpartitions.
   
   I've added this to the PR description:
   > If partitions are requested fast enough then the operator (chain) can receive data before being initialized. In particular, this can cause out-of-orderness: elements from the upstream followed by elements from operator state (not channel state).
   
   > I'd also like to see a test that shows why the previous way does not work. I'm assuming the modified UCITCase would show that, but it's also not easy to see. However, I'm also assuming that writing a dedicated test that depicts some race condition is probably quite hard, so I'd be fine without one.
   
   Yes, the modified UCITCase shows that, but only on a 2nd-3rd run. And yes, there is (was) a race condition between getting a response from the upstream and initializing the state, which is difficult to test deterministically.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] rkhachatryan commented on a change in pull request #14024: [FLINK-20079][task] Fix StreamTask initialization order

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #14024:
URL: https://github.com/apache/flink/pull/14024#discussion_r521220231



##########
File path: flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointITCase.java
##########
@@ -531,8 +534,10 @@ public void invoke(Long value, Context context) throws Exception {
 			state.numOutput++;
 
 			if (state.completedCheckpoints < minCheckpoints) {
-				// induce heavy backpressure until enough checkpoints have been written
-				Thread.sleep(0, 100_000);
+				// induce backpressure until enough checkpoints have been written
+				if (random.nextInt(1000) == 42) {

Review comment:
       > Why do we need to reduce the backpressure here?
   
   This is a commit from #13827. As far as I understand, the motivation was:
   1. to have higher backpressure (.1ms vs 1ms) for **some** parts of the stream
   2. to have unbalanced streams
   3. reduce running time (was ~1m, now ~10s)
   
   On average, the change decreases sleep time only by 60% (1-42/1000*10).
   
   > I'm worried that any solution on top of it will only work with a certain minimal flow.
   
   With 1K elements, the sink will sleep 23 times for 1ms, which I think is enough and 1K is a reasonable number of elements for this test.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot commented on pull request #14024: [FLINK-20079][task] Fix StreamTask initialization order

Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #14024:
URL: https://github.com/apache/flink/pull/14024#issuecomment-724996806


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "10e38b5e449d42aa2d7563a9e44f6eb1c53c9509",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "10e38b5e449d42aa2d7563a9e44f6eb1c53c9509",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 10e38b5e449d42aa2d7563a9e44f6eb1c53c9509 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #14024: [FLINK-20079][task] Fix StreamTask initialization order

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14024:
URL: https://github.com/apache/flink/pull/14024#issuecomment-724996806


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "10e38b5e449d42aa2d7563a9e44f6eb1c53c9509",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "10e38b5e449d42aa2d7563a9e44f6eb1c53c9509",
       "triggerType" : "PUSH"
     }, {
       "hash" : "26ec4e6904ae7ae4d307fc10c08bbbc264a820a5",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9427",
       "triggerID" : "26ec4e6904ae7ae4d307fc10c08bbbc264a820a5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1531ec1122918c4a30311af8fb3f2dcd9deaf774",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9463",
       "triggerID" : "1531ec1122918c4a30311af8fb3f2dcd9deaf774",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 10e38b5e449d42aa2d7563a9e44f6eb1c53c9509 UNKNOWN
   * 26ec4e6904ae7ae4d307fc10c08bbbc264a820a5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9427) 
   * 1531ec1122918c4a30311af8fb3f2dcd9deaf774 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9463) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org