You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/08/03 20:40:02 UTC

[GitHub] [flink] dmvk opened a new pull request #16695: [FLINK-21116][runtime] Harden DefaultDispatcherRunnerITCase#leaderCha…

dmvk opened a new pull request #16695:
URL: https://github.com/apache/flink/pull/16695


   ## What is the purpose of the change
   
   Fixing a flaky test. Test could get stuck due to hard to reproduce memory-visibility issue in TestingnJobManagerRunnerFactory and race condition between job recovery and dispatcher runner terminationn.
   
   Issue can be reproduced with following change.
   
   ```diff
   diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/dispatcher/TestingJobManagerRunnerFactory.java b/flink-runtime/src/test/java/org/apache/flink/runtime/dispatcher/TestingJobManagerRunnerFactory.java
   index 7b195f09294..47f9ac61734 100644
   --- a/flink-runtime/src/test/java/org/apache/flink/runtime/dispatcher/TestingJobManagerRunnerFactory.java
   +++ b/flink-runtime/src/test/java/org/apache/flink/runtime/dispatcher/TestingJobManagerRunnerFactory.java
   @@ -30,6 +30,9 @@ import org.apache.flink.runtime.rpc.RpcService;
    
    import javax.annotation.Nonnull;
    
   +import java.util.Collections;
   +import java.util.HashSet;
   +import java.util.Set;
    import java.util.concurrent.ArrayBlockingQueue;
    import java.util.concurrent.BlockingQueue;
    
   @@ -71,11 +74,22 @@ public class TestingJobManagerRunnerFactory implements JobManagerRunnerFactory {
            return testingJobManagerRunner;
        }
    
   +    private final Set<String> threads = Collections.synchronizedSet(new HashSet<>());
   +
        @Nonnull
        private TestingJobManagerRunner createTestingJobManagerRunner(JobGraph jobGraph) {
            final boolean blockingTermination;
    
   -        if (numBlockingJobManagerRunners > 0) {
   +        final boolean memoryFlap = threads.add(Thread.currentThread().getName()) && threads.size() > 1;
   +        System.out.println(
   +                "FLAP: "
   +                        + memoryFlap
   +                        + ", THREAD: "
   +                        + Thread.currentThread()
   +                        + ", INSTANCE: "
   +                        + System.identityHashCode(this)
   +                        + threads);
   +        if (numBlockingJobManagerRunners > 0 || memoryFlap) {
                numBlockingJobManagerRunners--;
                blockingTermination = true;
            } else {
   ```
   
   ## Brief change log
   
   - TestingJobManagerRunnerFactory is now thread safe
   - We're waiting for job recovery before closing DispatcherRunner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot commented on pull request #16695: [FLINK-21116][runtime] Harden DefaultDispatcherRunnerITCase#leaderCha…

Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #16695:
URL: https://github.com/apache/flink/pull/16695#issuecomment-892178848


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5fdeb3228f1dc9e24d9bc2aaf77425ffc7f14e78",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "5fdeb3228f1dc9e24d9bc2aaf77425ffc7f14e78",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5fdeb3228f1dc9e24d9bc2aaf77425ffc7f14e78 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] tillrohrmann closed pull request #16695: [FLINK-21116][runtime] Harden DefaultDispatcherRunnerITCase#leaderCha…

Posted by GitBox <gi...@apache.org>.
tillrohrmann closed pull request #16695:
URL: https://github.com/apache/flink/pull/16695


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #16695: [FLINK-21116][runtime] Harden DefaultDispatcherRunnerITCase#leaderCha…

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16695:
URL: https://github.com/apache/flink/pull/16695#issuecomment-892178848


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5fdeb3228f1dc9e24d9bc2aaf77425ffc7f14e78",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21442",
       "triggerID" : "5fdeb3228f1dc9e24d9bc2aaf77425ffc7f14e78",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5fdeb3228f1dc9e24d9bc2aaf77425ffc7f14e78 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21442) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #16695: [FLINK-21116][runtime] Harden DefaultDispatcherRunnerITCase#leaderCha…

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16695:
URL: https://github.com/apache/flink/pull/16695#issuecomment-892178848


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5fdeb3228f1dc9e24d9bc2aaf77425ffc7f14e78",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21442",
       "triggerID" : "5fdeb3228f1dc9e24d9bc2aaf77425ffc7f14e78",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5fdeb3228f1dc9e24d9bc2aaf77425ffc7f14e78 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21442) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #16695: [FLINK-21116][runtime] Harden DefaultDispatcherRunnerITCase#leaderCha…

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16695:
URL: https://github.com/apache/flink/pull/16695#issuecomment-892152551


   Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress of the review.
   
   
   ## Automated Checks
   Last check on commit 81a8a689769220965fa216e02cf7f84872da2f1d (Sat Aug 28 13:10:04 UTC 2021)
   
   **Warnings:**
    * No documentation files were touched! Remember to keep the Flink docs up to date!
   
   
   <sub>Mention the bot in a comment to re-run the automated checks.</sub>
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
    The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
    - `@flinkbot approve all` to approve all aspects
    - `@flinkbot approve-until architecture` to approve everything until `architecture`
    - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
    - `@flinkbot disapprove architecture` to remove an approval you gave earlier
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] dmvk commented on pull request #16695: [FLINK-21116][runtime] Harden DefaultDispatcherRunnerITCase#leaderCha…

Posted by GitBox <gi...@apache.org>.
dmvk commented on pull request #16695:
URL: https://github.com/apache/flink/pull/16695#issuecomment-892152358


   Also, during testing, I've seen a deadlock when giving up leadership (two threads competing for TestingLeaderElectionService lock, ... didn't save the thread dump 😢 ). I wasn't able to reproduce it again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot commented on pull request #16695: [FLINK-21116][runtime] Harden DefaultDispatcherRunnerITCase#leaderCha…

Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #16695:
URL: https://github.com/apache/flink/pull/16695#issuecomment-892152551


   Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress of the review.
   
   
   ## Automated Checks
   Last check on commit 5fdeb3228f1dc9e24d9bc2aaf77425ffc7f14e78 (Tue Aug 03 20:43:26 UTC 2021)
   
   **Warnings:**
    * No documentation files were touched! Remember to keep the Flink docs up to date!
    * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-21116).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work.
   
   
   <sub>Mention the bot in a comment to re-run the automated checks.</sub>
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
    The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
    - `@flinkbot approve all` to approve all aspects
    - `@flinkbot approve-until architecture` to approve everything until `architecture`
    - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
    - `@flinkbot disapprove architecture` to remove an approval you gave earlier
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] tillrohrmann closed pull request #16695: [FLINK-21116][runtime] Harden DefaultDispatcherRunnerITCase#leaderCha…

Posted by GitBox <gi...@apache.org>.
tillrohrmann closed pull request #16695:
URL: https://github.com/apache/flink/pull/16695


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org