You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/04/22 07:20:21 UTC

[GitHub] [flink] becketqin opened a new pull request #11856: [FLINK-17268][connector/common] Fix the bug of missing records during SplitFetcher wakeup.

becketqin opened a new pull request #11856:
URL: https://github.com/apache/flink/pull/11856


   ## What is the purpose of the change
   This patch fixes a bug that when `SplitFetcher.wakeup()` is invoked, the fetcher thread may be interrupted and thus lose the record.
   
   ## Brief change log
   1. Fix the wakeup handling in `SplitFetcher`
   2. Added unit tests for wakeup correctness.
   3. Fixed a bug in `MockSplitReader` to handle interruptions.
   
   ## Verifying this change
   
   Added unit test `SplitFetcherTest` for the correctness of wakeup. The tests keeps waking up a fetcher thread to ensure the result delivered by the fetcher thread is still in order without duplication.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (no)
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
     - The serializers: (no)
     - The runtime per-record code paths (performance sensitive): (yes)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no)
     - The S3 file system connector: (no)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (no)
     - If yes, how is the feature documented? (not applicable)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] StephanEwen edited a comment on pull request #11856: [FLINK-17268][connector/common] Fix the bug of missing records during SplitFetcher wakeup.

Posted by GitBox <gi...@apache.org>.
StephanEwen edited a comment on pull request #11856:
URL: https://github.com/apache/flink/pull/11856#issuecomment-619636516


   The changes look fine to me, +1 to merge this.
   
   Maybe some comments for separate improvements, as follow-ups:
     - The use of `AtomicBoolean` is a bit confusing. It looks like these are mainly `volatile boolean` to report status with a `happens-before` ordering/visibility.
   
     - The whole "interrupting"-based flow control is difficult to follow, and I believe very difficult to get right. One interrupt flag clearing/setting can make the difference between correctness, a deadlock, or a "hot loop lock". And the interrupt flags are hellishly hard to reason about, especially because they can be mutated by any component outside the fetcher.
   
       One way to solve this is by turning the fetch queue into a "wakeup-able queue". That pulls a good amount of the the wakeup logic out of the fetcher class, making that one easier to read and to test. The [Handover](https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/Handover.java) in the current KafkaConsumer is like a "wakeup-able size-one queue", as an example. We probably need something slightly different (wake up consumer of the queue). A simple notify/wait implementation is quite straightforward, and if we have time we can make this more efficient with Java's `LockSupport` classes.
   
     - The fact that this Interruption-based flow is hard to get right looks like it is at least part of the reason for the original bug, so I think it is worth thinking about.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on issue #11856: [FLINK-17268][connector/common] Fix the bug of missing records during SplitFetcher wakeup.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11856:
URL: https://github.com/apache/flink/pull/11856#issuecomment-617606082


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e5d05181bb89c1f1378b2010ee00b7993b662201",
       "status" : "CANCELED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/161385026",
       "triggerID" : "e5d05181bb89c1f1378b2010ee00b7993b662201",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e5d05181bb89c1f1378b2010ee00b7993b662201",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=46",
       "triggerID" : "e5d05181bb89c1f1378b2010ee00b7993b662201",
       "triggerType" : "PUSH"
     }, {
       "hash" : "043f306951e45f7d5503d512764ef1d7a8e96f34",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=67",
       "triggerID" : "043f306951e45f7d5503d512764ef1d7a8e96f34",
       "triggerType" : "PUSH"
     }, {
       "hash" : "043f306951e45f7d5503d512764ef1d7a8e96f34",
       "status" : "PENDING",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/161418491",
       "triggerID" : "043f306951e45f7d5503d512764ef1d7a8e96f34",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e5d05181bb89c1f1378b2010ee00b7993b662201 Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/161385026) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=46) 
   * 043f306951e45f7d5503d512764ef1d7a8e96f34 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/161418491) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=67) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on issue #11856: [FLINK-17268][connector/common] Fix the bug of missing records during SplitFetcher wakeup.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11856:
URL: https://github.com/apache/flink/pull/11856#issuecomment-617606082


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e5d05181bb89c1f1378b2010ee00b7993b662201",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/161385026",
       "triggerID" : "e5d05181bb89c1f1378b2010ee00b7993b662201",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e5d05181bb89c1f1378b2010ee00b7993b662201",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=46",
       "triggerID" : "e5d05181bb89c1f1378b2010ee00b7993b662201",
       "triggerType" : "PUSH"
     }, {
       "hash" : "043f306951e45f7d5503d512764ef1d7a8e96f34",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=67",
       "triggerID" : "043f306951e45f7d5503d512764ef1d7a8e96f34",
       "triggerType" : "PUSH"
     }, {
       "hash" : "043f306951e45f7d5503d512764ef1d7a8e96f34",
       "status" : "SUCCESS",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/161418491",
       "triggerID" : "043f306951e45f7d5503d512764ef1d7a8e96f34",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 043f306951e45f7d5503d512764ef1d7a8e96f34 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/161418491) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=67) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #11856: [FLINK-17268][connector/common] Fix the bug of missing records during SplitFetcher wakeup.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #11856:
URL: https://github.com/apache/flink/pull/11856#issuecomment-617606082


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e5d05181bb89c1f1378b2010ee00b7993b662201",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/161385026",
       "triggerID" : "e5d05181bb89c1f1378b2010ee00b7993b662201",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e5d05181bb89c1f1378b2010ee00b7993b662201",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=46",
       "triggerID" : "e5d05181bb89c1f1378b2010ee00b7993b662201",
       "triggerType" : "PUSH"
     }, {
       "hash" : "043f306951e45f7d5503d512764ef1d7a8e96f34",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=67",
       "triggerID" : "043f306951e45f7d5503d512764ef1d7a8e96f34",
       "triggerType" : "PUSH"
     }, {
       "hash" : "043f306951e45f7d5503d512764ef1d7a8e96f34",
       "status" : "SUCCESS",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/161418491",
       "triggerID" : "043f306951e45f7d5503d512764ef1d7a8e96f34",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 043f306951e45f7d5503d512764ef1d7a8e96f34 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=67) Unknown: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/161418491) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot commented on issue #11856: [FLINK-17268][connector/common] Fix the bug of missing records during SplitFetcher wakeup.

Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #11856:
URL: https://github.com/apache/flink/pull/11856#issuecomment-617606082


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e5d05181bb89c1f1378b2010ee00b7993b662201",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e5d05181bb89c1f1378b2010ee00b7993b662201",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e5d05181bb89c1f1378b2010ee00b7993b662201 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] becketqin commented on pull request #11856: [FLINK-17268][connector/common] Fix the bug of missing records during SplitFetcher wakeup.

Posted by GitBox <gi...@apache.org>.
becketqin commented on pull request #11856:
URL: https://github.com/apache/flink/pull/11856#issuecomment-619653062


   @StephanEwen Thanks for the review. You are right, the wakeup is difficult to make right. I found the major difficulty comes from the efforts to have "fine-grained" wakeup, i.e. avoid waking up the `SplitFetcher` unnecessarily, because we don't know if the call to `SplitFetcher.wakeup()` is always efficient. Maybe we don't really need to worry about this. If so, the code would be much more straightforward with a "coarse-grained" wakeup.
   
   I am curious about your suggestion of using `LockSupport`. Would be glad to hear the idea.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] becketqin commented on issue #11856: [FLINK-17268][connector/common] Fix the bug of missing records during SplitFetcher wakeup.

Posted by GitBox <gi...@apache.org>.
becketqin commented on issue #11856:
URL: https://github.com/apache/flink/pull/11856#issuecomment-617599793


   @StephanEwen will you have time to take a quick look? This may result in intermittent failure of the build.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on issue #11856: [FLINK-17268][connector/common] Fix the bug of missing records during SplitFetcher wakeup.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11856:
URL: https://github.com/apache/flink/pull/11856#issuecomment-617606082


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e5d05181bb89c1f1378b2010ee00b7993b662201",
       "status" : "CANCELED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/161385026",
       "triggerID" : "e5d05181bb89c1f1378b2010ee00b7993b662201",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e5d05181bb89c1f1378b2010ee00b7993b662201",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=46",
       "triggerID" : "e5d05181bb89c1f1378b2010ee00b7993b662201",
       "triggerType" : "PUSH"
     }, {
       "hash" : "043f306951e45f7d5503d512764ef1d7a8e96f34",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=67",
       "triggerID" : "043f306951e45f7d5503d512764ef1d7a8e96f34",
       "triggerType" : "PUSH"
     }, {
       "hash" : "043f306951e45f7d5503d512764ef1d7a8e96f34",
       "status" : "PENDING",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/161418491",
       "triggerID" : "043f306951e45f7d5503d512764ef1d7a8e96f34",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e5d05181bb89c1f1378b2010ee00b7993b662201 Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/161385026) Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=46) 
   * 043f306951e45f7d5503d512764ef1d7a8e96f34 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/161418491) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=67) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on issue #11856: [FLINK-17268][connector/common] Fix the bug of missing records during SplitFetcher wakeup.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11856:
URL: https://github.com/apache/flink/pull/11856#issuecomment-617606082


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e5d05181bb89c1f1378b2010ee00b7993b662201",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/161385026",
       "triggerID" : "e5d05181bb89c1f1378b2010ee00b7993b662201",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e5d05181bb89c1f1378b2010ee00b7993b662201",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=46",
       "triggerID" : "e5d05181bb89c1f1378b2010ee00b7993b662201",
       "triggerType" : "PUSH"
     }, {
       "hash" : "043f306951e45f7d5503d512764ef1d7a8e96f34",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=67",
       "triggerID" : "043f306951e45f7d5503d512764ef1d7a8e96f34",
       "triggerType" : "PUSH"
     }, {
       "hash" : "043f306951e45f7d5503d512764ef1d7a8e96f34",
       "status" : "SUCCESS",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/161418491",
       "triggerID" : "043f306951e45f7d5503d512764ef1d7a8e96f34",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 043f306951e45f7d5503d512764ef1d7a8e96f34 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/161418491) Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=67) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot commented on issue #11856: [FLINK-17268][connector/common] Fix the bug of missing records during SplitFetcher wakeup.

Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #11856:
URL: https://github.com/apache/flink/pull/11856#issuecomment-617600234


   Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress of the review.
   
   
   ## Automated Checks
   Last check on commit e5d05181bb89c1f1378b2010ee00b7993b662201 (Wed Apr 22 07:22:48 UTC 2020)
   
   **Warnings:**
    * No documentation files were touched! Remember to keep the Flink docs up to date!
   
   
   <sub>Mention the bot in a comment to re-run the automated checks.</sub>
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
    The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
    - `@flinkbot approve all` to approve all aspects
    - `@flinkbot approve-until architecture` to approve everything until `architecture`
    - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
    - `@flinkbot disapprove architecture` to remove an approval you gave earlier
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] becketqin commented on pull request #11856: [FLINK-17268][connector/common] Fix the bug of missing records during SplitFetcher wakeup.

Posted by GitBox <gi...@apache.org>.
becketqin commented on pull request #11856:
URL: https://github.com/apache/flink/pull/11856#issuecomment-619655010


   > The fact that this Interruption-based flow is hard to get right looks like it is at least part of the reason for the original bug, so I think it is worth thinking about.
   
   The interruption could happen in two cases:
   1. Thrown from `SplitFetcher.wakeup();`
   2. Thrown from `BlockingQueue.put()` in `FetchTask.run()`;
   
   There isn't much we can do in the first case, but we can probably have an implementation of `BlockingQueue` (just like the handover you mentioned) to avoid `InterruptedException` on `put()` in the second case.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] StephanEwen commented on pull request #11856: [FLINK-17268][connector/common] Fix the bug of missing records during SplitFetcher wakeup.

Posted by GitBox <gi...@apache.org>.
StephanEwen commented on pull request #11856:
URL: https://github.com/apache/flink/pull/11856#issuecomment-619636516


   The changes look fine to me, +1 to merge this.
   
   Maybe some comments for separate improvements, as follow-ups:
     - The use of `AtomicBoolean` is a bit confusing. It looks like these are mainly `volatile boolean` to report status with a `happens-before` ordering/visibility.
   
     - The whole "interrupting"-based flow control is difficult to follow, and I believe very difficult to get right. One interrupt flag clearing/setting can make the difference between correctness, a deadlock, or a "hot loop lock". And the interrupt flags are hellishly hard to reason about, especially because they can be mutated by any component outside the fetcher.
   
       One way to solve this is by turning the fetch queue into a "wakeup-able queue". That pulls a good amount of the the wakeup logic out of the fetcher class, making that one easier to read and to test. The [Handover](https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/Handover.java) in the current KafkaConsumer is like a "wakeup-able size-one queue", as an example. We probably need something slightly different (wake up consumer of the queue). A simple notify/wait implementation is quite straightforward, and if we have time we can make this more efficient with Java's `LockSupport` classes.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on issue #11856: [FLINK-17268][connector/common] Fix the bug of missing records during SplitFetcher wakeup.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11856:
URL: https://github.com/apache/flink/pull/11856#issuecomment-617606082


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e5d05181bb89c1f1378b2010ee00b7993b662201",
       "status" : "PENDING",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/161385026",
       "triggerID" : "e5d05181bb89c1f1378b2010ee00b7993b662201",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e5d05181bb89c1f1378b2010ee00b7993b662201",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=46",
       "triggerID" : "e5d05181bb89c1f1378b2010ee00b7993b662201",
       "triggerType" : "PUSH"
     }, {
       "hash" : "043f306951e45f7d5503d512764ef1d7a8e96f34",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "043f306951e45f7d5503d512764ef1d7a8e96f34",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e5d05181bb89c1f1378b2010ee00b7993b662201 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/161385026) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=46) 
   * 043f306951e45f7d5503d512764ef1d7a8e96f34 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on issue #11856: [FLINK-17268][connector/common] Fix the bug of missing records during SplitFetcher wakeup.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11856:
URL: https://github.com/apache/flink/pull/11856#issuecomment-617606082


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e5d05181bb89c1f1378b2010ee00b7993b662201",
       "status" : "PENDING",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/161385026",
       "triggerID" : "e5d05181bb89c1f1378b2010ee00b7993b662201",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e5d05181bb89c1f1378b2010ee00b7993b662201",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=46",
       "triggerID" : "e5d05181bb89c1f1378b2010ee00b7993b662201",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e5d05181bb89c1f1378b2010ee00b7993b662201 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/161385026) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=46) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org