You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/02/16 02:18:53 UTC

[GitHub] [flink] PatrickRen opened a new pull request #18790: [FLINK-26018][connector/common] Create per-split output on split addition in SourceOperator

PatrickRen opened a new pull request #18790:
URL: https://github.com/apache/flink/pull/18790


   ## What is the purpose of the change
   
   This PR fixes a bug when multiple splits are assigned to a source operator, records from the first fetch of the first split might mistakenly push watermark forward so that records from other splits are treated as late event. 
   
   
   ## Brief change log
   
   *(for example:)*
     - Fix typo of variable in SourceOperator
     - Create per-split output on split addition in SourceOperator
   
   ## Verifying this change
   
   This change added tests and can be verified as follows:
   
   - Added unit test for per-split watermark in source operator
   - Added IT case for testing `SourceReaderBase` working together with `SourceOperator` on per-split watermark
   - Added IT case for KafkaSource for testing per-split watermark
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (yes / **no**)
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
     - The serializers: (yes / **no** / don't know)
     - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know)
     - The S3 file system connector: (yes / **no** / don't know)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (yes / **no**)
     - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot commented on pull request #18790: [FLINK-26018][connector/common] Create per-split output on split addition in SourceOperator

Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #18790:
URL: https://github.com/apache/flink/pull/18790#issuecomment-1041026494


   Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress of the review.
   
   
   ## Automated Checks
   Last check on commit f11bc4e5c50b90d5410db10c4ae532ca888a3a48 (Wed Feb 16 02:23:38 UTC 2022)
   
   **Warnings:**
    * **1 pom.xml files were touched**: Check for build and licensing issues.
    * No documentation files were touched! Remember to keep the Flink docs up to date!
   
   
   <sub>Mention the bot in a comment to re-run the automated checks.</sub>
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
    The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
    - `@flinkbot approve all` to approve all aspects
    - `@flinkbot approve-until architecture` to approve everything until `architecture`
    - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
    - `@flinkbot disapprove architecture` to remove an approval you gave earlier
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] pnowojski commented on pull request #18790: [FLINK-26018][connector/common] Create per-split output on split addition in SourceOperator

Posted by GitBox <gi...@apache.org>.
pnowojski commented on pull request #18790:
URL: https://github.com/apache/flink/pull/18790#issuecomment-1065046228


   Just and update from my side, me and @AHeise chatted about this on Wednesday, and while we still think the ability to send global watermark from the source coordinator is an important one, and it could be used to solve this issue as well. However it would require an independent fix in every `SplitEnumerator`. While indeed we can fix this bug with this change. So +1 for the general idea of this PR for 1.15, and later we can work on the more general solution.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on pull request #18790: [FLINK-26018][connector/common] Create per-split output on split addition in SourceOperator

Posted by GitBox <gi...@apache.org>.
AHeise commented on pull request #18790:
URL: https://github.com/apache/flink/pull/18790#issuecomment-1067631387


   I'm [rerunning](http://codespeed.dak8s.net:8080/job/flink-benchmark-request/69) the benchmarks and would merge if there are no obvious things 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18790: [FLINK-26018][connector/common] Create per-split output on split addition in SourceOperator

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18790:
URL: https://github.com/apache/flink/pull/18790#issuecomment-1041025841


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31590",
       "triggerID" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31590",
       "triggerID" : "1041200353",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31590",
       "triggerID" : "1042708445",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "ebe36ecab291ff59275a8df96796a0c3800d2edb",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31744",
       "triggerID" : "ebe36ecab291ff59275a8df96796a0c3800d2edb",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f11bc4e5c50b90d5410db10c4ae532ca888a3a48 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31590) 
   * ebe36ecab291ff59275a8df96796a0c3800d2edb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31744) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18790: [FLINK-26018][connector/common] Create per-split output on split addition in SourceOperator

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18790:
URL: https://github.com/apache/flink/pull/18790#issuecomment-1041025841


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31590",
       "triggerID" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31590",
       "triggerID" : "1041200353",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31590",
       "triggerID" : "1042708445",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "ebe36ecab291ff59275a8df96796a0c3800d2edb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31744",
       "triggerID" : "ebe36ecab291ff59275a8df96796a0c3800d2edb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bdc3dca6e0c8fdb84c83d111e70d7adb866b07db",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33003",
       "triggerID" : "bdc3dca6e0c8fdb84c83d111e70d7adb866b07db",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * bdc3dca6e0c8fdb84c83d111e70d7adb866b07db Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33003) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] pnowojski commented on a change in pull request #18790: [FLINK-26018][connector/common] Create per-split output on split addition in SourceOperator

Posted by GitBox <gi...@apache.org>.
pnowojski commented on a change in pull request #18790:
URL: https://github.com/apache/flink/pull/18790#discussion_r809863683



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java
##########
@@ -516,7 +525,14 @@ public void handleOperatorEvent(OperatorEvent event) {
             checkWatermarkAlignment();
         } else if (event instanceof AddSplitEvent) {
             try {
-                sourceReader.addSplits(((AddSplitEvent<SplitT>) event).splits(splitSerializer));
+                List<SplitT> newSplits = ((AddSplitEvent<SplitT>) event).splits(splitSerializer);
+                if (operatingMode == OperatingMode.OUTPUT_NOT_INITIALIZED) {
+                    outputPendingSplits.addAll(newSplits);
+                } else {
+                    newSplits.forEach(
+                            split -> currentMainOutput.createOutputForSplit(split.splitId()));
+                }
+                sourceReader.addSplits(newSplits);

Review comment:
       I would already extract this code to a separate method. `handleAddSplitEvent`?

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java
##########
@@ -516,7 +525,14 @@ public void handleOperatorEvent(OperatorEvent event) {
             checkWatermarkAlignment();
         } else if (event instanceof AddSplitEvent) {
             try {
-                sourceReader.addSplits(((AddSplitEvent<SplitT>) event).splits(splitSerializer));
+                List<SplitT> newSplits = ((AddSplitEvent<SplitT>) event).splits(splitSerializer);
+                if (operatingMode == OperatingMode.OUTPUT_NOT_INITIALIZED) {
+                    outputPendingSplits.addAll(newSplits);
+                } else {
+                    newSplits.forEach(
+                            split -> currentMainOutput.createOutputForSplit(split.splitId()));

Review comment:
       `checkNotNull(currentMainOutput);`?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] pnowojski commented on pull request #18790: [FLINK-26018][connector/common] Create per-split output on split addition in SourceOperator

Posted by GitBox <gi...@apache.org>.
pnowojski commented on pull request #18790:
URL: https://github.com/apache/flink/pull/18790#issuecomment-1044267094


   I've started a benchmark request for this PR:
   http://codespeed.dak8s.net:8080/job/flink-benchmark-request/65/


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] PatrickRen commented on pull request #18790: [FLINK-26018][connector/common] Create per-split output on split addition in SourceOperator

Posted by GitBox <gi...@apache.org>.
PatrickRen commented on pull request #18790:
URL: https://github.com/apache/flink/pull/18790#issuecomment-1072086494


   @AHeise 
   BP on 1.15: #19110, CI has passed ✅ 
   BP on 1.14: #19128, seems that E2E doesn't even start 🤔 I'll keep monitoring


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18790: [FLINK-26018][connector/common] Create per-split output on split addition in SourceOperator

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18790:
URL: https://github.com/apache/flink/pull/18790#issuecomment-1041025841


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31590",
       "triggerID" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31590",
       "triggerID" : "1041200353",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31590",
       "triggerID" : "1042708445",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "ebe36ecab291ff59275a8df96796a0c3800d2edb",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31744",
       "triggerID" : "ebe36ecab291ff59275a8df96796a0c3800d2edb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bdc3dca6e0c8fdb84c83d111e70d7adb866b07db",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33003",
       "triggerID" : "bdc3dca6e0c8fdb84c83d111e70d7adb866b07db",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * ebe36ecab291ff59275a8df96796a0c3800d2edb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31744) 
   * bdc3dca6e0c8fdb84c83d111e70d7adb866b07db Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33003) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on a change in pull request #18790: [FLINK-26018][connector/common] Create per-split output on split addition in SourceOperator

Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #18790:
URL: https://github.com/apache/flink/pull/18790#discussion_r826149039



##########
File path: flink-connectors/flink-connector-base/src/test/java/org/apache/flink/connector/base/source/reader/SourceReaderBaseTest.java
##########
@@ -239,6 +259,83 @@ void testPollNextReturnMoreAvailableWhenAllSplitFetcherCloseWithLeftoverElementI
                 .isEqualTo(InputStatus.MORE_AVAILABLE);
     }
 
+    @ParameterizedTest(name = "Emit record before split addition: {0}")
+    @ValueSource(booleans = {true, false})
+    void testPerSplitWatermark(boolean emitRecordBeforeSplitAddition) throws Exception {
+        MockSplitReader mockSplitReader =
+                MockSplitReader.newBuilder()
+                        .setNumRecordsPerSplitPerFetch(3)
+                        .setBlockingFetch(true)
+                        .build();
+
+        MockSourceReader reader =
+                new MockSourceReader(
+                        new FutureCompletingBlockingQueue<>(),
+                        () -> mockSplitReader,
+                        new Configuration(),
+                        new TestingReaderContext());
+
+        SourceOperator<Integer, MockSourceSplit> sourceOperator =
+                createTestOperator(
+                        reader,
+                        WatermarkStrategy.forGenerator(
+                                (context) -> new OnEventWatermarkGenerator()),
+                        true);
+
+        MockSourceSplit splitA = new MockSourceSplit(0, 0, 3);
+        splitA.addRecord(100);
+        splitA.addRecord(200);
+        splitA.addRecord(300);
+
+        MockSourceSplit splitB = new MockSourceSplit(1, 0, 3);
+        splitB.addRecord(150);
+        splitB.addRecord(250);
+        splitB.addRecord(350);
+
+        WatermarkCollectingDataOutput output = new WatermarkCollectingDataOutput();
+
+        if (emitRecordBeforeSplitAddition) {
+            sourceOperator.emitNext(output);
+        }

Review comment:
       👍 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18790: [FLINK-26018][connector/common] Create per-split output on split addition in SourceOperator

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18790:
URL: https://github.com/apache/flink/pull/18790#issuecomment-1041025841


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31590",
       "triggerID" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31590",
       "triggerID" : "1041200353",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * f11bc4e5c50b90d5410db10c4ae532ca888a3a48 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31590) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18790: [FLINK-26018][connector/common] Create per-split output on split addition in SourceOperator

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18790:
URL: https://github.com/apache/flink/pull/18790#issuecomment-1041025841


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31590",
       "triggerID" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31590",
       "triggerID" : "1041200353",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31590",
       "triggerID" : "1042708445",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "ebe36ecab291ff59275a8df96796a0c3800d2edb",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31744",
       "triggerID" : "ebe36ecab291ff59275a8df96796a0c3800d2edb",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * ebe36ecab291ff59275a8df96796a0c3800d2edb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31744) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] PatrickRen commented on a change in pull request #18790: [FLINK-26018][connector/common] Create per-split output on split addition in SourceOperator

Posted by GitBox <gi...@apache.org>.
PatrickRen commented on a change in pull request #18790:
URL: https://github.com/apache/flink/pull/18790#discussion_r809646789



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java
##########
@@ -423,6 +422,16 @@ private DataInputStatus emitNextNotReading(DataOutput<OUT> output) throws Except
         }
     }
 
+    private void initializeMainOutput(DataOutput<OUT> output) {
+        currentMainOutput = eventTimeLogic.createMainOutput(output, this::onWatermarkEmitted);
+        initializeLatencyMarkerEmitter(output);
+        lastInvokedOutput = output;
+        outputPendingSplits.forEach(
+                split -> currentMainOutput.createOutputForSplit(split.splitId()));

Review comment:
       The output will be released only when the split is finished, so I think the logic is OK here, just advancing the output creation to the moment of split addition.
   
   One concern in my mind is about adding an invalid splits to source reader. For example a split whose starting offset is ahead of ending offset, is added to the source reader, and the implementation of SplitReader just "swallows" this invalid split instead of marking it as finished, there will be an output leak. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] PatrickRen commented on pull request #18790: [FLINK-26018][connector/common] Create per-split output on split addition in SourceOperator

Posted by GitBox <gi...@apache.org>.
PatrickRen commented on pull request #18790:
URL: https://github.com/apache/flink/pull/18790#issuecomment-1072086494


   @AHeise 
   BP on 1.15: #19110, CI has passed ✅ 
   BP on 1.14: #19128, seems that E2E doesn't even start 🤔 I'll keep monitoring


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] PatrickRen commented on a change in pull request #18790: [FLINK-26018][connector/common] Create per-split output on split addition in SourceOperator

Posted by GitBox <gi...@apache.org>.
PatrickRen commented on a change in pull request #18790:
URL: https://github.com/apache/flink/pull/18790#discussion_r809647311



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java
##########
@@ -516,7 +525,14 @@ public void handleOperatorEvent(OperatorEvent event) {
             checkWatermarkAlignment();
         } else if (event instanceof AddSplitEvent) {
             try {
-                sourceReader.addSplits(((AddSplitEvent<SplitT>) event).splits(splitSerializer));
+                List<SplitT> newSplits = ((AddSplitEvent<SplitT>) event).splits(splitSerializer);

Review comment:
       Yes
   
   https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java#L1426




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot commented on pull request #18790: [FLINK-26018][connector/common] Create per-split output on split addition in SourceOperator

Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #18790:
URL: https://github.com/apache/flink/pull/18790#issuecomment-1041025841


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f11bc4e5c50b90d5410db10c4ae532ca888a3a48 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] MartijnVisser commented on pull request #18790: [FLINK-26018][connector/common] Create per-split output on split addition in SourceOperator

Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on pull request #18790:
URL: https://github.com/apache/flink/pull/18790#issuecomment-1042708445


   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on pull request #18790: [FLINK-26018][connector/common] Create per-split output on split addition in SourceOperator

Posted by GitBox <gi...@apache.org>.
AHeise commented on pull request #18790:
URL: https://github.com/apache/flink/pull/18790#issuecomment-1064841625


   The analysis of the problem and the solution are great. 
   
   Ideally, I'd just embed it in `SourceReaderBase#addSplits` and eagerly call `ReaderOutput#createOutputForSplit` but unfortunately we don't have access to the main `ReaderOutput` there (something that we should change, but it's public and it's high noon :/). It still feels hacky to call `ReaderOutput#createOutputForSplit` twice but the contract says that it will only recreate it once, so I guess it's okay. Could you please create a follow-up ticket for that?
   
   I'll take an in-depth look at your solution now.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] fapaul commented on a change in pull request #18790: [FLINK-26018][connector/common] Create per-split output on split addition in SourceOperator

Posted by GitBox <gi...@apache.org>.
fapaul commented on a change in pull request #18790:
URL: https://github.com/apache/flink/pull/18790#discussion_r809110464



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java
##########
@@ -423,6 +422,16 @@ private DataInputStatus emitNextNotReading(DataOutput<OUT> output) throws Except
         }
     }
 
+    private void initializeMainOutput(DataOutput<OUT> output) {
+        currentMainOutput = eventTimeLogic.createMainOutput(output, this::onWatermarkEmitted);
+        initializeLatencyMarkerEmitter(output);
+        lastInvokedOutput = output;
+        outputPendingSplits.forEach(
+                split -> currentMainOutput.createOutputForSplit(split.splitId()));

Review comment:
       I am wondering whether it might be problematic that you initialize all the outputs already but do not release them.
   
   Previously the outputs were initialized one by one here [1] and released when moving to the next split[2]
   
   [1] https://github.com/apache/flink/blob/106280e10a96d729943985986198b942446197d9/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/reader/SourceReaderBase.java#L327
   [2] https://github.com/apache/flink/blob/106280e10a96d729943985986198b942446197d9/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/reader/SourceReaderBase.java#L195

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java
##########
@@ -516,7 +525,14 @@ public void handleOperatorEvent(OperatorEvent event) {
             checkWatermarkAlignment();
         } else if (event instanceof AddSplitEvent) {
             try {
-                sourceReader.addSplits(((AddSplitEvent<SplitT>) event).splits(splitSerializer));
+                List<SplitT> newSplits = ((AddSplitEvent<SplitT>) event).splits(splitSerializer);

Review comment:
       Is `handleOperatorEvent` executed by the mailbox thread as the other methods?

##########
File path: flink-connectors/flink-connector-base/src/test/java/org/apache/flink/connector/base/source/reader/SourceReaderBaseTest.java
##########
@@ -239,6 +257,77 @@ void testPollNextReturnMoreAvailableWhenAllSplitFetcherCloseWithLeftoverElementI
                 .isEqualTo(InputStatus.MORE_AVAILABLE);
     }
 
+    @Test
+    void testPerSplitWatermark() throws Exception {
+        MockSplitReader mockSplitReader =
+                MockSplitReader.newBuilder()
+                        .setNumRecordsPerSplitPerFetch(3)
+                        .setBlockingFetch(true)
+                        .build();
+
+        MockSourceReader reader =
+                new MockSourceReader(
+                        new FutureCompletingBlockingQueue<>(),
+                        () -> mockSplitReader,
+                        new Configuration(),
+                        new TestingReaderContext());
+
+        SourceOperator<Integer, MockSourceSplit> sourceOperator =
+                createTestOperator(
+                        reader,
+                        WatermarkStrategy.forGenerator(
+                                (context) -> new OnEventWatermarkGenerator()),
+                        true);
+
+        MockSourceSplit splitA = new MockSourceSplit(0, 0, 3);
+        splitA.addRecord(100);
+        splitA.addRecord(200);
+        splitA.addRecord(300);
+
+        MockSourceSplit splitB = new MockSourceSplit(1, 0, 3);
+        splitB.addRecord(150);
+        splitB.addRecord(250);
+        splitB.addRecord(350);
+
+        AddSplitEvent<MockSourceSplit> addSplitsEvent =
+                new AddSplitEvent<>(Arrays.asList(splitA, splitB), new MockSourceSplitSerializer());
+        sourceOperator.handleOperatorEvent(addSplitsEvent);
+        WatermarkCollectingDataOutput output = new WatermarkCollectingDataOutput();
+
+        // First 3 records from split A should not generate any watermarks
+        CommonTestUtils.waitUtil(

Review comment:
       Why do you need this test loop? Can't you call `emitNext` the correct amount of time?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] MartijnVisser commented on pull request #18790: [FLINK-26018][connector/common] Create per-split output on split addition in SourceOperator

Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on pull request #18790:
URL: https://github.com/apache/flink/pull/18790#issuecomment-1041200353


   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] PatrickRen commented on pull request #18790: [FLINK-26018][connector/common] Create per-split output on split addition in SourceOperator

Posted by GitBox <gi...@apache.org>.
PatrickRen commented on pull request #18790:
URL: https://github.com/apache/flink/pull/18790#issuecomment-1066825690


   @AHeise @pnowojski I updated the PR just now and CI has passed. Could you take another look when you are available? Thanks~


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18790: [FLINK-26018][connector/common] Create per-split output on split addition in SourceOperator

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18790:
URL: https://github.com/apache/flink/pull/18790#issuecomment-1041025841


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31590",
       "triggerID" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31590",
       "triggerID" : "1041200353",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31590",
       "triggerID" : "1042708445",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * f11bc4e5c50b90d5410db10c4ae532ca888a3a48 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31590) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] pnowojski commented on pull request #18790: [FLINK-26018][connector/common] Create per-split output on split addition in SourceOperator

Posted by GitBox <gi...@apache.org>.
pnowojski commented on pull request #18790:
URL: https://github.com/apache/flink/pull/18790#issuecomment-1046915190


   I've chatted offline with @fapaul , and we both reached very similar conclusions to what I've posted above. Generic solution to this problem seems to be also described in the FLIP-27 ( https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface#FLIP27:RefactorSourceInterface-OverallDesign ), with sending the global watermark event from the SourceCoordinator/SplitEnumerator to the SourceOperators.
   
   1. Source readers/operators start with the "global watermark"/"max allowed watermark" set to MIN_VALUE
   2. SourceCoordinator can decide when to bump the "global watermark"/"max allowed watermark". In this case, it should bump it to MAX_VALUE after the initial/starting set of splits has been generated and sent to readers.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on pull request #18790: [FLINK-26018][connector/common] Create per-split output on split addition in SourceOperator

Posted by GitBox <gi...@apache.org>.
AHeise commented on pull request #18790:
URL: https://github.com/apache/flink/pull/18790#issuecomment-1068435916


   Benchmark looks good. Merging. Can you please create backports?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18790: [FLINK-26018][connector/common] Create per-split output on split addition in SourceOperator

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18790:
URL: https://github.com/apache/flink/pull/18790#issuecomment-1041025841


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31590",
       "triggerID" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31590",
       "triggerID" : "1041200353",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31590",
       "triggerID" : "1042708445",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "ebe36ecab291ff59275a8df96796a0c3800d2edb",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31744",
       "triggerID" : "ebe36ecab291ff59275a8df96796a0c3800d2edb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bdc3dca6e0c8fdb84c83d111e70d7adb866b07db",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "bdc3dca6e0c8fdb84c83d111e70d7adb866b07db",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * ebe36ecab291ff59275a8df96796a0c3800d2edb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31744) 
   * bdc3dca6e0c8fdb84c83d111e70d7adb866b07db UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18790: [FLINK-26018][connector/common] Create per-split output on split addition in SourceOperator

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18790:
URL: https://github.com/apache/flink/pull/18790#issuecomment-1041025841


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31590",
       "triggerID" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f11bc4e5c50b90d5410db10c4ae532ca888a3a48 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31590) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18790: [FLINK-26018][connector/common] Create per-split output on split addition in SourceOperator

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18790:
URL: https://github.com/apache/flink/pull/18790#issuecomment-1041025841


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31590",
       "triggerID" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31590",
       "triggerID" : "1041200353",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * f11bc4e5c50b90d5410db10c4ae532ca888a3a48 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31590) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] PatrickRen commented on a change in pull request #18790: [FLINK-26018][connector/common] Create per-split output on split addition in SourceOperator

Posted by GitBox <gi...@apache.org>.
PatrickRen commented on a change in pull request #18790:
URL: https://github.com/apache/flink/pull/18790#discussion_r809649007



##########
File path: flink-connectors/flink-connector-base/src/test/java/org/apache/flink/connector/base/source/reader/SourceReaderBaseTest.java
##########
@@ -239,6 +257,77 @@ void testPollNextReturnMoreAvailableWhenAllSplitFetcherCloseWithLeftoverElementI
                 .isEqualTo(InputStatus.MORE_AVAILABLE);
     }
 
+    @Test
+    void testPerSplitWatermark() throws Exception {
+        MockSplitReader mockSplitReader =
+                MockSplitReader.newBuilder()
+                        .setNumRecordsPerSplitPerFetch(3)
+                        .setBlockingFetch(true)
+                        .build();
+
+        MockSourceReader reader =
+                new MockSourceReader(
+                        new FutureCompletingBlockingQueue<>(),
+                        () -> mockSplitReader,
+                        new Configuration(),
+                        new TestingReaderContext());
+
+        SourceOperator<Integer, MockSourceSplit> sourceOperator =
+                createTestOperator(
+                        reader,
+                        WatermarkStrategy.forGenerator(
+                                (context) -> new OnEventWatermarkGenerator()),
+                        true);
+
+        MockSourceSplit splitA = new MockSourceSplit(0, 0, 3);
+        splitA.addRecord(100);
+        splitA.addRecord(200);
+        splitA.addRecord(300);
+
+        MockSourceSplit splitB = new MockSourceSplit(1, 0, 3);
+        splitB.addRecord(150);
+        splitB.addRecord(250);
+        splitB.addRecord(350);
+
+        AddSplitEvent<MockSourceSplit> addSplitsEvent =
+                new AddSplitEvent<>(Arrays.asList(splitA, splitB), new MockSourceSplitSerializer());
+        sourceOperator.handleOperatorEvent(addSplitsEvent);
+        WatermarkCollectingDataOutput output = new WatermarkCollectingDataOutput();
+
+        // First 3 records from split A should not generate any watermarks
+        CommonTestUtils.waitUtil(

Review comment:
       `SplitReader` is working in the individual fetcher thread, so it's possible that an invoke of `emitNext` doesn't emit anything




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18790: [FLINK-26018][connector/common] Create per-split output on split addition in SourceOperator

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18790:
URL: https://github.com/apache/flink/pull/18790#issuecomment-1041025841


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31590",
       "triggerID" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31590",
       "triggerID" : "1041200353",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f11bc4e5c50b90d5410db10c4ae532ca888a3a48",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31590",
       "triggerID" : "1042708445",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "ebe36ecab291ff59275a8df96796a0c3800d2edb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ebe36ecab291ff59275a8df96796a0c3800d2edb",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f11bc4e5c50b90d5410db10c4ae532ca888a3a48 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31590) 
   * ebe36ecab291ff59275a8df96796a0c3800d2edb UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on a change in pull request #18790: [FLINK-26018][connector/common] Create per-split output on split addition in SourceOperator

Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #18790:
URL: https://github.com/apache/flink/pull/18790#discussion_r824445935



##########
File path: flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/reader/SourceReaderBase.java
##########
@@ -324,6 +324,8 @@ private SplitContext(String splitId, SplitStateT state) {
 
         SourceOutput<T> getOrCreateSplitOutput(ReaderOutput<T> mainOutput) {
             if (sourceOutput == null) {
+                // The split output should have been created when AddSplitsEvent was processed in
+                // SourceOperator. Here we just use this method to get the created output.

Review comment:
       ```suggestion
                   // SourceOperator. Here we just use this method to get the previously created output.
   ```

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java
##########
@@ -516,7 +525,14 @@ public void handleOperatorEvent(OperatorEvent event) {
             checkWatermarkAlignment();
         } else if (event instanceof AddSplitEvent) {
             try {
-                sourceReader.addSplits(((AddSplitEvent<SplitT>) event).splits(splitSerializer));
+                List<SplitT> newSplits = ((AddSplitEvent<SplitT>) event).splits(splitSerializer);
+                if (operatingMode == OperatingMode.OUTPUT_NOT_INITIALIZED) {
+                    outputPendingSplits.addAll(newSplits);
+                } else {
+                    newSplits.forEach(

Review comment:
       So I guess we have two cases:
   - bulk addition of initial splits before first `emitNext`
   - bulk addition of initial splits after first `emitNext`
   
   You should ensure that both cases are covered by tests. I'd also recommend to add inline (java)docs that spell out the cases explicitly. For that I'd probably extract this `forEach` to a private method (e.g., `setupInitialSplit` or find a better name).

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java
##########
@@ -423,6 +422,16 @@ private DataInputStatus emitNextNotReading(DataOutput<OUT> output) throws Except
         }
     }
 
+    private void initializeMainOutput(DataOutput<OUT> output) {
+        currentMainOutput = eventTimeLogic.createMainOutput(output, this::onWatermarkEmitted);
+        initializeLatencyMarkerEmitter(output);
+        lastInvokedOutput = output;
+        outputPendingSplits.forEach(
+                split -> currentMainOutput.createOutputForSplit(split.splitId()));

Review comment:
       For your concern, we should update the javadoc of SplitReader to say that you always have to release.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise merged pull request #18790: [FLINK-26018][connector/common] Create per-split output on split addition in SourceOperator

Posted by GitBox <gi...@apache.org>.
AHeise merged pull request #18790:
URL: https://github.com/apache/flink/pull/18790


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] PatrickRen commented on pull request #18790: [FLINK-26018][connector/common] Create per-split output on split addition in SourceOperator

Posted by GitBox <gi...@apache.org>.
PatrickRen commented on pull request #18790:
URL: https://github.com/apache/flink/pull/18790#issuecomment-1066562809


   Thanks @pnowojski @AHeise for the review! I'll update the PR ASAP. Hopefully I'll push my update today. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org