You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/02/10 12:00:31 UTC

[GitHub] [flink] pnowojski opened a new pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

pnowojski opened a new pull request #18702:
URL: https://github.com/apache/flink/pull/18702


   ## What is the purpose of the change
   
   This PR integrates with FLINK-24440 and it implements `SourceOperator` level handling of watermark alignment.
   
   ## Verifying this change
   
   This PR adds a dedicated unit test for the `SourceOperator` changes.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (yes / **no**)
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
     - The serializers: (yes / **no** / don't know)
     - The runtime per-record code paths (performance sensitive): (**yes** / no / don't know)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know)
     - The S3 file system connector: (yes / **no** / don't know)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (**yes** / no)
     - If yes, how is the feature documented? (not applicable / docs / JavaDocs / **not documented**)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18702:
URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118",
       "triggerID" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6b2d3af9ac7e956e358c6058809f49368d6c9c0a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6b2d3af9ac7e956e358c6058809f49368d6c9c0a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1c798c2c84e8285f7a7784038583a194bc42b3ea Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118) 
   * 6b2d3af9ac7e956e358c6058809f49368d6c9c0a UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] pnowojski commented on a change in pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
pnowojski commented on a change in pull request #18702:
URL: https://github.com/apache/flink/pull/18702#discussion_r803730641



##########
File path: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/source/CollectingDataOutput.java
##########
@@ -47,11 +47,16 @@ public void emitWatermarkStatus(WatermarkStatus watermarkStatus) throws Exceptio
 
     @Override
     public void emitRecord(StreamRecord<E> streamRecord) throws Exception {
-        events.add(streamRecord);
+        // Bypass issues with object re-use disabled by copying the record
+        events.add(streamRecord.copy(streamRecord.getValue()));

Review comment:
       `StreamRecord` instance is being re-used by the sources. While this `CollectingDataOutput` is collecting them. So `CollectingDataOutput` passed to a source that two emitted different records, would first emit end up with `events` list having two entries pointing to the same `StreamRecord` instance with the same value. The secondly emitted record would replace the previously collected value :(




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] dawidwys commented on a change in pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #18702:
URL: https://github.com/apache/flink/pull/18702#discussion_r803735949



##########
File path: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/SourceOperatorAlignmentTest.java
##########
@@ -0,0 +1,155 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package org.apache.flink.streaming.api.operators;
+
+import org.apache.flink.api.common.eventtime.Watermark;
+import org.apache.flink.api.common.eventtime.WatermarkGenerator;
+import org.apache.flink.api.common.eventtime.WatermarkOutput;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.connector.source.mocks.MockSourceSplit;
+import org.apache.flink.api.connector.source.mocks.MockSourceSplitSerializer;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.runtime.source.coordinator.SourceCoordinator.WatermarkAlignmentParams;
+import org.apache.flink.runtime.source.event.AddSplitEvent;
+import org.apache.flink.runtime.source.event.ReportedWatermarkEvent;
+import org.apache.flink.runtime.source.event.WatermarkAlignmentEvent;
+import org.apache.flink.streaming.api.operators.source.CollectingDataOutput;
+import org.apache.flink.streaming.runtime.io.DataInputStatus;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.contains;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+/** Unit test for {@link SourceOperator} watermark alignment. */
+@SuppressWarnings("serial")
+public class SourceOperatorAlignmentTest {

Review comment:
       sure, e.g. https://github.com/apache/flink/blob/master/flink-formats/flink-avro/src/test/java/org/apache/flink/formats/avro/AvroBulkFormatTest.java
   
   It's mostly about using different imports: e.g. `org.junit.Test` -> `org.junit.jupiter.api.Test` (the annotation names also changed slightly)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18702:
URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118",
       "triggerID" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2cc0ad9e6652642762f48c030a2ad5339881123b",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2cc0ad9e6652642762f48c030a2ad5339881123b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1c798c2c84e8285f7a7784038583a194bc42b3ea Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118) 
   * 2cc0ad9e6652642762f48c030a2ad5339881123b UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18702:
URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118",
       "triggerID" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f730132ffa59826d88df89d4bd6ac1e5c93639d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31165",
       "triggerID" : "f730132ffa59826d88df89d4bd6ac1e5c93639d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "065960d2962be7fc5df6098a93cab9e94df7b6ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31144",
       "triggerID" : "065960d2962be7fc5df6098a93cab9e94df7b6ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "41edf88f3c2942523f9f91e4c0a26265cf2e97ec",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31174",
       "triggerID" : "41edf88f3c2942523f9f91e4c0a26265cf2e97ec",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7aa690d4fb463b9f3bb6df7b26ad1b22472f8ea5",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31234",
       "triggerID" : "7aa690d4fb463b9f3bb6df7b26ad1b22472f8ea5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "161836b917f75ec4bc8a82e5a7d4abd601437fd8",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31248",
       "triggerID" : "161836b917f75ec4bc8a82e5a7d4abd601437fd8",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 41edf88f3c2942523f9f91e4c0a26265cf2e97ec Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31174) 
   * 7aa690d4fb463b9f3bb6df7b26ad1b22472f8ea5 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31234) 
   * 161836b917f75ec4bc8a82e5a7d4abd601437fd8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31248) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18702:
URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118",
       "triggerID" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f730132ffa59826d88df89d4bd6ac1e5c93639d3",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f730132ffa59826d88df89d4bd6ac1e5c93639d3",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1c798c2c84e8285f7a7784038583a194bc42b3ea Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118) 
   * f730132ffa59826d88df89d4bd6ac1e5c93639d3 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18702:
URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118",
       "triggerID" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f730132ffa59826d88df89d4bd6ac1e5c93639d3",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31165",
       "triggerID" : "f730132ffa59826d88df89d4bd6ac1e5c93639d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "065960d2962be7fc5df6098a93cab9e94df7b6ff",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31144",
       "triggerID" : "065960d2962be7fc5df6098a93cab9e94df7b6ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "41edf88f3c2942523f9f91e4c0a26265cf2e97ec",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "41edf88f3c2942523f9f91e4c0a26265cf2e97ec",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1c798c2c84e8285f7a7784038583a194bc42b3ea Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118) 
   * f730132ffa59826d88df89d4bd6ac1e5c93639d3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31165) 
   * 065960d2962be7fc5df6098a93cab9e94df7b6ff Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31144) 
   * 41edf88f3c2942523f9f91e4c0a26265cf2e97ec UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18702:
URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118",
       "triggerID" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1c798c2c84e8285f7a7784038583a194bc42b3ea Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18702:
URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118",
       "triggerID" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2cc0ad9e6652642762f48c030a2ad5339881123b",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2cc0ad9e6652642762f48c030a2ad5339881123b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1c798c2c84e8285f7a7784038583a194bc42b3ea Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118) 
   * 2cc0ad9e6652642762f48c030a2ad5339881123b UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18702:
URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118",
       "triggerID" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f730132ffa59826d88df89d4bd6ac1e5c93639d3",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31165",
       "triggerID" : "f730132ffa59826d88df89d4bd6ac1e5c93639d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "065960d2962be7fc5df6098a93cab9e94df7b6ff",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "065960d2962be7fc5df6098a93cab9e94df7b6ff",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1c798c2c84e8285f7a7784038583a194bc42b3ea Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118) 
   * f730132ffa59826d88df89d4bd6ac1e5c93639d3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31165) 
   * 065960d2962be7fc5df6098a93cab9e94df7b6ff UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18702:
URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118",
       "triggerID" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f730132ffa59826d88df89d4bd6ac1e5c93639d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31165",
       "triggerID" : "f730132ffa59826d88df89d4bd6ac1e5c93639d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "065960d2962be7fc5df6098a93cab9e94df7b6ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31144",
       "triggerID" : "065960d2962be7fc5df6098a93cab9e94df7b6ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "41edf88f3c2942523f9f91e4c0a26265cf2e97ec",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31174",
       "triggerID" : "41edf88f3c2942523f9f91e4c0a26265cf2e97ec",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7aa690d4fb463b9f3bb6df7b26ad1b22472f8ea5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7aa690d4fb463b9f3bb6df7b26ad1b22472f8ea5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 41edf88f3c2942523f9f91e4c0a26265cf2e97ec Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31174) 
   * 7aa690d4fb463b9f3bb6df7b26ad1b22472f8ea5 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18702:
URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118",
       "triggerID" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f730132ffa59826d88df89d4bd6ac1e5c93639d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31165",
       "triggerID" : "f730132ffa59826d88df89d4bd6ac1e5c93639d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "065960d2962be7fc5df6098a93cab9e94df7b6ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31144",
       "triggerID" : "065960d2962be7fc5df6098a93cab9e94df7b6ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "41edf88f3c2942523f9f91e4c0a26265cf2e97ec",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31174",
       "triggerID" : "41edf88f3c2942523f9f91e4c0a26265cf2e97ec",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7aa690d4fb463b9f3bb6df7b26ad1b22472f8ea5",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31234",
       "triggerID" : "7aa690d4fb463b9f3bb6df7b26ad1b22472f8ea5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "161836b917f75ec4bc8a82e5a7d4abd601437fd8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "161836b917f75ec4bc8a82e5a7d4abd601437fd8",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 41edf88f3c2942523f9f91e4c0a26265cf2e97ec Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31174) 
   * 7aa690d4fb463b9f3bb6df7b26ad1b22472f8ea5 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31234) 
   * 161836b917f75ec4bc8a82e5a7d4abd601437fd8 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18702:
URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118",
       "triggerID" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f730132ffa59826d88df89d4bd6ac1e5c93639d3",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31165",
       "triggerID" : "f730132ffa59826d88df89d4bd6ac1e5c93639d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "065960d2962be7fc5df6098a93cab9e94df7b6ff",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31144",
       "triggerID" : "065960d2962be7fc5df6098a93cab9e94df7b6ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "41edf88f3c2942523f9f91e4c0a26265cf2e97ec",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31174",
       "triggerID" : "41edf88f3c2942523f9f91e4c0a26265cf2e97ec",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f730132ffa59826d88df89d4bd6ac1e5c93639d3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31165) 
   * 065960d2962be7fc5df6098a93cab9e94df7b6ff Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31144) 
   * 41edf88f3c2942523f9f91e4c0a26265cf2e97ec Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31174) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18702:
URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118",
       "triggerID" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f730132ffa59826d88df89d4bd6ac1e5c93639d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31165",
       "triggerID" : "f730132ffa59826d88df89d4bd6ac1e5c93639d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "065960d2962be7fc5df6098a93cab9e94df7b6ff",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31144",
       "triggerID" : "065960d2962be7fc5df6098a93cab9e94df7b6ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "41edf88f3c2942523f9f91e4c0a26265cf2e97ec",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31174",
       "triggerID" : "41edf88f3c2942523f9f91e4c0a26265cf2e97ec",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 065960d2962be7fc5df6098a93cab9e94df7b6ff Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31144) 
   * 41edf88f3c2942523f9f91e4c0a26265cf2e97ec Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31174) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18702:
URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118",
       "triggerID" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "589c2ba8040ed9e7e00d4409a368a2249471f8e4",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "589c2ba8040ed9e7e00d4409a368a2249471f8e4",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1c798c2c84e8285f7a7784038583a194bc42b3ea Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118) 
   * 589c2ba8040ed9e7e00d4409a368a2249471f8e4 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18702:
URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118",
       "triggerID" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f730132ffa59826d88df89d4bd6ac1e5c93639d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31165",
       "triggerID" : "f730132ffa59826d88df89d4bd6ac1e5c93639d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "065960d2962be7fc5df6098a93cab9e94df7b6ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31144",
       "triggerID" : "065960d2962be7fc5df6098a93cab9e94df7b6ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "41edf88f3c2942523f9f91e4c0a26265cf2e97ec",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31174",
       "triggerID" : "41edf88f3c2942523f9f91e4c0a26265cf2e97ec",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7aa690d4fb463b9f3bb6df7b26ad1b22472f8ea5",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31234",
       "triggerID" : "7aa690d4fb463b9f3bb6df7b26ad1b22472f8ea5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 41edf88f3c2942523f9f91e4c0a26265cf2e97ec Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31174) 
   * 7aa690d4fb463b9f3bb6df7b26ad1b22472f8ea5 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31234) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] pnowojski commented on a change in pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
pnowojski commented on a change in pull request #18702:
URL: https://github.com/apache/flink/pull/18702#discussion_r804506477



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java
##########
@@ -529,4 +625,43 @@ public void forceStop() {
             forcedStopFuture.complete(null);
         }
     }
+
+    /**
+     * TODO: this wrapper introduces a small performance regression when used. Maybe we can avoid

Review comment:
       👍 done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18702:
URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118",
       "triggerID" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1c798c2c84e8285f7a7784038583a194bc42b3ea Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18702:
URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118",
       "triggerID" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1c798c2c84e8285f7a7784038583a194bc42b3ea Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18702:
URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118",
       "triggerID" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f730132ffa59826d88df89d4bd6ac1e5c93639d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31165",
       "triggerID" : "f730132ffa59826d88df89d4bd6ac1e5c93639d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "065960d2962be7fc5df6098a93cab9e94df7b6ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31144",
       "triggerID" : "065960d2962be7fc5df6098a93cab9e94df7b6ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "41edf88f3c2942523f9f91e4c0a26265cf2e97ec",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31174",
       "triggerID" : "41edf88f3c2942523f9f91e4c0a26265cf2e97ec",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7aa690d4fb463b9f3bb6df7b26ad1b22472f8ea5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31234",
       "triggerID" : "7aa690d4fb463b9f3bb6df7b26ad1b22472f8ea5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "161836b917f75ec4bc8a82e5a7d4abd601437fd8",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31248",
       "triggerID" : "161836b917f75ec4bc8a82e5a7d4abd601437fd8",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 161836b917f75ec4bc8a82e5a7d4abd601437fd8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31248) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] pnowojski commented on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
pnowojski commented on pull request #18702:
URL: https://github.com/apache/flink/pull/18702#issuecomment-1038909633


   Two test failures unrelated (one already fixed on the master https://issues.apache.org/jira/browse/FLINK-26036). Private build is green. Will merge in a second.
   
   https://dev.azure.com/pnowojski/Flink/_build/results?buildId=627&view=results


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] pnowojski commented on a change in pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
pnowojski commented on a change in pull request #18702:
URL: https://github.com/apache/flink/pull/18702#discussion_r804581514



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java
##########
@@ -438,6 +499,11 @@ public void snapshotState(StateSnapshotContext context) throws Exception {
     @Override
     public CompletableFuture<?> getAvailableFuture() {
         switch (operatingMode) {
+            case WAITING_FOR_ALIGNMENT:
+                return availabilityHelper.update(

Review comment:
       nope, good catch. I guess I got carried away coping pasting the code. This future can not ever be null.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18702:
URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118",
       "triggerID" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1c798c2c84e8285f7a7784038583a194bc42b3ea Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] dawidwys commented on a change in pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #18702:
URL: https://github.com/apache/flink/pull/18702#discussion_r803680107



##########
File path: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/SourceOperatorAlignmentTest.java
##########
@@ -0,0 +1,155 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package org.apache.flink.streaming.api.operators;
+
+import org.apache.flink.api.common.eventtime.Watermark;
+import org.apache.flink.api.common.eventtime.WatermarkGenerator;
+import org.apache.flink.api.common.eventtime.WatermarkOutput;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.connector.source.mocks.MockSourceSplit;
+import org.apache.flink.api.connector.source.mocks.MockSourceSplitSerializer;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.runtime.source.coordinator.SourceCoordinator.WatermarkAlignmentParams;
+import org.apache.flink.runtime.source.event.AddSplitEvent;
+import org.apache.flink.runtime.source.event.ReportedWatermarkEvent;
+import org.apache.flink.runtime.source.event.WatermarkAlignmentEvent;
+import org.apache.flink.streaming.api.operators.source.CollectingDataOutput;
+import org.apache.flink.streaming.runtime.io.DataInputStatus;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.contains;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+/** Unit test for {@link SourceOperator} watermark alignment. */
+@SuppressWarnings("serial")
+public class SourceOperatorAlignmentTest {

Review comment:
       Could we update it to JUnit 5?

##########
File path: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/SourceOperatorAlignmentTest.java
##########
@@ -0,0 +1,155 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package org.apache.flink.streaming.api.operators;
+
+import org.apache.flink.api.common.eventtime.Watermark;
+import org.apache.flink.api.common.eventtime.WatermarkGenerator;
+import org.apache.flink.api.common.eventtime.WatermarkOutput;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.connector.source.mocks.MockSourceSplit;
+import org.apache.flink.api.connector.source.mocks.MockSourceSplitSerializer;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.runtime.source.coordinator.SourceCoordinator.WatermarkAlignmentParams;
+import org.apache.flink.runtime.source.event.AddSplitEvent;
+import org.apache.flink.runtime.source.event.ReportedWatermarkEvent;
+import org.apache.flink.runtime.source.event.WatermarkAlignmentEvent;
+import org.apache.flink.streaming.api.operators.source.CollectingDataOutput;
+import org.apache.flink.streaming.runtime.io.DataInputStatus;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.contains;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+/** Unit test for {@link SourceOperator} watermark alignment. */
+@SuppressWarnings("serial")
+public class SourceOperatorAlignmentTest {
+
+    @Nullable private SourceOperatorTestContext context;
+    @Nullable private SourceOperator<Integer, MockSourceSplit> operator;
+
+    @Before
+    public void setup() throws Exception {
+        context =
+                new SourceOperatorTestContext(
+                        false,
+                        WatermarkStrategy.forGenerator(ctx -> new PunctuatedGenerator())
+                                .withTimestampAssigner((r, t) -> r),
+                        new WatermarkAlignmentParams(100, "group1", 1));
+        operator = context.getOperator();
+    }
+
+    @After
+    public void tearDown() throws Exception {
+        context.close();
+        context = null;
+        operator = null;
+    }
+
+    @Test
+    public void testHandleAddSplitsEvent() throws Exception {

Review comment:
       I guess we should update the test name ;)

##########
File path: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/source/CollectingDataOutput.java
##########
@@ -47,11 +47,16 @@ public void emitWatermarkStatus(WatermarkStatus watermarkStatus) throws Exceptio
 
     @Override
     public void emitRecord(StreamRecord<E> streamRecord) throws Exception {
-        events.add(streamRecord);
+        // Bypass issues with object re-use disabled by copying the record
+        events.add(streamRecord.copy(streamRecord.getValue()));

Review comment:
       out of curiosity, what was the issue here?

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java
##########
@@ -370,7 +413,14 @@ public DataInputStatus emitNext(DataOutput<OUT> output) throws Exception {
     private DataInputStatus emitNextNotReading(DataOutput<OUT> output) throws Exception {
         switch (operatingMode) {
             case OUTPUT_NOT_INITIALIZED:
-                currentMainOutput = eventTimeLogic.createMainOutput(output);
+                watermarkTrackingOutput = new WatermarkTrackingOutput<>(output);
+                if (watermarkAlignmentParams.isEnabled()) {
+                    processingTimeService.scheduleWithFixedDelay(
+                            this::emitLatestWatermark,
+                            watermarkAlignmentParams.getUpdateInterval(),
+                            watermarkAlignmentParams.getUpdateInterval());
+                }
+                currentMainOutput = eventTimeLogic.createMainOutput(watermarkTrackingOutput);

Review comment:
       thiis will most probably cause some tests to fail. 
   
   We should use the original `output` if watermark alignment is disabled.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18702:
URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118",
       "triggerID" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f730132ffa59826d88df89d4bd6ac1e5c93639d3",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31165",
       "triggerID" : "f730132ffa59826d88df89d4bd6ac1e5c93639d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "065960d2962be7fc5df6098a93cab9e94df7b6ff",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31144",
       "triggerID" : "065960d2962be7fc5df6098a93cab9e94df7b6ff",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1c798c2c84e8285f7a7784038583a194bc42b3ea Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118) 
   * f730132ffa59826d88df89d4bd6ac1e5c93639d3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31165) 
   * 065960d2962be7fc5df6098a93cab9e94df7b6ff Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31144) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18702:
URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1c798c2c84e8285f7a7784038583a194bc42b3ea UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] pnowojski commented on a change in pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
pnowojski commented on a change in pull request #18702:
URL: https://github.com/apache/flink/pull/18702#discussion_r804471668



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java
##########
@@ -529,4 +625,43 @@ public void forceStop() {
             forcedStopFuture.complete(null);
         }
     }
+
+    /**
+     * TODO: this wrapper introduces a small performance regression when used. Maybe we can avoid
+     * using it somehow?
+     */
+    private class WatermarkTrackingOutput<OUT> implements DataOutput<OUT> {
+        private final DataOutput<OUT> output;
+        private long lastWatermark;

Review comment:
       In that case this source would report `0` and I guess would never be blocked (as long as any actually emitted watermarks are positive). But maybe indeed it's a bit cleaner to use `org.apache.flink.streaming.api.watermark.Watermark#UNINITIALIZED` here.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] dawidwys commented on a change in pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #18702:
URL: https://github.com/apache/flink/pull/18702#discussion_r804523534



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java
##########
@@ -370,7 +411,17 @@ public DataInputStatus emitNext(DataOutput<OUT> output) throws Exception {
     private DataInputStatus emitNextNotReading(DataOutput<OUT> output) throws Exception {
         switch (operatingMode) {
             case OUTPUT_NOT_INITIALIZED:
-                currentMainOutput = eventTimeLogic.createMainOutput(output);
+                if (watermarkAlignmentParams.isEnabled()) {
+                    // Only wrap the output when watermark alignment is enabled, as otherwise this
+                    // introduces a small performance regression (probably because of an extra
+                    // virtual call)
+                    processingTimeService.scheduleWithFixedDelay(
+                            this::emitLatestWatermark,
+                            watermarkAlignmentParams.getUpdateInterval(),
+                            watermarkAlignmentParams.getUpdateInterval());
+                }
+                currentMainOutput =
+                        eventTimeLogic.createMainOutput(output, this::onWatermarkEmitted);

Review comment:
       Does it make sense to pass an empty method instead of `onWatermarkEmitted` if watermark alignment is disabled? Right now we execute some unnecessary logic on every watermark if the alignment is disabled.

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/source/WatermarkToDataOutput.java
##########
@@ -35,12 +36,21 @@
 public final class WatermarkToDataOutput implements WatermarkOutput {
 
     private final PushingAsyncDataInput.DataOutput<?> output;
+    private final TimestampsAndWatermarks.OnWatermarkEmitted watermarkEmitted;
     private long maxWatermarkSoFar;
     private boolean isIdle;
 
-    /** Creates a new WatermarkOutput against the given DataOutput. */
+    @VisibleForTesting
     public WatermarkToDataOutput(PushingAsyncDataInput.DataOutput<?> output) {
+        this(output, watermark -> {});
+    }
+
+    /** Creates a new WatermarkOutput against the given DataOutput. */
+    public WatermarkToDataOutput(
+            PushingAsyncDataInput.DataOutput<?> output,
+            TimestampsAndWatermarks.OnWatermarkEmitted watermarkEmitted) {
         this.output = checkNotNull(output);
+        this.watermarkEmitted = watermarkEmitted;

Review comment:
       nit: for the sake of consistency: `checkNotNull`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] pnowojski merged pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
pnowojski merged pull request #18702:
URL: https://github.com/apache/flink/pull/18702


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18702:
URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118",
       "triggerID" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f730132ffa59826d88df89d4bd6ac1e5c93639d3",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f730132ffa59826d88df89d4bd6ac1e5c93639d3",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1c798c2c84e8285f7a7784038583a194bc42b3ea Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118) 
   * f730132ffa59826d88df89d4bd6ac1e5c93639d3 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] dawidwys commented on a change in pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #18702:
URL: https://github.com/apache/flink/pull/18702#discussion_r804437340



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java
##########
@@ -529,4 +625,43 @@ public void forceStop() {
             forcedStopFuture.complete(null);
         }
     }
+
+    /**
+     * TODO: this wrapper introduces a small performance regression when used. Maybe we can avoid
+     * using it somehow?
+     */
+    private class WatermarkTrackingOutput<OUT> implements DataOutput<OUT> {
+        private final DataOutput<OUT> output;
+        private long lastWatermark;

Review comment:
       What is the initial value for the `lastWatermark`? What happens if there are no watermarks emitted by the reader?

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java
##########
@@ -529,4 +625,43 @@ public void forceStop() {
             forcedStopFuture.complete(null);
         }
     }
+
+    /**
+     * TODO: this wrapper introduces a small performance regression when used. Maybe we can avoid

Review comment:
       Unless we pass a callback in the `eventTimeLogic.createMainOutput`, after all it's a private stack.

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java
##########
@@ -325,8 +366,12 @@ private void stopInternalServices() {
 
     public CompletableFuture<Void> stop(StopMode mode) {
         switch (operatingMode) {
+            case WAITING_FOR_ALIGNMENT:
             case OUTPUT_NOT_INITIALIZED:
             case READING:
+                if (operatingMode == OperatingMode.WAITING_FOR_ALIGNMENT) {
+                    waitingForAlignmentFuture.complete(null);

Review comment:
       Don't know if better, but can't we instead combine the `waitingForAlignmentFuture` with `finished` in `getAvailableFuture`?
   
   Something like:
   ```
       public CompletableFuture<?> getAvailableFuture() {
   ....
               case WAITING_FOR_ALIGNMENT:
                   return availabilityHelper.update(waitingForAlignmentFuture);
               case OUTPUT_NOT_INITIALIZED:
               case READING:
                   return availabilityHelper.update(sourceReader.isAvailable());
   ```

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java
##########
@@ -529,4 +625,43 @@ public void forceStop() {
             forcedStopFuture.complete(null);
         }
     }
+
+    /**
+     * TODO: this wrapper introduces a small performance regression when used. Maybe we can avoid

Review comment:
       Ok, I think it would not work, because we call `checkWatermarkAlignment` in the `WatermarkTrackingOutput` :(




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] dawidwys commented on a change in pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #18702:
URL: https://github.com/apache/flink/pull/18702#discussion_r804520295



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java
##########
@@ -438,6 +499,11 @@ public void snapshotState(StateSnapshotContext context) throws Exception {
     @Override
     public CompletableFuture<?> getAvailableFuture() {
         switch (operatingMode) {
+            case WAITING_FOR_ALIGNMENT:
+                return availabilityHelper.update(

Review comment:
       nit: is it possible that we are in the `WAITING_FOR_ALIGNMENT` state and the future is `null`?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] pnowojski commented on a change in pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
pnowojski commented on a change in pull request #18702:
URL: https://github.com/apache/flink/pull/18702#discussion_r804581514



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java
##########
@@ -438,6 +499,11 @@ public void snapshotState(StateSnapshotContext context) throws Exception {
     @Override
     public CompletableFuture<?> getAvailableFuture() {
         switch (operatingMode) {
+            case WAITING_FOR_ALIGNMENT:
+                return availabilityHelper.update(

Review comment:
       nope, good catch. I guess I got carried away coping pasting the code.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18702:
URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118",
       "triggerID" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f730132ffa59826d88df89d4bd6ac1e5c93639d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31165",
       "triggerID" : "f730132ffa59826d88df89d4bd6ac1e5c93639d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "065960d2962be7fc5df6098a93cab9e94df7b6ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31144",
       "triggerID" : "065960d2962be7fc5df6098a93cab9e94df7b6ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "41edf88f3c2942523f9f91e4c0a26265cf2e97ec",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31174",
       "triggerID" : "41edf88f3c2942523f9f91e4c0a26265cf2e97ec",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7aa690d4fb463b9f3bb6df7b26ad1b22472f8ea5",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31234",
       "triggerID" : "7aa690d4fb463b9f3bb6df7b26ad1b22472f8ea5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "161836b917f75ec4bc8a82e5a7d4abd601437fd8",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31248",
       "triggerID" : "161836b917f75ec4bc8a82e5a7d4abd601437fd8",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 7aa690d4fb463b9f3bb6df7b26ad1b22472f8ea5 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31234) 
   * 161836b917f75ec4bc8a82e5a7d4abd601437fd8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31248) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] pnowojski commented on a change in pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
pnowojski commented on a change in pull request #18702:
URL: https://github.com/apache/flink/pull/18702#discussion_r804469710



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java
##########
@@ -325,8 +366,12 @@ private void stopInternalServices() {
 
     public CompletableFuture<Void> stop(StopMode mode) {
         switch (operatingMode) {
+            case WAITING_FOR_ALIGNMENT:
             case OUTPUT_NOT_INITIALIZED:
             case READING:
+                if (operatingMode == OperatingMode.WAITING_FOR_ALIGNMENT) {
+                    waitingForAlignmentFuture.complete(null);

Review comment:
       I was thinking about it, initially I decided against it, but on the second thought maybe indeed that's better.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18702:
URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118",
       "triggerID" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f730132ffa59826d88df89d4bd6ac1e5c93639d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31165",
       "triggerID" : "f730132ffa59826d88df89d4bd6ac1e5c93639d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "065960d2962be7fc5df6098a93cab9e94df7b6ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31144",
       "triggerID" : "065960d2962be7fc5df6098a93cab9e94df7b6ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "41edf88f3c2942523f9f91e4c0a26265cf2e97ec",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31174",
       "triggerID" : "41edf88f3c2942523f9f91e4c0a26265cf2e97ec",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 41edf88f3c2942523f9f91e4c0a26265cf2e97ec Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31174) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot commented on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #18702:
URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] pnowojski commented on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
pnowojski commented on pull request #18702:
URL: https://github.com/apache/flink/pull/18702#issuecomment-1035032817


   The initial version was causing a small performance regression. After a couple of optimisations there is no visible regression when the watermark alignment is not used.
   ```
   "Benchmark","Mode","Threads","Samples","Score","Score Error (99.9%)","Unit","Param: sourceType"
   "org.apache.flink.benchmark.InputBenchmark.mapRebalanceMapSink","thrpt",1,30,9936.451429,255.258997,"ops/ms",LEGACY
   "org.apache.flink.benchmark.InputBenchmark.mapRebalanceMapSink","thrpt",1,30,14651.620943,383.589770,"ops/ms",F27_UNBOUNDED
   "org.apache.flink.benchmark.InputBenchmark.mapSink","thrpt",1,30,15669.405419,199.914977,"ops/ms",LEGACY
   "org.apache.flink.benchmark.InputBenchmark.mapSink","thrpt",1,30,30679.668113,265.650771,"ops/ms",F27_UNBOUNDED
   ```
   However with watermark alignment enabled, most likely due to the `WatermarkTrackingOutput` wrapper, there is a small (under 3%) extra overhead visible in the `mapSink.F27_UNBOUNDED` benchmark.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18702:
URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118",
       "triggerID" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f730132ffa59826d88df89d4bd6ac1e5c93639d3",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31165",
       "triggerID" : "f730132ffa59826d88df89d4bd6ac1e5c93639d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "065960d2962be7fc5df6098a93cab9e94df7b6ff",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31144",
       "triggerID" : "065960d2962be7fc5df6098a93cab9e94df7b6ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "41edf88f3c2942523f9f91e4c0a26265cf2e97ec",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31174",
       "triggerID" : "41edf88f3c2942523f9f91e4c0a26265cf2e97ec",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1c798c2c84e8285f7a7784038583a194bc42b3ea Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118) 
   * f730132ffa59826d88df89d4bd6ac1e5c93639d3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31165) 
   * 065960d2962be7fc5df6098a93cab9e94df7b6ff Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31144) 
   * 41edf88f3c2942523f9f91e4c0a26265cf2e97ec Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31174) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18702:
URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118",
       "triggerID" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "425f2f16c73e8302ec0be19b6c9994cfccfee40d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "425f2f16c73e8302ec0be19b6c9994cfccfee40d",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1c798c2c84e8285f7a7784038583a194bc42b3ea Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118) 
   * 425f2f16c73e8302ec0be19b6c9994cfccfee40d UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18702:
URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118",
       "triggerID" : "1c798c2c84e8285f7a7784038583a194bc42b3ea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f730132ffa59826d88df89d4bd6ac1e5c93639d3",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31165",
       "triggerID" : "f730132ffa59826d88df89d4bd6ac1e5c93639d3",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1c798c2c84e8285f7a7784038583a194bc42b3ea Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118) 
   * f730132ffa59826d88df89d4bd6ac1e5c93639d3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31165) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] pnowojski commented on a change in pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
pnowojski commented on a change in pull request #18702:
URL: https://github.com/apache/flink/pull/18702#discussion_r803731352



##########
File path: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/SourceOperatorAlignmentTest.java
##########
@@ -0,0 +1,155 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package org.apache.flink.streaming.api.operators;
+
+import org.apache.flink.api.common.eventtime.Watermark;
+import org.apache.flink.api.common.eventtime.WatermarkGenerator;
+import org.apache.flink.api.common.eventtime.WatermarkOutput;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.connector.source.mocks.MockSourceSplit;
+import org.apache.flink.api.connector.source.mocks.MockSourceSplitSerializer;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.runtime.source.coordinator.SourceCoordinator.WatermarkAlignmentParams;
+import org.apache.flink.runtime.source.event.AddSplitEvent;
+import org.apache.flink.runtime.source.event.ReportedWatermarkEvent;
+import org.apache.flink.runtime.source.event.WatermarkAlignmentEvent;
+import org.apache.flink.streaming.api.operators.source.CollectingDataOutput;
+import org.apache.flink.streaming.runtime.io.DataInputStatus;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.contains;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+/** Unit test for {@link SourceOperator} watermark alignment. */
+@SuppressWarnings("serial")
+public class SourceOperatorAlignmentTest {

Review comment:
       Hmmm, how so? Could you give me some pointer to an example test I could based on?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] dawidwys commented on a change in pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #18702:
URL: https://github.com/apache/flink/pull/18702#discussion_r804434912



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java
##########
@@ -529,4 +625,43 @@ public void forceStop() {
             forcedStopFuture.complete(null);
         }
     }
+
+    /**
+     * TODO: this wrapper introduces a small performance regression when used. Maybe we can avoid

Review comment:
       An idea to drop the `WatermarkTrackingOutput` would be to make the `eventTimeLogic.createMainOutput` return e.g. `InternalReaderOutput` that would have the method `getLastWatermark`.
   
   I believe that's the approach Arvid took in his PoC.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] dawidwys commented on a change in pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #18702:
URL: https://github.com/apache/flink/pull/18702#discussion_r804429303



##########
File path: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/source/CollectingDataOutput.java
##########
@@ -47,11 +47,16 @@ public void emitWatermarkStatus(WatermarkStatus watermarkStatus) throws Exceptio
 
     @Override
     public void emitRecord(StreamRecord<E> streamRecord) throws Exception {
-        events.add(streamRecord);
+        // Bypass issues with object re-use disabled by copying the record
+        events.add(streamRecord.copy(streamRecord.getValue()));

Review comment:
       Oh, ok, I see. Thanks for the explanation!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] pnowojski commented on a change in pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment

Posted by GitBox <gi...@apache.org>.
pnowojski commented on a change in pull request #18702:
URL: https://github.com/apache/flink/pull/18702#discussion_r804579899



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java
##########
@@ -370,7 +411,17 @@ public DataInputStatus emitNext(DataOutput<OUT> output) throws Exception {
     private DataInputStatus emitNextNotReading(DataOutput<OUT> output) throws Exception {
         switch (operatingMode) {
             case OUTPUT_NOT_INITIALIZED:
-                currentMainOutput = eventTimeLogic.createMainOutput(output);
+                if (watermarkAlignmentParams.isEnabled()) {
+                    // Only wrap the output when watermark alignment is enabled, as otherwise this
+                    // introduces a small performance regression (probably because of an extra
+                    // virtual call)
+                    processingTimeService.scheduleWithFixedDelay(
+                            this::emitLatestWatermark,
+                            watermarkAlignmentParams.getUpdateInterval(),
+                            watermarkAlignmentParams.getUpdateInterval());
+                }
+                currentMainOutput =
+                        eventTimeLogic.createMainOutput(output, this::onWatermarkEmitted);

Review comment:
       Hard to make a call. It would be more descriptive to pass an empty callback, but it would be more complicated and `lastEmittedWatermark` would be not as consistent. As i can not decide which is better I would prefer leave it as it is.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org