You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/10/28 16:23:38 UTC
[GitHub] [flink] aljoscha opened a new pull request #13828: [FLINK-19835] Don't emit intermediate watermarks from sources in BATCH execution mode
aljoscha opened a new pull request #13828:
URL: https://github.com/apache/flink/pull/13828
## What is the purpose of the change
Currently, both sources and watermark/timestamp operators can emit watermarks that we don't really need. We only need a final watermark in BATCH execution mode.
**Note to reviewers: I'm not sure whether to add a new RuntimeMode enum or use the existing RuntimeExecutionMode for this.**
## Brief change log
* turn SourceTransformation into a logical transformation and only create the operator in the Translator
* make source aware of BATCH/STREAMING mode
## Verifying this change
* new tests that verify we don't emit watermarks in BATCH mode
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): yes
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? no
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13828: [FLINK-19835] Don't emit intermediate watermarks from sources in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13828:
URL: https://github.com/apache/flink/pull/13828#issuecomment-718083567
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cbef03730f1eef907bcc6aaacda3525410cb6080",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cbef03730f1eef907bcc6aaacda3525410cb6080",
"triggerType" : "PUSH"
}, {
"hash" : "996dc230ec95673342ae3f635e895e0925b8082e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8533",
"triggerID" : "996dc230ec95673342ae3f635e895e0925b8082e",
"triggerType" : "PUSH"
}, {
"hash" : "1e84ec3d86ab118ad30e6571fbe792da3e88b09f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8538",
"triggerID" : "1e84ec3d86ab118ad30e6571fbe792da3e88b09f",
"triggerType" : "PUSH"
}, {
"hash" : "0ece2b151e00211f414d167b4e2d43248baef1e8",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8588",
"triggerID" : "0ece2b151e00211f414d167b4e2d43248baef1e8",
"triggerType" : "PUSH"
}, {
"hash" : "cebe3fbe5418492a95cea17ef3141c8207a8d02a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8594",
"triggerID" : "cebe3fbe5418492a95cea17ef3141c8207a8d02a",
"triggerType" : "PUSH"
}, {
"hash" : "b2332fd1a5e22460e043267a3b7d296e59c804b6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b2332fd1a5e22460e043267a3b7d296e59c804b6",
"triggerType" : "PUSH"
}, {
"hash" : "9e3a6d81951df35dbf94e25e845e9a07ad007601",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8604",
"triggerID" : "9e3a6d81951df35dbf94e25e845e9a07ad007601",
"triggerType" : "PUSH"
}, {
"hash" : "098a3e0039824c161d1fe815a65e6a982f166681",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "098a3e0039824c161d1fe815a65e6a982f166681",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cbef03730f1eef907bcc6aaacda3525410cb6080 UNKNOWN
* 0ece2b151e00211f414d167b4e2d43248baef1e8 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8588)
* cebe3fbe5418492a95cea17ef3141c8207a8d02a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8594)
* b2332fd1a5e22460e043267a3b7d296e59c804b6 UNKNOWN
* 9e3a6d81951df35dbf94e25e845e9a07ad007601 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8604)
* 098a3e0039824c161d1fe815a65e6a982f166681 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on pull request #13828: [FLINK-19835] Don't emit intermediate watermarks from sources in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
dawidwys commented on pull request #13828:
URL: https://github.com/apache/flink/pull/13828#issuecomment-718456902
@kl0u Yes, #13073 covers emitting `MAX_WATERMARK` at the end of input in both `BATCH` and `STREAM`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13828: [FLINK-19835] Don't emit intermediate watermarks from sources in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13828:
URL: https://github.com/apache/flink/pull/13828#issuecomment-718083567
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cbef03730f1eef907bcc6aaacda3525410cb6080",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cbef03730f1eef907bcc6aaacda3525410cb6080",
"triggerType" : "PUSH"
}, {
"hash" : "996dc230ec95673342ae3f635e895e0925b8082e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8533",
"triggerID" : "996dc230ec95673342ae3f635e895e0925b8082e",
"triggerType" : "PUSH"
}, {
"hash" : "1e84ec3d86ab118ad30e6571fbe792da3e88b09f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8538",
"triggerID" : "1e84ec3d86ab118ad30e6571fbe792da3e88b09f",
"triggerType" : "PUSH"
}, {
"hash" : "0ece2b151e00211f414d167b4e2d43248baef1e8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8588",
"triggerID" : "0ece2b151e00211f414d167b4e2d43248baef1e8",
"triggerType" : "PUSH"
}, {
"hash" : "cebe3fbe5418492a95cea17ef3141c8207a8d02a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8594",
"triggerID" : "cebe3fbe5418492a95cea17ef3141c8207a8d02a",
"triggerType" : "PUSH"
}, {
"hash" : "b2332fd1a5e22460e043267a3b7d296e59c804b6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b2332fd1a5e22460e043267a3b7d296e59c804b6",
"triggerType" : "PUSH"
}, {
"hash" : "9e3a6d81951df35dbf94e25e845e9a07ad007601",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8604",
"triggerID" : "9e3a6d81951df35dbf94e25e845e9a07ad007601",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cbef03730f1eef907bcc6aaacda3525410cb6080 UNKNOWN
* 1e84ec3d86ab118ad30e6571fbe792da3e88b09f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8538)
* 0ece2b151e00211f414d167b4e2d43248baef1e8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8588)
* cebe3fbe5418492a95cea17ef3141c8207a8d02a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8594)
* b2332fd1a5e22460e043267a3b7d296e59c804b6 UNKNOWN
* 9e3a6d81951df35dbf94e25e845e9a07ad007601 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8604)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13828: [FLINK-19835] Don't emit intermediate watermarks from sources in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13828:
URL: https://github.com/apache/flink/pull/13828#issuecomment-718083567
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cbef03730f1eef907bcc6aaacda3525410cb6080",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cbef03730f1eef907bcc6aaacda3525410cb6080",
"triggerType" : "PUSH"
}, {
"hash" : "996dc230ec95673342ae3f635e895e0925b8082e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8533",
"triggerID" : "996dc230ec95673342ae3f635e895e0925b8082e",
"triggerType" : "PUSH"
}, {
"hash" : "1e84ec3d86ab118ad30e6571fbe792da3e88b09f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8538",
"triggerID" : "1e84ec3d86ab118ad30e6571fbe792da3e88b09f",
"triggerType" : "PUSH"
}, {
"hash" : "0ece2b151e00211f414d167b4e2d43248baef1e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8588",
"triggerID" : "0ece2b151e00211f414d167b4e2d43248baef1e8",
"triggerType" : "PUSH"
}, {
"hash" : "cebe3fbe5418492a95cea17ef3141c8207a8d02a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8594",
"triggerID" : "cebe3fbe5418492a95cea17ef3141c8207a8d02a",
"triggerType" : "PUSH"
}, {
"hash" : "b2332fd1a5e22460e043267a3b7d296e59c804b6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b2332fd1a5e22460e043267a3b7d296e59c804b6",
"triggerType" : "PUSH"
}, {
"hash" : "9e3a6d81951df35dbf94e25e845e9a07ad007601",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8604",
"triggerID" : "9e3a6d81951df35dbf94e25e845e9a07ad007601",
"triggerType" : "PUSH"
}, {
"hash" : "098a3e0039824c161d1fe815a65e6a982f166681",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8581",
"triggerID" : "098a3e0039824c161d1fe815a65e6a982f166681",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cbef03730f1eef907bcc6aaacda3525410cb6080 UNKNOWN
* cebe3fbe5418492a95cea17ef3141c8207a8d02a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8594)
* b2332fd1a5e22460e043267a3b7d296e59c804b6 UNKNOWN
* 9e3a6d81951df35dbf94e25e845e9a07ad007601 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8604)
* 098a3e0039824c161d1fe815a65e6a982f166681 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8581)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13828: [FLINK-19835] Don't emit intermediate watermarks from sources in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13828:
URL: https://github.com/apache/flink/pull/13828#issuecomment-718083567
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cbef03730f1eef907bcc6aaacda3525410cb6080",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cbef03730f1eef907bcc6aaacda3525410cb6080",
"triggerType" : "PUSH"
}, {
"hash" : "996dc230ec95673342ae3f635e895e0925b8082e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8533",
"triggerID" : "996dc230ec95673342ae3f635e895e0925b8082e",
"triggerType" : "PUSH"
}, {
"hash" : "1e84ec3d86ab118ad30e6571fbe792da3e88b09f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8538",
"triggerID" : "1e84ec3d86ab118ad30e6571fbe792da3e88b09f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cbef03730f1eef907bcc6aaacda3525410cb6080 UNKNOWN
* 1e84ec3d86ab118ad30e6571fbe792da3e88b09f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8538)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] SteNicholas commented on a change in pull request #13828: [FLINK-19835] Don't emit intermediate watermarks from sources in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
SteNicholas commented on a change in pull request #13828:
URL: https://github.com/apache/flink/pull/13828#discussion_r513842239
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java
##########
@@ -254,4 +269,10 @@ private void registerReader() {
ListState<SplitT> getReaderState() {
return readerState;
}
+
+ // Not really meant to be here, this is an example.
+ // TODO: Should we use RuntimeExecutionMode here? I don't like that that one has AUTOMATIC.
+ public enum RuntimeMode {
+ BATCH, STREAMING
Review comment:
@aljoscha @dawidwys @kl0u I have confusion about which choice should follow. As @aljoscha mentioned that he prefers to use more enums than boolean. And @dawidwys @kl0u prefers to use a boolean. I would like to follow your standard later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] kl0u commented on a change in pull request #13828: [FLINK-19835] Don't emit intermediate watermarks from sources in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
kl0u commented on a change in pull request #13828:
URL: https://github.com/apache/flink/pull/13828#discussion_r513675900
##########
File path: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/source/SourceOperatorEventTimeTest.java
##########
@@ -161,6 +199,7 @@ public void testPerSplitOutputEventWatermarks() throws Exception {
@SuppressWarnings("FinalPrivateMethod")
@SafeVarargs
private final List<Object> testSequenceOfEvents(
+ SourceOperator.RuntimeMode runtimeMode,
Review comment:
Here we use `final` for some args and nothing for others. So feel free to uniformize them the way you feel more comfortable.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] kl0u commented on a change in pull request #13828: [FLINK-19835] Don't emit intermediate watermarks from sources in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
kl0u commented on a change in pull request #13828:
URL: https://github.com/apache/flink/pull/13828#discussion_r514183535
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/SourceTransformationTranslator.java
##########
@@ -36,26 +38,29 @@
* @param <OUT> The type of the elements that this source produces.
*/
@Internal
-public class SourceTransformationTranslator<OUT>
- extends SimpleTransformationTranslator<OUT, SourceTransformation<OUT>> {
+public class SourceTransformationTranslator<OUT, SplitT extends SourceSplit, EnumChkT>
+ extends SimpleTransformationTranslator<OUT, SourceTransformation<OUT, SplitT, EnumChkT>> {
@Override
protected Collection<Integer> translateForBatchInternal(
- final SourceTransformation<OUT> transformation,
+ final SourceTransformation<OUT, SplitT, EnumChkT> transformation,
final Context context) {
- return translateInternal(transformation, context);
+
+ return translateInternal(transformation, context, false /* emit progressive watermarks */);
}
@Override
protected Collection<Integer> translateForStreamingInternal(
- final SourceTransformation<OUT> transformation,
+ final SourceTransformation<OUT, SplitT, EnumChkT> transformation,
final Context context) {
- return translateInternal(transformation, context);
+
+ return translateInternal(transformation, context, true /* emit progressive watermarks */);
}
private Collection<Integer> translateInternal(
- final SourceTransformation<OUT> transformation,
- final Context context) {
+ final SourceTransformation<OUT, SplitT, EnumChkT> transformation,
+ final Context context,
+ boolean emitProgressiveWatermarks) {
Review comment:
Why not simply `emitWatermarks`?
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/SourceTransformationTranslator.java
##########
@@ -36,26 +38,29 @@
* @param <OUT> The type of the elements that this source produces.
*/
@Internal
-public class SourceTransformationTranslator<OUT>
- extends SimpleTransformationTranslator<OUT, SourceTransformation<OUT>> {
+public class SourceTransformationTranslator<OUT, SplitT extends SourceSplit, EnumChkT>
+ extends SimpleTransformationTranslator<OUT, SourceTransformation<OUT, SplitT, EnumChkT>> {
@Override
protected Collection<Integer> translateForBatchInternal(
- final SourceTransformation<OUT> transformation,
+ final SourceTransformation<OUT, SplitT, EnumChkT> transformation,
final Context context) {
- return translateInternal(transformation, context);
+
+ return translateInternal(transformation, context, false /* emit progressive watermarks */);
Review comment:
Update the comment to reflect that we do not send Watermarks in this case.
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/SourceOperatorFactory.java
##########
@@ -51,19 +51,36 @@ Licensed to the Apache Software Foundation (ASF) under one
/** The event time setup (timestamp assigners, watermark generators, etc.). */
private final WatermarkStrategy<OUT> watermarkStrategy;
+ /**
+ * Whether to periodically emit watermarks as we go or only one final watermark at the end of
+ * input.
+ */
+ private final boolean emitProgressiveWatermarks;
Review comment:
Why not simply `emitWatermarks`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13828: [FLINK-19835] Don't emit intermediate watermarks from sources in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13828:
URL: https://github.com/apache/flink/pull/13828#issuecomment-718083567
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cbef03730f1eef907bcc6aaacda3525410cb6080",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cbef03730f1eef907bcc6aaacda3525410cb6080",
"triggerType" : "PUSH"
}, {
"hash" : "996dc230ec95673342ae3f635e895e0925b8082e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8533",
"triggerID" : "996dc230ec95673342ae3f635e895e0925b8082e",
"triggerType" : "PUSH"
}, {
"hash" : "1e84ec3d86ab118ad30e6571fbe792da3e88b09f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8538",
"triggerID" : "1e84ec3d86ab118ad30e6571fbe792da3e88b09f",
"triggerType" : "PUSH"
}, {
"hash" : "0ece2b151e00211f414d167b4e2d43248baef1e8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8588",
"triggerID" : "0ece2b151e00211f414d167b4e2d43248baef1e8",
"triggerType" : "PUSH"
}, {
"hash" : "cebe3fbe5418492a95cea17ef3141c8207a8d02a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8594",
"triggerID" : "cebe3fbe5418492a95cea17ef3141c8207a8d02a",
"triggerType" : "PUSH"
}, {
"hash" : "b2332fd1a5e22460e043267a3b7d296e59c804b6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b2332fd1a5e22460e043267a3b7d296e59c804b6",
"triggerType" : "PUSH"
}, {
"hash" : "9e3a6d81951df35dbf94e25e845e9a07ad007601",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9e3a6d81951df35dbf94e25e845e9a07ad007601",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cbef03730f1eef907bcc6aaacda3525410cb6080 UNKNOWN
* 1e84ec3d86ab118ad30e6571fbe792da3e88b09f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8538)
* 0ece2b151e00211f414d167b4e2d43248baef1e8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8588)
* cebe3fbe5418492a95cea17ef3141c8207a8d02a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8594)
* b2332fd1a5e22460e043267a3b7d296e59c804b6 UNKNOWN
* 9e3a6d81951df35dbf94e25e845e9a07ad007601 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13828: [FLINK-19835] Don't emit intermediate watermarks from sources in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13828:
URL: https://github.com/apache/flink/pull/13828#issuecomment-718083567
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cbef03730f1eef907bcc6aaacda3525410cb6080",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cbef03730f1eef907bcc6aaacda3525410cb6080",
"triggerType" : "PUSH"
}, {
"hash" : "996dc230ec95673342ae3f635e895e0925b8082e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8533",
"triggerID" : "996dc230ec95673342ae3f635e895e0925b8082e",
"triggerType" : "PUSH"
}, {
"hash" : "1e84ec3d86ab118ad30e6571fbe792da3e88b09f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8538",
"triggerID" : "1e84ec3d86ab118ad30e6571fbe792da3e88b09f",
"triggerType" : "PUSH"
}, {
"hash" : "0ece2b151e00211f414d167b4e2d43248baef1e8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0ece2b151e00211f414d167b4e2d43248baef1e8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cbef03730f1eef907bcc6aaacda3525410cb6080 UNKNOWN
* 1e84ec3d86ab118ad30e6571fbe792da3e88b09f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8538)
* 0ece2b151e00211f414d167b4e2d43248baef1e8 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] SteNicholas commented on a change in pull request #13828: [FLINK-19835] Don't emit intermediate watermarks from sources in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
SteNicholas commented on a change in pull request #13828:
URL: https://github.com/apache/flink/pull/13828#discussion_r513842239
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java
##########
@@ -254,4 +269,10 @@ private void registerReader() {
ListState<SplitT> getReaderState() {
return readerState;
}
+
+ // Not really meant to be here, this is an example.
+ // TODO: Should we use RuntimeExecutionMode here? I don't like that that one has AUTOMATIC.
+ public enum RuntimeMode {
+ BATCH, STREAMING
Review comment:
@aljoscha @dawidwys @kl0u I have confusion about which choice should follow. As @aljoscha mentioned that he prefers to use more enums than boolean. And @dawidwys @kl0u prefers to use a boolean. I would like to follow your standard way later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on a change in pull request #13828: [FLINK-19835] Don't emit intermediate watermarks from sources in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #13828:
URL: https://github.com/apache/flink/pull/13828#discussion_r514068419
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java
##########
@@ -254,4 +269,10 @@ private void registerReader() {
ListState<SplitT> getReaderState() {
return readerState;
}
+
+ // Not really meant to be here, this is an example.
+ // TODO: Should we use RuntimeExecutionMode here? I don't like that that one has AUTOMATIC.
+ public enum RuntimeMode {
+ BATCH, STREAMING
Review comment:
Just to be clear. I have no preference here. I just gave another option.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #13828: [FLINK-19835] Don't emit intermediate watermarks from sources in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #13828:
URL: https://github.com/apache/flink/pull/13828#discussion_r513588833
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java
##########
@@ -254,4 +269,10 @@ private void registerReader() {
ListState<SplitT> getReaderState() {
return readerState;
}
+
+ // Not really meant to be here, this is an example.
+ // TODO: Should we use RuntimeExecutionMode here? I don't like that that one has AUTOMATIC.
+ public enum RuntimeMode {
+ BATCH, STREAMING
Review comment:
I lately try to use more enums than boolean because it's easier to maintain and understand at a glance. Compare `doSomething(13, true, false)` vs. `doSomething(13, STREAMING, EXACTLY_ONCE)`. But yes, here it's just a "custom" boolean. 😅
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13828: [FLINK-19835] Don't emit intermediate watermarks from sources in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #13828:
URL: https://github.com/apache/flink/pull/13828#issuecomment-718050989
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit cbef03730f1eef907bcc6aaacda3525410cb6080 (Wed Oct 28 16:26:35 UTC 2020)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13828: [FLINK-19835] Don't emit intermediate watermarks from sources in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13828:
URL: https://github.com/apache/flink/pull/13828#issuecomment-718083567
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cbef03730f1eef907bcc6aaacda3525410cb6080",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cbef03730f1eef907bcc6aaacda3525410cb6080",
"triggerType" : "PUSH"
}, {
"hash" : "996dc230ec95673342ae3f635e895e0925b8082e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8533",
"triggerID" : "996dc230ec95673342ae3f635e895e0925b8082e",
"triggerType" : "PUSH"
}, {
"hash" : "1e84ec3d86ab118ad30e6571fbe792da3e88b09f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8538",
"triggerID" : "1e84ec3d86ab118ad30e6571fbe792da3e88b09f",
"triggerType" : "PUSH"
}, {
"hash" : "0ece2b151e00211f414d167b4e2d43248baef1e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8588",
"triggerID" : "0ece2b151e00211f414d167b4e2d43248baef1e8",
"triggerType" : "PUSH"
}, {
"hash" : "cebe3fbe5418492a95cea17ef3141c8207a8d02a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8594",
"triggerID" : "cebe3fbe5418492a95cea17ef3141c8207a8d02a",
"triggerType" : "PUSH"
}, {
"hash" : "b2332fd1a5e22460e043267a3b7d296e59c804b6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b2332fd1a5e22460e043267a3b7d296e59c804b6",
"triggerType" : "PUSH"
}, {
"hash" : "9e3a6d81951df35dbf94e25e845e9a07ad007601",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8604",
"triggerID" : "9e3a6d81951df35dbf94e25e845e9a07ad007601",
"triggerType" : "PUSH"
}, {
"hash" : "098a3e0039824c161d1fe815a65e6a982f166681",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8581",
"triggerID" : "098a3e0039824c161d1fe815a65e6a982f166681",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cbef03730f1eef907bcc6aaacda3525410cb6080 UNKNOWN
* b2332fd1a5e22460e043267a3b7d296e59c804b6 UNKNOWN
* 9e3a6d81951df35dbf94e25e845e9a07ad007601 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8604)
* 098a3e0039824c161d1fe815a65e6a982f166681 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8581)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13828: [FLINK-19835] Don't emit intermediate watermarks from sources in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13828:
URL: https://github.com/apache/flink/pull/13828#issuecomment-718083567
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cbef03730f1eef907bcc6aaacda3525410cb6080",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cbef03730f1eef907bcc6aaacda3525410cb6080",
"triggerType" : "PUSH"
}, {
"hash" : "996dc230ec95673342ae3f635e895e0925b8082e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8533",
"triggerID" : "996dc230ec95673342ae3f635e895e0925b8082e",
"triggerType" : "PUSH"
}, {
"hash" : "1e84ec3d86ab118ad30e6571fbe792da3e88b09f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8538",
"triggerID" : "1e84ec3d86ab118ad30e6571fbe792da3e88b09f",
"triggerType" : "PUSH"
}, {
"hash" : "0ece2b151e00211f414d167b4e2d43248baef1e8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8588",
"triggerID" : "0ece2b151e00211f414d167b4e2d43248baef1e8",
"triggerType" : "PUSH"
}, {
"hash" : "cebe3fbe5418492a95cea17ef3141c8207a8d02a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cebe3fbe5418492a95cea17ef3141c8207a8d02a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cbef03730f1eef907bcc6aaacda3525410cb6080 UNKNOWN
* 1e84ec3d86ab118ad30e6571fbe792da3e88b09f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8538)
* 0ece2b151e00211f414d167b4e2d43248baef1e8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8588)
* cebe3fbe5418492a95cea17ef3141c8207a8d02a UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on pull request #13828: [FLINK-19835] Don't emit intermediate watermarks from sources in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on pull request #13828:
URL: https://github.com/apache/flink/pull/13828#issuecomment-718485781
Thanks for the reviews! I believe I addressed all your comments. PTAL (Please take another look)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13828: [FLINK-19835] Don't emit intermediate watermarks from sources in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13828:
URL: https://github.com/apache/flink/pull/13828#issuecomment-718083567
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cbef03730f1eef907bcc6aaacda3525410cb6080",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cbef03730f1eef907bcc6aaacda3525410cb6080",
"triggerType" : "PUSH"
}, {
"hash" : "996dc230ec95673342ae3f635e895e0925b8082e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8533",
"triggerID" : "996dc230ec95673342ae3f635e895e0925b8082e",
"triggerType" : "PUSH"
}, {
"hash" : "1e84ec3d86ab118ad30e6571fbe792da3e88b09f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1e84ec3d86ab118ad30e6571fbe792da3e88b09f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cbef03730f1eef907bcc6aaacda3525410cb6080 UNKNOWN
* 996dc230ec95673342ae3f635e895e0925b8082e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8533)
* 1e84ec3d86ab118ad30e6571fbe792da3e88b09f UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on a change in pull request #13828: [FLINK-19835] Don't emit intermediate watermarks from sources in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #13828:
URL: https://github.com/apache/flink/pull/13828#discussion_r513587210
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java
##########
@@ -254,4 +269,10 @@ private void registerReader() {
ListState<SplitT> getReaderState() {
return readerState;
}
+
+ // Not really meant to be here, this is an example.
+ // TODO: Should we use RuntimeExecutionMode here? I don't like that that one has AUTOMATIC.
+ public enum RuntimeMode {
+ BATCH, STREAMING
Review comment:
Another option is to use a boolean flag? Will we ever have more modes?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] kl0u commented on a change in pull request #13828: [FLINK-19835] Don't emit intermediate watermarks from sources in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
kl0u commented on a change in pull request #13828:
URL: https://github.com/apache/flink/pull/13828#discussion_r513618837
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java
##########
@@ -254,4 +269,10 @@ private void registerReader() {
ListState<SplitT> getReaderState() {
return readerState;
}
+
+ // Not really meant to be here, this is an example.
+ // TODO: Should we use RuntimeExecutionMode here? I don't like that that one has AUTOMATIC.
+ public enum RuntimeMode {
+ BATCH, STREAMING
Review comment:
I also think that here we should either go with a boolean as @dawidwys suggested, or with the `RuntimeExecutionMode` that already exists, and not introduce yet another flag.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #13828: [FLINK-19835] Don't emit intermediate watermarks from sources in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #13828:
URL: https://github.com/apache/flink/pull/13828#discussion_r514195600
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/SourceTransformationTranslator.java
##########
@@ -36,26 +38,29 @@
* @param <OUT> The type of the elements that this source produces.
*/
@Internal
-public class SourceTransformationTranslator<OUT>
- extends SimpleTransformationTranslator<OUT, SourceTransformation<OUT>> {
+public class SourceTransformationTranslator<OUT, SplitT extends SourceSplit, EnumChkT>
+ extends SimpleTransformationTranslator<OUT, SourceTransformation<OUT, SplitT, EnumChkT>> {
@Override
protected Collection<Integer> translateForBatchInternal(
- final SourceTransformation<OUT> transformation,
+ final SourceTransformation<OUT, SplitT, EnumChkT> transformation,
final Context context) {
- return translateInternal(transformation, context);
+
+ return translateInternal(transformation, context, false /* emit progressive watermarks */);
}
@Override
protected Collection<Integer> translateForStreamingInternal(
- final SourceTransformation<OUT> transformation,
+ final SourceTransformation<OUT, SplitT, EnumChkT> transformation,
final Context context) {
- return translateInternal(transformation, context);
+
+ return translateInternal(transformation, context, true /* emit progressive watermarks */);
}
private Collection<Integer> translateInternal(
- final SourceTransformation<OUT> transformation,
- final Context context) {
+ final SourceTransformation<OUT, SplitT, EnumChkT> transformation,
+ final Context context,
+ boolean emitProgressiveWatermarks) {
Review comment:
It is to show that we emit watermarks as we go. Even in BATCH mode we still emit the final watermark.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on pull request #13828: [FLINK-19835] Don't emit intermediate watermarks from sources in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on pull request #13828:
URL: https://github.com/apache/flink/pull/13828#issuecomment-718636776
I changed this now to decouple BATCH/STREAMING mode from Watermarks/NoWatermarks because the connection might not be as clear cut in the future. This now uses a boolean field instead. I also renamed the `TimestampsAndWatermarks` implementations in a separate commit. PTAL.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #13828: [FLINK-19835] Don't emit intermediate watermarks from sources in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #13828:
URL: https://github.com/apache/flink/pull/13828#discussion_r514195353
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/SourceTransformationTranslator.java
##########
@@ -36,26 +38,29 @@
* @param <OUT> The type of the elements that this source produces.
*/
@Internal
-public class SourceTransformationTranslator<OUT>
- extends SimpleTransformationTranslator<OUT, SourceTransformation<OUT>> {
+public class SourceTransformationTranslator<OUT, SplitT extends SourceSplit, EnumChkT>
+ extends SimpleTransformationTranslator<OUT, SourceTransformation<OUT, SplitT, EnumChkT>> {
@Override
protected Collection<Integer> translateForBatchInternal(
- final SourceTransformation<OUT> transformation,
+ final SourceTransformation<OUT, SplitT, EnumChkT> transformation,
final Context context) {
- return translateInternal(transformation, context);
+
+ return translateInternal(transformation, context, false /* emit progressive watermarks */);
Review comment:
👌
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #13828: [FLINK-19835] Don't emit intermediate watermarks from sources in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #13828:
URL: https://github.com/apache/flink/pull/13828#discussion_r514087006
##########
File path: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/source/SourceOperatorEventTimeTest.java
##########
@@ -161,6 +199,7 @@ public void testPerSplitOutputEventWatermarks() throws Exception {
@SuppressWarnings("FinalPrivateMethod")
@SafeVarargs
private final List<Object> testSequenceOfEvents(
+ SourceOperator.RuntimeMode runtimeMode,
Review comment:
👌
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13828: [FLINK-19835] Don't emit intermediate watermarks from sources in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #13828:
URL: https://github.com/apache/flink/pull/13828#issuecomment-718083567
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cbef03730f1eef907bcc6aaacda3525410cb6080",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cbef03730f1eef907bcc6aaacda3525410cb6080",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cbef03730f1eef907bcc6aaacda3525410cb6080 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13828: [FLINK-19835] Don't emit intermediate watermarks from sources in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13828:
URL: https://github.com/apache/flink/pull/13828#issuecomment-718083567
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cbef03730f1eef907bcc6aaacda3525410cb6080",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cbef03730f1eef907bcc6aaacda3525410cb6080",
"triggerType" : "PUSH"
}, {
"hash" : "996dc230ec95673342ae3f635e895e0925b8082e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8533",
"triggerID" : "996dc230ec95673342ae3f635e895e0925b8082e",
"triggerType" : "PUSH"
}, {
"hash" : "1e84ec3d86ab118ad30e6571fbe792da3e88b09f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8538",
"triggerID" : "1e84ec3d86ab118ad30e6571fbe792da3e88b09f",
"triggerType" : "PUSH"
}, {
"hash" : "0ece2b151e00211f414d167b4e2d43248baef1e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8588",
"triggerID" : "0ece2b151e00211f414d167b4e2d43248baef1e8",
"triggerType" : "PUSH"
}, {
"hash" : "cebe3fbe5418492a95cea17ef3141c8207a8d02a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8594",
"triggerID" : "cebe3fbe5418492a95cea17ef3141c8207a8d02a",
"triggerType" : "PUSH"
}, {
"hash" : "b2332fd1a5e22460e043267a3b7d296e59c804b6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b2332fd1a5e22460e043267a3b7d296e59c804b6",
"triggerType" : "PUSH"
}, {
"hash" : "9e3a6d81951df35dbf94e25e845e9a07ad007601",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8604",
"triggerID" : "9e3a6d81951df35dbf94e25e845e9a07ad007601",
"triggerType" : "PUSH"
}, {
"hash" : "098a3e0039824c161d1fe815a65e6a982f166681",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8581",
"triggerID" : "098a3e0039824c161d1fe815a65e6a982f166681",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cbef03730f1eef907bcc6aaacda3525410cb6080 UNKNOWN
* b2332fd1a5e22460e043267a3b7d296e59c804b6 UNKNOWN
* 098a3e0039824c161d1fe815a65e6a982f166681 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8581)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #13828: [FLINK-19835] Don't emit intermediate watermarks from sources in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #13828:
URL: https://github.com/apache/flink/pull/13828#discussion_r514086559
##########
File path: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/source/SourceOperatorEventTimeTest.java
##########
@@ -133,24 +158,37 @@ public void testPerSplitOutputEventWatermarks() throws Exception {
(output) -> output.createOutputForSplit("two").collect(0, 200L)
);
- assertThat(result, contains(
- new Watermark(100L),
- new Watermark(150L),
- new Watermark(200L)
- ));
+ assertWatermarksOrEmpty(
+ result,
+ new Watermark(100L),
+ new Watermark(150L),
+ new Watermark(200L));
}
// ------------------------------------------------------------------------
// test execution helpers
// ------------------------------------------------------------------------
+ /**
+ * Asserts that the given expected watermarks are present in the actual watermarks in STREAMING
+ * mode. Otherwise, asserts that the list of actual watermarks is empty in BATCH mode.
+ */
+ private void assertWatermarksOrEmpty(List<Watermark> actualWatermarks, Watermark... expectedWatermarks) {
+ if (runtimeMode == SourceOperator.RuntimeMode.STREAMING) {
+ assertThat(actualWatermarks, contains(expectedWatermarks));
+ } else {
+ assertThat(actualWatermarks, empty());
+ }
+ }
+
@SuppressWarnings("FinalPrivateMethod")
@SafeVarargs
private final List<Watermark> testSequenceOfWatermarks(
+ SourceOperator.RuntimeMode runtimeMode,
Review comment:
👌
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13828: [FLINK-19835] Don't emit intermediate watermarks from sources in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13828:
URL: https://github.com/apache/flink/pull/13828#issuecomment-718083567
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cbef03730f1eef907bcc6aaacda3525410cb6080",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cbef03730f1eef907bcc6aaacda3525410cb6080",
"triggerType" : "PUSH"
}, {
"hash" : "996dc230ec95673342ae3f635e895e0925b8082e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "996dc230ec95673342ae3f635e895e0925b8082e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cbef03730f1eef907bcc6aaacda3525410cb6080 UNKNOWN
* 996dc230ec95673342ae3f635e895e0925b8082e UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] kl0u commented on a change in pull request #13828: [FLINK-19835] Don't emit intermediate watermarks from sources in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
kl0u commented on a change in pull request #13828:
URL: https://github.com/apache/flink/pull/13828#discussion_r513675008
##########
File path: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/source/SourceOperatorEventTimeTest.java
##########
@@ -133,24 +158,37 @@ public void testPerSplitOutputEventWatermarks() throws Exception {
(output) -> output.createOutputForSplit("two").collect(0, 200L)
);
- assertThat(result, contains(
- new Watermark(100L),
- new Watermark(150L),
- new Watermark(200L)
- ));
+ assertWatermarksOrEmpty(
+ result,
+ new Watermark(100L),
+ new Watermark(150L),
+ new Watermark(200L));
}
// ------------------------------------------------------------------------
// test execution helpers
// ------------------------------------------------------------------------
+ /**
+ * Asserts that the given expected watermarks are present in the actual watermarks in STREAMING
+ * mode. Otherwise, asserts that the list of actual watermarks is empty in BATCH mode.
+ */
+ private void assertWatermarksOrEmpty(List<Watermark> actualWatermarks, Watermark... expectedWatermarks) {
+ if (runtimeMode == SourceOperator.RuntimeMode.STREAMING) {
+ assertThat(actualWatermarks, contains(expectedWatermarks));
+ } else {
+ assertThat(actualWatermarks, empty());
+ }
+ }
+
@SuppressWarnings("FinalPrivateMethod")
@SafeVarargs
private final List<Watermark> testSequenceOfWatermarks(
+ SourceOperator.RuntimeMode runtimeMode,
Review comment:
Just for uniformity I would add the `final`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on pull request #13828: [FLINK-19835] Don't emit intermediate watermarks from sources in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on pull request #13828:
URL: https://github.com/apache/flink/pull/13828#issuecomment-718049071
bcc @StephanEwen
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13828: [FLINK-19835] Don't emit intermediate watermarks from sources in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13828:
URL: https://github.com/apache/flink/pull/13828#issuecomment-718083567
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cbef03730f1eef907bcc6aaacda3525410cb6080",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cbef03730f1eef907bcc6aaacda3525410cb6080",
"triggerType" : "PUSH"
}, {
"hash" : "996dc230ec95673342ae3f635e895e0925b8082e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8533",
"triggerID" : "996dc230ec95673342ae3f635e895e0925b8082e",
"triggerType" : "PUSH"
}, {
"hash" : "1e84ec3d86ab118ad30e6571fbe792da3e88b09f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8538",
"triggerID" : "1e84ec3d86ab118ad30e6571fbe792da3e88b09f",
"triggerType" : "PUSH"
}, {
"hash" : "0ece2b151e00211f414d167b4e2d43248baef1e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8588",
"triggerID" : "0ece2b151e00211f414d167b4e2d43248baef1e8",
"triggerType" : "PUSH"
}, {
"hash" : "cebe3fbe5418492a95cea17ef3141c8207a8d02a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8594",
"triggerID" : "cebe3fbe5418492a95cea17ef3141c8207a8d02a",
"triggerType" : "PUSH"
}, {
"hash" : "b2332fd1a5e22460e043267a3b7d296e59c804b6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b2332fd1a5e22460e043267a3b7d296e59c804b6",
"triggerType" : "PUSH"
}, {
"hash" : "9e3a6d81951df35dbf94e25e845e9a07ad007601",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8604",
"triggerID" : "9e3a6d81951df35dbf94e25e845e9a07ad007601",
"triggerType" : "PUSH"
}, {
"hash" : "098a3e0039824c161d1fe815a65e6a982f166681",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8581",
"triggerID" : "098a3e0039824c161d1fe815a65e6a982f166681",
"triggerType" : "PUSH"
}, {
"hash" : "aa503b5eaed5991fa4bfcc5fa32fbb538cf6f6ea",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aa503b5eaed5991fa4bfcc5fa32fbb538cf6f6ea",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cbef03730f1eef907bcc6aaacda3525410cb6080 UNKNOWN
* b2332fd1a5e22460e043267a3b7d296e59c804b6 UNKNOWN
* 098a3e0039824c161d1fe815a65e6a982f166681 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8581)
* aa503b5eaed5991fa4bfcc5fa32fbb538cf6f6ea UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13828: [FLINK-19835] Don't emit intermediate watermarks from sources in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13828:
URL: https://github.com/apache/flink/pull/13828#issuecomment-718083567
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cbef03730f1eef907bcc6aaacda3525410cb6080",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cbef03730f1eef907bcc6aaacda3525410cb6080",
"triggerType" : "PUSH"
}, {
"hash" : "996dc230ec95673342ae3f635e895e0925b8082e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8533",
"triggerID" : "996dc230ec95673342ae3f635e895e0925b8082e",
"triggerType" : "PUSH"
}, {
"hash" : "1e84ec3d86ab118ad30e6571fbe792da3e88b09f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8538",
"triggerID" : "1e84ec3d86ab118ad30e6571fbe792da3e88b09f",
"triggerType" : "PUSH"
}, {
"hash" : "0ece2b151e00211f414d167b4e2d43248baef1e8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8588",
"triggerID" : "0ece2b151e00211f414d167b4e2d43248baef1e8",
"triggerType" : "PUSH"
}, {
"hash" : "cebe3fbe5418492a95cea17ef3141c8207a8d02a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8594",
"triggerID" : "cebe3fbe5418492a95cea17ef3141c8207a8d02a",
"triggerType" : "PUSH"
}, {
"hash" : "b2332fd1a5e22460e043267a3b7d296e59c804b6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b2332fd1a5e22460e043267a3b7d296e59c804b6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cbef03730f1eef907bcc6aaacda3525410cb6080 UNKNOWN
* 1e84ec3d86ab118ad30e6571fbe792da3e88b09f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8538)
* 0ece2b151e00211f414d167b4e2d43248baef1e8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8588)
* cebe3fbe5418492a95cea17ef3141c8207a8d02a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8594)
* b2332fd1a5e22460e043267a3b7d296e59c804b6 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org