You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/04/09 10:34:57 UTC
[GitHub] [beam] mxm opened a new pull request #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
mxm opened a new pull request #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362
The Flink Runner's ImpulseSourceFunction does not emit a final watermark, unless the
`--shutdownSourcesOnFinalWatermark` flag has been specified (the flag is used in
tests to shutdown the pipeline after reading all data). Most pipelines will be
long-running and thus do not specify the flag.
Not sending out the final watermark causes GroupByKey to hold back the data of
event time windows until the pipeline is shut down (the final watermark is
always emitted on pipeline shutdown which is why using the above flag works).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-616202637
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-616211644
Run Java Flink PortableValidatesRunner Streaming
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-613518742
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-616202575
Run Flink ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-611572653
> Just to confirm: Emitting the final watermark won't terminate the source function and therefore checkpointing will still succeed?
Yes, the two are unrelated to each other.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-615226436
Third party licensing checks fail in: https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Commit/4209/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm edited a comment on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm edited a comment on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-613469832
Unrelated test failure:
```
15:55:21 FAILURE: Build failed with an exception.
15:55:21
15:55:21 * What went wrong:
15:55:21 Execution failed for task ':sdks:java:container:generateThirdPartyLicenses'.
15:55:21 > Process 'command './sdks/java/container/license_scripts/license_script.sh'' finished with non-zero exit value 1
```
https://builds.apache.org/job/beam_PreCommit_Java_Commit/10841/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-615300858
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-613083686
Run Java Flink PortableValidatesRunner Streaming
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm removed a comment on issue #11362: [BEAM-9733] Always
let ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm removed a comment on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-615217301
Flink Runner Nexmark Tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-613579619
> @mxm these are significant changes after the PR was approved. Questions:
I'm aware of that. I've tried to keep the necessary changes at a minimum. I'm happy to hear your feedback.
> 1. What test coverage do we have for ensuring that watermarks don't bypass in-progress elements?
We have tests in `ExecutableStageDoFnOperatorTest` (`outputsAreTaggedCorrectly`, `testEnsureDeferredStateCleanupTimerFiring`) and through our integration tests. Admittedly, it would be good to dedicate a test specifically to watermark behavior.
> 2. Do these changes affect how the main input watermark interacts with the side input watermark?
Effectively, side inputs in portability were broken before because (a) the side input watermark hold was abused by the the portable operator (b) only `processWatermark` was overridden but for proper support for side inputs we have to override `processWatermark1` (1 is the main input when we have side inputs, `DoFnoperator#processWatermark` calls `processWatermark1` when we do not have side inputs, but `processWatermark` is not called when we have side input, only `processWatermark1`).
> 3. Will the added watermark logging affect the usefulness of debug logging for other investigations (I had in the past removed it after done debugging issues)
It greatly helped me to debug the current behavior and develop the solution. I find it immensely helpful and would like to keep it if further debugging is necessary. It is very useful to have debug information already built-in, instead of having to add it manually every time (which in any case will be possible).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-615340854
There is still an issue with the bundle timeout timer which only became visible on Jenkins. Need to look into this but the principle approach holds here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-612030833
I need to rewrite the FlinkSavepointTest since it assumed different semantics. I think we can use the recently introduced timer output timestamp feature. However, I realized the portable operator needs a slight adjustment to fully support holding back the output timestamp correctly at all times. This is trickier than in the non-portable operator with respect to timers setting new timers; we do not have a direct feedback loop as we have in the non-portable operator. We need an additional check when a timer sets a new timer with a timer output timestamp because we only get to fire that timer after we started a new bundle. Thus, we can't always advance the watermark after we finish bundle execution.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-611596479
Any plans on how to make the flag less confusing?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-615216155
I apologize for the delay here. Turns out we had a bit of work to do with regards to handling watermark and timers in portability. Please see the PR description for an update on what has been changed. @tweise @iemejia Please have a look. I'm sorry that there are more changes to review now but I'm convinced the implementation is now correct and robust. I'm also expecting considerable performance gains for portability.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] tweise commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
tweise commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-613628419
> We have tests in `ExecutableStageDoFnOperatorTest` (`outputsAreTaggedCorrectly`, `testEnsureDeferredStateCleanupTimerFiring`) and through our integration tests. Admittedly, it would be good to dedicate a test specifically to watermark behavior.
There is some coverage here: https://github.com/apache/beam/blob/8db19a4645b8588ce9e046637b7619815169bdb1/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperatorTest.java#L376
It covers only finish bundle on close though. But a similarly fine-grained test should do (integration tests generally don't provide a signal for this).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-613439530
Run Java Flink PortableValidatesRunner Streaming
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-616168234
Run Flink ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-612613900
Run Java Flink PortableValidatesRunner Streaming
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm removed a comment on issue #11362: [BEAM-9733] Always
let ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm removed a comment on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-615216506
Run Nexmark Flink
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-611457979
Run Java Flink PortableValidatesRunner Streaming
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm removed a comment on issue #11362: [BEAM-9733] Always
let ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm removed a comment on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-615275254
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-616168266
Run Java Flink PortableValidatesRunner Streaming
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-615216365
Run Java Flink PortableValidatesRunner Streaming
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-612031071
> Any plans on how to make the flag less confusing?
We could deprecate the flag and add an alias for backwards-compatibility.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-613083609
Failure was caused by https://github.com/apache/beam/pull/11314.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-613469690
I've had another go at the solution because I wasn't entirely satisfied. This allowed me to remove dependencies on the base DoFnOperator, e.g. the abuse of the watermark hold intended for side inputs for holding back the watermark in portable bundles.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-616211695
Run Flink Runner Nexmark Tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-612618733
Test case failing has been failing before these changes. I'll look into the cause. https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming_PR/191/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-613084788
Tests should be passing now, though I'll give this another thought over night.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-615217389
Run Flink Runner Nexmark Tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-616168344
Run Flink Runner Nexmark Tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-613469832
Unrelated test failure:
```
15:55:21 FAILURE: Build failed with an exception.
15:55:21
15:55:21 * What went wrong:
15:55:21 Execution failed for task ':sdks:java:container:generateThirdPartyLicenses'.
15:55:21 > Process 'command './sdks/java/container/license_scripts/license_script.sh'' finished with non-zero exit value 1
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-615216506
Run Nexmark Flink
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-616211669
Run Flink ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-615226504
Run Python2_PVR_Flink PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-615216315
Run Flink ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] ibzib commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
ibzib commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-614951136
What remains to be done on this PR? Test coverage? I'd like to get this merged as soon as possible to unblock the 2.21 release.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-611575865
The flag `shutdownSourcesOnFinalWatermark` might be confusing because it does not actually shutdown on final watermark but instead shuts down the source function which causes Flink to emit the final watermark.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] tweise commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
tweise commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-611559308
Just to confirm: Emitting the final watermark won't terminate the source function and therefore checkpointing will still succeed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-615275254
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11362: [BEAM-9733] Always let
ImpulseSourceFunction emit a final watermark
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11362: [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
URL: https://github.com/apache/beam/pull/11362#issuecomment-615217301
Flink Runner Nexmark Tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services