You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/04/08 16:57:27 UTC
[GitHub] [beam] rehmanmuradali opened a new pull request #11350: [BEAM-1589]
Added @onWindowExpiration annotation.
rehmanmuradali opened a new pull request #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350
Added @onWindowExpiration annotation for annotating a user defined method (similar to a method annotated with @ontimer), which allows user to do some work before garbage collection. For example, users can use this annotation to write a method that extracts a value saved in a state before garbage collection.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lukecwik edited a comment on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
lukecwik edited a comment on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-611749077
Any discussion about adding support for @OnWindowExpiration for portable runners?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615110263
Run Spark StructuredStreaming ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] reuvenlax commented on a change in pull request #11350:
[BEAM-1589] Added @onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on a change in pull request #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#discussion_r406508585
##########
File path: runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
##########
@@ -841,6 +847,238 @@ public BundleFinalizer bundleFinalizer() {
}
}
+ /**
+ * A concrete implementation of {@link DoFnInvoker.ArgumentProvider} used for running a {@link
+ * DoFn} on window expiration.
+ */
+ private class OnWindowExpirationArgumentProvider extends DoFn<InputT, OutputT>.OnTimerContext
Review comment:
Why is this based on OnTimerContext?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615108171
Run Direct ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615107990
Run Apex ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615109364
Run Samza ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615109070
Run Gearpump ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615109922
Run Spark ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615107852
Run Apex ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615112122
Run Java Spark PortableValidatesRunner Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615110544
Run Dataflow PortabilityApi ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aaltay removed a comment on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
aaltay removed a comment on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-614754539
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615109364
Run Samza ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615109070
Run Gearpump ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lukecwik commented on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
lukecwik commented on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-611200240
trigger tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615111395
Run Java Flink PortableValidatesRunner Streaming
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rehmanmuradali commented on a change in pull request #11350:
[BEAM-1589] Added @onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
rehmanmuradali commented on a change in pull request #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#discussion_r409602692
##########
File path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingModeExecutionContext.java
##########
@@ -591,7 +591,7 @@ public void flushState() {
timerId,
"",
cleanupTime,
- cleanupTime,
Review comment:
@reuvenlax added in ReduceFnRunner.java
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rehmanmuradali commented on a change in pull request #11350:
[BEAM-1589] Added @onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
rehmanmuradali commented on a change in pull request #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#discussion_r409602437
##########
File path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingModeExecutionContext.java
##########
@@ -591,7 +591,7 @@ public void flushState() {
timerId,
"",
cleanupTime,
- cleanupTime,
+ window.maxTimestamp(),
Review comment:
@steveniemitz cleanupTime.minus(1L) added
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lukecwik commented on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
lukecwik commented on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-611749077
Any discussion about adding support for @OnWindowExpiration for portable execution?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615108071
Run Apex ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615108171
Run Direct ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615108467
Run Flink ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615112001
Run Java Spark PortableValidatesRunner Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615110022
Run Spark ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615107852
Run Apex ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aaltay commented on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
aaltay commented on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-614754539
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] reuvenlax commented on a change in pull request #11350:
[BEAM-1589] Added @onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on a change in pull request #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#discussion_r406508479
##########
File path: runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunner.java
##########
@@ -52,6 +52,12 @@ void onTimer(
*/
void finishBundle();
+ /**
+ * Calls a {@link DoFn DoFn's} {@link DoFn.OnWindowExpiration @OnWindowExpiration} method and
+ * performs additional task, such as extracts a value saved in a state before garbage collection.
+ */
+ void onWindowExpiration(BoundedWindow window, Instant timestamp, TimeDomain timeDomain);
Review comment:
What do timestamp and timeDomain mean in this context?
Also presumably you do want to be able to access the key in onWindowExpiration
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aaltay removed a comment on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
aaltay removed a comment on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-614754706
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] steveniemitz commented on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-611310160
I think you'll need runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SimpleParDoFn.java as well for the streaming dataflow path.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615109200
Run Samza ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615109138
Run Gearpump ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rehmanmuradali commented on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
rehmanmuradali commented on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-614783089
R: @reuvenlax
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615107990
Run Apex ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615109458
Run Samza ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615110544
Run Dataflow PortabilityApi ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615108799
Run Flink ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615108354
Run Direct ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615108467
Run Flink ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615109200
Run Samza ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] steveniemitz commented on a change in pull request #11350:
[BEAM-1589] Added @onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on a change in pull request #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#discussion_r407479202
##########
File path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingModeExecutionContext.java
##########
@@ -591,7 +591,7 @@ public void flushState() {
timerId,
"",
cleanupTime,
- cleanupTime,
+ window.maxTimestamp(),
Review comment:
this should be `window.maxTimestamp() + allowedLateness` I think? cleanupTime here is passed in as `maxTimestamp + allowedLateness + 1ms`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615109922
Run Spark ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615110180
Run Spark StructuredStreaming ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rehmanmuradali commented on a change in pull request #11350:
[BEAM-1589] Added @onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
rehmanmuradali commented on a change in pull request #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#discussion_r406841025
##########
File path: runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunner.java
##########
@@ -52,6 +52,12 @@ void onTimer(
*/
void finishBundle();
+ /**
+ * Calls a {@link DoFn DoFn's} {@link DoFn.OnWindowExpiration @OnWindowExpiration} method and
+ * performs additional task, such as extracts a value saved in a state before garbage collection.
+ */
+ void onWindowExpiration(BoundedWindow window, Instant timestamp, TimeDomain timeDomain);
Review comment:
@reuvenlax timestamp added to keep the time of outputs from OnWindowExpiration. This would be window.maxTimestamp()
timeDomain removed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615110180
Run Spark StructuredStreaming ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615111288
Run Java Flink PortableValidatesRunner Streaming
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rehmanmuradali commented on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
rehmanmuradali commented on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-611625991
R: @reuvenlax could you please do an overall review for Direct Runner and let me know if I am on the right track?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615110804
Run Java Flink PortableValidatesRunner Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aaltay commented on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
aaltay commented on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-614754706
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rehmanmuradali commented on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
rehmanmuradali commented on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-611580540
> I think you'll need runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SimpleParDoFn.java as well for the streaming dataflow path.
@steveniemitz I am on it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615110655
Run Dataflow PortabilityApi ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] reuvenlax commented on a change in pull request #11350:
[BEAM-1589] Added @onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on a change in pull request #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#discussion_r406510585
##########
File path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingModeExecutionContext.java
##########
@@ -591,7 +591,7 @@ public void flushState() {
timerId,
"",
cleanupTime,
- cleanupTime,
Review comment:
We also seem to set GC timers in ReduceFnRunner.java. @kennknowles do you know why we have both?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] robertwb commented on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
robertwb commented on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-612966451
Does this need to be annotated with a new requirement? https://github.com/apache/beam/blob/release-2.21.0/model/pipeline/src/main/proto/beam_runner_api.proto#L1371
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11350: [BEAM-1589] Added
@onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#issuecomment-615110955
Run Java Flink PortableValidatesRunner Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rehmanmuradali commented on a change in pull request #11350:
[BEAM-1589] Added @onWindowExpiration annotation.
Posted by GitBox <gi...@apache.org>.
rehmanmuradali commented on a change in pull request #11350: [BEAM-1589] Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/11350#discussion_r406839599
##########
File path: runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
##########
@@ -841,6 +847,238 @@ public BundleFinalizer bundleFinalizer() {
}
}
+ /**
+ * A concrete implementation of {@link DoFnInvoker.ArgumentProvider} used for running a {@link
+ * DoFn} on window expiration.
+ */
+ private class OnWindowExpirationArgumentProvider extends DoFn<InputT, OutputT>.OnTimerContext
Review comment:
@reuvenlax OnWindowExpirationContext added.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services