You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/08/31 02:11:40 UTC
[GitHub] [beam] tweise opened a new pull request #12733: [BEAM-10760] Optimize state cleanup for global window in portable Flink runner
tweise opened a new pull request #12733:
URL: https://github.com/apache/beam/pull/12733
Cleanup timers per key lead to potentially unbounded state/checkpoint size growth with a global window in streaming mode.
Instead of timers, we can use the final watermark as the barrier to perform state cleanup.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace
--- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tweise commented on pull request #12733: [BEAM-10760] Optimize state cleanup for global window in portable Flink runner
Posted by GitBox <gi...@apache.org>.
tweise commented on pull request #12733:
URL: https://github.com/apache/beam/pull/12733#issuecomment-685174323
Unrelated error in `Portable_Python ("Run Portable_Python PreCommit") `
```
13:57:33 Execution failed for task ':sdks:python:container:py2:docker'.
13:57:33 > Process 'command 'docker'' finished with non-zero exit value 1
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tweise commented on pull request #12733: [BEAM-10760] Optimize state cleanup for global window in portable Flink runner
Posted by GitBox <gi...@apache.org>.
tweise commented on pull request #12733:
URL: https://github.com/apache/beam/pull/12733#issuecomment-685128393
@mxm PTAL
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mxm commented on a change in pull request #12733: [BEAM-10760] Optimize state cleanup for global window in portable Flink runner
Posted by GitBox <gi...@apache.org>.
mxm commented on a change in pull request #12733:
URL: https://github.com/apache/beam/pull/12733#discussion_r480015675
##########
File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
##########
@@ -592,12 +593,37 @@ public long applyOutputWatermarkHold(long currentOutputWatermark, long potential
return currentOutputWatermark;
}
} else {
+ // Check if the final watermark was triggered to perform state cleanup for global window
+ if (potentialOutputWatermark == Long.MAX_VALUE
+ && currentOutputWatermark < Long.MAX_VALUE
+ && isStateful) {
+ cleanupGlobalWindowState();
+ }
Review comment:
```suggestion
// Check if the final watermark was triggered to perform state cleanup for global window
if (potentialOutputWatermark > BoundedWindow.maxTimestamp().getMillis()
&& currentOutputWatermark <= BoundedWindow.maxTimestamp().getMillis()) {
cleanupGlobalWindowState();
}
```
Would change the condition to specifically target the end of the global window. There is also no need to include the statefulness in the condition because if there is no state, it also won't be cleaned up.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] je-ik commented on pull request #12733: [BEAM-10760] Optimize state cleanup for global window in portable Flink runner
Posted by GitBox <gi...@apache.org>.
je-ik commented on pull request #12733:
URL: https://github.com/apache/beam/pull/12733#issuecomment-685453050
This PR fixes the issue for portable runner only, am I right? Can we do the same for non-portable runner?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rakeshcusat commented on pull request #12733: [BEAM-10760] Optimize state cleanup for global window in portable Flink runner
Posted by GitBox <gi...@apache.org>.
rakeshcusat commented on pull request #12733:
URL: https://github.com/apache/beam/pull/12733#issuecomment-685282245
👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mxm commented on a change in pull request #12733: [BEAM-10760] Optimize state cleanup for global window in portable Flink runner
Posted by GitBox <gi...@apache.org>.
mxm commented on a change in pull request #12733:
URL: https://github.com/apache/beam/pull/12733#discussion_r481938984
##########
File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
##########
@@ -592,12 +593,35 @@ public long applyOutputWatermarkHold(long currentOutputWatermark, long potential
return currentOutputWatermark;
}
} else {
+ // Check if the final watermark was triggered to perform state cleanup for global window
+ if (potentialOutputWatermark > BoundedWindow.TIMESTAMP_MAX_VALUE.getMillis()
+ && currentOutputWatermark <= BoundedWindow.TIMESTAMP_MAX_VALUE.getMillis()) {
+ cleanupGlobalWindowState();
+ }
Review comment:
This logic should be moved to `DoFnOperator#emitWatermark`. It can then also be simplified.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tweise commented on pull request #12733: [BEAM-10760] Optimize state cleanup for global window in portable Flink runner
Posted by GitBox <gi...@apache.org>.
tweise commented on pull request #12733:
URL: https://github.com/apache/beam/pull/12733#issuecomment-683514633
This change appears to work as expected, based on e2e test.
I have not worked on the unit tests yet (couple will fail).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mxm merged pull request #12733: [BEAM-10760] Optimize state cleanup for global window in portable Flink runner
Posted by GitBox <gi...@apache.org>.
mxm merged pull request #12733:
URL: https://github.com/apache/beam/pull/12733
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mxm commented on pull request #12733: [BEAM-10760] Optimize state cleanup for global window in portable Flink runner
Posted by GitBox <gi...@apache.org>.
mxm commented on pull request #12733:
URL: https://github.com/apache/beam/pull/12733#issuecomment-685452105
Run Portable_Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mxm commented on pull request #12733: [BEAM-10760] Optimize state cleanup for global window in portable Flink runner
Posted by GitBox <gi...@apache.org>.
mxm commented on pull request #12733:
URL: https://github.com/apache/beam/pull/12733#issuecomment-685730748
Followed up on this via https://github.com/apache/beam/pull/12759.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org