You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/03/25 17:23:47 UTC
[GitHub] [beam] mxm opened a new pull request #11220:
[BEAM-9573][release-2.0] Correct computing of watermark hold for timer
output timestamp
mxm opened a new pull request #11220: [BEAM-9573][release-2.0] Correct computing of watermark hold for timer output timestamp
URL: https://github.com/apache/beam/pull/11220
### Backport of #11200.
With the introduction of timer output timestamps, a new watermark hold had been
added to the Flink Runner. The watermark computation works on the keyed state
backend which computes a key-scoped watermark hold and not the desired
operator-wide watermark hold.
Computation: https://github.com/apache/beam/blob/b564239081e9351c56fb0e7d263495b95dd3f8f3/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java#L1140
Key-scoped state: https://github.com/apache/beam/blob/b564239081e9351c56fb0e7d263495b95dd3f8f3/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java#L1130
The solution is to iterate over all available state backend keys.
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11220: [BEAM-9573][release-2.20]
Correct computing of watermark hold for timer output timestamp
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11220: [BEAM-9573][release-2.20] Correct computing of watermark hold for timer output timestamp
URL: https://github.com/apache/beam/pull/11220#issuecomment-604679224
Error prone errors in the build:
```
20:42:09 * What went wrong:
20:42:09 Execution failed for task ':model:pipeline:compileJava'.
20:42:09 > Could not resolve all files for configuration ':model:pipeline:errorprone'.
20:42:09 > Could not resolve com.google.errorprone:error_prone_core:latest.release.
20:42:09 Required by:
20:42:09 project :model:pipeline
20:42:09 > Could not resolve com.google.errorprone:error_prone_core:2.3.4-SNAPSHOT.
20:42:09 > Unable to load Maven meta-data from https://oss.sonatype.org/content/repositories/staging/com/google/errorprone/error_prone_core/2.3.4-SNAPSHOT/maven-metadata.xml.
20:42:09 > Could not HEAD 'https://oss.sonatype.org/content/repositories/staging/com/google/errorprone/error_prone_core/2.3.4-SNAPSHOT/maven-metadata.xml'. Received status code 502 from server: Bad Gateway
20:42:09
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11220: [BEAM-9573][release-2.20]
Correct computing of watermark hold for timer output timestamp
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11220: [BEAM-9573][release-2.20] Correct computing of watermark hold for timer output timestamp
URL: https://github.com/apache/beam/pull/11220#issuecomment-604501694
I've already tested those, but running them again to make sure.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11220: [BEAM-9573][release-2.20]
Correct computing of watermark hold for timer output timestamp
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11220: [BEAM-9573][release-2.20] Correct computing of watermark hold for timer output timestamp
URL: https://github.com/apache/beam/pull/11220#issuecomment-604880488
Awesome. Thanks for the patience!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11220: [BEAM-9573][release-2.20]
Correct computing of watermark hold for timer output timestamp
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11220: [BEAM-9573][release-2.20] Correct computing of watermark hold for timer output timestamp
URL: https://github.com/apache/beam/pull/11220#issuecomment-604679825
Run Flink Runner Nexmark Tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11220: [BEAM-9573][release-2.20]
Correct computing of watermark hold for timer output timestamp
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11220: [BEAM-9573][release-2.20] Correct computing of watermark hold for timer output timestamp
URL: https://github.com/apache/beam/pull/11220#issuecomment-604501440
Run Flink ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] amaliujia commented on issue #11220:
[BEAM-9573][release-2.20] Correct computing of watermark hold for timer
output timestamp
Posted by GitBox <gi...@apache.org>.
amaliujia commented on issue #11220: [BEAM-9573][release-2.20] Correct computing of watermark hold for timer output timestamp
URL: https://github.com/apache/beam/pull/11220#issuecomment-604152523
@mxm any additional tests should run?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm merged pull request #11220: [BEAM-9573][release-2.20]
Correct computing of watermark hold for timer output timestamp
Posted by GitBox <gi...@apache.org>.
mxm merged pull request #11220: [BEAM-9573][release-2.20] Correct computing of watermark hold for timer output timestamp
URL: https://github.com/apache/beam/pull/11220
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] amaliujia commented on issue #11220:
[BEAM-9573][release-2.20] Correct computing of watermark hold for timer
output timestamp
Posted by GitBox <gi...@apache.org>.
amaliujia commented on issue #11220: [BEAM-9573][release-2.20] Correct computing of watermark hold for timer output timestamp
URL: https://github.com/apache/beam/pull/11220#issuecomment-604611247
Sounds good to me. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] amaliujia edited a comment on issue #11220:
[BEAM-9573][release-2.20] Correct computing of watermark hold for timer
output timestamp
Posted by GitBox <gi...@apache.org>.
amaliujia edited a comment on issue #11220: [BEAM-9573][release-2.20] Correct computing of watermark hold for timer output timestamp
URL: https://github.com/apache/beam/pull/11220#issuecomment-604729513
Great! all tests have passed!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11220: [BEAM-9573][release-2.20]
Correct computing of watermark hold for timer output timestamp
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11220: [BEAM-9573][release-2.20] Correct computing of watermark hold for timer output timestamp
URL: https://github.com/apache/beam/pull/11220#issuecomment-604590270
I have backported both fixes now: BEAM-9573 and BEAM-9566 as they really only make sense together. One fixes a bug, the other one a performance regression.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11220: [BEAM-9573][release-2.20]
Correct computing of watermark hold for timer output timestamp
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11220: [BEAM-9573][release-2.20] Correct computing of watermark hold for timer output timestamp
URL: https://github.com/apache/beam/pull/11220#issuecomment-604501285
Run Flink Runner Nexmark Tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] amaliujia commented on issue #11220:
[BEAM-9573][release-2.20] Correct computing of watermark hold for timer
output timestamp
Posted by GitBox <gi...@apache.org>.
amaliujia commented on issue #11220: [BEAM-9573][release-2.20] Correct computing of watermark hold for timer output timestamp
URL: https://github.com/apache/beam/pull/11220#issuecomment-603985830
LGTM after necessary tests pass
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11220: [BEAM-9573][release-2.20]
Correct computing of watermark hold for timer output timestamp
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11220: [BEAM-9573][release-2.20] Correct computing of watermark hold for timer output timestamp
URL: https://github.com/apache/beam/pull/11220#issuecomment-604589905
Run Flink Runner Nexmark Tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11220: [BEAM-9573][release-2.20]
Correct computing of watermark hold for timer output timestamp
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11220: [BEAM-9573][release-2.20] Correct computing of watermark hold for timer output timestamp
URL: https://github.com/apache/beam/pull/11220#issuecomment-604679458
Run Portable_Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11220: [BEAM-9573][release-2.20]
Correct computing of watermark hold for timer output timestamp
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11220: [BEAM-9573][release-2.20] Correct computing of watermark hold for timer output timestamp
URL: https://github.com/apache/beam/pull/11220#issuecomment-604589973
Run Flink ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11220: [BEAM-9573][release-2.20]
Correct computing of watermark hold for timer output timestamp
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11220: [BEAM-9573][release-2.20] Correct computing of watermark hold for timer output timestamp
URL: https://github.com/apache/beam/pull/11220#issuecomment-604679733
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11220: [BEAM-9573][release-2.20]
Correct computing of watermark hold for timer output timestamp
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11220: [BEAM-9573][release-2.20] Correct computing of watermark hold for timer output timestamp
URL: https://github.com/apache/beam/pull/11220#issuecomment-604679909
Run Flink ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] amaliujia commented on issue #11220:
[BEAM-9573][release-2.20] Correct computing of watermark hold for timer
output timestamp
Posted by GitBox <gi...@apache.org>.
amaliujia commented on issue #11220: [BEAM-9573][release-2.20] Correct computing of watermark hold for timer output timestamp
URL: https://github.com/apache/beam/pull/11220#issuecomment-604729513
Great! all tests have pass!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services