You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/04/08 14:07:15 UTC
[GitHub] [beam] mxm opened a new pull request #11346: [BEAM-9580] Allow
Flink 1.10 processing timers to finish on pipeline shutdown
mxm opened a new pull request #11346: [BEAM-9580] Allow Flink 1.10 processing timers to finish on pipeline shutdown
URL: https://github.com/apache/beam/pull/11346
Due to the new mailbox operator architecture in Flink 1.10, processing remaining
timers during operator shutdown can't be achieved by releasing the checkpoint
lock anymore. Timers are only drained after the close() method completes.
However, at this point new timers also cannot be set anymore.
This doesn't play well with some of Beam's code (e.g. SDF) which assumes that
timers can set new timers which are guaranteed to fire at the end of the
pipeline. With the current processing model of Flink 1.10 this cannot be
guaranteed. This has manifesting in the Nexmark and ValidatesRunner tests.
While I have considered other possibilities, this was the only way to restore
the timer behavior we support with Beam and have been implementing with the
previous versions of the Flink Runner.
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11346: [BEAM-9580] Allow Flink 1.10
processing timers to finish on pipeline shutdown
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11346: [BEAM-9580] Allow Flink 1.10 processing timers to finish on pipeline shutdown
URL: https://github.com/apache/beam/pull/11346#issuecomment-611015543
Run Java Flink PortableValidatesRunner Streaming
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11346: [BEAM-9580] Allow Flink 1.10
processing timers to finish on pipeline shutdown
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11346: [BEAM-9580] Allow Flink 1.10 processing timers to finish on pipeline shutdown
URL: https://github.com/apache/beam/pull/11346#issuecomment-611016121
Run Flink ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11346: [BEAM-9580] Allow Flink 1.10
processing timers to finish on pipeline shutdown
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11346: [BEAM-9580] Allow Flink 1.10 processing timers to finish on pipeline shutdown
URL: https://github.com/apache/beam/pull/11346#issuecomment-610988986
Run Java Flink PortableValidatesRunner Streaming
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11346: [BEAM-9580] Allow Flink 1.10
processing timers to finish on pipeline shutdown
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11346: [BEAM-9580] Allow Flink 1.10 processing timers to finish on pipeline shutdown
URL: https://github.com/apache/beam/pull/11346#issuecomment-611032805
Unrelated MongoDB failure in https://builds.apache.org/job/beam_PreCommit_Java_Commit/10758/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11346: [BEAM-9580] Allow Flink 1.10
processing timers to finish on pipeline shutdown
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11346: [BEAM-9580] Allow Flink 1.10 processing timers to finish on pipeline shutdown
URL: https://github.com/apache/beam/pull/11346#issuecomment-611015467
Run Flink Runner Nexmark Tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11346: [BEAM-9580] Allow Flink 1.10
processing timers to finish on pipeline shutdown
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11346: [BEAM-9580] Allow Flink 1.10 processing timers to finish on pipeline shutdown
URL: https://github.com/apache/beam/pull/11346#issuecomment-611052832
Run Java Flink PortableValidatesRunner Streaming
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11346: [BEAM-9580] Allow Flink 1.10
processing timers to finish on pipeline shutdown
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11346: [BEAM-9580] Allow Flink 1.10 processing timers to finish on pipeline shutdown
URL: https://github.com/apache/beam/pull/11346#issuecomment-611015371
Run Flink ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11346: [BEAM-9580] Allow Flink 1.10
processing timers to finish on pipeline shutdown
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11346: [BEAM-9580] Allow Flink 1.10 processing timers to finish on pipeline shutdown
URL: https://github.com/apache/beam/pull/11346#issuecomment-611052767
Run Flink ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm removed a comment on issue #11346: [BEAM-9580] Allow
Flink 1.10 processing timers to finish on pipeline shutdown
Posted by GitBox <gi...@apache.org>.
mxm removed a comment on issue #11346: [BEAM-9580] Allow Flink 1.10 processing timers to finish on pipeline shutdown
URL: https://github.com/apache/beam/pull/11346#issuecomment-611016121
Run Flink ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11346: [BEAM-9580] Allow Flink
1.10 processing timers to finish on pipeline shutdown
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11346: [BEAM-9580] Allow Flink 1.10 processing timers to finish on pipeline shutdown
URL: https://github.com/apache/beam/pull/11346#issuecomment-610983488
Run Flink Runner Nexmark Tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11346: [BEAM-9580] Allow Flink
1.10 processing timers to finish on pipeline shutdown
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11346: [BEAM-9580] Allow Flink 1.10 processing timers to finish on pipeline shutdown
URL: https://github.com/apache/beam/pull/11346#issuecomment-610983140
Run Flink ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm merged pull request #11346: [BEAM-9580] Allow Flink 1.10
processing timers to finish on pipeline shutdown
Posted by GitBox <gi...@apache.org>.
mxm merged pull request #11346: [BEAM-9580] Allow Flink 1.10 processing timers to finish on pipeline shutdown
URL: https://github.com/apache/beam/pull/11346
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11346: [BEAM-9580] Allow Flink 1.10
processing timers to finish on pipeline shutdown
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11346: [BEAM-9580] Allow Flink 1.10 processing timers to finish on pipeline shutdown
URL: https://github.com/apache/beam/pull/11346#issuecomment-610992075
Run Seed Job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services