You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2019/02/15 12:45:00 UTC
[jira] [Work logged] (BEAM-6678) FlinkRunner does not checkpoint
partition view of watermark holds
[ https://issues.apache.org/jira/browse/BEAM-6678?focusedWorklogId=199203&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-199203 ]
ASF GitHub Bot logged work on BEAM-6678:
----------------------------------------
Author: ASF GitHub Bot
Created on: 15/Feb/19 12:44
Start Date: 15/Feb/19 12:44
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #7849: [BEAM-6678] Persist watermark holds view in Flink checkpoints
URL: https://github.com/apache/beam/pull/7849
The FlinkRunner maintains a partition view of the pending watermark holds. This
view is used to hold back the watermark at the operator. It must be persisted,
otherwise elements may be late after recovering a Beam job.
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | ---
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) </br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/) | --- | --- | ---
See [.test-infra/jenkins/README](../.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 199203)
Time Spent: 10m
Remaining Estimate: 0h
> FlinkRunner does not checkpoint partition view of watermark holds
> -----------------------------------------------------------------
>
> Key: BEAM-6678
> URL: https://issues.apache.org/jira/browse/BEAM-6678
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
> Reporter: Maximilian Michels
> Assignee: Maximilian Michels
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The FlinkRunner does not persist its view of the per-partition Watermark holds. This can lead to elements to be considered late after restoring from a savepoint or resuming a failed pipeline.
> Similar to the approach in BEAM-6650, we can recover the Watermarks by iterating through the keys of the state backend during recovery.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)