You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Pablo Estrada (Jira)" <ji...@apache.org> on 2021/02/03 01:56:00 UTC
[jira] [Comment Edited] (BEAM-9640) Track PCollection watermark
across bundle executions
[ https://issues.apache.org/jira/browse/BEAM-9640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17277627#comment-17277627 ]
Pablo Estrada edited comment on BEAM-9640 at 2/3/21, 1:55 AM:
--------------------------------------------------------------
As of right now, most tests are passing (all tests pass if I remove assertion checks for MIN and MAX watermarks before and after execution of stages).
Failing test:
{code:java}
FAIL: test_draining_sdf_with_sdf_initiated_checkpointing (apache_beam.runners.portability.fn_api_runner.fn_runner_test.FnApiRunnerTest)
{code}
This test includes an SDF expansion with support for drain. The reason it fails is that there is some sort of delayed application on `ref_PCollection_PCollection_8_truncate_restriction`. I am investigating this.
Shape of the pipeline (main inputs in bold links):
!graph.png|width=893,height=436!
To investigate:
* The PR adding support for drain in Python: [https://github.com/apache/beam/pull/12289/]
*
was (Author: pabloem):
As of right now, most tests are passing (all tests pass if I remove assertion checks for MIN and MAX watermarks before and after execution of stages).
Failing test:
{code:java}
FAIL: test_draining_sdf_with_sdf_initiated_checkpointing (apache_beam.runners.portability.fn_api_runner.fn_runner_test.FnApiRunnerTest)
{code}
This test includes an SDF expansion with support for drain. The reason it fails is that there is some sort of delayed application on `ref_PCollection_PCollection_8_truncate_restriction`. I am investigating this.
Shape of the pipeline:
!graph.png|width=893,height=436!
To investigate:
* The PR adding support for drain in Python: [https://github.com/apache/beam/pull/12289/]
*
> Track PCollection watermark across bundle executions
> ----------------------------------------------------
>
> Key: BEAM-9640
> URL: https://issues.apache.org/jira/browse/BEAM-9640
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
> Reporter: Pablo Estrada
> Priority: P3
> Attachments: graph.png
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> This can be done without relying on the watermark manager for execution.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)