[jira] [Commented] (BEAM-9640) Track PCollection watermark across bundle executions

["Pablo Estrada (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-9640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17277627#comment-17277627 ] 

Pablo Estrada commented on BEAM-9640:
-------------------------------------

As of right now, most tests are passing (all tests pass if I remove assertion checks for MIN and MAX watermarks before and after execution of stages).

 

Failing test:
{code:java}
FAIL: test_draining_sdf_with_sdf_initiated_checkpointing (apache_beam.runners.portability.fn_api_runner.fn_runner_test.FnApiRunnerTest)
{code}
 This test includes an SDF expansion with support for drain. The reason it fails is that there is some sort of delayed application on `ref_PCollection_PCollection_8_truncate_restriction`. I am investigating this.

Shape of the pipeline:

!graph.png|width=893,height=436!

 

To investigate:
 * The PR adding support for drain in Python: [https://github.com/apache/beam/pull/12289/]
 *

> Track PCollection watermark across bundle executions
> ----------------------------------------------------
>
>                 Key: BEAM-9640
>                 URL: https://issues.apache.org/jira/browse/BEAM-9640
>             Project: Beam
>          Issue Type: Sub-task
>          Components: sdk-py-core
>            Reporter: Pablo Estrada
>            Priority: P3
>         Attachments: graph.png
>
>          Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> This can be done without relying on the watermark manager for execution.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)