[jira] [Commented] (BEAM-849) Redesign PipelineResult API

["Eugene Kirpichov (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890855#comment-15890855 ] 

Eugene Kirpichov commented on BEAM-849:
---------------------------------------

I disagree that "unbounded pipelines" can't finish successfully.

- Dataflow runner supports draining of pipelines, which leads to successful termination.
- It is possible to run a pipeline like Create.of(1, 2, 3) + ParDo(do nothing) using a streaming runner, and it should terminate rather than hang.
- One might ask "why run such a pipeline with a streaming runner", but it makes a lot more sense if the ParDo is splittable. E.g. Create.of(filename) + ParDo(tail file) + ParDo(process records) could use the low-latency capabilities of a streaming runner, but successfully terminate when the file is somehow "finalized". As a more mundane example - tests in https://github.com/apache/beam/blob/master/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SplittableDoFnTest.java should pass in streaming runners as well as batch runners.
- "Unbounded pipeline" is in general not a Beam concept - we should have a batch/streaming-agnostic meaning of "finished" in "waitUntilFinished". I propose the one that Dataflow runner uses for deciding when drain is completed: "all watermarks have progressed to infinity".

> Redesign PipelineResult API
> ---------------------------
>
>                 Key: BEAM-849
>                 URL: https://issues.apache.org/jira/browse/BEAM-849
>             Project: Beam
>          Issue Type: Improvement
>          Components: sdk-java-core
>            Reporter: Pei He
>
> Current state: 
> Jira https://issues.apache.org/jira/browse/BEAM-443 addresses waitUntilFinish() and cancel(). 
> However, there are additional work around PipelineResult: 
> need clearly defined contract and verification across all runners 
> need to revisit how to handle metrics/aggregators 
> need to be able to get logs



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)