You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "David Cavazos (Jira)" <ji...@apache.org> on 2021/06/04 18:18:00 UTC

[jira] [Created] (BEAM-12455) Have a --wait_until_finish=[True|False] pipeline option

David Cavazos created BEAM-12455:
------------------------------------

             Summary: Have a --wait_until_finish=[True|False] pipeline option
                 Key: BEAM-12455
                 URL: https://issues.apache.org/jira/browse/BEAM-12455
             Project: Beam
          Issue Type: Improvement
          Components: sdk-py-core
            Reporter: David Cavazos


Currently, if we use the `with` statement to run a pipeline, it waits until finish by default with no way to configure it. The only way to run without waiting is by calling `pipeline.run()` explicitly without the `with` statement context. This is not consistent and users now have to be aware of two different ways of calling.

Ideally, we would like to define pipelines with the `with` statement always to have a "single way to do things". Running without waiting is useful, for example, to run a pipeline in Cloud Build, Cloud Run, templates, or other serverless frameworks.

Right now the only way to do it is by changing the code to remove the `with` statement and calling `pipeline.run()` manually, which might not be always possible. With a pipeline option we can just pass that from the command line arguments, or even when testing.

I'm opening this for Python since many Python samples would greatly benefit from this, but I can see this being useful in Java and Go as well.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)