(beam) branch master updated: Document yaml pipeline options (#30490)

[da...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

damccorm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
     new 3c46415a3ce Document yaml pipeline options (#30490)
3c46415a3ce is described below

commit 3c46415a3cebc7337ff5fd0d7969836dd2e58694
Author: Danny McCormick <da...@google.com>
AuthorDate: Tue Mar 5 15:14:21 2024 -0500

    Document yaml pipeline options (#30490)
---
 .../www/site/content/en/documentation/sdks/yaml.md | 40 ++++++++++++++++++++++
 1 file changed, 40 insertions(+)

diff --git a/website/www/site/content/en/documentation/sdks/yaml.md b/website/www/site/content/en/documentation/sdks/yaml.md
index ca87d302380..d9c3b8db51b 100644
--- a/website/www/site/content/en/documentation/sdks/yaml.md
+++ b/website/www/site/content/en/documentation/sdks/yaml.md
@@ -366,6 +366,8 @@ pipeline:
       config:
         topic: anotherPubSubTopic
         format: json
+options:
+  streaming: true
 ```
 
 Rather than using an explicit `WindowInto` operation, one may instead tag a
@@ -392,6 +394,8 @@ pipeline:
       config:
         topic: anotherPubSubTopic
         format: json
+options:
+  streaming: true
 ```
 
 Note that the `Sql` operation itself is often a from of aggregation, and
@@ -417,6 +421,8 @@ pipeline:
       config:
         topic: anotherPubSubTopic
         format: json
+options:
+  streaming: true
 ```
 
 The specified windowing is applied to all inputs, in this case resulting in
@@ -448,6 +454,8 @@ pipeline:
       windowing:
         type: fixed
         size: 60s
+options:
+  streaming: true
 ```
 
 For a transform with no inputs, the specified windowing is instead applied to
@@ -473,6 +481,8 @@ pipeline:
       config:
         topic: anotherPubSubTopic
         format: json
+options:
+  streaming: true
 ```
 
 One can also specify windowing at the top level of a pipeline (or composite),
@@ -499,6 +509,8 @@ pipeline:
   windowing:
     type: fixed
     size: 60
+options:
+  streaming: true
 ```
 
 Note that all these windowing specifications are compatible with the `source`
@@ -530,6 +542,9 @@ pipeline:
     windowing:
       type: fixed
       size: 5m
+
+options:
+  streaming: true
 ```
 
 
@@ -584,6 +599,31 @@ providers:
        MyCustomTransform: "pkg.subpkg.PTransformClassOrCallable"
 ```
 
+## Pipeline Options
+
+[Pipeline options](https://beam.apache.org/documentation/programming-guide/#configuring-pipeline-options)
+are used to configure different aspects of your pipeline, such as the pipeline runner that will execute
+your pipeline and any runner-specific configuration required by the chosen runner. To set pipeline options,
+append an options block at the end of your yaml file. For example:
+
+```
+pipeline:
+  type: chain
+  transforms:
+    - type: ReadFromPubSub
+      config:
+        topic: myPubSubTopic
+        format: ...
+        schema: ...
+    ...
+    - type: WriteToPubSub
+      config:
+        topic: anotherPubSubTopic
+        format: json
+options:
+  streaming: true
+```
+
 ## Other Resources
 
 * [Example pipelines](https://gist.github.com/robertwb/2cb26973f1b1203e8f5f8f88c5764da0)