You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by ke...@apache.org on 2025/06/03 15:54:10 UTC
(beam) branch master updated: add extra_transforms block documentation to chain transform documentation (#35101)
This is an automated email from the ASF dual-hosted git repository.
kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 64ea938877c add extra_transforms block documentation to chain transform documentation (#35101)
64ea938877c is described below
commit 64ea938877c6c0ed6ab02abff25dc00340ac1a05
Author: Derrick Williams <de...@google.com>
AuthorDate: Tue Jun 3 11:54:03 2025 -0400
add extra_transforms block documentation to chain transform documentation (#35101)
---
.../www/site/content/en/documentation/sdks/yaml.md | 46 ++++++++++++++++++++++
1 file changed, 46 insertions(+)
diff --git a/website/www/site/content/en/documentation/sdks/yaml.md b/website/www/site/content/en/documentation/sdks/yaml.md
index 3d266ade616..73d1eebaae9 100644
--- a/website/www/site/content/en/documentation/sdks/yaml.md
+++ b/website/www/site/content/en/documentation/sdks/yaml.md
@@ -273,6 +273,52 @@ pipeline:
path: /path/to/output.json
```
+If a `chain` pipeline has required error consumption or needs additional
+transforms not supported in a typical `chain` context, use an
+`extra_transforms` block.
+
+```
+pipeline:
+ type: chain
+ transforms:
+ - type: ReadFromCsv
+ config:
+ path: /path/to/input*.csv
+
+ - type: MapToFields
+ name: SomeStep
+ config:
+ language: python
+ fields:
+ col1: col1
+ # This could raise a divide-by-zero error.
+ ratio: col2 / col3
+ error_handling:
+ output: errors
+
+ - type: MapToFields
+ name: AnotherStep
+ config:
+ language: python
+ fields:
+ col1: col1
+ # This could raise a divide-by-zero error.
+ inverse_ratio: 1 / ratio
+ error_handling:
+ output: errors
+
+ - type: WriteToJson
+ config:
+ path: /path/to/output.json
+
+ extra_transforms:
+ - type: WriteToJson
+ name: WriteErrors
+ input: [SomeStep.errors, AnotherStep.errors]
+ config:
+ path: /path/to/errors.json
+```
+
### Source and sink transforms
As syntactic sugar, you can name the first and last transforms in your pipeline