You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2026/01/08 23:40:51 UTC
(beam) branch asf-site updated: Publishing website 2026/01/08 23:40:44 at commit ef0a03c
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 44a958a3205 Publishing website 2026/01/08 23:40:44 at commit ef0a03c
44a958a3205 is described below
commit 44a958a3205ce7f0ab45f26f5237f620c325d553
Author: runner <ru...@main-runner-vplcm-hvszh>
AuthorDate: Thu Jan 8 23:40:44 2026 +0000
Publishing website 2026/01/08 23:40:44 at commit ef0a03c
---
.../documentation/runners/flink/index.html | 2 +-
.../documentation/sdks/yaml-errors/index.html | 45 +++++++++++++++++++++-
website/generated-content/sitemap.xml | 2 +-
3 files changed, 45 insertions(+), 4 deletions(-)
diff --git a/website/generated-content/documentation/runners/flink/index.html b/website/generated-content/documentation/runners/flink/index.html
index 1ddafb7da3d..ad74fc6e516 100644
--- a/website/generated-content/documentation/runners/flink/index.html
+++ b/website/generated-content/documentation/runners/flink/index.html
@@ -124,7 +124,7 @@ and will not work on remote clusters.
See <a href=/documentation/runtime/sdk-harness-config/>here</a> for details.</p><h2 id=additional-information-and-caveats>Additional information and caveats</h2><h3 id=monitoring-your-job>Monitoring your job</h3><p>You can monitor a running Flink job using the Flink JobManager Dashboard or its Rest interfaces. By default, this is available at port <code>8081</code> of the JobManager node. If you have a Flink installation on your local machine that would be <code>http://localhost:8081</co [...]
Many sources like <code>PubSubIO</code> rely on their checkpoints to be acknowledged which can only be done when checkpointing is enabled for the <code>FlinkRunner</code>. To enable checkpointing, please set <span class=language-java><code>checkpointingInterval</code></span><span class=language-py><code>checkpointing_interval</code></span> to the desired checkpointing interval in milliseconds.</p><h2 id=pipeline-options-for-the-flink-runner>Pipeline options for the Flink Runner</h2><p>Wh [...]
<a href=https://beam.apache.org/releases/javadoc/2.70.0/index.html?org/apache/beam/runners/flink/FlinkPipelineOptions.html>FlinkPipelineOptions</a>
-reference class:</p><div class=language-java><table class="table table-bordered"><tr><td><code>allowNonRestoredState</code></td><td>Flag indicating whether non restored state is allowed if the savepoint contains state for an operator that is no longer part of the pipeline.</td><td>Default: <code>false</code></td></tr><tr><td><code>attachedMode</code></td><td>Specifies if the pipeline is submitted in attached or detached mode</td><td>Default: <code>true</code></td></tr><tr><td><code>autoB [...]
+reference class:</p><div class=language-java><table class="table table-bordered"><tr><td><code>allowNonRestoredState</code></td><td>Flag indicating whether non restored state is allowed if the savepoint contains state for an operator that is no longer part of the pipeline.</td><td>Default: <code>false</code></td></tr><tr><td><code>attachedMode</code></td><td>Specifies if the pipeline is submitted in attached or detached mode</td><td>Default: <code>true</code></td></tr><tr><td><code>autoB [...]
<a href=https://beam.apache.org/releases/javadoc/2.70.0/index.html?org/apache/beam/sdk/options/PipelineOptions.html>PipelineOptions</a>
reference.</p><h2 id=flink-version-compatibility>Flink Version Compatibility</h2><p>The Flink cluster version has to match the minor version used by the FlinkRunner.
The minor version is the first two numbers in the version string, e.g. in <code>1.18.0</code> the
diff --git a/website/generated-content/documentation/sdks/yaml-errors/index.html b/website/generated-content/documentation/sdks/yaml-errors/index.html
index 5f2c20ca7ee..39e345edb45 100644
--- a/website/generated-content/documentation/sdks/yaml-errors/index.html
+++ b/website/generated-content/documentation/sdks/yaml-errors/index.html
@@ -35,7 +35,7 @@
<img class=banner-img-mobile src=/images/banners/tour-of-beam/tour-of-beam-mobile.png alt="Start Tour of Beam"></a></div><div class=swiper-slide><a href=https://beam.apache.org/documentation/ml/overview/><img class=banner-img-desktop src=/images/banners/machine-learning/machine-learning-desktop.jpg alt="Machine Learning">
<img class=banner-img-mobile src=/images/banners/machine-learning/machine-learning-mobile.jpg alt="Machine Learning"></a></div></div><div class=swiper-pagination></div><div class=swiper-button-prev></div><div class=swiper-button-next></div></div><script src=/js/swiper-bundle.min.min.e0e8f81b0b15728d35ff73c07f42ddbb17a108d6f23df4953cb3e60df7ade675.js></script>
<script src=/js/sliders/top-banners.min.afa7d0a19acf7a3b28ca369490b3d401a619562a2a4c9612577be2f66a4b9855.js></script>
-<script>function showSearch(){addPlaceholder();var e,t=document.querySelector(".searchBar");t.classList.remove("disappear"),e=document.querySelector("#iconsBar"),e.classList.add("disappear")}function addPlaceholder(){$("input:text").attr("placeholder","What are you looking for?")}function endSearch(){var e,t=document.querySelector(".searchBar");t.classList.add("disappear"),e=document.querySelector("#iconsBar"),e.classList.remove("disappear")}function blockScroll(){$("body").toggleClass(" [...]
+<script>function showSearch(){addPlaceholder();var e,t=document.querySelector(".searchBar");t.classList.remove("disappear"),e=document.querySelector("#iconsBar"),e.classList.add("disappear")}function addPlaceholder(){$("input:text").attr("placeholder","What are you looking for?")}function endSearch(){var e,t=document.querySelector(".searchBar");t.classList.add("disappear"),e=document.querySelector("#iconsBar"),e.classList.remove("disappear")}function blockScroll(){$("body").toggleClass(" [...]
data that is malformatted, doesn’t handle the proper preconditions, or otherwise
breaks during processing. Generally any such record will cause the pipeline to
permanently fail, but often it is desirable to allow the pipeline to continue,
@@ -204,7 +204,48 @@ in an <code>extra_transforms</code> block.</p><div class=highlight><pre tabindex
</span></span></span><span class=line><span class=cl><span class=w> </span><span class=nt>input</span><span class=p>:</span><span class=w> </span><span class=p>[</span><span class=l>SomeStep.errors, AnotherStep.errors]</span><span class=w>
</span></span></span><span class=line><span class=cl><span class=w> </span><span class=nt>config</span><span class=p>:</span><span class=w>
</span></span></span><span class=line><span class=cl><span class=w> </span><span class=nt>path</span><span class=p>:</span><span class=w> </span><span class=l>/path/to/errors.json</span><span class=w>
-</span></span></span></code></pre></div><p>See YAML schema <a href=https://beam.apache.org/documentation/sdks/yaml-schema/>info</a> for another use of error_handling in a schema context.</p></div></div><footer class=footer><div class=footer__contained><div class=footer__cols><div class="footer__cols__col footer__cols__col__logos"><div class=footer__cols__col__logo><img src=/images/beam_logo_circle.svg class=footer__logo alt="Beam logo"></div><div class=footer__cols__col__logo><img src=/i [...]
+</span></span></span></code></pre></div><h2 id=error-handling-with-custom-providers>Error Handling with Custom Providers</h2><p>Custom transforms, such as those defined in separate YAML files via a <code>YamlProvider</code>, can also expose error outputs from their underlying transforms.</p><p>Consider a file <code>my_transforms.yaml</code> that defines a <code>RaiseElementToPower</code> transform:</p><div class=highlight><pre tabindex=0 class=chroma><code class=language-yaml data-lang=y [...]
+</span></span></span><span class=line><span class=cl><span class=w></span>- <span class=nt>type</span><span class=p>:</span><span class=w> </span><span class=l>yaml</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span><span class=nt>transforms</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span><span class=nt>RaiseElementToPower</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span><span class=nt>config_schema</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span><span class=nt>properties</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span><span class=nt>n</span><span class=p>:</span><span class=w> </span>{<span class=nt>type</span><span class=p>:</span><span class=w> </span><span class=l>integer}</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span><span class=nt>body</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span><span class=nt>type</span><span class=p>:</span><span class=w> </span><span class=l>MapToFields</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span><span class=nt>config</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span><span class=nt>language</span><span class=p>:</span><span class=w> </span><span class=l>python</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span><span class=nt>append</span><span class=p>:</span><span class=w> </span><span class=kc>true</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span><span class=nt>fields</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span><span class=nt>power</span><span class=p>:</span><span class=w> </span><span class=s2>"element ** {{n}}"</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span><span class=c># This transform internally defines and exposes an error output.</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span><span class=nt>error_handling</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span><span class=nt>output</span><span class=p>:</span><span class=w> </span><span class=l>my_error</span><span class=w>
+</span></span></span></code></pre></div><p>This transform takes a numeric element and raises it to the power of <code>n</code>. If the element is not a number, it will produce an error. The error output from the internal <code>MapToFields</code> is named <code>my_error</code>. This error output is automatically exposed by the <code>RaiseElementToPower</code> transform.</p><p>When using this transform in a pipeline, you can access this error output and handle it. The main output of the tr [...]
+</span></span></span><span class=line><span class=cl><span class=w> </span><span class=nt>transforms</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span>- <span class=nt>type</span><span class=p>:</span><span class=w> </span><span class=l>Create</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span><span class=nt>config</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span><span class=nt>elements</span><span class=p>:</span><span class=w> </span><span class=p>[</span><span class=m>2</span><span class=p>,</span><span class=w> </span><span class=s1>'bad'</span><span class=p>,</span><span class=w> </span><span class=m>3</span><span class=p>]</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span>- <span class=nt>type</span><span class=p>:</span><span class=w> </span><span class=l>RaiseElementToPower</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span><span class=nt>input</span><span class=p>:</span><span class=w> </span><span class=l>Create</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span><span class=nt>config</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span><span class=nt>n</span><span class=p>:</span><span class=w> </span><span class=m>2</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span>- <span class=nt>type</span><span class=p>:</span><span class=w> </span><span class=l>WriteToJson</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span><span class=nt>name</span><span class=p>:</span><span class=w> </span><span class=l>WriteGood</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span><span class=c># The main output contains successfully processed elements.</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span><span class=nt>input</span><span class=p>:</span><span class=w> </span><span class=l>RaiseElementToPower</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span><span class=nt>config</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span><span class=nt>path</span><span class=p>:</span><span class=w> </span><span class=l>/path/to/good</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span>- <span class=nt>type</span><span class=p>:</span><span class=w> </span><span class=l>WriteToJson</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span><span class=nt>name</span><span class=p>:</span><span class=w> </span><span class=l>WriteBad</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span><span class=c># The error output is accessed by its name.</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span><span class=nt>input</span><span class=p>:</span><span class=w> </span><span class=l>RaiseElementToPower.my_error</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span><span class=nt>config</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span><span class=nt>path</span><span class=p>:</span><span class=w> </span><span class=l>/path/to/bad</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span><span class=nt>providers</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w> </span>- <span class=nt>include</span><span class=p>:</span><span class=w> </span><span class=l>my_transforms.yaml</span><span class=w>
+</span></span></span></code></pre></div><p>In this example, the pipeline separates the good and bad records coming from the custom <code>RaiseElementToPower</code> transform. The good records are written to one location, and the error records are written to another.</p><p>A pipeline will fail at construction time if an error output is declared (either in a built-in transform or a custom one) but not consumed. This helps ensure that all error paths are considered.</p><p>See YAML schema <a [...]
<a href=https://www.apache.org>The Apache Software Foundation</a>
| <a href=/privacy_policy>Privacy Policy</a>
| <a href=/feed.xml>RSS Feed</a><br><br>Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation. All other products or name brands are trademarks of their respective holders, including The Apache Software Foundation.</div></div><div class="footer__cols__col footer__cols__col__logos"><div class=footer__cols__col--group><div class=footer__cols__col__logo><a href=https://github.com/apache/beam><im [...]
\ No newline at end of file
diff --git a/website/generated-content/sitemap.xml b/website/generated-content/sitemap.xml
index 5cf7d1600b6..92271dd7588 100644
--- a/website/generated-content/sitemap.xml
+++ b/website/generated-content/sitemap.xml
@@ -1 +1 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/case-studies/albertsons/</loc><lastmod>2026-01-08T10:36:08-05:00</lastmod></url><url><loc>/blog/beam-2.70.0/</loc><lastmod>2026-01-08T10:36:08-05:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2026-01-08T10:36:08-05:00</lastmod></url><url><loc>/blog/</loc><lastmod>2026-01-08T10:36:08-05:00</lastmod></url><url [...]
\ No newline at end of file
+<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/case-studies/albertsons/</loc><lastmod>2026-01-08T18:26:55-05:00</lastmod></url><url><loc>/blog/beam-2.70.0/</loc><lastmod>2026-01-08T18:26:55-05:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2026-01-08T18:26:55-05:00</lastmod></url><url><loc>/blog/</loc><lastmod>2026-01-08T18:26:55-05:00</lastmod></url><url [...]
\ No newline at end of file