You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2021/10/13 06:06:46 UTC
[beam] branch asf-site updated: Publishing website 2021/10/13
06:06:04 at commit 71d7213
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new d0596f6 Publishing website 2021/10/13 06:06:04 at commit 71d7213
d0596f6 is described below
commit d0596f6f645e53f75a02bf54291dd3807e323c46
Author: jenkins <bu...@apache.org>
AuthorDate: Wed Oct 13 06:06:04 2021 +0000
Publishing website 2021/10/13 06:06:04 at commit 71d7213
---
website/generated-content/documentation/index.xml | 121 ++++++++++++++++++++-
.../documentation/programming-guide/index.html | 36 +++---
website/generated-content/sitemap.xml | 2 +-
3 files changed, 138 insertions(+), 21 deletions(-)
diff --git a/website/generated-content/documentation/index.xml b/website/generated-content/documentation/index.xml
index e6c7c3d..73601a9 100644
--- a/website/generated-content/documentation/index.xml
+++ b/website/generated-content/documentation/index.xml
@@ -9600,12 +9600,17 @@ state. Merging windows are not currently supported when using state and timers.&
care must be taken to remember that the elements in input PCollection have no guaranteed order and to ensure that the
program logic is resilient to this. Unit tests written using the DirectRunner will shuffle the order of element
processing, and are recommended to test for correctness.</p>
-<p class="language-java">In Java DoFn declares states to be accessed by creating final <code>StateSpec</code> member variables representing each state. Each
+<p class="language-java">In Java, DoFn declares states to be accessed by creating final <code>StateSpec</code> member variables representing each state. Each
state must be named using the <code>StateId</code> annotation; this name is unique to a ParDo in the graph and has no relation
to other nodes in the graph. A <code>DoFn</code> can declare multiple state variables.</p>
-<p class="language-py">In Python DoFn declares states to be accessed by creating <code>StateSpec</code> class member variables representing each state. Each
+<p class="language-py">In Python, DoFn declares states to be accessed by creating <code>StateSpec</code> class member variables representing each state. Each
<code>StateSpec</code> is initialized with a name, this name is unique to a ParDo in the graph and has no relation
to other nodes in the graph. A <code>DoFn</code> can declare multiple state variables.</p>
+<span class="language-go">
+<blockquote>
+<p><strong>Note:</strong> The Beam SDK for Go does not yet support a State and Timer API. See <a href="https://issues.apache.org/jira/browse/BEAM-10660">BEAM-10660</a> to contribute.</p>
+</blockquote>
+</span>
<h3 id="types-of-state">11.1. Types of state</h3>
<p>Beam provides several types of state:</p>
<h4 id="valuestate">ValueState</h4>
@@ -9661,6 +9666,14 @@ accumulates the number of elements seen.</p>
<span class="o">|</span> <span class="s1">&#39;state pardo&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">ReadModifyWriteStateDoFn</span><span class="p">()))</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="nx">This</span> <span class="nx">is</span> <span class="nx">not</span> <span class="nx">supported</span> <span class="nx">yet</span><span class="p">,</span> <span class="nx">see</span> <span class="nx">BEAM</span><span class="o">-</span><span class="mf">10660.</span></code></pre></div>
+</div>
+</div>
<h4 id="combiningstate">CombiningState</h4>
<p><code>CombiningState</code> allows you to create a state object that is updated using a Beam combiner. For example, the previous
<code>ValueState</code> example could be rewritten to use <code>CombiningState</code></p>
@@ -9692,6 +9705,14 @@ accumulates the number of elements seen.</p>
<span class="o">|</span> <span class="s1">&#39;Combine state pardo&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">CombiningStateDofn</span><span class="p">()))</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="nx">This</span> <span class="nx">is</span> <span class="nx">not</span> <span class="nx">supported</span> <span class="nx">yet</span><span class="p">,</span> <span class="nx">see</span> <span class="nx">BEAM</span><span class="o">-</span><span class="mf">10660.</span></code></pre></div>
+</div>
+</div>
<h4 id="bagstate">BagState</h4>
<p>A common use case for state is to accumulate multiple elements. <code>BagState</code> allows for accumulating an unordered set
of elements. This allows for addition of elements to the collection without requiring the reading of the entire
@@ -9737,6 +9758,14 @@ bags larger than available memory.</p>
<span class="o">|</span> <span class="s1">&#39;Bag state pardo&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">BagStateDoFn</span><span class="p">()))</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="nx">This</span> <span class="nx">is</span> <span class="nx">not</span> <span class="nx">supported</span> <span class="nx">yet</span><span class="p">,</span> <span class="nx">see</span> <span class="nx">BEAM</span><span class="o">-</span><span class="mf">10660.</span></code></pre></div>
+</div>
+</div>
<h3 id="deferred-state-reads">11.2. Deferred state reads</h3>
<p>When a <code>DoFn</code> contains multiple state specifications, reading each one in order can be slow. Calling the <code>read()</code> function
on a state can cause the runner to perform a blocking read. Performing multiple blocking reads in sequence adds latency
@@ -9771,6 +9800,14 @@ runner can prefetch all of the states necessary. For example:</p>
<div class="highlight"><pre class="chroma"><code class="language-py" data-lang="py"><span class="n">This</span> <span class="ow">is</span> <span class="ow">not</span> <span class="n">supported</span> <span class="n">yet</span><span class="p">,</span> <span class="n">see</span> <span class="n">BEAM</span><span class="o">-</span><span class="mf">11506.</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="nx">This</span> <span class="nx">is</span> <span class="nx">not</span> <span class="nx">supported</span> <span class="nx">yet</span><span class="p">,</span> <span class="nx">see</span> <span class="nx">BEAM</span><span class="o">-</span><span class="mf">10660.</span></code></pre></div>
+</div>
+</div>
<p>If however there are code paths in which the states are not fetched, then annotating with @AlwaysFetched will add
unnecessary fetching for those paths. In this case, the readLater method allows the runner to know that the state will
be read in the future, allowing multiple state reads to be batched together.</p>
@@ -9857,6 +9894,14 @@ allows for event-time aggregations.</p>
<span class="o">|</span> <span class="s1">&#39;EventTime timer pardo&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">EventTimerDoFn</span><span class="p">()))</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="nx">This</span> <span class="nx">is</span> <span class="nx">not</span> <span class="nx">supported</span> <span class="nx">yet</span><span class="p">,</span> <span class="nx">see</span> <span class="nx">BEAM</span><span class="o">-</span><span class="mf">10660.</span></code></pre></div>
+</div>
+</div>
<h4 id="processing-time-timers">11.3.2. Processing-time timers</h4>
<p>Processing-time timers fire when the real wall-clock time passes. This is often used to create larger batches of data
before processing. It can also be used to schedule events that should occur at a specific time. Just like with
@@ -9905,6 +9950,14 @@ to the current time. In Java, the <code>Timer.offset</code> and <code>T
<span class="o">|</span> <span class="s1">&#39;ProcessingTime timer pardo&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">ProcessingTimerDoFn</span><span class="p">()))</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="nx">This</span> <span class="nx">is</span> <span class="nx">not</span> <span class="nx">supported</span> <span class="nx">yet</span><span class="p">,</span> <span class="nx">see</span> <span class="nx">BEAM</span><span class="o">-</span><span class="mf">10660.</span></code></pre></div>
+</div>
+</div>
<h4 id="dynamic-timer-tags">11.3.3. Dynamic timer tags</h4>
<p>Beam also supports dynamically setting a timer tag using <code>TimerMap</code> in the Java SDK. This allows for setting multiple different timers
in a <code>DoFn</code> and allowing for the timer tags to be dynamically chosen - e.g. based on data in the input elements. A
@@ -9962,6 +10015,14 @@ tag is an empty string if not specified.</p>
<span class="o">|</span> <span class="s1">&#39;ProcessingTime timer pardo&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">TimerDoFn</span><span class="p">()))</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="nx">This</span> <span class="nx">is</span> <span class="nx">not</span> <span class="nx">supported</span> <span class="nx">yet</span><span class="p">,</span> <span class="nx">see</span> <span class="nx">BEAM</span><span class="o">-</span><span class="mf">10660.</span></code></pre></div>
+</div>
+</div>
<h4 id="timer-output-timestamps">11.3.4. Timer output timestamps</h4>
<p>By default, event-time timers will hold the output watermark of the <code>ParDo</code> to the timestamp of the timer. This means
that if a timer is set to 12pm, any windowed aggregations or event-time timers later in the pipeline graph that finish
@@ -10071,6 +10132,14 @@ past the timestamp of the minimum element. The following code demonstrates this.
<div class="highlight"><pre class="chroma"><code class="language-py" data-lang="py"><span class="n">Timer</span> <span class="n">output</span> <span class="n">timestamps</span> <span class="ow">is</span> <span class="ow">not</span> <span class="n">yet</span> <span class="n">supported</span> <span class="ow">in</span> <span class="n">Python</span> <span class="n">SDK</span><span class="o">.</span> <span class="n [...]
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="nx">This</span> <span class="nx">is</span> <span class="nx">not</span> <span class="nx">supported</span> <span class="nx">yet</span><span class="p">,</span> <span class="nx">see</span> <span class="nx">BEAM</span><span class="o">-</span><span class="mf">10660.</span></code></pre></div>
+</div>
+</div>
<h3 id="garbage-collecting-state">11.4. Garbage collecting state</h3>
<p>Per-key state needs to be garbage collected, or eventually the increasing size of state may negatively impact
performance. There are two common strategies for garbage collecting state.</p>
@@ -10116,6 +10185,14 @@ garbage-collection strategy.</p>
<span class="o">|</span> <span class="s1">&#39;DoFn&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">StateDoFn</span><span class="p">()))</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="nx">This</span> <span class="nx">is</span> <span class="nx">not</span> <span class="nx">supported</span> <span class="nx">yet</span><span class="p">,</span> <span class="nx">see</span> <span class="nx">BEAM</span><span class="o">-</span><span class="mf">10660.</span></code></pre></div>
+</div>
+</div>
<p>This <code>ParDo</code> stores state per day. Once the pipeline is done processing data for a given day, all the state for that
day is garbage collected.</p>
<h5 id="using-timers-for-garbage-collection">11.4.1. <strong>Using timers For garbage collection</strong></h5>
@@ -10191,6 +10268,14 @@ This can be done by updating a timer that garbage collects state. For example<
<span class="o">|</span> <span class="s1">&#39;User DoFn&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">UserDoFn</span><span class="p">()))</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="nx">This</span> <span class="nx">is</span> <span class="nx">not</span> <span class="nx">supported</span> <span class="nx">yet</span><span class="p">,</span> <span class="nx">see</span> <span class="nx">BEAM</span><span class="o">-</span><span class="mf">10660.</span></code></pre></div>
+</div>
+</div>
<h3 id="state-timers-examples">11.5. State and timers examples</h3>
<p>Following are some example uses of state and timers</p>
<h4 id="joining-clicks-and-views">11.5.1. Joining clicks and views</h4>
@@ -10749,6 +10834,14 @@ resource utilization.</p>
<span class="k">return</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="nx">This</span> <span class="nx">is</span> <span class="nx">not</span> <span class="nx">supported</span> <span class="nx">yet</span><span class="p">,</span> <span class="nx">see</span> <span class="nx">BEAM</span><span class="o">-</span><span class="mf">11104.</span></code></pre></div>
+</div>
+</div>
<h3 id="runner-initiated-split">12.4. Runner-initiated split</h3>
<p>A runner at any time may attempt to split a restriction while it is being processed. This allows the
runner to either pause processing of the restriction so that other work may be done (common for
@@ -10961,6 +11054,14 @@ watermark estimator implementation. You can also provide your own watermark esti
<span class="n">watermark_estimator</span><span class="o">.</span><span class="n">current_watermark</span><span class="p">()</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="nx">This</span> <span class="nx">is</span> <span class="nx">not</span> <span class="nx">supported</span> <span class="nx">yet</span><span class="p">,</span> <span class="nx">see</span> <span class="nx">BEAM</span><span class="o">-</span><span class="mf">11105.</span></code></pre></div>
+</div>
+</div>
<h3 id="truncating-during-drain">12.6. Truncating during drain</h3>
<p>Runners which support draining pipelines need the ability to drain SDFs; otherwise, the
pipeline may never stop. By default, bounded restrictions process the remainder of the restriction while
@@ -10997,6 +11098,14 @@ provider.</p>
<span class="k">return</span> <span class="n">restriction</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="nx">This</span> <span class="nx">is</span> <span class="nx">not</span> <span class="nx">supported</span> <span class="nx">yet</span><span class="p">,</span> <span class="nx">see</span> <span class="nx">BEAM</span><span class="o">-</span><span class="mf">11106.</span></code></pre></div>
+</div>
+</div>
<h3 id="bundle-finalization">12.7. Bundle finalization</h3>
<p>Bundle finalization enables a <code>DoFn</code> to perform side effects by registering a callback.
The callback is invoked once the runner has acknowledged that it has durably persisted the output.
@@ -11033,6 +11142,14 @@ use case.</p>
<span class="n">bundle_finalizer</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">my_callback_func</span><span class="p">)</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="nx">This</span> <span class="nx">is</span> <span class="nx">not</span> <span class="nx">supported</span> <span class="nx">yet</span><span class="p">,</span> <span class="nx">see</span> <span class="nx">BEAM</span><span class="o">-</span><span class="mf">10976.</span></code></pre></div>
+</div>
+</div>
<h2 id="multi-language-pipelines">13. Multi-language pipelines</h2>
<p>Beam allows you to combine transforms written in any supported SDK language (currently, Java and Python) and use them in one multi-language pipeline. This capability makes it easy to provide new functionality simultaneously in different Apache Beam SDKs through a single cross-language transform. For example, the <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/kafka.py">Apache Kafka connector</a> and <a href="https://github.com/apache/beam/blo [...]
<p>Pipelines that use transforms from more than one SDK-language are known as <em>multi-language pipelines</em>.</p>
diff --git a/website/generated-content/documentation/programming-guide/index.html b/website/generated-content/documentation/programming-guide/index.html
index 81b826f..224974a 100644
--- a/website/generated-content/documentation/programming-guide/index.html
+++ b/website/generated-content/documentation/programming-guide/index.html
@@ -3090,11 +3090,11 @@ the stateful operator. For example, using a combiner to preaggregate data, and t
state. Merging windows are not currently supported when using state and timers.</p><p>Sometimes stateful processing is used to implement state-machine style processing inside a <code>DoFn</code>. When doing this,
care must be taken to remember that the elements in input PCollection have no guaranteed order and to ensure that the
program logic is resilient to this. Unit tests written using the DirectRunner will shuffle the order of element
-processing, and are recommended to test for correctness.</p><p class=language-java>In Java DoFn declares states to be accessed by creating final <code>StateSpec</code> member variables representing each state. Each
+processing, and are recommended to test for correctness.</p><p class=language-java>In Java, DoFn declares states to be accessed by creating final <code>StateSpec</code> member variables representing each state. Each
state must be named using the <code>StateId</code> annotation; this name is unique to a ParDo in the graph and has no relation
-to other nodes in the graph. A <code>DoFn</code> can declare multiple state variables.</p><p class=language-py>In Python DoFn declares states to be accessed by creating <code>StateSpec</code> class member variables representing each state. Each
+to other nodes in the graph. A <code>DoFn</code> can declare multiple state variables.</p><p class=language-py>In Python, DoFn declares states to be accessed by creating <code>StateSpec</code> class member variables representing each state. Each
<code>StateSpec</code> is initialized with a name, this name is unique to a ParDo in the graph and has no relation
-to other nodes in the graph. A <code>DoFn</code> can declare multiple state variables.</p><h3 id=types-of-state>11.1. Types of state</h3><p>Beam provides several types of state:</p><h4 id=valuestate>ValueState</h4><p>A ValueState is a scalar state value. For each key in the input, a ValueState will store a typed value that can be
+to other nodes in the graph. A <code>DoFn</code> can declare multiple state variables.</p><span class=language-go><blockquote><p><strong>Note:</strong> The Beam SDK for Go does not yet support a State and Timer API. See <a href=https://issues.apache.org/jira/browse/BEAM-10660>BEAM-10660</a> to contribute.</p></blockquote></span><h3 id=types-of-state>11.1. Types of state</h3><p>Beam provides several types of state:</p><h4 id=valuestate>ValueState</h4><p>A ValueState is a scalar state valu [...]
read and modified inside the DoFn’s <code>@ProcessElement</code> or <code>@OnTimer</code> methods. If the type of the ValueState has a coder
registered, then Beam will automatically infer the coder for the state value. Otherwise, a coder can be explicitly
specified when creating the ValueState. For example, the following ParDo creates a single state variable that
@@ -3122,7 +3122,7 @@ accumulates the number of elements seen.</p><p>Note: <code>ValueState</code> is
<span class=n>state</span><span class=o>.</span><span class=n>write</span><span class=p>(</span><span class=n>current_value</span><span class=o>+</span><span class=mi>1</span><span class=p>)</span>
<span class=n>_</span> <span class=o>=</span> <span class=p>(</span><span class=n>p</span> <span class=o>|</span> <span class=s1>'Read per user'</span> <span class=o>>></span> <span class=n>ReadPerUser</span><span class=p>()</span>
- <span class=o>|</span> <span class=s1>'state pardo'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span class=n>ReadModifyWriteStateDoFn</span><span class=p>()))</span></code></pre></div></div></div><h4 id=combiningstate>CombiningState</h4><p><code>CombiningState</code> allows you to create a state object that is updated using a Beam combiner. For example, the previous
+ <span class=o>|</span> <span class=s1>'state pardo'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span class=n>ReadModifyWriteStateDoFn</span><span class=p>()))</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/co [...]
<code>ValueState</code> example could be rewritten to use <code>CombiningState</code></p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=n>PCollection</span><span class=o><</span><span class=n>KV</span><span class=o><</span><span cl [...]
<span class=n>perUser</span><span class=o>.</span><span class=na>apply</span><span class=o>(</span><span class=n>ParDo</span><span class=o>.</span><span class=na>of</span><span class=o>(</span><span class=k>new</span> <span class=n>DoFn</span><span class=o><</span><span class=n>KV</span><span class=o><</span><span class=n>String</span><span class=o>,</span> <span class=n>ValueT</span><span class=o>>,</span> <span class=n>OutputT</span><span class=o>>()</span> <span class=o>{</span>
<span class=nd>@StateId</span><span class=o>(</span><span class=s>"state"</span><span class=o>)</span> <span class=kd>private</span> <span class=kd>final</span> <span class=n>StateSpec</span><span class=o><</span><span class=n>CombiningState</span><span class=o><</span><span class=n>Integer</span><span class=o>,</span> <span class=kt>int</span><span class=o>[],</span> <span class=n>Integer</span><span class=o>>></span> <span class=n>numElements</span> <span class=o> [...]
@@ -3138,7 +3138,7 @@ accumulates the number of elements seen.</p><p>Note: <code>ValueState</code> is
<span class=n>state</span><span class=o>.</span><span class=n>add</span><span class=p>(</span><span class=mi>1</span><span class=p>)</span>
<span class=n>_</span> <span class=o>=</span> <span class=p>(</span><span class=n>p</span> <span class=o>|</span> <span class=s1>'Read per user'</span> <span class=o>>></span> <span class=n>ReadPerUser</span><span class=p>()</span>
- <span class=o>|</span> <span class=s1>'Combine state pardo'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span class=n>CombiningStateDofn</span><span class=p>()))</span></code></pre></div></div></div><h4 id=bagstate>BagState</h4><p>A common use case for state is to accumulate multiple elements. <code>BagState</code> allows for accumulating an unordered set
+ <span class=o>|</span> <span class=s1>'Combine state pardo'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span class=n>CombiningStateDofn</span><span class=p>()))</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/ [...]
of elements. This allows for addition of elements to the collection without requiring the reading of the entire
collection first, which is an efficiency gain. In addition, runners that support paged reads can allow individual
bags larger than available memory.</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=n>PCollection</span><span class=o><</span><span class=n>KV</span><span class=o><</span><span class=n>String</span><span class=o>,</span> <span cla [...]
@@ -3168,7 +3168,7 @@ bags larger than available memory.</p><div class="language-java snippet"><div cl
<span class=n>state</span><span class=o>.</span><span class=n>clear</span><span class=p>()</span>
<span class=n>_</span> <span class=o>=</span> <span class=p>(</span><span class=n>p</span> <span class=o>|</span> <span class=s1>'Read per user'</span> <span class=o>>></span> <span class=n>ReadPerUser</span><span class=p>()</span>
- <span class=o>|</span> <span class=s1>'Bag state pardo'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span class=n>BagStateDoFn</span><span class=p>()))</span></code></pre></div></div></div><h3 id=deferred-state-reads>11.2. Deferred state reads</h3><p>When a <code>DoFn</code> contains multiple state specifications, reading each one in order can be slow. Calling the <code>read()</code> function
+ <span class=o>|</span> <span class=s1>'Bag state pardo'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span class=n>BagStateDoFn</span><span class=p>()))</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon. [...]
on a state can cause the runner to perform a blocking read. Performing multiple blocking reads in sequence adds latency
to element processing. If you know that a state will always be read, you can annotate it as @AlwaysFetched, and then the
runner can prefetch all of the states necessary. For example:</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=n>PCollection</span><span class=o><</span><span class=n>KV</span><span class=o><</span><span class=n>String</span><span [...]
@@ -3185,7 +3185,7 @@ runner can prefetch all of the states necessary. For example:</p><div class="lan
<span class=n>state2</span><span class=o>.</span><span class=na>read</span><span class=o>();</span>
<span class=n>state3</span><span class=o>.</span><span class=na>read</span><span class=o>();</span>
<span class=o>}</span>
-<span class=o>}));</span></code></pre></div></div></div><div class="language-py snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-py data-lang=py><span class=n>This</span> <span class=ow>is</span> <span class=ow>not</span> <span class=n>supported</span> <span class=n>yet</span><span class=p>,</spa [...]
+<span class=o>}));</span></code></pre></div></div></div><div class="language-py snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-py data-lang=py><span class=n>This</span> <span class=ow>is</span> <span class=ow>not</span> <span class=n>supported</span> <span class=n>yet</span><span class=p>,</spa [...]
unnecessary fetching for those paths. In this case, the readLater method allows the runner to know that the state will
be read in the future, allowing multiple state reads to be batched together.</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=n>PCollection</span><span class=o><</span><span class=n>KV</span><span class=o><</span><span class=n>Str [...]
<span class=n>perUser</span><span class=o>.</span><span class=na>apply</span><span class=o>(</span><span class=n>ParDo</span><span class=o>.</span><span class=na>of</span><span class=o>(</span><span class=k>new</span> <span class=n>DoFn</span><span class=o><</span><span class=n>KV</span><span class=o><</span><span class=n>String</span><span class=o>,</span> <span class=n>ValueT</span><span class=o>>,</span> <span class=n>OutputT</span><span class=o>>()</span> <span class=o>{</span>
@@ -3249,7 +3249,7 @@ allows for event-time aggregations.</p><div class="language-java snippet"><div c
<span class=n>state</span><span class=o>.</span><span class=n>clear</span><span class=p>()</span>
<span class=n>_</span> <span class=o>=</span> <span class=p>(</span><span class=n>p</span> <span class=o>|</span> <span class=s1>'Read per user'</span> <span class=o>>></span> <span class=n>ReadPerUser</span><span class=p>()</span>
- <span class=o>|</span> <span class=s1>'EventTime timer pardo'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span class=n>EventTimerDoFn</span><span class=p>()))</span></code></pre></div></div></div><h4 id=processing-time-timers>11.3.2. Processing-time timers</h4><p>Processing-time timers fire when the real wall-clock time passes. This is often used to create larger batches of data
+ <span class=o>|</span> <span class=s1>'EventTime timer pardo'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span class=n>EventTimerDoFn</span><span class=p>()))</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/co [...]
before processing. It can also be used to schedule events that should occur at a specific time. Just like with
event-time timers, processing-time timers are per key - each key has a separate copy of the timer.</p><p>While processing-time timers can be set to an absolute timestamp, it is very common to set them to an offset relative
to the current time. In Java, the <code>Timer.offset</code> and <code>Timer.setRelative</code> methods can be used to accomplish this.</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=n>PCollection</span><span class=o><</span><span c [...]
@@ -3283,7 +3283,7 @@ to the current time. In Java, the <code>Timer.offset</code> and <code>Timer.setR
<span class=n>state</span><span class=o>.</span><span class=n>clear</span><span class=p>()</span>
<span class=n>_</span> <span class=o>=</span> <span class=p>(</span><span class=n>p</span> <span class=o>|</span> <span class=s1>'Read per user'</span> <span class=o>>></span> <span class=n>ReadPerUser</span><span class=p>()</span>
- <span class=o>|</span> <span class=s1>'ProcessingTime timer pardo'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span class=n>ProcessingTimerDoFn</span><span class=p>()))</span></code></pre></div></div></div><h4 id=dynamic-timer-tags>11.3.3. Dynamic timer tags</h4><p>Beam also supports dynamically setting a timer tag using <code>TimerMap</code> in the Java SDK. This allows for setting mu [...]
+ <span class=o>|</span> <span class=s1>'ProcessingTime timer pardo'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span class=n>ProcessingTimerDoFn</span><span class=p>()))</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src= [...]
in a <code>DoFn</code> and allowing for the timer tags to be dynamically chosen - e.g. based on data in the input elements. A
timer with a specific tag can only be set to a single timestamp, so setting the timer again has the effect of
overwriting the previous expiration time for the timer with that tag. Each <code>TimerMap</code> is identified with a timer family
@@ -3326,7 +3326,7 @@ tag is an empty string if not specified.</p><div class="language-java snippet"><
<span class=k>yield</span> <span class=p>(</span><span class=n>timer_tag</span><span class=p>,</span> <span class=s1>'fired'</span><span class=p>)</span>
<span class=n>_</span> <span class=o>=</span> <span class=p>(</span><span class=n>p</span> <span class=o>|</span> <span class=s1>'Read per user'</span> <span class=o>>></span> <span class=n>ReadPerUser</span><span class=p>()</span>
- <span class=o>|</span> <span class=s1>'ProcessingTime timer pardo'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span class=n>TimerDoFn</span><span class=p>()))</span></code></pre></div></div></div><h4 id=timer-output-timestamps>11.3.4. Timer output timestamps</h4><p>By default, event-time timers will hold the output watermark of the <code>ParDo</code> to the timestamp of the timer. This means
+ <span class=o>|</span> <span class=s1>'ProcessingTime timer pardo'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span class=n>TimerDoFn</span><span class=p>()))</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/co [...]
that if a timer is set to 12pm, any windowed aggregations or event-time timers later in the pipeline graph that finish
after 12pm will not expire. The timestamp of the timer is also the default output timestamp for the timer callback. This
means that any elements output from the onTimer method will have a timestamp equal to the timestamp of the timer firing.
@@ -3413,7 +3413,7 @@ past the timestamp of the minimum element. The following code demonstrates this.
<span class=c1>// Note that the timer has now fired.
</span><span class=c1></span> <span class=n>timerTimestamp</span><span class=o>.</span><span class=na>clear</span><span class=o>();</span>
<span class=o>}</span>
-<span class=o>}));</span></code></pre></div></div></div><div class="language-py snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-py data-lang=py><span class=n>Timer</span> <span class=n>output</span> <span class=n>timestamps</span> <span class=ow>is</span> <span class=ow>not</span> <span class=n> [...]
+<span class=o>}));</span></code></pre></div></div></div><div class="language-py snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-py data-lang=py><span class=n>Timer</span> <span class=n>output</span> <span class=n>timestamps</span> <span class=ow>is</span> <span class=ow>not</span> <span class=n> [...]
performance. There are two common strategies for garbage collecting state.</p><h5 id=using-windows-for-garbage-collection>11.4.1. <strong>Using windows for garbage collection</strong></h5><p>All state and timers for a key is scoped to the window it is in. This means that depending on the timestamp of the
input element the ParDo will see different values for the state depending on the window that element falls into. In
addition, once the input watermark passes the end of the window, the runner should garbage collect all state for that
@@ -3439,7 +3439,7 @@ garbage-collection strategy.</p><p>For example, given the following:</p><div cla
<span class=n>_</span> <span class=o>=</span> <span class=p>(</span><span class=n>p</span> <span class=o>|</span> <span class=s1>'Read per user'</span> <span class=o>>></span> <span class=n>ReadPerUser</span><span class=p>()</span>
<span class=o>|</span> <span class=s1>'Windowing'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>WindowInto</span><span class=p>(</span><span class=n>FixedWindows</span><span class=p>(</span><span class=mi>60</span> <span class=o>*</span> <span class=mi>60</span> <span class=o>*</span> <span class=mi>24</span><span class=p>))</span>
- <span class=o>|</span> <span class=s1>'DoFn'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span class=n>StateDoFn</span><span class=p>()))</span></code></pre></div></div></div><p>This <code>ParDo</code> stores state per day. Once the pipeline is done processing data for a given day, all the state for that
+ <span class=o>|</span> <span class=s1>'DoFn'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span class=n>StateDoFn</span><span class=p>()))</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div c [...]
day is garbage collected.</p><h5 id=using-timers-for-garbage-collection>11.4.1. <strong>Using timers For garbage collection</strong></h5><p>In some cases, it is difficult to find a windowing strategy that models the desired garbage-collection strategy. For
example, a common desire is to garbage collect state for a key once no activity has been seen on the key for some time.
This can be done by updating a timer that garbage collects state. For example</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=n>PCollection</span><span class=o><</span><span class=n>KV</span><span class=o><</span><span class=n>St [...]
@@ -3505,7 +3505,7 @@ This can be done by updating a timer that garbage collects state. For example</p
<span class=n>_</span> <span class=o>=</span> <span class=p>(</span><span class=n>p</span> <span class=o>|</span> <span class=s1>'Read per user'</span> <span class=o>>></span> <span class=n>ReadPerUser</span><span class=p>()</span>
- <span class=o>|</span> <span class=s1>'User DoFn'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span class=n>UserDoFn</span><span class=p>()))</span></code></pre></div></div></div><h3 id=state-timers-examples>11.5. State and timers examples</h3><p>Following are some example uses of state and timers</p><h4 id=joining-clicks-and-views>11.5.1. Joining clicks and views</h4><p>In this example [...]
+ <span class=o>|</span> <span class=s1>'User DoFn'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span class=n>UserDoFn</span><span class=p>()))</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><d [...]
a stream of views, representing suggested product links displayed to the user on the home page, and a stream of
clicks, representing actual user clicks on these links. The goal of the pipeline is to join click events with view
events, outputting a new joined event that contains information from both events. Each link has a unique identifier
@@ -3906,7 +3906,7 @@ resource utilization.</p><div class="language-java snippet"><div class="notebook
<span class=k>except</span> <span class=n>TimeoutError</span><span class=p>:</span>
<span class=c1># Set a longer delay in case we are being throttled.</span>
<span class=n>restriction_tracker</span><span class=o>.</span><span class=n>defer_remainder</span><span class=p>(</span><span class=n>timestamp</span><span class=o>.</span><span class=n>Duration</span><span class=p>(</span><span class=n>seconds</span><span class=o>=</span><span class=mi>60</span><span class=p>))</span>
- <span class=k>return</span></code></pre></div></div></div><h3 id=runner-initiated-split>12.4. Runner-initiated split</h3><p>A runner at any time may attempt to split a restriction while it is being processed. This allows the
+ <span class=k>return</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-go data-lang=go><span class=nx>This</span> <span class=nx>is</span> <span class=nx>not</span> <span class=nx>supported</span> <span class=nx>yet</span><span c [...]
runner to either pause processing of the restriction so that other work may be done (common for
unbounded restrictions to limit the amount of output and/or improve latency) or split the restriction
into two pieces, increasing the available parallelism within the system. Different runners (e.g.,
@@ -4080,7 +4080,7 @@ watermark estimator implementation. You can also provide your own watermark esti
<span class=n>watermark_estimator</span><span class=o>=</span><span class=n>beam</span><span class=o>.</span><span class=n>DoFn</span><span class=o>.</span><span class=n>WatermarkEstimatorParam</span><span class=p>(</span>
<span class=n>MyWatermarkEstimatorProvider</span><span class=p>())):</span>
<span class=c1># The current watermark can be inspected.</span>
- <span class=n>watermark_estimator</span><span class=o>.</span><span class=n>current_watermark</span><span class=p>()</span></code></pre></div></div></div><h3 id=truncating-during-drain>12.6. Truncating during drain</h3><p>Runners which support draining pipelines need the ability to drain SDFs; otherwise, the
+ <span class=n>watermark_estimator</span><span class=o>.</span><span class=n>current_watermark</span><span class=p>()</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-go data-lang=go><span class=nx>This</span> <span class=nx>is</span [...]
pipeline may never stop. By default, bounded restrictions process the remainder of the restriction while
unbounded restrictions finish processing at the next SDF-initiated checkpoint or runner-initiated split.
You are able to override this default behavior by defining the appropriate method on the restriction
@@ -4098,7 +4098,7 @@ provider.</p><div class="language-java snippet"><div class="notebook-skip code-s
<span class=k>if</span> <span class=s2>"optional"</span> <span class=ow>in</span> <span class=n>file_name</span><span class=p>:</span>
<span class=c1># Skip optional files</span>
<span class=k>return</span> <span class=bp>None</span>
- <span class=k>return</span> <span class=n>restriction</span></code></pre></div></div></div><h3 id=bundle-finalization>12.7. Bundle finalization</h3><p>Bundle finalization enables a <code>DoFn</code> to perform side effects by registering a callback.
+ <span class=k>return</span> <span class=n>restriction</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-go data-lang=go><span class=nx>This</span> <span class=nx>is</span> <span class=nx>not</span> <span class=nx>supported</span> <sp [...]
The callback is invoked once the runner has acknowledged that it has durably persisted the output.
For example, a message queue might need to acknowledge messages that it has ingested into the pipeline.
Bundle finalization is not limited to SDFs but is called out here since this is the primary
@@ -4117,7 +4117,7 @@ use case.</p><div class="language-java snippet"><div class="notebook-skip code-s
<span class=c1># Register callback function for this bundle that performs the side</span>
<span class=c1># effect.</span>
- <span class=n>bundle_finalizer</span><span class=o>.</span><span class=n>register</span><span class=p>(</span><span class=n>my_callback_func</span><span class=p>)</span></code></pre></div></div></div><h2 id=multi-language-pipelines>13. Multi-language pipelines</h2><p>Beam allows you to combine transforms written in any supported SDK language (currently, Java and Python) and use them in one multi-language pipeline. This capability makes it easy to provide new functionality simultaneou [...]
+ <span class=n>bundle_finalizer</span><span class=o>.</span><span class=n>register</span><span class=p>(</span><span class=n>my_callback_func</span><span class=p>)</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-go data-lang=go><spa [...]
abstract static class Builder<K, V>
implements ExternalTransformBuilder<External.Configuration, PBegin, PCollection<KV<K, V>>> {
abstract Builder<K, V> setConsumerConfig(Map<String, Object> config);
@@ -4245,7 +4245,7 @@ expansionAddr := "localhost:8097"
outT := beam.UnnamedOutput(typex.New(reflectx.String))
res := beam.CrossLanguage(s, urn, payload, expansionAddr, beam.UnnamedInput(inputPCol), outT)
</code></pre></div></div></li><li><p>After the job has been submitted to the Beam runner, shutdown the expansion service by
-terminating the expansion service process.</p></li></ol><h3 id=x-lang-transform-runner-support>13.3. Runner Support</h3><p>Currently, portable runners such as Flink, Spark, and the Direct runner can be used with multi-language pipelines.</p><p>Google Cloud Dataflow supports multi-language pipelines through the Dataflow Runner v2 backend architecture.</p><div class=feedback><p class=update>Last updated on 2021/10/07</p><h3>Have you found everything you were looking for?</h3><p class=descr [...]
+terminating the expansion service process.</p></li></ol><h3 id=x-lang-transform-runner-support>13.3. Runner Support</h3><p>Currently, portable runners such as Flink, Spark, and the Direct runner can be used with multi-language pipelines.</p><p>Google Cloud Dataflow supports multi-language pipelines through the Dataflow Runner v2 backend architecture.</p><div class=feedback><p class=update>Last updated on 2021/10/12</p><h3>Have you found everything you were looking for?</h3><p class=descr [...]
<a href=http://www.apache.org>The Apache Software Foundation</a>
| <a href=/privacy_policy>Privacy Policy</a>
| <a href=/feed.xml>RSS Feed</a><br><br>Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation. All other products or name brands are trademarks of their respective holders, including The Apache Software Foundation.</div></div></div></div></footer></body></html>
\ No newline at end of file
diff --git a/website/generated-content/sitemap.xml b/website/generated-content/sitemap.xml
index 217d5af..4e6db2c 100644
--- a/website/generated-content/sitemap.xml
+++ b/website/generated-content/sitemap.xml
@@ -1 +1 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/beam-2.33.0/</loc><lastmod>2021-10-11T18:22:03-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2021-10-11T18:22:03-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2021-10-11T18:22:03-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2021-10-11T18:22:03-07:00</lastmod></url><url><loc>/blog/b [...]
\ No newline at end of file
+<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/beam-2.33.0/</loc><lastmod>2021-10-11T18:22:03-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2021-10-11T18:22:03-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2021-10-11T18:22:03-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2021-10-11T18:22:03-07:00</lastmod></url><url><loc>/blog/b [...]
\ No newline at end of file