You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2020/07/16 00:03:18 UTC
[beam] branch asf-site updated: Publishing website 2020/07/16
00:03:08 at commit b753c1a
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 7ed0c7b Publishing website 2020/07/16 00:03:08 at commit b753c1a
7ed0c7b is described below
commit 7ed0c7b51df697aaf184d3cb9399535e73570323
Author: jenkins <us...@infra.apache.org>
AuthorDate: Thu Jul 16 00:03:08 2020 +0000
Publishing website 2020/07/16 00:03:08 at commit b753c1a
---
website/generated-content/documentation/index.xml | 443 +++++++++++++++++++--
.../python/aggregation/cogroupbykey/index.html | 33 +-
.../python/aggregation/combineglobally/index.html | 172 +++++++-
.../python/elementwise/filter/index.html | 12 +-
.../python/elementwise/flatmap/index.html | 18 +-
.../transforms/python/elementwise/keys/index.html | 2 +-
.../python/elementwise/kvswap/index.html | 2 +-
.../transforms/python/elementwise/map/index.html | 16 +-
.../transforms/python/elementwise/pardo/index.html | 6 +-
.../python/elementwise/partition/index.html | 6 +-
.../transforms/python/elementwise/regex/index.html | 18 +-
.../python/elementwise/tostring/index.html | 6 +-
.../python/elementwise/values/index.html | 2 +-
.../python/elementwise/withtimestamps/index.html | 6 +-
website/generated-content/sitemap.xml | 2 +-
15 files changed, 654 insertions(+), 90 deletions(-)
diff --git a/website/generated-content/documentation/index.xml b/website/generated-content/documentation/index.xml
index aef0c4a..bdb7d1c 100644
--- a/website/generated-content/documentation/index.xml
+++ b/website/generated-content/documentation/index.xml
@@ -6743,15 +6743,26 @@ See the License for the specific language governing permissions and
limitations under the License.
-->
<h1 id="cogroupbykey">CoGroupByKey</h1>
-<table align="left">
-<a target="_blank" class="button"
-href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.CoGroupByKey">
-<img src="https://beam.apache.org/images/logos/sdks/python.png" width="20px" height="20px"
-alt="Pydoc" />
-Pydoc
-</a>
+<script type="text/javascript">
+localStorage.setItem("language", "language-py")
+</script>
+<table align="left" style="margin-right:1em">
+<td>
+<a
+class="button"
+target="_blank"
+href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.CoGroupByKey"
+><img
+src="https://beam.apache.org/images/logos/sdks/python.png"
+width="32px"
+height="32px"
+alt="Pydoc"
+/>
+Pydoc</a
+>
+</td>
</table>
-<br><br>
+<p><br><br><br></p>
<p>Aggregates all input elements by their key and allows downstream processing
to consume all values associated with the key. While <code>GroupByKey</code> performs
this operation over a single input collection and thus a single type of input
@@ -6760,12 +6771,66 @@ the result for each key is a tuple of the values associated with that key in
each input collection.</p>
<p>See more information in the <a href="/documentation/programming-guide/#cogroupbykey">Beam Programming Guide</a>.</p>
<h2 id="examples">Examples</h2>
-<p>See <a href="https://issues.apache.org/jira/browse/BEAM-7390">BEAM-7390</a> for updates.</p>
+<p>In the following example, we create a pipeline with two <code>PCollection</code>s of produce, one with icons and one with durations, both with a common key of the produce name.
+Then, we apply <code>CoGroupByKey</code> to join both <code>PCollection</code>s using their keys.</p>
+<p><code>CoGroupByKey</code> expects a dictionary of named keyed <code>PCollection</code>s, and produces elements joined by their keys.
+The values of each output element are dictionaries where the names correspond to the input dictionary, with lists of all the values found for that key.</p>
+<div class=language-py>
+<div class="highlight"><pre class="chroma"><code class="language-py" data-lang="py"><span class="kn">import</span> <span class="nn">apache_beam</span> <span class="kn">as</span> <span class="nn">beam</span>
+<span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> <span class="k">as</span> <span class="n">pipeline</span><span class="p">:</span>
+<span class="n">icon_pairs</span> <span class="o">=</span> <span class="n">pipeline</span> <span class="o">|</span> <span class="s1">&#39;Create icons&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span>
+<span class="p">(</span><span class="s1">&#39;Apple&#39;</span><span class="p">,</span> <span class="s1">&#39;π&#39;</span><span class="p">),</span>
+<span class="p">(</span><span class="s1">&#39;Apple&#39;</span><span class="p">,</span> <span class="s1">&#39;π&#39;</span><span class="p">),</span>
+<span class="p">(</span><span class="s1">&#39;Eggplant&#39;</span><span class="p">,</span> <span class="s1">&#39;π&#39;</span><span class="p">),</span>
+<span class="p">(</span><span class="s1">&#39;Tomato&#39;</span><span class="p">,</span> <span class="s1">&#39;π
&#39;</span><span class="p">),</span>
+<span class="p">])</span>
+<span class="n">duration_pairs</span> <span class="o">=</span> <span class="n">pipeline</span> <span class="o">|</span> <span class="s1">&#39;Create durations&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span>
+<span class="p">(</span><span class="s1">&#39;Apple&#39;</span><span class="p">,</span> <span class="s1">&#39;perennial&#39;</span><span class="p">),</span>
+<span class="p">(</span><span class="s1">&#39;Carrot&#39;</span><span class="p">,</span> <span class="s1">&#39;biennial&#39;</span><span class="p">),</span>
+<span class="p">(</span><span class="s1">&#39;Tomato&#39;</span><span class="p">,</span> <span class="s1">&#39;perennial&#39;</span><span class="p">),</span>
+<span class="p">(</span><span class="s1">&#39;Tomato&#39;</span><span class="p">,</span> <span class="s1">&#39;annual&#39;</span><span class="p">),</span>
+<span class="p">])</span>
+<span class="n">plants</span> <span class="o">=</span> <span class="p">(({</span>
+<span class="s1">&#39;icons&#39;</span><span class="p">:</span> <span class="n">icon_pairs</span><span class="p">,</span> <span class="s1">&#39;durations&#39;</span><span class="p">:</span> <span class="n">duration_pairs</span>
+<span class="p">})</span>
+<span class="o">|</span> <span class="s1">&#39;Merge&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">CoGroupByKey</span><span class="p">()</span>
+<span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">print</span><span class="p">))</span></code></pre></div>
+</div>
+<p class="notebook-skip">Output:</p>
+<div class=notebook-skip>
+<pre><code>(&#39;Apple&#39;, {&#39;icons&#39;: [&#39;π&#39;, &#39;π&#39;], &#39;durations&#39;: [&#39;perennial&#39;]})
+(&#39;Carrot&#39;, {&#39;icons&#39;: [], &#39;durations&#39;: [&#39;biennial&#39;]})
+(&#39;Tomato&#39;, {&#39;icons&#39;: [&#39;π
&#39;], &#39;durations&#39;: [&#39;perennial&#39;, &#39;annual&#39;]})
+(&#39;Eggplant&#39;, {&#39;icons&#39;: [&#39;π&#39;], &#39;durations&#39;: []})</code></pre>
+</div>
+<table align="left" style="margin-right:1em" class=".language-py" >
+<td>
+<a class="button" target="_blank" href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/aggregation/cogroupbykey.py"><img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png" width="32px" height="32px" alt="View source code" /> View source code</a>
+</td>
+</table>
+<p><br><br><br></p>
<h2 id="related-transforms">Related transforms</h2>
<ul>
<li><a href="/documentation/transforms/python/aggregation/combineglobally">CombineGlobally</a> to combine elements.</li>
<li><a href="/documentation/transforms/python/aggregation/groupbykey">GroupByKey</a> takes one input collection.</li>
-</ul></description></item><item><title>Documentation: Combine</title><link>/documentation/transforms/java/aggregation/combine/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/transforms/java/aggregation/combine/</guid><description>
+</ul>
+<table align="left" style="margin-right:1em">
+<td>
+<a
+class="button"
+target="_blank"
+href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.CoGroupByKey"
+><img
+src="https://beam.apache.org/images/logos/sdks/python.png"
+width="32px"
+height="32px"
+alt="Pydoc"
+/>
+Pydoc</a
+>
+</td>
+</table>
+<p><br><br><br></p></description></item><item><title>Documentation: Combine</title><link>/documentation/transforms/java/aggregation/combine/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/transforms/java/aggregation/combine/</guid><description>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -6847,19 +6912,306 @@ See the License for the specific language governing permissions and
limitations under the License.
-->
<h1 id="combineglobally">CombineGlobally</h1>
-<table align="left">
-<a target="_blank" class="button"
-href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineGlobally">
-<img src="https://beam.apache.org/images/logos/sdks/python.png" width="20px" height="20px"
-alt="Pydoc" />
-Pydoc
-</a>
+<script type="text/javascript">
+localStorage.setItem("language", "language-py")
+</script>
+<table align="left" style="margin-right:1em">
+<td>
+<a
+class="button"
+target="_blank"
+href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineGlobally"
+><img
+src="https://beam.apache.org/images/logos/sdks/python.png"
+width="32px"
+height="32px"
+alt="Pydoc"
+/>
+Pydoc</a
+>
+</td>
</table>
-<br><br>
+<p><br><br><br></p>
<p>Combines all elements in a collection.</p>
<p>See more information in the <a href="/documentation/programming-guide/#combine">Beam Programming Guide</a>.</p>
<h2 id="examples">Examples</h2>
-<p>See <a href="https://issues.apache.org/jira/browse/BEAM-7390">BEAM-7390</a> for updates.</p>
+<p>In the following examples, we create a pipeline with a <code>PCollection</code> of produce.
+Then, we apply <code>CombineGlobally</code> in multiple ways to combine all the elements in the <code>PCollection</code>.</p>
+<p><code>CombineGlobally</code> accepts a function that takes an <code>iterable</code> of elements as an input, and combines them to return a single element.</p>
+<h3 id="example-1-combining-with-a-function">Example 1: Combining with a function</h3>
+<p>We define a function <code>get_common_items</code> which takes an <code>iterable</code> of sets as an input, and calculates the intersection (common items) of those sets.</p>
+<div class=language-py>
+<div class="highlight"><pre class="chroma"><code class="language-py" data-lang="py"><span class="kn">import</span> <span class="nn">apache_beam</span> <span class="kn">as</span> <span class="nn">beam</span>
+<span class="k">def</span> <span class="nf">get_common_items</span><span class="p">(</span><span class="n">sets</span><span class="p">):</span>
+<span class="c1"># set.intersection() takes multiple sets as separete arguments.</span>
+<span class="c1"># We unpack the `sets` list into multiple arguments with the * operator.</span>
+<span class="c1"># The combine transform might give us an empty list of `sets`,</span>
+<span class="c1"># so we use a list with an empty set as a default value.</span>
+<span class="k">return</span> <span class="nb">set</span><span class="o">.</span><span class="n">intersection</span><span class="p">(</span><span class="o">*</span><span class="p">(</span><span class="n">sets</span> <span class="ow">or</span> <span class="p">[</span><span class="nb">set</span><span class="p">()]))</span>
+<span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> <span class="k">as</span> <span class="n">pipeline</span><span class="p">:</span>
+<span class="n">common_items</span> <span class="o">=</span> <span class="p">(</span>
+<span class="n">pipeline</span>
+<span class="o">|</span> <span class="s1">&#39;Create produce&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span>
+<span class="p">{</span><span class="s1">&#39;π&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π&#39;</span><span class="p">,</span> <span class="s1">&#39;π
&#39;</span><span class="p">,</span> <span class="s1">&#39;πΆοΈ&#39;</span><span class="p">},</span>
+<span class="p">{</span><span class="s1">&#39;π&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π
&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">},</span>
+<span class="p">{</span><span class="s1">&#39;π&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π&#39;</span><span class="p">,</span> <span class="s1">&#39;π
&#39;</span><span class="p">,</span> <span class="s1">&#39;π&#39;</span><span class="p">},</span>
+<span class="p">{</span><span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π½&#39;</span><span class="p">,</span> <span class="s1">&#39;π
&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯₯&#39;</span><span class="p">},</span>
+<span class="p">])</span>
+<span class="o">|</span> <span class="s1">&#39;Get common items&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">CombineGlobally</span><span class="p">(</span><span class="n">get_common_items</span><span class="p">)</span>
+<span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">print</span><span class="p">))</span></code></pre></div>
+</div>
+<p class="notebook-skip">Output:</p>
+<div class=notebook-skip>
+<pre><code>{&#39;π
&#39;, &#39;π₯&#39;}</code></pre>
+</div>
+<table align="left" style="margin-right:1em" class=".language-py" >
+<td>
+<a class="button" target="_blank" href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineglobally.py"><img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png" width="32px" height="32px" alt="View source code" /> View source code</a>
+</td>
+</table>
+<p><br><br><br></p>
+<h3 id="example-2-combining-with-a-lambda-function">Example 2: Combining with a lambda function</h3>
+<p>We can also use lambda functions to simplify <strong>Example 1</strong>.</p>
+<div class=language-py>
+<div class="highlight"><pre class="chroma"><code class="language-py" data-lang="py"><span class="kn">import</span> <span class="nn">apache_beam</span> <span class="kn">as</span> <span class="nn">beam</span>
+<span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> <span class="k">as</span> <span class="n">pipeline</span><span class="p">:</span>
+<span class="n">common_items</span> <span class="o">=</span> <span class="p">(</span>
+<span class="n">pipeline</span>
+<span class="o">|</span> <span class="s1">&#39;Create produce&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span>
+<span class="p">{</span><span class="s1">&#39;π&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π&#39;</span><span class="p">,</span> <span class="s1">&#39;π
&#39;</span><span class="p">,</span> <span class="s1">&#39;πΆοΈ&#39;</span><span class="p">},</span>
+<span class="p">{</span><span class="s1">&#39;π&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π
&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">},</span>
+<span class="p">{</span><span class="s1">&#39;π&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π&#39;</span><span class="p">,</span> <span class="s1">&#39;π
&#39;</span><span class="p">,</span> <span class="s1">&#39;π&#39;</span><span class="p">},</span>
+<span class="p">{</span><span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π½&#39;</span><span class="p">,</span> <span class="s1">&#39;π
&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯₯&#39;</span><span class="p">},</span>
+<span class="p">])</span>
+<span class="o">|</span> <span class="s1">&#39;Get common items&#39;</span> <span class="o">&gt;&gt;</span>
+<span class="n">beam</span><span class="o">.</span><span class="n">CombineGlobally</span><span class="p">(</span><span class="k">lambda</span> <span class="n">sets</span><span class="p">:</span> <span class="nb">set</span><span class="o">.</span><span class="n">intersection</span><span class="p">(</span><span class="o">*</span><span class="p">(</span><span class="n">sets</span> <span class="ow">or</ [...]
+<span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">print</span><span class="p">))</span></code></pre></div>
+</div>
+<p class="notebook-skip">Output:</p>
+<div class=notebook-skip>
+<pre><code>{&#39;π
&#39;, &#39;π₯&#39;}</code></pre>
+</div>
+<table align="left" style="margin-right:1em" class=".language-py" >
+<td>
+<a class="button" target="_blank" href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineglobally.py"><img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png" width="32px" height="32px" alt="View source code" /> View source code</a>
+</td>
+</table>
+<p><br><br><br></p>
+<h3 id="example-3-combining-with-multiple-arguments">Example 3: Combining with multiple arguments</h3>
+<p>You can pass functions with multiple arguments to <code>CombineGlobally</code>.
+They are passed as additional positional arguments or keyword arguments to the function.</p>
+<p>In this example, the lambda function takes <code>sets</code> and <code>exclude</code> as arguments.</p>
+<div class=language-py>
+<div class="highlight"><pre class="chroma"><code class="language-py" data-lang="py"><span class="kn">import</span> <span class="nn">apache_beam</span> <span class="kn">as</span> <span class="nn">beam</span>
+<span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> <span class="k">as</span> <span class="n">pipeline</span><span class="p">:</span>
+<span class="n">common_items_with_exceptions</span> <span class="o">=</span> <span class="p">(</span>
+<span class="n">pipeline</span>
+<span class="o">|</span> <span class="s1">&#39;Create produce&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span>
+<span class="p">{</span><span class="s1">&#39;π&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π&#39;</span><span class="p">,</span> <span class="s1">&#39;π
&#39;</span><span class="p">,</span> <span class="s1">&#39;πΆοΈ&#39;</span><span class="p">},</span>
+<span class="p">{</span><span class="s1">&#39;π&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π
&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">},</span>
+<span class="p">{</span><span class="s1">&#39;π&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π&#39;</span><span class="p">,</span> <span class="s1">&#39;π
&#39;</span><span class="p">,</span> <span class="s1">&#39;π&#39;</span><span class="p">},</span>
+<span class="p">{</span><span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π½&#39;</span><span class="p">,</span> <span class="s1">&#39;π
&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯₯&#39;</span><span class="p">},</span>
+<span class="p">])</span>
+<span class="o">|</span> <span class="s1">&#39;Get common items with exceptions&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">CombineGlobally</span><span class="p">(</span>
+<span class="k">lambda</span> <span class="n">sets</span><span class="p">,</span> <span class="n">exclude</span><span class="p">:</span> \
+<span class="nb">set</span><span class="o">.</span><span class="n">intersection</span><span class="p">(</span><span class="o">*</span><span class="p">(</span><span class="n">sets</span> <span class="ow">or</span> <span class="p">[</span><span class="nb">set</span><span class="p">()]))</span> <span class="o">-</span> <span class="n">exclude</span><span class="p">,</span>
+<span class="n">exclude</span><span class="o">=</span><span class="p">{</span><span class="s1">&#39;π₯&#39;</span><span class="p">})</span>
+<span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">print</span><span class="p">)</span>
+<span class="p">)</span></code></pre></div>
+</div>
+<p class="notebook-skip">Output:</p>
+<div class=notebook-skip>
+<pre><code>{&#39;π
&#39;}</code></pre>
+</div>
+<table align="left" style="margin-right:1em" class=".language-py" >
+<td>
+<a class="button" target="_blank" href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineglobally.py"><img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png" width="32px" height="32px" alt="View source code" /> View source code</a>
+</td>
+</table>
+<p><br><br><br></p>
+<h3 id="example-4-combining-with-side-inputs-as-singletons">Example 4: Combining with side inputs as singletons</h3>
+<p>If the <code>PCollection</code> has a single value, such as the average from another computation,
+passing the <code>PCollection</code> as a <em>singleton</em> accesses that value.</p>
+<p>In this example, we pass a <code>PCollection</code> the value <code>'π₯'</code> as a singleton.
+We then use that value to exclude specific items.</p>
+<div class=language-py>
+<div class="highlight"><pre class="chroma"><code class="language-py" data-lang="py"><span class="kn">import</span> <span class="nn">apache_beam</span> <span class="kn">as</span> <span class="nn">beam</span>
+<span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> <span class="k">as</span> <span class="n">pipeline</span><span class="p">:</span>
+<span class="n">single_exclude</span> <span class="o">=</span> <span class="n">pipeline</span> <span class="o">|</span> <span class="s1">&#39;Create single_exclude&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span><span class="s1">&#39;π₯&#39;</span><span class="p">])</span>
+<span class="n">common_items_with_exceptions</span> <span class="o">=</span> <span class="p">(</span>
+<span class="n">pipeline</span>
+<span class="o">|</span> <span class="s1">&#39;Create produce&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span>
+<span class="p">{</span><span class="s1">&#39;π&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π&#39;</span><span class="p">,</span> <span class="s1">&#39;π
&#39;</span><span class="p">,</span> <span class="s1">&#39;πΆοΈ&#39;</span><span class="p">},</span>
+<span class="p">{</span><span class="s1">&#39;π&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π
&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">},</span>
+<span class="p">{</span><span class="s1">&#39;π&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π&#39;</span><span class="p">,</span> <span class="s1">&#39;π
&#39;</span><span class="p">,</span> <span class="s1">&#39;π&#39;</span><span class="p">},</span>
+<span class="p">{</span><span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π½&#39;</span><span class="p">,</span> <span class="s1">&#39;π
&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯₯&#39;</span><span class="p">},</span>
+<span class="p">])</span>
+<span class="o">|</span> <span class="s1">&#39;Get common items with exceptions&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">CombineGlobally</span><span class="p">(</span>
+<span class="k">lambda</span> <span class="n">sets</span><span class="p">,</span> <span class="n">single_exclude</span><span class="p">:</span> \
+<span class="nb">set</span><span class="o">.</span><span class="n">intersection</span><span class="p">(</span><span class="o">*</span><span class="p">(</span><span class="n">sets</span> <span class="ow">or</span> <span class="p">[</span><span class="nb">set</span><span class="p">()]))</span> <span class="o">-</span> <span class="p">{</span><span class="n">single_exclude</span><span class="p">},</span>
+<span class="n">single_exclude</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">pvalue</span><span class="o">.</span><span class="n">AsSingleton</span><span class="p">(</span><span class="n">single_exclude</span><span class="p">))</span>
+<span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">print</span><span class="p">)</span>
+<span class="p">)</span></code></pre></div>
+</div>
+<p class="notebook-skip">Output:</p>
+<div class=notebook-skip>
+<pre><code>{&#39;π
&#39;}</code></pre>
+</div>
+<table align="left" style="margin-right:1em" class=".language-py" >
+<td>
+<a class="button" target="_blank" href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineglobally.py"><img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png" width="32px" height="32px" alt="View source code" /> View source code</a>
+</td>
+</table>
+<p><br><br><br></p>
+<h3 id="example-5-combining-with-side-inputs-as-iterators">Example 5: Combining with side inputs as iterators</h3>
+<p>If the <code>PCollection</code> has multiple values, pass the <code>PCollection</code> as an <em>iterator</em>.
+This accesses elements lazily as they are needed,
+so it is possible to iterate over large <code>PCollection</code>s that won&rsquo;t fit into memory.</p>
+<div class=language-py>
+<div class="highlight"><pre class="chroma"><code class="language-py" data-lang="py"><span class="kn">import</span> <span class="nn">apache_beam</span> <span class="kn">as</span> <span class="nn">beam</span>
+<span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> <span class="k">as</span> <span class="n">pipeline</span><span class="p">:</span>
+<span class="n">exclude</span> <span class="o">=</span> <span class="n">pipeline</span> <span class="o">|</span> <span class="s1">&#39;Create exclude&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span><span class="s1">&#39;π₯&#39;</span><span class="p">])</span>
+<span class="n">common_items_with_exceptions</span> <span class="o">=</span> <span class="p">(</span>
+<span class="n">pipeline</span>
+<span class="o">|</span> <span class="s1">&#39;Create produce&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span>
+<span class="p">{</span><span class="s1">&#39;π&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π&#39;</span><span class="p">,</span> <span class="s1">&#39;π
&#39;</span><span class="p">,</span> <span class="s1">&#39;πΆοΈ&#39;</span><span class="p">},</span>
+<span class="p">{</span><span class="s1">&#39;π&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π
&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">},</span>
+<span class="p">{</span><span class="s1">&#39;π&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π&#39;</span><span class="p">,</span> <span class="s1">&#39;π
&#39;</span><span class="p">,</span> <span class="s1">&#39;π&#39;</span><span class="p">},</span>
+<span class="p">{</span><span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π½&#39;</span><span class="p">,</span> <span class="s1">&#39;π
&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯₯&#39;</span><span class="p">},</span>
+<span class="p">])</span>
+<span class="o">|</span> <span class="s1">&#39;Get common items with exceptions&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">CombineGlobally</span><span class="p">(</span>
+<span class="k">lambda</span> <span class="n">sets</span><span class="p">,</span> <span class="n">exclude</span><span class="p">:</span> \
+<span class="nb">set</span><span class="o">.</span><span class="n">intersection</span><span class="p">(</span><span class="o">*</span><span class="p">(</span><span class="n">sets</span> <span class="ow">or</span> <span class="p">[</span><span class="nb">set</span><span class="p">()]))</span> <span class="o">-</span> <span class="nb">set</span><span class="p">(</span><span class="n">exclude</span>< [...]
+<span class="n">exclude</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">pvalue</span><span class="o">.</span><span class="n">AsIter</span><span class="p">(</span><span class="n">exclude</span><span class="p">))</span>
+<span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">print</span><span class="p">)</span>
+<span class="p">)</span></code></pre></div>
+</div>
+<p class="notebook-skip">Output:</p>
+<div class=notebook-skip>
+<pre><code>{&#39;π
&#39;}</code></pre>
+</div>
+<table align="left" style="margin-right:1em" class=".language-py" >
+<td>
+<a class="button" target="_blank" href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineglobally.py"><img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png" width="32px" height="32px" alt="View source code" /> View source code</a>
+</td>
+</table>
+<p><br><br><br></p>
+<blockquote>
+<p><strong>Note</strong>: You can pass the <code>PCollection</code> as a <em>list</em> with <code>beam.pvalue.AsList(pcollection)</code>,
+but this requires that all the elements fit into memory.</p>
+</blockquote>
+<h3 id="example-6-combining-with-side-inputs-as-dictionaries">Example 6: Combining with side inputs as dictionaries</h3>
+<p>If a <code>PCollection</code> is small enough to fit into memory, then that <code>PCollection</code> can be passed as a <em>dictionary</em>.
+Each element must be a <code>(key, value)</code> pair.
+Note that all the elements of the <code>PCollection</code> must fit into memory for this.
+If the <code>PCollection</code> won&rsquo;t fit into memory, use <code>beam.pvalue.AsIter(pcollection)</code> instead.</p>
+<div class=language-py>
+<div class="highlight"><pre class="chroma"><code class="language-py" data-lang="py"><span class="kn">import</span> <span class="nn">apache_beam</span> <span class="kn">as</span> <span class="nn">beam</span>
+<span class="k">def</span> <span class="nf">get_custom_common_items</span><span class="p">(</span><span class="n">sets</span><span class="p">,</span> <span class="n">options</span><span class="p">):</span>
+<span class="n">sets</span> <span class="o">=</span> <span class="n">sets</span> <span class="ow">or</span> <span class="p">[</span><span class="nb">set</span><span class="p">()]</span>
+<span class="n">common_items</span> <span class="o">=</span> <span class="nb">set</span><span class="o">.</span><span class="n">intersection</span><span class="p">(</span><span class="o">*</span><span class="n">sets</span><span class="p">)</span>
+<span class="n">common_items</span> <span class="o">|=</span> <span class="n">options</span><span class="p">[</span><span class="s1">&#39;include&#39;</span><span class="p">]</span> <span class="c1"># union</span>
+<span class="n">common_items</span> <span class="o">&amp;=</span> <span class="n">options</span><span class="p">[</span><span class="s1">&#39;exclude&#39;</span><span class="p">]</span> <span class="c1"># intersection</span>
+<span class="k">return</span> <span class="n">common_items</span>
+<span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> <span class="k">as</span> <span class="n">pipeline</span><span class="p">:</span>
+<span class="n">options</span> <span class="o">=</span> <span class="n">pipeline</span> <span class="o">|</span> <span class="s1">&#39;Create options&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span>
+<span class="p">(</span><span class="s1">&#39;exclude&#39;</span><span class="p">,</span> <span class="p">{</span><span class="s1">&#39;π₯&#39;</span><span class="p">}),</span>
+<span class="p">(</span><span class="s1">&#39;include&#39;</span><span class="p">,</span> <span class="p">{</span><span class="s1">&#39;π&#39;</span><span class="p">,</span> <span class="s1">&#39;π½&#39;</span><span class="p">}),</span>
+<span class="p">])</span>
+<span class="n">custom_common_items</span> <span class="o">=</span> <span class="p">(</span>
+<span class="n">pipeline</span>
+<span class="o">|</span> <span class="s1">&#39;Create produce&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span>
+<span class="p">{</span><span class="s1">&#39;π&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π&#39;</span><span class="p">,</span> <span class="s1">&#39;π
&#39;</span><span class="p">,</span> <span class="s1">&#39;πΆοΈ&#39;</span><span class="p">},</span>
+<span class="p">{</span><span class="s1">&#39;π&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π
&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">},</span>
+<span class="p">{</span><span class="s1">&#39;π&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π&#39;</span><span class="p">,</span> <span class="s1">&#39;π
&#39;</span><span class="p">,</span> <span class="s1">&#39;π&#39;</span><span class="p">},</span>
+<span class="p">{</span><span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π½&#39;</span><span class="p">,</span> <span class="s1">&#39;π
&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯₯&#39;</span><span class="p">},</span>
+<span class="p">])</span>
+<span class="o">|</span> <span class="s1">&#39;Get common items&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">CombineGlobally</span><span class="p">(</span>
+<span class="n">get_custom_common_items</span><span class="p">,</span> <span class="n">options</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">pvalue</span><span class="o">.</span><span class="n">AsDict</span><span class="p">(</span><span class="n">options</span><span class="p">))</span>
+<span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">print</span><span class="p">))</span></code></pre></div>
+</div>
+<p class="notebook-skip">Output:</p>
+<div class=notebook-skip>
+<pre><code>{&#39;π
&#39;, &#39;π&#39;, &#39;π½&#39;}</code></pre>
+</div>
+<table align="left" style="margin-right:1em" class=".language-py" >
+<td>
+<a class="button" target="_blank" href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineglobally.py"><img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png" width="32px" height="32px" alt="View source code" /> View source code</a>
+</td>
+</table>
+<p><br><br><br></p>
+<h3 id="example-7-combining-with-a-combinefn">Example 7: Combining with a <code>CombineFn</code></h3>
+<p>The more general way to combine elements, and the most flexible, is with a class that inherits from <code>CombineFn</code>.</p>
+<ul>
+<li>
+<p><a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn.create_accumulator"><code>CombineFn.create_accumulator()</code></a>:
+This creates an empty accumulator.
+For example, an empty accumulator for a sum would be <code>0</code>, while an empty accumulator for a product (multiplication) would be <code>1</code>.</p>
+</li>
+<li>
+<p><a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn.add_input"><code>CombineFn.add_input()</code></a>:
+Called <em>once per element</em>.
+Takes an accumulator and an input element, combines them and returns the updated accumulator.</p>
+</li>
+<li>
+<p><a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn.merge_accumulators"><code>CombineFn.merge_accumulators()</code></a>:
+Multiple accumulators could be processed in parallel, so this function helps merging them into a single accumulator.</p>
+</li>
+<li>
+<p><a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn.extract_output"><code>CombineFn.extract_output()</code></a>:
+It allows to do additional calculations before extracting a result.</p>
+</li>
+</ul>
+<div class=language-py>
+<div class="highlight"><pre class="chroma"><code class="language-py" data-lang="py"><span class="kn">import</span> <span class="nn">apache_beam</span> <span class="kn">as</span> <span class="nn">beam</span>
+<span class="k">class</span> <span class="nc">PercentagesFn</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">CombineFn</span><span class="p">):</span>
+<span class="k">def</span> <span class="nf">create_accumulator</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+<span class="k">return</span> <span class="p">{}</span>
+<span class="k">def</span> <span class="nf">add_input</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">accumulator</span><span class="p">,</span> <span class="nb">input</span><span class="p">):</span>
+<span class="c1"># accumulator == {}</span>
+<span class="c1"># input == &#39;π₯&#39;</span>
+<span class="k">if</span> <span class="nb">input</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">accumulator</span><span class="p">:</span>
+<span class="n">accumulator</span><span class="p">[</span><span class="nb">input</span><span class="p">]</span> <span class="o">=</span> <span class="mi">0</span> <span class="c1"># {&#39;π₯&#39;: 0}</span>
+<span class="n">accumulator</span><span class="p">[</span><span class="nb">input</span><span class="p">]</span> <span class="o">+=</span> <span class="mi">1</span> <span class="c1"># {&#39;π₯&#39;: 1}</span>
+<span class="k">return</span> <span class="n">accumulator</span>
+<span class="k">def</span> <span class="nf">merge_accumulators</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">accumulators</span><span class="p">):</span>
+<span class="c1"># accumulators == [</span>
+<span class="c1"># {&#39;π₯&#39;: 1, &#39;π
&#39;: 2},</span>
+<span class="c1"># {&#39;π₯&#39;: 1, &#39;π
&#39;: 1, &#39;π&#39;: 1},</span>
+<span class="c1"># {&#39;π₯&#39;: 1, &#39;π
&#39;: 3},</span>
+<span class="c1"># ]</span>
+<span class="n">merged</span> <span class="o">=</span> <span class="p">{}</span>
+<span class="k">for</span> <span class="n">accum</span> <span class="ow">in</span> <span class="n">accumulators</span><span class="p">:</span>
+<span class="k">for</span> <span class="n">item</span><span class="p">,</span> <span class="n">count</span> <span class="ow">in</span> <span class="n">accum</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
+<span class="k">if</span> <span class="n">item</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">merged</span><span class="p">:</span>
+<span class="n">merged</span><span class="p">[</span><span class="n">item</span><span class="p">]</span> <span class="o">=</span> <span class="mi">0</span>
+<span class="n">merged</span><span class="p">[</span><span class="n">item</span><span class="p">]</span> <span class="o">+=</span> <span class="n">count</span>
+<span class="c1"># merged == {&#39;π₯&#39;: 3, &#39;π
&#39;: 6, &#39;π&#39;: 1}</span>
+<span class="k">return</span> <span class="n">merged</span>
+<span class="k">def</span> <span class="nf">extract_output</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">accumulator</span><span class="p">):</span>
+<span class="c1"># accumulator == {&#39;π₯&#39;: 3, &#39;π
&#39;: 6, &#39;π&#39;: 1}</span>
+<span class="n">total</span> <span class="o">=</span> <span class="nb">sum</span><span class="p">(</span><span class="n">accumulator</span><span class="o">.</span><span class="n">values</span><span class="p">())</span> <span class="c1"># 10</span>
+<span class="n">percentages</span> <span class="o">=</span> <span class="p">{</span><span class="n">item</span><span class="p">:</span> <span class="n">count</span> <span class="o">/</span> <span class="n">total</span> <span class="k">for</span> <span class="n">item</span><span class="p">,</span> <span class="n">count</span> <span class="ow">in</span> <span class="n">accumulator</span><span class="o">. [...]
+<span class="c1"># percentages == {&#39;π₯&#39;: 0.3, &#39;π
&#39;: 0.6, &#39;π&#39;: 0.1}</span>
+<span class="k">return</span> <span class="n">percentages</span>
+<span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> <span class="k">as</span> <span class="n">pipeline</span><span class="p">:</span>
+<span class="n">percentages</span> <span class="o">=</span> <span class="p">(</span>
+<span class="n">pipeline</span>
+<span class="o">|</span> <span class="s1">&#39;Create produce&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">(</span>
+<span class="p">[</span><span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π
&#39;</span><span class="p">,</span> <span class="s1">&#39;π
&#39;</span><span class="p">,</span> <span class="s1">&#39;π₯&#39;</span><span class="p">,</span> <span class="s1">&#39;π&#39;</span><span class="p">,</span> <span class="s1">&#39;π
&#39;</span><span clas [...]
+<span class="o">|</span> <span class="s1">&#39;Get percentages&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">CombineGlobally</span><span class="p">(</span><span class="n">PercentagesFn</span><span class="p">())</span>
+<span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">print</span><span class="p">))</span></code></pre></div>
+</div>
+<p class="notebook-skip">Output:</p>
+<div class=notebook-skip>
+<pre><code>{&#39;π₯&#39;: 0.3, &#39;π
&#39;: 0.6, &#39;π&#39;: 0.1}</code></pre>
+</div>
+<table align="left" style="margin-right:1em" class=".language-py" >
+<td>
+<a class="button" target="_blank" href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineglobally.py"><img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png" width="32px" height="32px" alt="View source code" /> View source code</a>
+</td>
+</table>
+<p><br><br><br></p>
<h2 id="related-transforms">Related transforms</h2>
<p>You can use the following combiner transforms:</p>
<ul>
@@ -6867,7 +7219,24 @@ Pydoc
<li><a href="/documentation/transforms/python/aggregation/count">Count</a></li>
<li><a href="/documentation/transforms/python/aggregation/top">Top</a></li>
<li><a href="/documentation/transforms/python/aggregation/sample">Sample</a></li>
-</ul></description></item><item><title>Documentation: CombineWithContext</title><link>/documentation/transforms/java/aggregation/combinewithcontext/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/transforms/java/aggregation/combinewithcontext/</guid><description>
+</ul>
+<table align="left" style="margin-right:1em">
+<td>
+<a
+class="button"
+target="_blank"
+href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineGlobally"
+><img
+src="https://beam.apache.org/images/logos/sdks/python.png"
+width="32px"
+height="32px"
+alt="Pydoc"
+/>
+Pydoc</a
+>
+</td>
+</table>
+<p><br><br><br></p></description></item><item><title>Documentation: CombineWithContext</title><link>/documentation/transforms/java/aggregation/combinewithcontext/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/transforms/java/aggregation/combinewithcontext/</guid><description>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -7948,7 +8317,7 @@ Then, we apply <code>Filter</code> in multiple ways to filter out produce
<span class="o">|</span> <span class="s1">&#39;Filter perennials&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Filter</span><span class="p">(</span><span class="n">is_perennial</span><span class="p">)</span>
<span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">print</span><span class="p">))</span></code></pre></div>
</div>
-<p class="notebook-skip">Output <code>PCollection</code> after <code>Filter</code>:</p>
+<p class="notebook-skip">Output:</p>
<div class=notebook-skip>
<pre><code>{&#39;icon&#39;: &#39;π&#39;, &#39;name&#39;: &#39;Strawberry&#39;, &#39;duration&#39;: &#39;perennial&#39;}
{&#39;icon&#39;: &#39;π&#39;, &#39;name&#39;: &#39;Eggplant&#39;, &#39;duration&#39;: &#39;perennial&#39;}
@@ -7993,7 +8362,7 @@ Then, we apply <code>Filter</code> in multiple ways to filter out produce
<span class="n">beam</span><span class="o">.</span><span class="n">Filter</span><span class="p">(</span><span class="k">lambda</span> <span class="n">plant</span><span class="p">:</span> <span class="n">plant</span><span class="p">[</span><span class="s1">&#39;duration&#39;</span><span class="p">]</span> <span class="o">==</span> <span class="s1">&#39;perennial&#39;</span><span class="p">)</span>
<span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">print</span><span class="p">))</span></code></pre></div>
</div>
-<p class="notebook-skip">Output <code>PCollection</code> after <code>Filter</code>:</p>
+<p class="notebook-skip">Output:</p>
<div class=notebook-skip>
<pre><code>{&#39;icon&#39;: &#39;π&#39;, &#39;name&#39;: &#39;Strawberry&#39;, &#39;duration&#39;: &#39;perennial&#39;}
{&#39;icon&#39;: &#39;π&#39;, &#39;name&#39;: &#39;Eggplant&#39;, &#39;duration&#39;: &#39;perennial&#39;}
@@ -8041,7 +8410,7 @@ They are passed as additional positional arguments or keyword arguments to the f
<span class="o">|</span> <span class="s1">&#39;Filter perennials&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Filter</span><span class="p">(</span><span class="n">has_duration</span><span class="p">,</span> <span class="s1">&#39;perennial&#39;</span><span class="p">)</span>
<span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">print</span><span class="p">))</span></code></pre></div>
</div>
-<p class="notebook-skip">Output <code>PCollection</code> after <code>Filter</code>:</p>
+<p class="notebook-skip">Output:</p>
<div class=notebook-skip>
<pre><code>{&#39;icon&#39;: &#39;π&#39;, &#39;name&#39;: &#39;Strawberry&#39;, &#39;duration&#39;: &#39;perennial&#39;}
{&#39;icon&#39;: &#39;π&#39;, &#39;name&#39;: &#39;Eggplant&#39;, &#39;duration&#39;: &#39;perennial&#39;}
@@ -8093,7 +8462,7 @@ We then use that value to filter out perennials.</p>
<span class="p">)</span>
<span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">print</span><span class="p">))</span></code></pre></div>
</div>
-<p class="notebook-skip">Output <code>PCollection</code> after <code>Filter</code>:</p>
+<p class="notebook-skip">Output:</p>
<div class=notebook-skip>
<pre><code>{&#39;icon&#39;: &#39;π&#39;, &#39;name&#39;: &#39;Strawberry&#39;, &#39;duration&#39;: &#39;perennial&#39;}
{&#39;icon&#39;: &#39;π&#39;, &#39;name&#39;: &#39;Eggplant&#39;, &#39;duration&#39;: &#39;perennial&#39;}
@@ -8148,7 +8517,7 @@ so it is possible to iterate over large <code>PCollection</code>s that won
<span class="p">)</span>
<span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">print</span><span class="p">))</span></code></pre></div>
</div>
-<p class="notebook-skip">Output <code>PCollection</code> after <code>Filter</code>:</p>
+<p class="notebook-skip">Output:</p>
<div class=notebook-skip>
<pre><code>{&#39;icon&#39;: &#39;π&#39;, &#39;name&#39;: &#39;Strawberry&#39;, &#39;duration&#39;: &#39;perennial&#39;}
{&#39;icon&#39;: &#39;π₯&#39;, &#39;name&#39;: &#39;Carrot&#39;, &#39;duration&#39;: &#39;biennial&#39;}
@@ -8209,7 +8578,7 @@ If the <code>PCollection</code> won&rsquo;t fit into memory, use <c
<span class="p">)</span>
<span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">print</span><span class="p">))</span></code></pre></div>
</div>
-<p class="notebook-skip">Output <code>PCollection</code> after <code>Filter</code>:</p>
+<p class="notebook-skip">Output:</p>
<div class=notebook-skip>
<pre><code>{&#39;icon&#39;: &#39;π&#39;, &#39;name&#39;: &#39;Strawberry&#39;, &#39;duration&#39;: &#39;perennial&#39;}
{&#39;icon&#39;: &#39;π&#39;, &#39;name&#39;: &#39;Eggplant&#39;, &#39;duration&#39;: &#39;perennial&#39;}
@@ -8304,7 +8673,7 @@ This pipeline splits the input element using whitespaces, creating a list of zer
<span class="o">|</span> <span class="s1">&#39;Split words&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">FlatMap</span><span class="p">(</span><span class="nb">str</span><span class="o">.</span><span class="n">split</span><span class="p">)</span>
<span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">print</span><span class="p">))</span></code></pre></div>
</div>
-<p class="notebook-skip">Output <code>PCollection</code> after <code>FlatMap</code>:</p>
+<p class="notebook-skip">Output:</p>
<div class=notebook-skip>
<pre><code>πStrawberry
π₯Carrot
@@ -8339,7 +8708,7 @@ This pipeline splits the input element using whitespaces, creating a list of zer
<span class="o">|</span> <span class="s1">&#39;Split words&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">FlatMap</span><span class="p">(</span><span class="n">split_words</span><span class="p">)</span>
<span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">print</span><span class="p">))</span></code></pre></div>
</div>
-<p class="notebook-skip">Output <code>PCollection</code> after <code>FlatMap</code>:</p>
+<p class="notebook-skip">Output:</p>
<div class=notebook-skip>
<pre><code>πStrawberry
π₯Carrot
@@ -8374,7 +8743,7 @@ We use a lambda function that returns the same input element it received.</p>
<span class="o">|</span> <span class="s1">&#39;Flatten lists&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">FlatMap</span><span class="p">(</span><span class="k">lambda</span> <span class="n">elements</span><span class="p">:</span> <span class="n">elements</span><span class="p">)</span>
<span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">print</span><span class="p">))</span></code></pre></div>
</div>
-<p class="notebook-skip">Output <code>PCollection</code> after <code>FlatMap</code>:</p>
+<p class="notebook-skip">Output:</p>
<div class=notebook-skip>
<pre><code>πStrawberry
π₯Carrot
@@ -8412,7 +8781,7 @@ Each yielded result in the generator is an element in the resulting <code>PCo
<span class="o">|</span> <span class="s1">&#39;Flatten lists&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">FlatMap</span><span class="p">(</span><span class="n">generate_elements</span><span class="p">)</span>
<span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">print</span><span class="p">))</span></code></pre></div>
</div>
-<p class="notebook-skip">Output <code>PCollection</code> after <code>FlatMap</code>:</p>
+<p class="notebook-skip">Output:</p>
<div class=notebook-skip>
<pre><code>πStrawberry
π₯Carrot
@@ -8453,7 +8822,7 @@ you can use <code>FlatMapTuple</code> to unpack them into different functi
<span class="o">|</span> <span class="s1">&#39;Format&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">FlatMapTuple</span><span class="p">(</span><span class="n">format_plant</span><span class="p">)</span>
<span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">print</span><span class="p">))</span></code></pre></div>
</div>
-<p class="notebook-skip">Output <code>PCollection</code> after <code>FlatMapTuple</code>:</p>
+<p class="notebook-skip">Output:</p>
<div class=notebook-skip>
<pre><code>πStrawberry
π₯Carrot
@@ -8490,7 +8859,7 @@ They are passed as additional positional arguments or keyword arguments to the f
<span class="o">|</span> <span class="s1">&#39;Split words&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">FlatMap</span><span class="p">(</span><span class="n">split_words</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="s1">&#39;,&#39;</span><span class="p">)</span>
<span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">print</span><span class="p">))</span></code></pre></div>
</div>
-<p class="notebook-skip">Output <code>PCollection</code> after <code>FlatMap</code>:</p>
+<p class="notebook-skip">Output:</p>
<div class=notebook-skip>
<pre><code>πStrawberry
π₯Carrot
@@ -8531,7 +8900,7 @@ We then use that value as the delimiter for the <code>str.split</code> met
<span class="p">)</span>
<span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">print</span><span class="p">))</span></code></pre></div>
</div>
-<p class="notebook-skip">Output <code>PCollection</code> after <code>FlatMap</code>:</p>
+<p class="notebook-skip">Output:</p>
<div class=notebook-skip>
<pre><code>πStrawberry
π₯Carrot
@@ -8591,7 +8960,7 @@ so it is possible to iterate over large <code>PCollection</code>s that won
<span class="p">)</span>
<span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">print</span><span class="p">))</span></code></pre></div>
</div>
-<p class="notebook-skip">Output <code>PCollection</code> after <code>FlatMap</code>:</p>
+<p class="notebook-skip">Output:</p>
<div class=notebook-skip>
<pre><code>{&#39;icon&#39;: &#39;π&#39;, &#39;name&#39;: &#39;Strawberry&#39;, &#39;duration&#39;: &#39;perennial&#39;}
{&#39;icon&#39;: &#39;π₯&#39;, &#39;name&#39;: &#39;Carrot&#39;, &#39;duration&#39;: &#39;biennial&#39;}
@@ -8655,7 +9024,7 @@ If the <code>PCollection</code> won&rsquo;t fit into memory, use <c
<span class="p">)</span>
<span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">print</span><span class="p">))</span></code></pre></div>
</div>
-<p class="notebook-skip">Output <code>PCollection</code> after <code>FlatMap</code>:</p>
+<p class="notebook-skip">Output:</p>
<div class=notebook-skip>
<pre><code>{&#39;icon&#39;: &#39;π&#39;, &#39;name&#39;: &#39;Strawberry&#39;, &#39;duration&#39;: &#39;perennial&#39;}
{&#39;icon&#39;: &#39;π₯&#39;, &#39;name&#39;: &#39;Carrot&#39;, &#39;duration&#39;: &#39;biennial&#39;}
@@ -10224,7 +10593,7 @@ Then, we apply <code>Keys</code> to extract the keys and discard the value
<span class="o">|</span> <span class="s1">&#39;Keys&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Keys</span><span class="p">()</span>
<span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">print</span><span class="p">))</span></code></pre></div>
</div>
-<p class="notebook-skip">Output <code>PCollection</code> after <code>Keys</code>:</p>
+<p class="notebook-skip">Output:</p>
<div class=notebook-skip>
<pre><code>π
π₯
@@ -10351,7 +10720,7 @@ Then, we apply <code>KvSwap</code> to swap the keys and values.</p>
<span class="o">|</span> <span class="s1">&#39;Key-Value swap&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">KvSwap</span><span class="p">()</span>
<span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">print</span><span class="p">))</span></code></pre></div>
</div>
-<p class="notebook-skip">Output <code>PCollection</code> after <code>KvSwap</code>:</p>
+<p class="notebook-skip">Output:</p>
<div class=notebook-skip>
<pre><code>(&#39;Strawberry&#39;, &#39;π&#39;)
(&#39;Carrot&#39;, &#39;π₯&#39;)
diff --git a/website/generated-content/documentation/transforms/python/aggregation/cogroupbykey/index.html b/website/generated-content/documentation/transforms/python/aggregation/cogroupbykey/index.html
index 15f9fe9..088bbee 100644
--- a/website/generated-content/documentation/transforms/python/aggregation/cogroupbykey/index.html
+++ b/website/generated-content/documentation/transforms/python/aggregation/cogroupbykey/index.html
@@ -1,13 +1,40 @@
<!doctype html><html lang=en class=no-js><head><meta charset=utf-8><meta http-equiv=x-ua-compatible content="IE=edge"><meta name=viewport content="width=device-width,initial-scale=1"><title>CoGroupByKey</title><meta name=description content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific [...]
<span class=sr-only>Toggle navigation</span>
<span class=icon-bar></span><span class=icon-bar></span><span class=icon-bar></span></button>
-<a href=/ class=navbar-brand><img alt=Brand style=height:25px src=/images/beam_logo_navbar.png></a></div><div class="navbar-mask closed"></div><div id=navbar class="navbar-container closed"><ul class="nav navbar-nav"><li><a href=/get-started/beam-overview/>Get Started</a></li><li><a href=/documentation/>Documentation</a></li><li><a href=/documentation/sdks/java/>Languages</a></li><li><a href=/documentation/runners/capability-matrix/>RUNNERS</a></li><li><a href=/roadmap/>Roadmap</a></li>< [...]
-Pydoc</a></table><br><br><p>Aggregates all input elements by their key and allows downstream processing
+<a href=/ class=navbar-brand><img alt=Brand style=height:25px src=/images/beam_logo_navbar.png></a></div><div class="navbar-mask closed"></div><div id=navbar class="navbar-container closed"><ul class="nav navbar-nav"><li><a href=/get-started/beam-overview/>Get Started</a></li><li><a href=/documentation/>Documentation</a></li><li><a href=/documentation/sdks/java/>Languages</a></li><li><a href=/documentation/runners/capability-matrix/>RUNNERS</a></li><li><a href=/roadmap/>Roadmap</a></li>< [...]
+Pydoc</a></td></table><p><br><br><br></p><p>Aggregates all input elements by their key and allows downstream processing
to consume all values associated with the key. While <code>GroupByKey</code> performs
this operation over a single input collection and thus a single type of input
values, <code>CoGroupByKey</code> operates over multiple input collections. As a result,
the result for each key is a tuple of the values associated with that key in
-each input collection.</p><p>See more information in the <a href=/documentation/programming-guide/#cogroupbykey>Beam Programming Guide</a>.</p><h2 id=examples>Examples</h2><p>See <a href=https://issues.apache.org/jira/browse/BEAM-7390>BEAM-7390</a> for updates.</p><h2 id=related-transforms>Related transforms</h2><ul><li><a href=/documentation/transforms/python/aggregation/combineglobally>CombineGlobally</a> to combine elements.</li><li><a href=/documentation/transforms/python/aggregation [...]
+each input collection.</p><p>See more information in the <a href=/documentation/programming-guide/#cogroupbykey>Beam Programming Guide</a>.</p><h2 id=examples>Examples</h2><p>In the following example, we create a pipeline with two <code>PCollection</code>s of produce, one with icons and one with durations, both with a common key of the produce name.
+Then, we apply <code>CoGroupByKey</code> to join both <code>PCollection</code>s using their keys.</p><p><code>CoGroupByKey</code> expects a dictionary of named keyed <code>PCollection</code>s, and produces elements joined by their keys.
+The values of each output element are dictionaries where the names correspond to the input dictionary, with lists of all the values found for that key.</p><div class=language-py><div class=highlight><pre class=chroma><code class=language-py data-lang=py><span class=kn>import</span> <span class=nn>apache_beam</span> <span class=kn>as</span> <span class=nn>beam</span>
+
+<span class=k>with</span> <span class=n>beam</span><span class=o>.</span><span class=n>Pipeline</span><span class=p>()</span> <span class=k>as</span> <span class=n>pipeline</span><span class=p>:</span>
+ <span class=n>icon_pairs</span> <span class=o>=</span> <span class=n>pipeline</span> <span class=o>|</span> <span class=s1>'Create icons'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Create</span><span class=p>([</span>
+ <span class=p>(</span><span class=s1>'Apple'</span><span class=p>,</span> <span class=s1>'π'</span><span class=p>),</span>
+ <span class=p>(</span><span class=s1>'Apple'</span><span class=p>,</span> <span class=s1>'π'</span><span class=p>),</span>
+ <span class=p>(</span><span class=s1>'Eggplant'</span><span class=p>,</span> <span class=s1>'π'</span><span class=p>),</span>
+ <span class=p>(</span><span class=s1>'Tomato'</span><span class=p>,</span> <span class=s1>'π
'</span><span class=p>),</span>
+ <span class=p>])</span>
+
+ <span class=n>duration_pairs</span> <span class=o>=</span> <span class=n>pipeline</span> <span class=o>|</span> <span class=s1>'Create durations'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Create</span><span class=p>([</span>
+ <span class=p>(</span><span class=s1>'Apple'</span><span class=p>,</span> <span class=s1>'perennial'</span><span class=p>),</span>
+ <span class=p>(</span><span class=s1>'Carrot'</span><span class=p>,</span> <span class=s1>'biennial'</span><span class=p>),</span>
+ <span class=p>(</span><span class=s1>'Tomato'</span><span class=p>,</span> <span class=s1>'perennial'</span><span class=p>),</span>
+ <span class=p>(</span><span class=s1>'Tomato'</span><span class=p>,</span> <span class=s1>'annual'</span><span class=p>),</span>
+ <span class=p>])</span>
+
+ <span class=n>plants</span> <span class=o>=</span> <span class=p>(({</span>
+ <span class=s1>'icons'</span><span class=p>:</span> <span class=n>icon_pairs</span><span class=p>,</span> <span class=s1>'durations'</span><span class=p>:</span> <span class=n>duration_pairs</span>
+ <span class=p>})</span>
+ <span class=o>|</span> <span class=s1>'Merge'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>CoGroupByKey</span><span class=p>()</span>
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>('Apple', {'icons': ['π', 'π'], 'durations': ['perennial']})
+('Carrot', {'icons': [], 'durations': ['biennial']})
+('Tomato', {'icons': ['π
'], 'durations': ['perennial', 'annual']})
+('Eggplant', {'icons': ['π'], 'durations': []})</code></pre></div><table align=left style=margin-right:1em class=.language-py><td><a class=button target=_blank href=https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/aggregation/cogroupbykey.py><img src=https://www.tensorflow.org/images/GitHub-Mark-32px.png width=32px height=32px alt="View source code"> View source code</a></td></table><p><br><br><br></p><h2 id=r [...]
+Pydoc</a></td></table><p><br><br><br></p></div></div><footer class=footer><div class=footer__contained><div class=footer__cols><div class=footer__cols__col><div class=footer__cols__col__logo><img src=/images/beam_logo_circle.svg class=footer__logo alt="Beam logo"></div><div class=footer__cols__col__logo><img src=/images/apache_logo_circle.svg class=footer__logo alt="Apache logo"></div></div><div class="footer__cols__col footer__cols__col--md"><div class=footer__cols__col__title>Start</di [...]
<a href=http://www.apache.org>The Apache Software Foundation</a>
| <a href=/privacy_policy>Privacy Policy</a>
| <a href=/feed.xml>RSS Feed</a><br><br>Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation. All other products or name brands are trademarks of their respective holders, including The Apache Software Foundation.</div></footer></body></html>
\ No newline at end of file
diff --git a/website/generated-content/documentation/transforms/python/aggregation/combineglobally/index.html b/website/generated-content/documentation/transforms/python/aggregation/combineglobally/index.html
index 059be0d..f7d8280 100644
--- a/website/generated-content/documentation/transforms/python/aggregation/combineglobally/index.html
+++ b/website/generated-content/documentation/transforms/python/aggregation/combineglobally/index.html
@@ -1,8 +1,176 @@
<!doctype html><html lang=en class=no-js><head><meta charset=utf-8><meta http-equiv=x-ua-compatible content="IE=edge"><meta name=viewport content="width=device-width,initial-scale=1"><title>CombineGlobally</title><meta name=description content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specif [...]
<span class=sr-only>Toggle navigation</span>
<span class=icon-bar></span><span class=icon-bar></span><span class=icon-bar></span></button>
-<a href=/ class=navbar-brand><img alt=Brand style=height:25px src=/images/beam_logo_navbar.png></a></div><div class="navbar-mask closed"></div><div id=navbar class="navbar-container closed"><ul class="nav navbar-nav"><li><a href=/get-started/beam-overview/>Get Started</a></li><li><a href=/documentation/>Documentation</a></li><li><a href=/documentation/sdks/java/>Languages</a></li><li><a href=/documentation/runners/capability-matrix/>RUNNERS</a></li><li><a href=/roadmap/>Roadmap</a></li>< [...]
-Pydoc</a></table><br><br><p>Combines all elements in a collection.</p><p>See more information in the <a href=/documentation/programming-guide/#combine>Beam Programming Guide</a>.</p><h2 id=examples>Examples</h2><p>See <a href=https://issues.apache.org/jira/browse/BEAM-7390>BEAM-7390</a> for updates.</p><h2 id=related-transforms>Related transforms</h2><p>You can use the following combiner transforms:</p><ul><li><a href=/documentation/transforms/python/aggregation/mean>Mean</a></li><li><a [...]
+<a href=/ class=navbar-brand><img alt=Brand style=height:25px src=/images/beam_logo_navbar.png></a></div><div class="navbar-mask closed"></div><div id=navbar class="navbar-container closed"><ul class="nav navbar-nav"><li><a href=/get-started/beam-overview/>Get Started</a></li><li><a href=/documentation/>Documentation</a></li><li><a href=/documentation/sdks/java/>Languages</a></li><li><a href=/documentation/runners/capability-matrix/>RUNNERS</a></li><li><a href=/roadmap/>Roadmap</a></li>< [...]
+Pydoc</a></td></table><p><br><br><br></p><p>Combines all elements in a collection.</p><p>See more information in the <a href=/documentation/programming-guide/#combine>Beam Programming Guide</a>.</p><h2 id=examples>Examples</h2><p>In the following examples, we create a pipeline with a <code>PCollection</code> of produce.
+Then, we apply <code>CombineGlobally</code> in multiple ways to combine all the elements in the <code>PCollection</code>.</p><p><code>CombineGlobally</code> accepts a function that takes an <code>iterable</code> of elements as an input, and combines them to return a single element.</p><h3 id=example-1-combining-with-a-function>Example 1: Combining with a function</h3><p>We define a function <code>get_common_items</code> which takes an <code>iterable</code> of sets as an input, and calcul [...]
+
+<span class=k>def</span> <span class=nf>get_common_items</span><span class=p>(</span><span class=n>sets</span><span class=p>):</span>
+ <span class=c1># set.intersection() takes multiple sets as separete arguments.</span>
+ <span class=c1># We unpack the `sets` list into multiple arguments with the * operator.</span>
+ <span class=c1># The combine transform might give us an empty list of `sets`,</span>
+ <span class=c1># so we use a list with an empty set as a default value.</span>
+ <span class=k>return</span> <span class=nb>set</span><span class=o>.</span><span class=n>intersection</span><span class=p>(</span><span class=o>*</span><span class=p>(</span><span class=n>sets</span> <span class=ow>or</span> <span class=p>[</span><span class=nb>set</span><span class=p>()]))</span>
+
+<span class=k>with</span> <span class=n>beam</span><span class=o>.</span><span class=n>Pipeline</span><span class=p>()</span> <span class=k>as</span> <span class=n>pipeline</span><span class=p>:</span>
+ <span class=n>common_items</span> <span class=o>=</span> <span class=p>(</span>
+ <span class=n>pipeline</span>
+ <span class=o>|</span> <span class=s1>'Create produce'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Create</span><span class=p>([</span>
+ <span class=p>{</span><span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π
'</span><span class=p>,</span> <span class=s1>'πΆοΈ'</span><span class=p>},</span>
+ <span class=p>{</span><span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π
'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>},</span>
+ <span class=p>{</span><span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π
'</span><span class=p>,</span> <span class=s1>'π'</span><span class=p>},</span>
+ <span class=p>{</span><span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π½'</span><span class=p>,</span> <span class=s1>'π
'</span><span class=p>,</span> <span class=s1>'π₯₯'</span><span class=p>},</span>
+ <span class=p>])</span>
+ <span class=o>|</span> <span class=s1>'Get common items'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>CombineGlobally</span><span class=p>(</span><span class=n>get_common_items</span><span class=p>)</span>
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>{'π
', 'π₯'}</code></pre></div><table align=left style=margin-right:1em class=.language-py><td><a class=button target=_blank href=https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippe [...]
+
+<span class=k>with</span> <span class=n>beam</span><span class=o>.</span><span class=n>Pipeline</span><span class=p>()</span> <span class=k>as</span> <span class=n>pipeline</span><span class=p>:</span>
+ <span class=n>common_items</span> <span class=o>=</span> <span class=p>(</span>
+ <span class=n>pipeline</span>
+ <span class=o>|</span> <span class=s1>'Create produce'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Create</span><span class=p>([</span>
+ <span class=p>{</span><span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π
'</span><span class=p>,</span> <span class=s1>'πΆοΈ'</span><span class=p>},</span>
+ <span class=p>{</span><span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π
'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>},</span>
+ <span class=p>{</span><span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π
'</span><span class=p>,</span> <span class=s1>'π'</span><span class=p>},</span>
+ <span class=p>{</span><span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π½'</span><span class=p>,</span> <span class=s1>'π
'</span><span class=p>,</span> <span class=s1>'π₯₯'</span><span class=p>},</span>
+ <span class=p>])</span>
+ <span class=o>|</span> <span class=s1>'Get common items'</span> <span class=o>>></span>
+ <span class=n>beam</span><span class=o>.</span><span class=n>CombineGlobally</span><span class=p>(</span><span class=k>lambda</span> <span class=n>sets</span><span class=p>:</span> <span class=nb>set</span><span class=o>.</span><span class=n>intersection</span><span class=p>(</span><span class=o>*</span><span class=p>(</span><span class=n>sets</span> <span class=ow>or</span> <span class=p>[</span><span class=nb>set</span><span class=p>()])))</span>
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>{'π
', 'π₯'}</code></pre></div><table align=left style=margin-right:1em class=.language-py><td><a class=button target=_blank href=https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippe [...]
+They are passed as additional positional arguments or keyword arguments to the function.</p><p>In this example, the lambda function takes <code>sets</code> and <code>exclude</code> as arguments.</p><div class=language-py><div class=highlight><pre class=chroma><code class=language-py data-lang=py><span class=kn>import</span> <span class=nn>apache_beam</span> <span class=kn>as</span> <span class=nn>beam</span>
+
+<span class=k>with</span> <span class=n>beam</span><span class=o>.</span><span class=n>Pipeline</span><span class=p>()</span> <span class=k>as</span> <span class=n>pipeline</span><span class=p>:</span>
+ <span class=n>common_items_with_exceptions</span> <span class=o>=</span> <span class=p>(</span>
+ <span class=n>pipeline</span>
+ <span class=o>|</span> <span class=s1>'Create produce'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Create</span><span class=p>([</span>
+ <span class=p>{</span><span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π
'</span><span class=p>,</span> <span class=s1>'πΆοΈ'</span><span class=p>},</span>
+ <span class=p>{</span><span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π
'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>},</span>
+ <span class=p>{</span><span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π
'</span><span class=p>,</span> <span class=s1>'π'</span><span class=p>},</span>
+ <span class=p>{</span><span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π½'</span><span class=p>,</span> <span class=s1>'π
'</span><span class=p>,</span> <span class=s1>'π₯₯'</span><span class=p>},</span>
+ <span class=p>])</span>
+ <span class=o>|</span> <span class=s1>'Get common items with exceptions'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>CombineGlobally</span><span class=p>(</span>
+ <span class=k>lambda</span> <span class=n>sets</span><span class=p>,</span> <span class=n>exclude</span><span class=p>:</span> \
+ <span class=nb>set</span><span class=o>.</span><span class=n>intersection</span><span class=p>(</span><span class=o>*</span><span class=p>(</span><span class=n>sets</span> <span class=ow>or</span> <span class=p>[</span><span class=nb>set</span><span class=p>()]))</span> <span class=o>-</span> <span class=n>exclude</span><span class=p>,</span>
+ <span class=n>exclude</span><span class=o>=</span><span class=p>{</span><span class=s1>'π₯'</span><span class=p>})</span>
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>)</span>
+ <span class=p>)</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>{'π
'}</code></pre></div><table align=left style=margin-right:1em class=.language-py><td><a class=button target=_blank href=https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineglobally.py><img src=https://www.tensorflow.org/images/GitHub-Mark-32px.png width=32px height=32px alt="View source code"> View [...]
+passing the <code>PCollection</code> as a <em>singleton</em> accesses that value.</p><p>In this example, we pass a <code>PCollection</code> the value <code>'π₯'</code> as a singleton.
+We then use that value to exclude specific items.</p><div class=language-py><div class=highlight><pre class=chroma><code class=language-py data-lang=py><span class=kn>import</span> <span class=nn>apache_beam</span> <span class=kn>as</span> <span class=nn>beam</span>
+
+<span class=k>with</span> <span class=n>beam</span><span class=o>.</span><span class=n>Pipeline</span><span class=p>()</span> <span class=k>as</span> <span class=n>pipeline</span><span class=p>:</span>
+ <span class=n>single_exclude</span> <span class=o>=</span> <span class=n>pipeline</span> <span class=o>|</span> <span class=s1>'Create single_exclude'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Create</span><span class=p>([</span><span class=s1>'π₯'</span><span class=p>])</span>
+
+ <span class=n>common_items_with_exceptions</span> <span class=o>=</span> <span class=p>(</span>
+ <span class=n>pipeline</span>
+ <span class=o>|</span> <span class=s1>'Create produce'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Create</span><span class=p>([</span>
+ <span class=p>{</span><span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π
'</span><span class=p>,</span> <span class=s1>'πΆοΈ'</span><span class=p>},</span>
+ <span class=p>{</span><span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π
'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>},</span>
+ <span class=p>{</span><span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π
'</span><span class=p>,</span> <span class=s1>'π'</span><span class=p>},</span>
+ <span class=p>{</span><span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π½'</span><span class=p>,</span> <span class=s1>'π
'</span><span class=p>,</span> <span class=s1>'π₯₯'</span><span class=p>},</span>
+ <span class=p>])</span>
+ <span class=o>|</span> <span class=s1>'Get common items with exceptions'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>CombineGlobally</span><span class=p>(</span>
+ <span class=k>lambda</span> <span class=n>sets</span><span class=p>,</span> <span class=n>single_exclude</span><span class=p>:</span> \
+ <span class=nb>set</span><span class=o>.</span><span class=n>intersection</span><span class=p>(</span><span class=o>*</span><span class=p>(</span><span class=n>sets</span> <span class=ow>or</span> <span class=p>[</span><span class=nb>set</span><span class=p>()]))</span> <span class=o>-</span> <span class=p>{</span><span class=n>single_exclude</span><span class=p>},</span>
+ <span class=n>single_exclude</span><span class=o>=</span><span class=n>beam</span><span class=o>.</span><span class=n>pvalue</span><span class=o>.</span><span class=n>AsSingleton</span><span class=p>(</span><span class=n>single_exclude</span><span class=p>))</span>
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>)</span>
+ <span class=p>)</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>{'π
'}</code></pre></div><table align=left style=margin-right:1em class=.language-py><td><a class=button target=_blank href=https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineglobally.py><img src=https://www.tensorflow.org/images/GitHub-Mark-32px.png width=32px height=32px alt="View source code"> View [...]
+This accesses elements lazily as they are needed,
+so it is possible to iterate over large <code>PCollection</code>s that won’t fit into memory.</p><div class=language-py><div class=highlight><pre class=chroma><code class=language-py data-lang=py><span class=kn>import</span> <span class=nn>apache_beam</span> <span class=kn>as</span> <span class=nn>beam</span>
+
+<span class=k>with</span> <span class=n>beam</span><span class=o>.</span><span class=n>Pipeline</span><span class=p>()</span> <span class=k>as</span> <span class=n>pipeline</span><span class=p>:</span>
+ <span class=n>exclude</span> <span class=o>=</span> <span class=n>pipeline</span> <span class=o>|</span> <span class=s1>'Create exclude'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Create</span><span class=p>([</span><span class=s1>'π₯'</span><span class=p>])</span>
+
+ <span class=n>common_items_with_exceptions</span> <span class=o>=</span> <span class=p>(</span>
+ <span class=n>pipeline</span>
+ <span class=o>|</span> <span class=s1>'Create produce'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Create</span><span class=p>([</span>
+ <span class=p>{</span><span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π
'</span><span class=p>,</span> <span class=s1>'πΆοΈ'</span><span class=p>},</span>
+ <span class=p>{</span><span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π
'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>},</span>
+ <span class=p>{</span><span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π
'</span><span class=p>,</span> <span class=s1>'π'</span><span class=p>},</span>
+ <span class=p>{</span><span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π½'</span><span class=p>,</span> <span class=s1>'π
'</span><span class=p>,</span> <span class=s1>'π₯₯'</span><span class=p>},</span>
+ <span class=p>])</span>
+ <span class=o>|</span> <span class=s1>'Get common items with exceptions'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>CombineGlobally</span><span class=p>(</span>
+ <span class=k>lambda</span> <span class=n>sets</span><span class=p>,</span> <span class=n>exclude</span><span class=p>:</span> \
+ <span class=nb>set</span><span class=o>.</span><span class=n>intersection</span><span class=p>(</span><span class=o>*</span><span class=p>(</span><span class=n>sets</span> <span class=ow>or</span> <span class=p>[</span><span class=nb>set</span><span class=p>()]))</span> <span class=o>-</span> <span class=nb>set</span><span class=p>(</span><span class=n>exclude</span><span class=p>),</span>
+ <span class=n>exclude</span><span class=o>=</span><span class=n>beam</span><span class=o>.</span><span class=n>pvalue</span><span class=o>.</span><span class=n>AsIter</span><span class=p>(</span><span class=n>exclude</span><span class=p>))</span>
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>)</span>
+ <span class=p>)</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>{'π
'}</code></pre></div><table align=left style=margin-right:1em class=.language-py><td><a class=button target=_blank href=https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineglobally.py><img src=https://www.tensorflow.org/images/GitHub-Mark-32px.png width=32px height=32px alt="View source code"> View [...]
+but this requires that all the elements fit into memory.</p></blockquote><h3 id=example-6-combining-with-side-inputs-as-dictionaries>Example 6: Combining with side inputs as dictionaries</h3><p>If a <code>PCollection</code> is small enough to fit into memory, then that <code>PCollection</code> can be passed as a <em>dictionary</em>.
+Each element must be a <code>(key, value)</code> pair.
+Note that all the elements of the <code>PCollection</code> must fit into memory for this.
+If the <code>PCollection</code> won’t fit into memory, use <code>beam.pvalue.AsIter(pcollection)</code> instead.</p><div class=language-py><div class=highlight><pre class=chroma><code class=language-py data-lang=py><span class=kn>import</span> <span class=nn>apache_beam</span> <span class=kn>as</span> <span class=nn>beam</span>
+
+<span class=k>def</span> <span class=nf>get_custom_common_items</span><span class=p>(</span><span class=n>sets</span><span class=p>,</span> <span class=n>options</span><span class=p>):</span>
+ <span class=n>sets</span> <span class=o>=</span> <span class=n>sets</span> <span class=ow>or</span> <span class=p>[</span><span class=nb>set</span><span class=p>()]</span>
+ <span class=n>common_items</span> <span class=o>=</span> <span class=nb>set</span><span class=o>.</span><span class=n>intersection</span><span class=p>(</span><span class=o>*</span><span class=n>sets</span><span class=p>)</span>
+ <span class=n>common_items</span> <span class=o>|=</span> <span class=n>options</span><span class=p>[</span><span class=s1>'include'</span><span class=p>]</span> <span class=c1># union</span>
+ <span class=n>common_items</span> <span class=o>&=</span> <span class=n>options</span><span class=p>[</span><span class=s1>'exclude'</span><span class=p>]</span> <span class=c1># intersection</span>
+ <span class=k>return</span> <span class=n>common_items</span>
+
+<span class=k>with</span> <span class=n>beam</span><span class=o>.</span><span class=n>Pipeline</span><span class=p>()</span> <span class=k>as</span> <span class=n>pipeline</span><span class=p>:</span>
+ <span class=n>options</span> <span class=o>=</span> <span class=n>pipeline</span> <span class=o>|</span> <span class=s1>'Create options'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Create</span><span class=p>([</span>
+ <span class=p>(</span><span class=s1>'exclude'</span><span class=p>,</span> <span class=p>{</span><span class=s1>'π₯'</span><span class=p>}),</span>
+ <span class=p>(</span><span class=s1>'include'</span><span class=p>,</span> <span class=p>{</span><span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π½'</span><span class=p>}),</span>
+ <span class=p>])</span>
+
+ <span class=n>custom_common_items</span> <span class=o>=</span> <span class=p>(</span>
+ <span class=n>pipeline</span>
+ <span class=o>|</span> <span class=s1>'Create produce'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Create</span><span class=p>([</span>
+ <span class=p>{</span><span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π
'</span><span class=p>,</span> <span class=s1>'πΆοΈ'</span><span class=p>},</span>
+ <span class=p>{</span><span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π
'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>},</span>
+ <span class=p>{</span><span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π
'</span><span class=p>,</span> <span class=s1>'π'</span><span class=p>},</span>
+ <span class=p>{</span><span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π½'</span><span class=p>,</span> <span class=s1>'π
'</span><span class=p>,</span> <span class=s1>'π₯₯'</span><span class=p>},</span>
+ <span class=p>])</span>
+ <span class=o>|</span> <span class=s1>'Get common items'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>CombineGlobally</span><span class=p>(</span>
+ <span class=n>get_custom_common_items</span><span class=p>,</span> <span class=n>options</span><span class=o>=</span><span class=n>beam</span><span class=o>.</span><span class=n>pvalue</span><span class=o>.</span><span class=n>AsDict</span><span class=p>(</span><span class=n>options</span><span class=p>))</span>
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>{'π
', 'π', 'π½'}</code></pre></div><table align=left style=margin-right:1em class=.language-py><td><a class=button target=_blank href=https://github.com/apache/beam/blob/master/sdks/python/apache_beam/ex [...]
+This creates an empty accumulator.
+For example, an empty accumulator for a sum would be <code>0</code>, while an empty accumulator for a product (multiplication) would be <code>1</code>.</p></li><li><p><a href=https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn.add_input><code>CombineFn.add_input()</code></a>:
+Called <em>once per element</em>.
+Takes an accumulator and an input element, combines them and returns the updated accumulator.</p></li><li><p><a href=https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn.merge_accumulators><code>CombineFn.merge_accumulators()</code></a>:
+Multiple accumulators could be processed in parallel, so this function helps merging them into a single accumulator.</p></li><li><p><a href=https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn.extract_output><code>CombineFn.extract_output()</code></a>:
+It allows to do additional calculations before extracting a result.</p></li></ul><div class=language-py><div class=highlight><pre class=chroma><code class=language-py data-lang=py><span class=kn>import</span> <span class=nn>apache_beam</span> <span class=kn>as</span> <span class=nn>beam</span>
+
+<span class=k>class</span> <span class=nc>PercentagesFn</span><span class=p>(</span><span class=n>beam</span><span class=o>.</span><span class=n>CombineFn</span><span class=p>):</span>
+ <span class=k>def</span> <span class=nf>create_accumulator</span><span class=p>(</span><span class=bp>self</span><span class=p>):</span>
+ <span class=k>return</span> <span class=p>{}</span>
+
+ <span class=k>def</span> <span class=nf>add_input</span><span class=p>(</span><span class=bp>self</span><span class=p>,</span> <span class=n>accumulator</span><span class=p>,</span> <span class=nb>input</span><span class=p>):</span>
+ <span class=c1># accumulator == {}</span>
+ <span class=c1># input == 'π₯'</span>
+ <span class=k>if</span> <span class=nb>input</span> <span class=ow>not</span> <span class=ow>in</span> <span class=n>accumulator</span><span class=p>:</span>
+ <span class=n>accumulator</span><span class=p>[</span><span class=nb>input</span><span class=p>]</span> <span class=o>=</span> <span class=mi>0</span> <span class=c1># {'π₯': 0}</span>
+ <span class=n>accumulator</span><span class=p>[</span><span class=nb>input</span><span class=p>]</span> <span class=o>+=</span> <span class=mi>1</span> <span class=c1># {'π₯': 1}</span>
+ <span class=k>return</span> <span class=n>accumulator</span>
+
+ <span class=k>def</span> <span class=nf>merge_accumulators</span><span class=p>(</span><span class=bp>self</span><span class=p>,</span> <span class=n>accumulators</span><span class=p>):</span>
+ <span class=c1># accumulators == [</span>
+ <span class=c1># {'π₯': 1, 'π
': 2},</span>
+ <span class=c1># {'π₯': 1, 'π
': 1, 'π': 1},</span>
+ <span class=c1># {'π₯': 1, 'π
': 3},</span>
+ <span class=c1># ]</span>
+ <span class=n>merged</span> <span class=o>=</span> <span class=p>{}</span>
+ <span class=k>for</span> <span class=n>accum</span> <span class=ow>in</span> <span class=n>accumulators</span><span class=p>:</span>
+ <span class=k>for</span> <span class=n>item</span><span class=p>,</span> <span class=n>count</span> <span class=ow>in</span> <span class=n>accum</span><span class=o>.</span><span class=n>items</span><span class=p>():</span>
+ <span class=k>if</span> <span class=n>item</span> <span class=ow>not</span> <span class=ow>in</span> <span class=n>merged</span><span class=p>:</span>
+ <span class=n>merged</span><span class=p>[</span><span class=n>item</span><span class=p>]</span> <span class=o>=</span> <span class=mi>0</span>
+ <span class=n>merged</span><span class=p>[</span><span class=n>item</span><span class=p>]</span> <span class=o>+=</span> <span class=n>count</span>
+ <span class=c1># merged == {'π₯': 3, 'π
': 6, 'π': 1}</span>
+ <span class=k>return</span> <span class=n>merged</span>
+
+ <span class=k>def</span> <span class=nf>extract_output</span><span class=p>(</span><span class=bp>self</span><span class=p>,</span> <span class=n>accumulator</span><span class=p>):</span>
+ <span class=c1># accumulator == {'π₯': 3, 'π
': 6, 'π': 1}</span>
+ <span class=n>total</span> <span class=o>=</span> <span class=nb>sum</span><span class=p>(</span><span class=n>accumulator</span><span class=o>.</span><span class=n>values</span><span class=p>())</span> <span class=c1># 10</span>
+ <span class=n>percentages</span> <span class=o>=</span> <span class=p>{</span><span class=n>item</span><span class=p>:</span> <span class=n>count</span> <span class=o>/</span> <span class=n>total</span> <span class=k>for</span> <span class=n>item</span><span class=p>,</span> <span class=n>count</span> <span class=ow>in</span> <span class=n>accumulator</span><span class=o>.</span><span class=n>items</span><span class=p>()}</span>
+ <span class=c1># percentages == {'π₯': 0.3, 'π
': 0.6, 'π': 0.1}</span>
+ <span class=k>return</span> <span class=n>percentages</span>
+
+<span class=k>with</span> <span class=n>beam</span><span class=o>.</span><span class=n>Pipeline</span><span class=p>()</span> <span class=k>as</span> <span class=n>pipeline</span><span class=p>:</span>
+ <span class=n>percentages</span> <span class=o>=</span> <span class=p>(</span>
+ <span class=n>pipeline</span>
+ <span class=o>|</span> <span class=s1>'Create produce'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Create</span><span class=p>(</span>
+ <span class=p>[</span><span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π
'</span><span class=p>,</span> <span class=s1>'π
'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π
'</span><span class=p>,</span> <span class=s1>'π
'</span><span class=p>,</span> <span class=s1>'π
'</span><span class=p>,</span> <span class= [...]
+ <span class=o>|</span> <span class=s1>'Get percentages'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>CombineGlobally</span><span class=p>(</span><span class=n>PercentagesFn</span><span class=p>())</span>
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>{'π₯': 0.3, 'π
': 0.6, 'π': 0.1}</code></pre></div><table align=left style=margin-right:1em class=.language-py><td><a class=button target=_blank href=https://github.com/apache/beam/blob/master/sdks/python [...]
+Pydoc</a></td></table><p><br><br><br></p></div></div><footer class=footer><div class=footer__contained><div class=footer__cols><div class=footer__cols__col><div class=footer__cols__col__logo><img src=/images/beam_logo_circle.svg class=footer__logo alt="Beam logo"></div><div class=footer__cols__col__logo><img src=/images/apache_logo_circle.svg class=footer__logo alt="Apache logo"></div></div><div class="footer__cols__col footer__cols__col--md"><div class=footer__cols__col__title>Start</di [...]
<a href=http://www.apache.org>The Apache Software Foundation</a>
| <a href=/privacy_policy>Privacy Policy</a>
| <a href=/feed.xml>RSS Feed</a><br><br>Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation. All other products or name brands are trademarks of their respective holders, including The Apache Software Foundation.</div></footer></body></html>
\ No newline at end of file
diff --git a/website/generated-content/documentation/transforms/python/elementwise/filter/index.html b/website/generated-content/documentation/transforms/python/elementwise/filter/index.html
index 9477b10..e131547 100644
--- a/website/generated-content/documentation/transforms/python/elementwise/filter/index.html
+++ b/website/generated-content/documentation/transforms/python/elementwise/filter/index.html
@@ -31,7 +31,7 @@ Then, we apply <code>Filter</code> in multiple ways to filter out produce by the
<span class=p>},</span>
<span class=p>])</span>
<span class=o>|</span> <span class=s1>'Filter perennials'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Filter</span><span class=p>(</span><span class=n>is_perennial</span><span class=p>)</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>Filter</code>:</p><div class=notebook-skip><pre><code>{'icon': 'π', 'name': 'Strawberry', 'duration': 'perennial'}
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>{'icon': 'π', 'name': 'Strawberry', 'duration': 'perennial'}
{'icon': 'π', 'name': 'Eggplant', 'duration': 'perennial'}
{'icon': 'π₯', 'name': 'Potato', 'duration': 'perennial'}</code></pre></div><table align=left style=margin-right:1em class=".language-py .notebook-skip"><td><a class=button target=_blank href=https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/documentation/transforms/python/elementwise/filter-py.ipynb><img src=https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png width=32px height=32px alt [...]
@@ -57,7 +57,7 @@ Then, we apply <code>Filter</code> in multiple ways to filter out produce by the
<span class=p>])</span>
<span class=o>|</span> <span class=s1>'Filter perennials'</span> <span class=o>>></span>
<span class=n>beam</span><span class=o>.</span><span class=n>Filter</span><span class=p>(</span><span class=k>lambda</span> <span class=n>plant</span><span class=p>:</span> <span class=n>plant</span><span class=p>[</span><span class=s1>'duration'</span><span class=p>]</span> <span class=o>==</span> <span class=s1>'perennial'</span><span class=p>)</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>Filter</code>:</p><div class=notebook-skip><pre><code>{'icon': 'π', 'name': 'Strawberry', 'duration': 'perennial'}
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>{'icon': 'π', 'name': 'Strawberry', 'duration': 'perennial'}
{'icon': 'π', 'name': 'Eggplant', 'duration': 'perennial'}
{'icon': 'π₯', 'name': 'Potato', 'duration': 'perennial'}</code></pre></div><table align=left style=margin-right:1em class=".language-py .notebook-skip"><td><a class=button target=_blank href=https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/documentation/transforms/python/elementwise/filter-py.ipynb><img src=https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png width=32px height=32px alt [...]
They are passed as additional positional arguments or keyword arguments to the function.</p><p>In this example, <code>has_duration</code> takes <code>plant</code> and <code>duration</code> as arguments.</p><div class=language-py><div class=highlight><pre class=chroma><code class=language-py data-lang=py><span class=kn>import</span> <span class=nn>apache_beam</span> <span class=kn>as</span> <span class=nn>beam</span>
@@ -86,7 +86,7 @@ They are passed as additional positional arguments or keyword arguments to the f
<span class=p>},</span>
<span class=p>])</span>
<span class=o>|</span> <span class=s1>'Filter perennials'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Filter</span><span class=p>(</span><span class=n>has_duration</span><span class=p>,</span> <span class=s1>'perennial'</span><span class=p>)</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>Filter</code>:</p><div class=notebook-skip><pre><code>{'icon': 'π', 'name': 'Strawberry', 'duration': 'perennial'}
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>{'icon': 'π', 'name': 'Strawberry', 'duration': 'perennial'}
{'icon': 'π', 'name': 'Eggplant', 'duration': 'perennial'}
{'icon': 'π₯', 'name': 'Potato', 'duration': 'perennial'}</code></pre></div><table align=left style=margin-right:1em class=".language-py .notebook-skip"><td><a class=button target=_blank href=https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/documentation/transforms/python/elementwise/filter-py.ipynb><img src=https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png width=32px height=32px alt [...]
passing the <code>PCollection</code> as a <em>singleton</em> accesses that value.</p><p>In this example, we pass a <code>PCollection</code> the value <code>'perennial'</code> as a singleton.
@@ -119,7 +119,7 @@ We then use that value to filter out perennials.</p><div class=language-py><div
<span class=n>duration</span><span class=p>:</span> <span class=n>plant</span><span class=p>[</span><span class=s1>'duration'</span><span class=p>]</span> <span class=o>==</span> <span class=n>duration</span><span class=p>,</span>
<span class=n>duration</span><span class=o>=</span><span class=n>beam</span><span class=o>.</span><span class=n>pvalue</span><span class=o>.</span><span class=n>AsSingleton</span><span class=p>(</span><span class=n>perennial</span><span class=p>),</span>
<span class=p>)</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>Filter</code>:</p><div class=notebook-skip><pre><code>{'icon': 'π', 'name': 'Strawberry', 'duration': 'perennial'}
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>{'icon': 'π', 'name': 'Strawberry', 'duration': 'perennial'}
{'icon': 'π', 'name': 'Eggplant', 'duration': 'perennial'}
{'icon': 'π₯', 'name': 'Potato', 'duration': 'perennial'}</code></pre></div><table align=left style=margin-right:1em class=".language-py .notebook-skip"><td><a class=button target=_blank href=https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/documentation/transforms/python/elementwise/filter-py.ipynb><img src=https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png width=32px height=32px alt [...]
This accesses elements lazily as they are needed,
@@ -156,7 +156,7 @@ so it is possible to iterate over large <code>PCollection</code>s that won&rsquo
<span class=n>valid_durations</span><span class=p>:</span> <span class=n>plant</span><span class=p>[</span><span class=s1>'duration'</span><span class=p>]</span> <span class=ow>in</span> <span class=n>valid_durations</span><span class=p>,</span>
<span class=n>valid_durations</span><span class=o>=</span><span class=n>beam</span><span class=o>.</span><span class=n>pvalue</span><span class=o>.</span><span class=n>AsIter</span><span class=p>(</span><span class=n>valid_durations</span><span class=p>),</span>
<span class=p>)</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>Filter</code>:</p><div class=notebook-skip><pre><code>{'icon': 'π', 'name': 'Strawberry', 'duration': 'perennial'}
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>{'icon': 'π', 'name': 'Strawberry', 'duration': 'perennial'}
{'icon': 'π₯', 'name': 'Carrot', 'duration': 'biennial'}
{'icon': 'π', 'name': 'Eggplant', 'duration': 'perennial'}
{'icon': 'π
', 'name': 'Tomato', 'duration': 'annual'}</code></pre></div><table align=left style=margin-right:1em class=".language-py .notebook-skip"><td><a class=button target=_blank href=https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/documentation/transforms/python/elementwise/filter-py.ipynb><img src=https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png width=32px height=32px alt="R [...]
@@ -196,7 +196,7 @@ If the <code>PCollection</code> won’t fit into memory, use <code>beam.pval
<span class=n>keep_duration</span><span class=p>:</span> <span class=n>keep_duration</span><span class=p>[</span><span class=n>plant</span><span class=p>[</span><span class=s1>'duration'</span><span class=p>]],</span>
<span class=n>keep_duration</span><span class=o>=</span><span class=n>beam</span><span class=o>.</span><span class=n>pvalue</span><span class=o>.</span><span class=n>AsDict</span><span class=p>(</span><span class=n>keep_duration</span><span class=p>),</span>
<span class=p>)</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>Filter</code>:</p><div class=notebook-skip><pre><code>{'icon': 'π', 'name': 'Strawberry', 'duration': 'perennial'}
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>{'icon': 'π', 'name': 'Strawberry', 'duration': 'perennial'}
{'icon': 'π', 'name': 'Eggplant', 'duration': 'perennial'}
{'icon': 'π₯', 'name': 'Potato', 'duration': 'perennial'}</code></pre></div><table align=left style=margin-right:1em class=".language-py .notebook-skip"><td><a class=button target=_blank href=https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/documentation/transforms/python/elementwise/filter-py.ipynb><img src=https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png width=32px height=32px alt [...]
each input it might produce zero or more outputs.</li><li><a href=/documentation/transforms/python/elementwise/pardo>ParDo</a> is the most general elementwise mapping
diff --git a/website/generated-content/documentation/transforms/python/elementwise/flatmap/index.html b/website/generated-content/documentation/transforms/python/elementwise/flatmap/index.html
index 02ceb03..78048a9 100644
--- a/website/generated-content/documentation/transforms/python/elementwise/flatmap/index.html
+++ b/website/generated-content/documentation/transforms/python/elementwise/flatmap/index.html
@@ -16,7 +16,7 @@ This pipeline splits the input element using whitespaces, creating a list of zer
<span class=s1>'π
Tomato π₯Potato'</span><span class=p>,</span>
<span class=p>])</span>
<span class=o>|</span> <span class=s1>'Split words'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>FlatMap</span><span class=p>(</span><span class=nb>str</span><span class=o>.</span><span class=n>split</span><span class=p>)</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>FlatMap</code>:</p><div class=notebook-skip><pre><code>πStrawberry
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>πStrawberry
π₯Carrot
πEggplant
π
Tomato
@@ -33,7 +33,7 @@ This pipeline splits the input element using whitespaces, creating a list of zer
<span class=s1>'π
Tomato,π₯Potato'</span><span class=p>,</span>
<span class=p>])</span>
<span class=o>|</span> <span class=s1>'Split words'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>FlatMap</span><span class=p>(</span><span class=n>split_words</span><span class=p>)</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>FlatMap</code>:</p><div class=notebook-skip><pre><code>πStrawberry
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>πStrawberry
π₯Carrot
πEggplant
π
Tomato
@@ -49,7 +49,7 @@ We use a lambda function that returns the same input element it received.</p><di
<span class=p>[</span><span class=s1>'π
Tomato'</span><span class=p>,</span> <span class=s1>'π₯Potato'</span><span class=p>],</span>
<span class=p>])</span>
<span class=o>|</span> <span class=s1>'Flatten lists'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>FlatMap</span><span class=p>(</span><span class=k>lambda</span> <span class=n>elements</span><span class=p>:</span> <span class=n>elements</span><span class=p>)</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>FlatMap</code>:</p><div class=notebook-skip><pre><code>πStrawberry
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>πStrawberry
π₯Carrot
πEggplant
π
Tomato
@@ -69,7 +69,7 @@ Each yielded result in the generator is an element in the resulting <code>PColle
<span class=p>[</span><span class=s1>'π
Tomato'</span><span class=p>,</span> <span class=s1>'π₯Potato'</span><span class=p>],</span>
<span class=p>])</span>
<span class=o>|</span> <span class=s1>'Flatten lists'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>FlatMap</span><span class=p>(</span><span class=n>generate_elements</span><span class=p>)</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>FlatMap</code>:</p><div class=notebook-skip><pre><code>πStrawberry
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>πStrawberry
π₯Carrot
πEggplant
π
Tomato
@@ -92,7 +92,7 @@ you can use <code>FlatMapTuple</code> to unpack them into different function arg
<span class=p>(</span><span class=bp>None</span><span class=p>,</span> <span class=s1>'Invalid'</span><span class=p>),</span>
<span class=p>])</span>
<span class=o>|</span> <span class=s1>'Format'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>FlatMapTuple</span><span class=p>(</span><span class=n>format_plant</span><span class=p>)</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>FlatMapTuple</code>:</p><div class=notebook-skip><pre><code>πStrawberry
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>πStrawberry
π₯Carrot
πEggplant
π
Tomato
@@ -110,7 +110,7 @@ They are passed as additional positional arguments or keyword arguments to the f
<span class=s1>'π
Tomato,π₯Potato'</span><span class=p>,</span>
<span class=p>])</span>
<span class=o>|</span> <span class=s1>'Split words'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>FlatMap</span><span class=p>(</span><span class=n>split_words</span><span class=p>,</span> <span class=n>delimiter</span><span class=o>=</span><span class=s1>','</span><span class=p>)</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>FlatMap</code>:</p><div class=notebook-skip><pre><code>πStrawberry
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>πStrawberry
π₯Carrot
πEggplant
π
Tomato
@@ -132,7 +132,7 @@ We then use that value as the delimiter for the <code>str.split</code> method.</
<span class=n>delimiter</span><span class=p>:</span> <span class=n>text</span><span class=o>.</span><span class=n>split</span><span class=p>(</span><span class=n>delimiter</span><span class=p>),</span>
<span class=n>delimiter</span><span class=o>=</span><span class=n>beam</span><span class=o>.</span><span class=n>pvalue</span><span class=o>.</span><span class=n>AsSingleton</span><span class=p>(</span><span class=n>delimiter</span><span class=p>),</span>
<span class=p>)</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>FlatMap</code>:</p><div class=notebook-skip><pre><code>πStrawberry
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>πStrawberry
π₯Carrot
πEggplant
π
Tomato
@@ -175,7 +175,7 @@ so it is possible to iterate over large <code>PCollection</code>s that won&rsquo
<span class=n>normalize_and_validate_durations</span><span class=p>,</span>
<span class=n>valid_durations</span><span class=o>=</span><span class=n>beam</span><span class=o>.</span><span class=n>pvalue</span><span class=o>.</span><span class=n>AsIter</span><span class=p>(</span><span class=n>valid_durations</span><span class=p>),</span>
<span class=p>)</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>FlatMap</code>:</p><div class=notebook-skip><pre><code>{'icon': 'π', 'name': 'Strawberry', 'duration': 'perennial'}
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>{'icon': 'π', 'name': 'Strawberry', 'duration': 'perennial'}
{'icon': 'π₯', 'name': 'Carrot', 'duration': 'biennial'}
{'icon': 'π', 'name': 'Eggplant', 'duration': 'perennial'}
{'icon': 'π
', 'name': 'Tomato', 'duration': 'annual'}</code></pre></div><table align=left style=margin-right:1em class=".language-py .notebook-skip"><td><a class=button target=_blank href=https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/documentation/transforms/python/elementwise/flatmap-py.ipynb><img src=https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png width=32px height=32px alt=" [...]
@@ -219,7 +219,7 @@ If the <code>PCollection</code> won’t fit into memory, use <code>beam.pval
<span class=n>replace_duration_if_valid</span><span class=p>,</span>
<span class=n>durations</span><span class=o>=</span><span class=n>beam</span><span class=o>.</span><span class=n>pvalue</span><span class=o>.</span><span class=n>AsDict</span><span class=p>(</span><span class=n>durations</span><span class=p>),</span>
<span class=p>)</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>FlatMap</code>:</p><div class=notebook-skip><pre><code>{'icon': 'π', 'name': 'Strawberry', 'duration': 'perennial'}
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>{'icon': 'π', 'name': 'Strawberry', 'duration': 'perennial'}
{'icon': 'π₯', 'name': 'Carrot', 'duration': 'biennial'}
{'icon': 'π', 'name': 'Eggplant', 'duration': 'perennial'}
{'icon': 'π
', 'name': 'Tomato', 'duration': 'annual'}</code></pre></div><table align=left style=margin-right:1em class=".language-py .notebook-skip"><td><a class=button target=_blank href=https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/documentation/transforms/python/elementwise/flatmap-py.ipynb><img src=https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png width=32px height=32px alt=" [...]
diff --git a/website/generated-content/documentation/transforms/python/elementwise/keys/index.html b/website/generated-content/documentation/transforms/python/elementwise/keys/index.html
index 90e90be..dc8942e 100644
--- a/website/generated-content/documentation/transforms/python/elementwise/keys/index.html
+++ b/website/generated-content/documentation/transforms/python/elementwise/keys/index.html
@@ -16,7 +16,7 @@ Then, we apply <code>Keys</code> to extract the keys and discard the values.</p>
<span class=p>(</span><span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'Potato'</span><span class=p>),</span>
<span class=p>])</span>
<span class=o>|</span> <span class=s1>'Keys'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Keys</span><span class=p>()</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>Keys</code>:</p><div class=notebook-skip><pre><code>π
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>π
π₯
π
π
diff --git a/website/generated-content/documentation/transforms/python/elementwise/kvswap/index.html b/website/generated-content/documentation/transforms/python/elementwise/kvswap/index.html
index 73d1e1a..92b6c0a 100644
--- a/website/generated-content/documentation/transforms/python/elementwise/kvswap/index.html
+++ b/website/generated-content/documentation/transforms/python/elementwise/kvswap/index.html
@@ -17,7 +17,7 @@ Then, we apply <code>KvSwap</code> to swap the keys and values.</p><div class=la
<span class=p>(</span><span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'Potato'</span><span class=p>),</span>
<span class=p>])</span>
<span class=o>|</span> <span class=s1>'Key-Value swap'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>KvSwap</span><span class=p>()</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>KvSwap</code>:</p><div class=notebook-skip><pre><code>('Strawberry', 'π')
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>('Strawberry', 'π')
('Carrot', 'π₯')
('Eggplant', 'π')
('Tomato', 'π
')
diff --git a/website/generated-content/documentation/transforms/python/elementwise/map/index.html b/website/generated-content/documentation/transforms/python/elementwise/map/index.html
index b24a33e..62b09f7 100644
--- a/website/generated-content/documentation/transforms/python/elementwise/map/index.html
+++ b/website/generated-content/documentation/transforms/python/elementwise/map/index.html
@@ -17,7 +17,7 @@ It strips the input element’s whitespaces, including newlines and tabs.</p
<span class=s1>' π₯Potato </span><span class=se>\n</span><span class=s1>'</span><span class=p>,</span>
<span class=p>])</span>
<span class=o>|</span> <span class=s1>'Strip'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=nb>str</span><span class=o>.</span><span class=n>strip</span><span class=p>)</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>Map</code>:</p><div class=notebook-skip><pre><code>πStrawberry
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>πStrawberry
π₯Carrot
πEggplant
π
Tomato
@@ -37,7 +37,7 @@ It strips the input element’s whitespaces, including newlines and tabs.</p
<span class=s1>'# π₯Potato</span><span class=se>\n</span><span class=s1>'</span><span class=p>,</span>
<span class=p>])</span>
<span class=o>|</span> <span class=s1>'Strip header'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=n>strip_header_and_newline</span><span class=p>)</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>Map</code>:</p><div class=notebook-skip><pre><code>πStrawberry
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>πStrawberry
π₯Carrot
πEggplant
π
Tomato
@@ -54,7 +54,7 @@ It strips the input element’s whitespaces, including newlines and tabs.</p
<span class=s1>'# π₯Potato</span><span class=se>\n</span><span class=s1>'</span><span class=p>,</span>
<span class=p>])</span>
<span class=o>|</span> <span class=s1>'Strip header'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>lambda</span> <span class=n>text</span><span class=p>:</span> <span class=n>text</span><span class=o>.</span><span class=n>strip</span><span class=p>(</span><span class=s1>'# </span><span class=se>\n</span><span class=s1>'</span><span class=p>))</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>Map</code>:</p><div class=notebook-skip><pre><code>πStrawberry
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>πStrawberry
π₯Carrot
πEggplant
π
Tomato
@@ -75,7 +75,7 @@ They are passed as additional positional arguments or keyword arguments to the f
<span class=s1>'# π₯Potato</span><span class=se>\n</span><span class=s1>'</span><span class=p>,</span>
<span class=p>])</span>
<span class=o>|</span> <span class=s1>'Strip header'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=n>strip</span><span class=p>,</span> <span class=n>chars</span><span class=o>=</span><span class=s1>'# </span><span class=se>\n</span><span class=s1>'</span><span class=p>)</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>Map</code>:</p><div class=notebook-skip><pre><code>πStrawberry
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>πStrawberry
π₯Carrot
πEggplant
π
Tomato
@@ -94,7 +94,7 @@ you can use <code>MapTuple</code> to unpack them into different function argumen
<span class=p>])</span>
<span class=o>|</span> <span class=s1>'Format'</span> <span class=o>>></span>
<span class=n>beam</span><span class=o>.</span><span class=n>MapTuple</span><span class=p>(</span><span class=k>lambda</span> <span class=n>icon</span><span class=p>,</span> <span class=n>plant</span><span class=p>:</span> <span class=s1>'{}{}'</span><span class=o>.</span><span class=n>format</span><span class=p>(</span><span class=n>icon</span><span class=p>,</span> <span class=n>plant</span><span class=p>))</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>MapTuple</code>:</p><div class=notebook-skip><pre><code>πStrawberry
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>πStrawberry
π₯Carrot
πEggplant
π
Tomato
@@ -119,7 +119,7 @@ We then use that value as the characters for the <code>str.strip</code> method.<
<span class=n>chars</span><span class=p>:</span> <span class=n>text</span><span class=o>.</span><span class=n>strip</span><span class=p>(</span><span class=n>chars</span><span class=p>),</span>
<span class=n>chars</span><span class=o>=</span><span class=n>beam</span><span class=o>.</span><span class=n>pvalue</span><span class=o>.</span><span class=n>AsSingleton</span><span class=p>(</span><span class=n>chars</span><span class=p>),</span>
<span class=p>)</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>Map</code>:</p><div class=notebook-skip><pre><code>πStrawberry
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>πStrawberry
π₯Carrot
πEggplant
π
Tomato
@@ -144,7 +144,7 @@ so it is possible to iterate over large <code>PCollection</code>s that won&rsquo
<span class=n>chars</span><span class=p>:</span> <span class=n>text</span><span class=o>.</span><span class=n>strip</span><span class=p>(</span><span class=s1>''</span><span class=o>.</span><span class=n>join</span><span class=p>(</span><span class=n>chars</span><span class=p>)),</span>
<span class=n>chars</span><span class=o>=</span><span class=n>beam</span><span class=o>.</span><span class=n>pvalue</span><span class=o>.</span><span class=n>AsIter</span><span class=p>(</span><span class=n>chars</span><span class=p>),</span>
<span class=p>)</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>Map</code>:</p><div class=notebook-skip><pre><code>πStrawberry
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>πStrawberry
π₯Carrot
πEggplant
π
Tomato
@@ -188,7 +188,7 @@ If the <code>PCollection</code> won’t fit into memory, use <code>beam.pval
<span class=n>replace_duration</span><span class=p>,</span>
<span class=n>durations</span><span class=o>=</span><span class=n>beam</span><span class=o>.</span><span class=n>pvalue</span><span class=o>.</span><span class=n>AsDict</span><span class=p>(</span><span class=n>durations</span><span class=p>),</span>
<span class=p>)</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>Map</code>:</p><div class=notebook-skip><pre><code>{'icon': 'π', 'name': 'Strawberry', 'duration': 'perennial'}
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>{'icon': 'π', 'name': 'Strawberry', 'duration': 'perennial'}
{'icon': 'π₯', 'name': 'Carrot', 'duration': 'biennial'}
{'icon': 'π', 'name': 'Eggplant', 'duration': 'perennial'}
{'icon': 'π
', 'name': 'Tomato', 'duration': 'annual'}
diff --git a/website/generated-content/documentation/transforms/python/elementwise/pardo/index.html b/website/generated-content/documentation/transforms/python/elementwise/pardo/index.html
index f26762a..4a73dd7 100644
--- a/website/generated-content/documentation/transforms/python/elementwise/pardo/index.html
+++ b/website/generated-content/documentation/transforms/python/elementwise/pardo/index.html
@@ -28,7 +28,7 @@ and it can yield zero or more output elements.</p><div class=language-py><div cl
<span class=s1>'π
Tomato,π₯Potato'</span><span class=p>,</span>
<span class=p>])</span>
<span class=o>|</span> <span class=s1>'Split words'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span class=n>SplitWords</span><span class=p>(</span><span class=s1>','</span><span class=p>))</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>ParDo</code>:</p><div class=notebook-skip><pre><code>πStrawberry
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>πStrawberry
π₯Carrot
πEggplant
π
Tomato
@@ -72,7 +72,7 @@ object.</li></ul><div class=language-py><div class=highlight><pre class=chroma><
<span class=o>|</span>
<span class=s1>'Fixed 30sec windows'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>WindowInto</span><span class=p>(</span><span class=n>beam</span><span class=o>.</span><span class=n>window</span><span class=o>.</span><span class=n>FixedWindows</span><span class=p>(</span><span class=mi>30</span><span class=p>))</span>
<span class=o>|</span> <span class=s1>'Analyze element'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span class=n>AnalyzeElement</span><span class=p>())</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip><code>stdout</code> output:</p><div class=notebook-skip><pre><code># timestamp
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code># timestamp
type(timestamp) -> <class 'apache_beam.utils.timestamp.Timestamp'>
timestamp.micros -> 1584675660000000
timestamp.to_rfc3339() -> '2020-03-20T03:41:00Z'
@@ -139,7 +139,7 @@ For example, if the worker crashes, <code>teardown</code> might not be called.</
<span class=n>pipeline</span>
<span class=o>|</span> <span class=s1>'Create inputs'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Create</span><span class=p>([</span><span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'π'</span><span class=p>,</span> <span class=s1>'π
'</span><span class=p>,</span> <span class=s1>'π₯'</span><span class=p>])</span>
<span class=o>|</span> <span class=s1>'DoFn methods'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span class=n>DoFnMethods</span><span class=p>())</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip><code>stdout</code> output:</p><div class=notebook-skip><pre><code>__init__
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>__init__
setup
start_bundle
* process: π
diff --git a/website/generated-content/documentation/transforms/python/elementwise/partition/index.html b/website/generated-content/documentation/transforms/python/elementwise/partition/index.html
index 70f6569..b9ec5fc 100644
--- a/website/generated-content/documentation/transforms/python/elementwise/partition/index.html
+++ b/website/generated-content/documentation/transforms/python/elementwise/partition/index.html
@@ -35,7 +35,7 @@ We partition the <code>PCollection</code> into one <code>PCollection</code> for
<span class=n>biennials</span> <span class=o>|</span> <span class=s1>'Biennials'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span>
<span class=k>lambda</span> <span class=n>x</span><span class=p>:</span> <span class=k>print</span><span class=p>(</span><span class=s1>'biennial: {}'</span><span class=o>.</span><span class=n>format</span><span class=p>(</span><span class=n>x</span><span class=p>)))</span>
<span class=n>perennials</span> <span class=o>|</span> <span class=s1>'Perennials'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span>
- <span class=k>lambda</span> <span class=n>x</span><span class=p>:</span> <span class=k>print</span><span class=p>(</span><span class=s1>'perennial: {}'</span><span class=o>.</span><span class=n>format</span><span class=p>(</span><span class=n>x</span><span class=p>)))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code>s:</p><div class=notebook-skip><pre><code>perennial: {'icon': 'π', 'name': 'Strawberry',  [...]
+ <span class=k>lambda</span> <span class=n>x</span><span class=p>:</span> <span class=k>print</span><span class=p>(</span><span class=s1>'perennial: {}'</span><span class=o>.</span><span class=n>format</span><span class=p>(</span><span class=n>x</span><span class=p>)))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>perennial: {'icon': 'π', 'name': 'Strawberry', 'duration': 'pere [...]
biennial: {'icon': 'π₯', 'name': 'Carrot', 'duration': 'biennial'}
perennial: {'icon': 'π', 'name': 'Eggplant', 'duration': 'perennial'}
annual: {'icon': 'π
', 'name': 'Tomato', 'duration': 'annual'}
@@ -63,7 +63,7 @@ perennial: {'icon': 'π₯', 'name': 'Potato', &#
<span class=n>biennials</span> <span class=o>|</span> <span class=s1>'Biennials'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span>
<span class=k>lambda</span> <span class=n>x</span><span class=p>:</span> <span class=k>print</span><span class=p>(</span><span class=s1>'biennial: {}'</span><span class=o>.</span><span class=n>format</span><span class=p>(</span><span class=n>x</span><span class=p>)))</span>
<span class=n>perennials</span> <span class=o>|</span> <span class=s1>'Perennials'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span>
- <span class=k>lambda</span> <span class=n>x</span><span class=p>:</span> <span class=k>print</span><span class=p>(</span><span class=s1>'perennial: {}'</span><span class=o>.</span><span class=n>format</span><span class=p>(</span><span class=n>x</span><span class=p>)))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code>s:</p><div class=notebook-skip><pre><code>perennial: {'icon': 'π', 'name': 'Strawberry',  [...]
+ <span class=k>lambda</span> <span class=n>x</span><span class=p>:</span> <span class=k>print</span><span class=p>(</span><span class=s1>'perennial: {}'</span><span class=o>.</span><span class=n>format</span><span class=p>(</span><span class=n>x</span><span class=p>)))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>perennial: {'icon': 'π', 'name': 'Strawberry', 'duration': 'pere [...]
biennial: {'icon': 'π₯', 'name': 'Carrot', 'duration': 'biennial'}
perennial: {'icon': 'π', 'name': 'Eggplant', 'duration': 'perennial'}
annual: {'icon': 'π
', 'name': 'Tomato', 'duration': 'annual'}
@@ -112,7 +112,7 @@ You might want to adapt the bucket assignment to use a more appropriate or rando
<span class=p>)</span>
<span class=n>train_dataset</span> <span class=o>|</span> <span class=s1>'Train'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>lambda</span> <span class=n>x</span><span class=p>:</span> <span class=k>print</span><span class=p>(</span><span class=s1>'train: {}'</span><span class=o>.</span><span class=n>format</span><span class=p>(</span><span class=n>x</span><span class=p>)) [...]
- <span class=n>test_dataset</span> <span class=o>|</span> <span class=s1>'Test'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>lambda</span> <span class=n>x</span><span class=p>:</span> <span class=k>print</span><span class=p>(</span><span class=s1>'test: {}'</span><span class=o>.</span><span class=n>format</span><span class=p>(</span><span class=n>x</span><span class=p>)))</ [...]
+ <span class=n>test_dataset</span> <span class=o>|</span> <span class=s1>'Test'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>lambda</span> <span class=n>x</span><span class=p>:</span> <span class=k>print</span><span class=p>(</span><span class=s1>'test: {}'</span><span class=o>.</span><span class=n>format</span><span class=p>(</span><span class=n>x</span><span class=p>)))</ [...]
train: {'icon': 'π₯', 'name': 'Carrot', 'duration': 'biennial'}
test: {'icon': 'π', 'name': 'Eggplant', 'duration': 'perennial'}
test: {'icon': 'π
', 'name': 'Tomato', 'duration': 'annual'}
diff --git a/website/generated-content/documentation/transforms/python/elementwise/regex/index.html b/website/generated-content/documentation/transforms/python/elementwise/regex/index.html
index da13891..361bacd 100644
--- a/website/generated-content/documentation/transforms/python/elementwise/regex/index.html
+++ b/website/generated-content/documentation/transforms/python/elementwise/regex/index.html
@@ -39,7 +39,7 @@ To match until the end of the string, add <code>'$'</code> at the end of the reg
<span class=s1>'invalid, π, format'</span><span class=p>,</span>
<span class=p>])</span>
<span class=o>|</span> <span class=s1>'Parse plants'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Regex</span><span class=o>.</span><span class=n>matches</span><span class=p>(</span><span class=n>regex</span><span class=p>)</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>Regex.matches</code>:</p><div class=notebook-skip><pre><code>π, Strawberry, perennial
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>π, Strawberry, perennial
π₯, Carrot, biennial
π, Eggplant, perennial
π
, Tomato, annual
@@ -65,7 +65,7 @@ To match until the end of the string, add <code>'$'</code> at the end of the reg
<span class=s1>'invalid, π, format'</span><span class=p>,</span>
<span class=p>])</span>
<span class=o>|</span> <span class=s1>'Parse plants'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Regex</span><span class=o>.</span><span class=n>all_matches</span><span class=p>(</span><span class=n>regex</span><span class=p>)</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>Regex.all_matches</code>:</p><div class=notebook-skip><pre><code>['π, Strawberry, perennial', 'π', 'Strawberry', 'perennial']
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>['π, Strawberry, perennial', 'π', 'Strawberry', 'perennial']
['π₯, Carrot, biennial', 'π₯', 'Carrot', 'biennial']
['π, Eggplant, perennial', 'π', 'Eggplant', 'perennial']
['π
, Tomato, annual', 'π
', 'Tomato', 'annual']
@@ -92,7 +92,7 @@ To match until the end of the string, add <code>'$'</code> at the end of the reg
<span class=s1>'invalid, π, format'</span><span class=p>,</span>
<span class=p>])</span>
<span class=o>|</span> <span class=s1>'Parse plants'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Regex</span><span class=o>.</span><span class=n>matches_kv</span><span class=p>(</span><span class=n>regex</span><span class=p>,</span> <span class=n>keyGroup</span><span class=o>=</span><span class=s1>'icon'</span><span class=p>)</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>Regex.matches_kv</code>:</p><div class=notebook-skip><pre><code>('π', 'π, Strawberry, perennial')
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>('π', 'π, Strawberry, perennial')
('π₯', 'π₯, Carrot, biennial')
('π', 'π, Eggplant, perennial')
('π
', 'π
, Tomato, annual')
@@ -117,7 +117,7 @@ To match until the end of the string, add <code>'$'</code> at the end of the reg
<span class=s1>'# π₯, Potato, perennial'</span><span class=p>,</span>
<span class=p>])</span>
<span class=o>|</span> <span class=s1>'Parse plants'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Regex</span><span class=o>.</span><span class=n>find</span><span class=p>(</span><span class=n>regex</span><span class=p>)</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>Regex.find</code>:</p><div class=notebook-skip><pre><code>π, Strawberry, perennial
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>π, Strawberry, perennial
π₯, Carrot, biennial
π, Eggplant, perennial
π
, Tomato, annual
@@ -142,7 +142,7 @@ To match until the end of the string, add <code>'$'</code> at the end of the reg
<span class=s1>'# π₯, Potato, perennial'</span><span class=p>,</span>
<span class=p>])</span>
<span class=o>|</span> <span class=s1>'Parse plants'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Regex</span><span class=o>.</span><span class=n>find_all</span><span class=p>(</span><span class=n>regex</span><span class=p>)</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>Regex.find_all</code>:</p><div class=notebook-skip><pre><code>['π, Strawberry, perennial']
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>['π, Strawberry, perennial']
['π₯, Carrot, biennial']
['π, Eggplant, perennial', 'π, Banana, perennial']
['π
, Tomato, annual', 'π, Watermelon, annual']
@@ -168,7 +168,7 @@ To match until the end of the string, add <code>'$'</code> at the end of the reg
<span class=s1>'# π₯, Potato, perennial'</span><span class=p>,</span>
<span class=p>])</span>
<span class=o>|</span> <span class=s1>'Parse plants'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Regex</span><span class=o>.</span><span class=n>find_kv</span><span class=p>(</span><span class=n>regex</span><span class=p>,</span> <span class=n>keyGroup</span><span class=o>=</span><span class=s1>'icon'</span><span class=p>)</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>Regex.find_kv</code>:</p><div class=notebook-skip><pre><code>('π', 'π, Strawberry, perennial')
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>('π', 'π, Strawberry, perennial')
('π₯', 'π₯, Carrot, biennial')
('π', 'π, Eggplant, perennial')
('π', 'π, Banana, perennial')
@@ -190,7 +190,7 @@ on the <code>replacement</code>.</p><div class=language-py><div class=highlight>
<span class=s1>'π₯ : Potato : perennial'</span><span class=p>,</span>
<span class=p>])</span>
<span class=o>|</span> <span class=s1>'To CSV'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Regex</span><span class=o>.</span><span class=n>replace_all</span><span class=p>(</span><span class=sa>r</span><span class=s1>'\s*:\s*'</span><span class=p>,</span> <span class=s1>','</span><span class=p>)</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>Regex.replace_all</code>:</p><div class=notebook-skip><pre><code>π,Strawberry,perennial
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>π,Strawberry,perennial
π₯,Carrot,biennial
π,Eggplant,perennial
π
,Tomato,annual
@@ -210,7 +210,7 @@ on the <code>replacement</code>.</p><div class=language-py><div class=highlight>
<span class=s1>'π₯, Potato, perennial'</span><span class=p>,</span>
<span class=p>])</span>
<span class=o>|</span> <span class=s1>'As dictionary'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Regex</span><span class=o>.</span><span class=n>replace_first</span><span class=p>(</span><span class=sa>r</span><span class=s1>'\s*,\s*'</span><span class=p>,</span> <span class=s1>': '</span><span class=p>)</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>Regex.replace_first</code>:</p><div class=notebook-skip><pre><code>π: Strawberry, perennial
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>π: Strawberry, perennial
π₯: Carrot, biennial
π: Eggplant, perennial
π
: Tomato, annual
@@ -228,7 +228,7 @@ The argument <code>outputEmpty</code> is set to <code>False</code> by default, b
<span class=s1>'π₯ : Potato : perennial'</span><span class=p>,</span>
<span class=p>])</span>
<span class=o>|</span> <span class=s1>'Parse plants'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Regex</span><span class=o>.</span><span class=n>split</span><span class=p>(</span><span class=sa>r</span><span class=s1>'\s*:\s*'</span><span class=p>)</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>Regex.split</code>:</p><div class=notebook-skip><pre><code>['π', 'Strawberry', 'perennial']
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>['π', 'Strawberry', 'perennial']
['π₯', 'Carrot', 'biennial']
['π', 'Eggplant', 'perennial']
['π
', 'Tomato', 'annual']
diff --git a/website/generated-content/documentation/transforms/python/elementwise/tostring/index.html b/website/generated-content/documentation/transforms/python/elementwise/tostring/index.html
index 648f776..f2ac7bd 100644
--- a/website/generated-content/documentation/transforms/python/elementwise/tostring/index.html
+++ b/website/generated-content/documentation/transforms/python/elementwise/tostring/index.html
@@ -19,7 +19,7 @@ You can specify a different delimiter using the <code>delimiter</code> argument.
<span class=p>(</span><span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'Potato'</span><span class=p>),</span>
<span class=p>])</span>
<span class=o>|</span> <span class=s1>'To string'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ToString</span><span class=o>.</span><span class=n>Kvs</span><span class=p>()</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>ToString</code>:</p><div class=notebook-skip><pre><code>π,Strawberry
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>π,Strawberry
π₯,Carrot
π,Eggplant
π
,Tomato
@@ -37,7 +37,7 @@ The string output will be equivalent to <code>str(element)</code>.</p><div class
<span class=p>[</span><span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'Potato'</span><span class=p>,</span> <span class=s1>'perennial'</span><span class=p>],</span>
<span class=p>])</span>
<span class=o>|</span> <span class=s1>'To string'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ToString</span><span class=o>.</span><span class=n>Element</span><span class=p>()</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>ToString</code>:</p><div class=notebook-skip><pre><code>['π', 'Strawberry', 'perennial']
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>['π', 'Strawberry', 'perennial']
['π₯', 'Carrot', 'biennial']
['π', 'Eggplant', 'perennial']
['π
', 'Tomato', 'annual']
@@ -57,7 +57,7 @@ The string output will be equivalent to <code>iterable.join(delimiter)</code>.</
<span class=p>[</span><span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'Potato'</span><span class=p>,</span> <span class=s1>'perennial'</span><span class=p>],</span>
<span class=p>])</span>
<span class=o>|</span> <span class=s1>'To string'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ToString</span><span class=o>.</span><span class=n>Iterables</span><span class=p>()</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>ToString</code>:</p><div class=notebook-skip><pre><code>π,Strawberry,perennial
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>π,Strawberry,perennial
π₯,Carrot,biennial
π,Eggplant,perennial
π
,Tomato,annual
diff --git a/website/generated-content/documentation/transforms/python/elementwise/values/index.html b/website/generated-content/documentation/transforms/python/elementwise/values/index.html
index 7901a53..375b4a2 100644
--- a/website/generated-content/documentation/transforms/python/elementwise/values/index.html
+++ b/website/generated-content/documentation/transforms/python/elementwise/values/index.html
@@ -16,7 +16,7 @@ Then, we apply <code>Values</code> to extract the values and discard the keys.</
<span class=p>(</span><span class=s1>'π₯'</span><span class=p>,</span> <span class=s1>'Potato'</span><span class=p>),</span>
<span class=p>])</span>
<span class=o>|</span> <span class=s1>'Values'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>Values</span><span class=p>()</span>
- <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after <code>Values</code>:</p><div class=notebook-skip><pre><code>Strawberry
+ <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>))</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>Strawberry
Carrot
Eggplant
Tomato
diff --git a/website/generated-content/documentation/transforms/python/elementwise/withtimestamps/index.html b/website/generated-content/documentation/transforms/python/elementwise/withtimestamps/index.html
index 88e274f..6f1718d 100644
--- a/website/generated-content/documentation/transforms/python/elementwise/withtimestamps/index.html
+++ b/website/generated-content/documentation/transforms/python/elementwise/withtimestamps/index.html
@@ -25,7 +25,7 @@ in the form of seconds.</p><div class=language-py><div class=highlight><pre clas
<span class=k>lambda</span> <span class=n>plant</span><span class=p>:</span> <span class=n>beam</span><span class=o>.</span><span class=n>window</span><span class=o>.</span><span class=n>TimestampedValue</span><span class=p>(</span><span class=n>plant</span><span class=p>,</span> <span class=n>plant</span><span class=p>[</span><span class=s1>'season'</span><span class=p>]))</span>
<span class=o>|</span> <span class=s1>'Get timestamp'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span class=n>GetTimestamp</span><span class=p>())</span>
<span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>)</span>
- <span class=p>)</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after getting the timestamps:</p><div class=notebook-skip><pre><code>2020-04-01 00:00:00 - Strawberry
+ <span class=p>)</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>2020-04-01 00:00:00 - Strawberry
2020-06-01 00:00:00 - Carrot
2020-03-01 00:00:00 - Artichoke
2020-05-01 00:00:00 - Tomato
@@ -68,7 +68,7 @@ These numbers have to be converted to a <em>“seconds”</em> equivalen
<span class=n>beam</span><span class=o>.</span><span class=n>window</span><span class=o>.</span><span class=n>TimestampedValue</span><span class=p>(</span><span class=n>plant</span><span class=p>,</span> <span class=n>plant</span><span class=p>[</span><span class=s1>'event_id'</span><span class=p>]))</span>
<span class=o>|</span> <span class=s1>'Get timestamp'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span class=n>GetTimestamp</span><span class=p>())</span>
<span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>)</span>
- <span class=p>)</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after getting the timestamps:</p><div class=notebook-skip><pre><code>1 - Strawberry
+ <span class=p>)</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>1 - Strawberry
4 - Carrot
2 - Artichoke
3 - Tomato
@@ -95,7 +95,7 @@ Workers might have time deltas, so using this method is not a reliable way to do
<span class=n>beam</span><span class=o>.</span><span class=n>window</span><span class=o>.</span><span class=n>TimestampedValue</span><span class=p>(</span><span class=n>plant</span><span class=p>,</span> <span class=n>time</span><span class=o>.</span><span class=n>time</span><span class=p>()))</span>
<span class=o>|</span> <span class=s1>'Get timestamp'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span class=n>GetTimestamp</span><span class=p>())</span>
<span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>)</span>
- <span class=p>)</span></code></pre></div></div><p class=notebook-skip>Output <code>PCollection</code> after getting the timestamps:</p><div class=notebook-skip><pre><code>2020-03-20 20:12:42.145594 - Strawberry
+ <span class=p>)</span></code></pre></div></div><p class=notebook-skip>Output:</p><div class=notebook-skip><pre><code>2020-03-20 20:12:42.145594 - Strawberry
2020-03-20 20:12:42.145827 - Carrot
2020-03-20 20:12:42.145962 - Artichoke
2020-03-20 20:12:42.146093 - Tomato
diff --git a/website/generated-content/sitemap.xml b/website/generated-content/sitemap.xml
index b7e5795..037f4cc9 100644
--- a/website/generated-content/sitemap.xml
+++ b/website/generated-content/sitemap.xml
@@ -1 +1 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/beam-2.22.0/</loc><lastmod>2020-06-08T14:13:37-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2020-06-08T14:13:37-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2020-06-08T14:13:37-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2020-06-08T14:13:37-07:00</lastmod></url><url><loc>/blog/b [...]
\ No newline at end of file
+<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/beam-2.22.0/</loc><lastmod>2020-06-08T14:13:37-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2020-06-08T14:13:37-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2020-06-08T14:13:37-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2020-06-08T14:13:37-07:00</lastmod></url><url><loc>/blog/b [...]
\ No newline at end of file