You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2022/07/09 22:20:31 UTC
[beam] branch asf-site updated: Publishing website 2022/07/09 22:20:23 at commit f641c24
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new c071f5dd777 Publishing website 2022/07/09 22:20:23 at commit f641c24
c071f5dd777 is described below
commit c071f5dd777d2f34eaa3458e7e136d3011304581
Author: jenkins <bu...@apache.org>
AuthorDate: Sat Jul 9 22:20:23 2022 +0000
Publishing website 2022/07/09 22:20:23 at commit f641c24
---
website/generated-content/documentation/index.xml | 644 ++++++++++++++++++++-
.../documentation/programming-guide/index.html | 478 ++++++++++++---
website/generated-content/sitemap.xml | 2 +-
3 files changed, 1026 insertions(+), 98 deletions(-)
diff --git a/website/generated-content/documentation/index.xml b/website/generated-content/documentation/index.xml
index 16920ccfee8..9b91a7b9570 100644
--- a/website/generated-content/documentation/index.xml
+++ b/website/generated-content/documentation/index.xml
@@ -4216,10 +4216,12 @@ programming guide, take a look at the
<li data-type="language-java" class="active">Java SDK</li>
<li data-type="language-py">Python SDK</li>
<li data-type="language-go">Go SDK</li>
+<li data-type="language-typescript">Typescript SDK</li>
</ul>
</nav>
<p class="language-py">The Python SDK supports Python 3.7, 3.8, and 3.9.</p>
<p class="language-go">The Go SDK supports Go v1.18+. SDK release 2.32.0 is the last experimental version.</p>
+<p class="language-typescript">The Typescript SDK supports Node v16+ and is still experimental.</p>
<h2 id="overview">1. Overview</h2>
<p>To use Beam, you need to first create a driver program using the classes in one
of the Beam SDKs. Your driver program <em>defines</em> your pipeline, including all of
@@ -4307,6 +4309,10 @@ class <code>Pipeline</code> (typically in the <code>main()</code> fu
your pipeline&rsquo;s configuration options programmatically, but it&rsquo;s often easier to
set the options ahead of time (or read them from the command line) and pass them
to the <code>Pipeline</code> object when you create the object.</p>
+<span class="language-typescript">
+A Pipeline in the Typescript API is simply a function that will be called
+with a single `root` object and is passed to a Runner's `run` method.
+</span>
<div class='language-java snippet'>
<div class="notebook-skip code-snippet">
<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
@@ -4340,6 +4346,17 @@ to the <code>Pipeline</code> object when you create the object.</p>
</span><span class="c1"></span><span class="nx">pipeline</span><span class="p">,</span> <span class="nx">scope</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">NewPipelineWithRoot</span><span class="p">()</span></code></pre></div>
</div>
</div>
+<div class='language-typescript snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-typescript" data-lang="typescript"><span class="nx">await</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">createRunner</span><span class="p">().</span><span class="nx">run</span><span class="p">(</span><span class="kd">function</span> <span class="nx">pipeline</span><span class="p">(</span><span class="nx">root</span><s [...]
+<span class="c1">// Use root to build a pipeline.
+</span><span class="c1"></span><span class="p">});</span>
+</code></pre></div>
+</div>
+</div>
<h3 id="configuring-pipeline-options">2.1. Configuring pipeline options</h3>
<p>Use the pipeline options to configure different aspects of your pipeline, such
as the pipeline runner that will execute your pipeline and any runner-specific
@@ -4357,6 +4374,9 @@ you can use to set fields in <code>PipelineOptions</code> using command-li
as demonstrated in the following example code:</p>
<p class="language-go">Use Go flags to parse command line arguments to configure your pipeline. Flags must be parsed
before <code>beam.Init()</code> is called.</p>
+<p class="language-typescript">Any Javascript object can be used as pipeline options.
+One can either construct one manually, but it is also common to pass an object
+created from command line options such as <code>yargs.argv</code>.</p>
<div class='language-java snippet'>
<div class="notebook-skip code-snippet">
<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
@@ -4385,6 +4405,20 @@ before <code>beam.Init()</code> is called.</p>
</span><span class="c1"></span><span class="nx">flag</span><span class="p">.</span><span class="nf">Parse</span><span class="p">()</span></code></pre></div>
</div>
</div>
+<div class='language-typescript snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-typescript" data-lang="typescript"><span class="kr">const</span> <span class="nx">pipeline_options</span> <span class="o">=</span> <span class="p">{</span>
+<span class="nx">runner</span><span class="o">:</span> <span class="s2">&#34;default&#34;</span><span class="p">,</span>
+<span class="nx">project</span><span class="o">:</span> <span class="s2">&#34;my_project&#34;</span><span class="p">,</span>
+<span class="p">};</span>
+<span class="kr">const</span> <span class="nx">runner</span> <span class="o">=</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">createRunner</span><span class="p">(</span><span class="nx">pipeline_options</span><span class="p">);</span>
+<span class="kr">const</span> <span class="nx">runnerFromCommandLineOptions</span> <span class="o">=</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">createRunner</span><span class="p">(</span><span class="nx">yargs</span><span class="p">.</span><span class="nx">argv</span><span class="p">);</span>
+</code></pre></div>
+</div>
+</div>
<p>This interprets command-line arguments that follow the format:</p>
<pre><code>--&lt;option&gt;=&lt;value&gt;
</code></pre><span class="language-java">
@@ -4403,9 +4437,10 @@ options.</p>
</blockquote>
<h4 id="creating-custom-options">2.1.2. Creating custom options</h4>
<p>You can add your own custom options in addition to the standard
-<code>PipelineOptions</code>. To add your own options, define an interface with getter and
-setter methods for each option, as in the following example for
-adding <code>input</code> and <code>output</code> custom options:</p>
+<code>PipelineOptions</code>.
+<p class="language-java">To add your own options, define an interface with getter and
+setter methods for each option.</p>
+The following example shows how to add <code>input</code> and <code>output</code> custom options:</p>
<div class='language-java snippet'>
<div class="notebook-skip code-snippet">
<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
@@ -4428,8 +4463,8 @@ adding <code>input</code> and <code>output</code> custom options:<
<span class="k">class</span> <span class="nc">MyOptions</span><span class="p">(</span><span class="n">PipelineOptions</span><span class="p">):</span>
<span class="nd">@classmethod</span>
<span class="k">def</span> <span class="nf">_add_argparse_args</span><span class="p">(</span><span class="bp">cls</span><span class="p">,</span> <span class="n">parser</span><span class="p">):</span>
-<span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--input-file&#39;</span><span class="p">)</span>
-<span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--output-path&#39;</span><span class="p">)</span></code></pre></div>
+<span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--input&#39;</span><span class="p">)</span>
+<span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--output&#39;</span><span class="p">)</span></code></pre></div>
</div>
</div>
<div class='language-go snippet'>
@@ -4444,9 +4479,20 @@ adding <code>input</code> and <code>output</code> custom options:<
<span class="p">)</span></code></pre></div>
</div>
</div>
+<div class='language-typescript snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-typescript" data-lang="typescript"><span class="kr">const</span> <span class="nx">options</span> <span class="o">=</span> <span class="nx">yargs</span><span class="p">.</span><span class="nx">argv</span><span class="p">;</span> <span class="c1">// Or an alternative command-line parsing library.
+</span><span class="c1"></span>
+<span class="c1">// Use options.input and options.output during pipeline construction.
+</span></code></pre></div>
+</div>
+</div>
<p>You can also specify a description, which appears when a user passes <code>--help</code> as
a command-line argument, and a default value.</p>
-<p>You set the description and default value using annotations, as follows:</p>
+<p class="language-java language-py langauge-go">You set the description and default value using annotations, as follows:</p>
<div class='language-java snippet'>
<div class="notebook-skip code-snippet">
<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
@@ -4474,13 +4520,11 @@ a command-line argument, and a default value.</p>
<span class="nd">@classmethod</span>
<span class="k">def</span> <span class="nf">_add_argparse_args</span><span class="p">(</span><span class="bp">cls</span><span class="p">,</span> <span class="n">parser</span><span class="p">):</span>
<span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span>
-<span class="s1">&#39;--input-file&#39;</span><span class="p">,</span>
+<span class="s1">&#39;--input&#39;</span><span class="p">,</span>
<span class="n">default</span><span class="o">=</span><span class="s1">&#39;gs://dataflow-samples/shakespeare/kinglear.txt&#39;</span><span class="p">,</span>
<span class="n">help</span><span class="o">=</span><span class="s1">&#39;The file path for the input text to process.&#39;</span><span class="p">)</span>
<span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span>
-<span class="s1">&#39;--output-path&#39;</span><span class="p">,</span>
-<span class="n">required</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
-<span class="n">help</span><span class="o">=</span><span class="s1">&#39;The path prefix for output files.&#39;</span><span class="p">)</span></code></pre></div>
+<span class="s1">&#39;--output&#39;</span><span class="p">,</span> <span class="n">required</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;The path prefix for output files.&#39;</span><span class="p">)</span></code></pre></div>
</div>
</div>
<div class='language-go snippet'>
@@ -4494,6 +4538,8 @@ a command-line argument, and a default value.</p>
<span class="p">)</span></code></pre></div>
</div>
</div>
+<p class="language-py">For Python, you can also simply parse your custom options with argparse; there
+is no need to create a separate PipelineOptions subclass.</p>
<p class="language-java">It&rsquo;s recommended that you register your interface with <code>PipelineOptionsFactory</code>
and then pass the interface when creating the <code>PipelineOptions</code> object. When you
register your interface with <code>PipelineOptionsFactory</code>, the <code>--help</code> can find
@@ -4543,12 +4589,16 @@ represent the data records in that source.</p>
transform to the <code>Pipeline</code> object itself.
<span class="language-java"><code>TextIO.Read</code></span>
<span class="language-py"><code>io.TextFileSource</code></span>
-<span class="language-go"><code>textio.Read</code></span>, for example, reads from an
+<span class="language-go"><code>textio.Read</code></span>
+<span class="language-typescript"><code>textio.ReadFromText</code></span>,
+for example, reads from an
external text file and returns a <code>PCollection</code> whose elements are of type
<code>String</code>, each <code>String</code> represents one line from the text file. Here&rsquo;s how you
would apply <span class="language-java"><code>TextIO.Read</code></span>
<span class="language-py"><code>io.TextFileSource</code></span>
-<span class="language-go"><code>textio.Read</code></span> to your <code>Pipeline</code> to create
+<span class="language-go"><code>textio.Read</code></span>
+<span class="language-typescript"><code>textio.ReadFromText</code></span>
+to your <code>Pipeline</code> <span class="language-typescript">root</span> to create
a <code>PCollection</code>:</p>
<div class='language-java snippet'>
<div class="notebook-skip code-snippet">
@@ -4587,6 +4637,19 @@ a <code>PCollection</code>:</p>
</span><span class="c1"></span><span class="nx">lines</span> <span class="o">:=</span> <span class="nx">textio</span><span class="p">.</span><span class="nf">Read</span><span class="p">(</span><span class="nx">scope</span><span class="p">,</span> <span class="s">&#34;gs://some/inputData.txt&#34;</span><span class="p">)</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="nx">async</span> <span class="nx">function</span> <span class="nf">pipeline</span><span class="p">(</span><span class="nx">root</span><span class="p">:</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">Root</span><span class="p">)</span> <span class="p">{</span>
+<span class="c1">// Note that textio.ReadFromText is an AsyncPTransform.
+</span><span class="c1"></span> <span class="kd">const</span> <span class="nx">pcoll</span><span class="p">:</span> <span class="nx">PCollection</span><span class="p">&lt;</span><span class="kt">string</span><span class="p">&gt;</span> <span class="p">=</span> <span class="nx">await</span> <span class="nx">root</span><span class="p">.</span><span class="nf">asyncApply</span><span class="p">(</span>
+<span class="nx">textio</span><span class="p">.</span><span class="nf">ReadFromText</span><span class="p">(</span><span class="s">&#34;path/to/text_pattern&#34;</span><span class="p">)</span>
+<span class="p">);</span>
+<span class="p">}</span></code></pre></div>
+</div>
+</div>
<p>See the <a href="#pipeline-io">section on I/O</a> to learn more about how to read from the
various data sources supported by the Beam SDK.</p>
<h4 id="creating-pcollection-in-memory">3.1.2. Creating a PCollection from in-memory data</h4>
@@ -4601,8 +4664,13 @@ Beam-provided <code>Create</code> transform. Much like a data adapter&
itself.</p>
<p class="language-go">To create a <code>PCollection</code> from an in-memory <code>slice</code>, you use the Beam-provided
<code>beam.CreateList</code> transform. Pass the pipeline <code>scope</code>, and the <code>slice</code> to this transform.</p>
+<p class="language-typescript">To create a <code>PCollection</code> from an in-memory <code>array</code>, you use the Beam-provided
+<code>Create</code> transform. Apply this transform directly to your <code>Root</code> object.</p>
<p>The following example code shows how to create a <code>PCollection</code> from an in-memory
-<span class="language-java"><code>List</code></span><span class="language-py"><code>list</code></span><span class="language-go"><code>slice</code></span>:</p>
+<span class="language-java"><code>List</code></span>
+<span class="language-py"><code>list</code></span>
+<span class="language-go"><code>slice</code></span>
+<span class="language-typescript"><code>array</code></span>:</p>
<div class='language-java snippet'>
<div class="notebook-skip code-snippet">
<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
@@ -4662,6 +4730,24 @@ itself.</p>
</span><span class="c1"></span><span class="nx">linesPCol</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">CreateList</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">lines</span><span class="p">)</span></code></pre></div>
</div>
</div>
+<div class='language-typescript snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-typescript" data-lang="typescript"><span class="kd">function</span> <span class="nx">pipeline</span><span class="p">(</span><span class="nx">root</span>: <span class="kt">beam.Root</span><span class="p">)</span> <span class="p">{</span>
+<span class="kr">const</span> <span class="nx">pcoll</span> <span class="o">=</span> <span class="nx">root</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span>
+<span class="nx">beam</span><span class="p">.</span><span class="nx">create</span><span class="p">([</span>
+<span class="s2">&#34;To be, or not to be: that is the question: &#34;</span><span class="p">,</span>
+<span class="s2">&#34;Whether &#39;tis nobler in the mind to suffer &#34;</span><span class="p">,</span>
+<span class="s2">&#34;The slings and arrows of outrageous fortune, &#34;</span><span class="p">,</span>
+<span class="s2">&#34;Or to take arms against a sea of troubles, &#34;</span><span class="p">,</span>
+<span class="p">])</span>
+<span class="p">);</span>
+<span class="p">}</span>
+</code></pre></div>
+</div>
+</div>
<h3 id="pcollection-characteristics">3.2. PCollection characteristics</h3>
<p>A <code>PCollection</code> is owned by the specific <code>Pipeline</code> object for which it is
created; multiple pipelines cannot share a <code>PCollection</code>.
@@ -4793,7 +4879,17 @@ the transform itself as an argument, and the operation returns the output
<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="p">[</span><span class="nx">Output</span> <span class="nx">PCollection</span><span class="p">]</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">scope</span><span class="p">,</span> <span class="p">[</span>< [...]
</div>
</div>
-<p class="language-java language-py">Because Beam uses a generic <code>apply</code> method for <code>PCollection</code>, you can both chain
+<div class='language-typescript snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-typescript" data-lang="typescript"><span class="p">[</span><span class="nx">Output</span> <span class="nx">PCollection</span><span class="p">]</span> <span class="o">=</span> <span class="p">[</span><span class="nx">Input</span> <span class="nx">PCollection</span><span class="p">].</span><span class="nx">apply</span><span class="p">([</span><span [...]
+<span class="p">[</span><span class="nx">Output</span> <span class="nx">PCollection</span><span class="p">]</span> <span class="o">=</span> <span class="nx">await</span> <span class="p">[</span><span class="nx">Input</span> <span class="nx">PCollection</span><span class="p">].</span><span class="nx">asyncApply</span><span class="p">([</span><span class="nx">AsyncTransform</span><span class="p">])</span>
+</code></pre></div>
+</div>
+</div>
+<p class="language-java language-py language-typescript">Because Beam uses a generic <code>apply</code> method for <code>PCollection</code>, you can both chain
transforms sequentially and also apply transforms that contain other transforms
nested within (called <a href="#composite-transforms">composite transforms</a> in the Beam
SDKs).</p>
@@ -4841,6 +4937,17 @@ For example, you can successively call transforms on PCollections to modify the
<span class="p">[</span><span class="nx">Final</span> <span class="nx">Output</span> <span class="nx">PCollection</span><span class="p">]</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">scope</span><span class="p">,</span> <span class="p">[</span><span class="nx">Third</span> <span class="nx">Tran [...]
</div>
</div>
+<div class='language-typescript snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-typescript" data-lang="typescript"><span class="p">[</span><span class="nx">Final</span> <span class="nx">Output</span> <span class="nx">PCollection</span><span class="p">]</span> <span class="o">=</span> <span class="p">[</span><span class="nx">Initial</span> <span class="nx">Input</span> <span class="nx">PCollection</span><span class="p">].</span>& [...]
+<span class="p">.</span><span class="nx">apply</span><span class="p">([</span><span class="nx">Second</span> <span class="nx">Transform</span><span class="p">])</span>
+<span class="p">.</span><span class="nx">apply</span><span class="p">([</span><span class="nx">Third</span> <span class="nx">Transform</span><span class="p">])</span>
+</code></pre></div>
+</div>
+</div>
<p>The graph of this pipeline looks like the following:</p>
<p><img src="/images/design-your-pipeline-linear.svg" alt="This linear pipeline starts with one input collection, sequentially appliesthree transforms, and ends with one output collection."></p>
<p><em>Figure 1: A linear pipeline with three sequential transforms.</em></p>
@@ -4878,6 +4985,17 @@ a branching pipeline, like so:</p>
<span class="p">[</span><span class="nx">PCollection</span> <span class="nx">of</span> <span class="sc">&#39;B&#39;</span> <span class="nx">names</span><span class="p">]</span> <span class="p">=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">scope</span><span class="p">,</span> <span class="p">[</span><span class= [...]
</div>
</div>
+<div class='language-typescript snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-typescript" data-lang="typescript"><span class="p">[</span><span class="nx">PCollection</span> <span class="nx">of</span> <span class="nx">database</span> <span class="nx">table</span> <span class="nx">rows</span><span class="p">]</span> <span class="o">=</span> <span class="p">[</span><span class="nx">Database</span> <span class="nx">Table</span> &l [...]
+<span class="p">[</span><span class="nx">PCollection</span> <span class="nx">of</span> <span class="s1">&#39;A&#39;</span> <span class="nx">names</span><span class="p">]</span> <span class="o">=</span> <span class="p">[</span><span class="nx">PCollection</span> <span class="nx">of</span> <span class="nx">database</span> <span class="nx">table</span> <span class="nx">rows</span><span class="p">].</ [...]
+<span class="p">[</span><span class="nx">PCollection</span> <span class="nx">of</span> <span class="s1">&#39;B&#39;</span> <span class="nx">names</span><span class="p">]</span> <span class="o">=</span> <span class="p">[</span><span class="nx">PCollection</span> <span class="nx">of</span> <span class="nx">database</span> <span class="nx">table</span> <span class="nx">rows</span><span class="p">].</ [...]
+</code></pre></div>
+</div>
+</div>
<p>The graph of this branching pipeline looks like the following:</p>
<p><img src="/images/design-your-pipeline-multiple-pcollections.svg" alt="This pipeline applies two transforms to a single input collection. Eachtransform produces an output collection."></p>
<p><em>Figure 2: A branching pipeline. Two transforms are applied to a single
@@ -4886,6 +5004,18 @@ PCollection of database table rows.</em></p>
nest multiple transforms inside a single, larger transform. Composite transforms
are particularly useful for building a reusable sequence of simple steps that
get used in a lot of different places.</p>
+<p class="language-python">The pipe syntax allows one to apply PTransforms to <code>tuple</code>s and <code>dict</code>s of
+PCollections as well for those transforms accepting multiple inputs (such as
+<code>Flatten</code> and <code>CoGroupByKey</code>).</p>
+<p class="language-typescript">PTransforms can also be applied to any <code>PValue</code>, which include the Root object,
+PCollections, arrays of <code>PValue</code>s, and objects with <code>PValue</code> values.
+One can apply transforms to these composite types by wrapping them with
+<code>beam.P</code>, e.g.
+<code>beam.P({left: pcollA, right: pcollB}).apply(transformExpectingTwoPCollections)</code>.</p>
+<p class="language-typescript">PTransforms come in two flavors, synchronous and asynchronous, depending on
+whether their <em>application</em>* involves asynchronous invocations.
+An <code>AsyncTransform</code> must be applied with <code>asyncApply</code> and returns a <code>Promise</code>
+which must be awaited before further pipeline construction.</p>
<h3 id="core-beam-transforms">4.2. Core Beam transforms</h3>
<p>Beam provides the following core transforms, each of which represents a different
processing paradigm:</p>
@@ -4897,6 +5027,8 @@ processing paradigm:</p>
<li><code>Flatten</code></li>
<li><code>Partition</code></li>
</ul>
+<p class="language-typescript">The Typescript SDK provides some of the most basic of these transforms
+as methods on <code>PCollection</code> itself.</p>
<h4 id="pardo">4.2.1. ParDo</h4>
<p><code>ParDo</code> is a Beam transform for generic parallel processing. The <code>ParDo</code>
processing paradigm is similar to the &ldquo;Map&rdquo; phase of a
@@ -5021,6 +5153,25 @@ as shown in the following example code:</p>
<span class="nx">wordLengths</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="o">&amp;</span><span class="nx">ComputeWordLengthFn</span><span class="p">{},</span> <span class="nx">words</span><span class="p">)</span></code></pre></div>
</div>
</div>
+<div class='language-typescript snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-typescript" data-lang="typescript"><span class="err">#</span> <span class="nx">The</span> <span class="nx">input</span> <span class="nx">PCollection</span> <span class="nx">of</span> <span class="nx">Strings</span><span class="p">.</span>
+<span class="kr">const</span> <span class="nx">words</span> : <span class="kt">PCollection</span><span class="o">&lt;</span><span class="kt">string</span><span class="o">&gt;</span> <span class="o">=</span> <span class="p">...</span>
+<span class="err">#</span> <span class="nx">The</span> <span class="nx">DoFn</span> <span class="nx">to</span> <span class="nx">perform</span> <span class="nx">on</span> <span class="nx">each</span> <span class="nx">element</span> <span class="k">in</span> <span class="nx">the</span> <span class="nx">input</span> <span class="nx">PCollection</span><span class="p">.</span>
+<span class="kd">function</span> <span class="nx">computeWordLengthFn</span><span class="p">()</span><span class="o">:</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">DoFn</span><span class="o">&lt;</span><span class="kt">string</span><span class="p">,</span> <span class="kt">number</span><span class="o">&gt;</span> <span class="p">{</span>
+<span class="k">return</span> <span class="p">{</span>
+<span class="nx">process</span>: <span class="kt">function</span><span class="o">*</span> <span class="p">(</span><span class="nx">element</span><span class="p">)</span> <span class="p">{</span>
+<span class="nx">yield</span> <span class="nx">element</span><span class="p">.</span><span class="nx">length</span><span class="p">;</span>
+<span class="p">},</span>
+<span class="p">};</span>
+<span class="p">}</span>
+<span class="kr">const</span> <span class="nx">result</span> <span class="o">=</span> <span class="nx">words</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nx">parDo</span><span class="p">(</span><span class="nx">computeWordLengthFn</span><span class="p">()));</span>
+</code></pre></div>
+</div>
+</div>
<p>In the example, our input <code>PCollection</code> contains <span class="language-java language-py"><code>String</code></span>
<span class="language-go"><code>string</code></span> values. We apply a
<code>ParDo</code> transform that specifies a function (<code>ComputeWordLengthFn</code>) to compute
@@ -5089,7 +5240,7 @@ provides a method for emitting elements. The parameter types must match the inpu
and output types of your <code>DoFn</code> or the framework will raise an error. Note: <code>@Element</code> and
<code>OutputReceiver</code> were introduced in Beam 2.5.0; if using an earlier release of Beam, a
<code>ProcessContext</code> parameter should be used instead.</p>
-<p class="language-py">Inside your <code>DoFn</code> subclass, you&rsquo;ll write a method <code>process</code> where you provide
+<p class="language-py language-typescript">Inside your <code>DoFn</code> subclass, you&rsquo;ll write a method <code>process</code> where you provide
the actual processing logic. You don&rsquo;t need to manually extract the elements
from the input collection; the Beam SDKs handle that for you. Your <code>process</code> method
should accept an argument <code>element</code>, which is the input element, and return an
@@ -5149,6 +5300,21 @@ or the framework will raise an error.</p>
<span class="p">}</span></code></pre></div>
</div>
</div>
+<div class='language-typescript snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-typescript" data-lang="typescript"><span class="kd">function</span> <span class="nx">computeWordLengthFn</span><span class="p">()</span><span class="o">:</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">DoFn</span><span class="o">&lt;</span><span class="kt">string</span><span class="p">,</span> <span class="kt">number</ [...]
+<span class="k">return</span> <span class="p">{</span>
+<span class="nx">process</span>: <span class="kt">function</span><span class="o">*</span> <span class="p">(</span><span class="nx">element</span><span class="p">)</span> <span class="p">{</span>
+<span class="nx">yield</span> <span class="nx">element</span><span class="p">.</span><span class="nx">length</span><span class="p">;</span>
+<span class="p">},</span>
+<span class="p">};</span>
+<span class="p">}</span>
+</code></pre></div>
+</div>
+</div>
<p class="language-go">Simple DoFns can also be written as functions.</p>
<div class='language-go snippet'>
<div class="notebook-skip code-snippet">
@@ -5205,7 +5371,7 @@ elements from the input collection).</li>
that value in any way.</li>
</ul>
</span>
-<span class="language-py">
+<span class="language-py language-typescript">
<ul>
<li>You should not in any way modify the <code>element</code> argument provided to the
<code>process</code> method, or any side inputs.</li>
@@ -5226,12 +5392,14 @@ that value in any way.</li>
<code>ParDo</code> by providing a lightweight <code>DoFn</code> in-line, as
<span class="language-java">an anonymous inner class instance</span>
<span class="language-py">a lambda function</span>
-<span class="language-go">an anonymous function</span>.</p>
+<span class="language-go">an anonymous function</span>
+<span class="language-typescript">a function passed to <code>PCollection.map</code> or <code>PCollection.flatMap</code></span>.</p>
<p>Here&rsquo;s the previous example, <code>ParDo</code> with <code>ComputeLengthWordsFn</code>, with the
<code>DoFn</code> specified as
<span class="language-java">an anonymous inner class instance</span>
<span class="language-py">a lambda function</span>
-<span class="language-go">an anonymous function</span>:</p>
+<span class="language-go">an anonymous function</span>
+<span class="language-typescript">a function</span>:</p>
<div class='language-java snippet'>
<div class="notebook-skip code-snippet">
<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
@@ -5275,15 +5443,26 @@ that value in any way.</li>
<span class="p">},</span> <span class="nx">words</span><span class="p">)</span></code></pre></div>
</div>
</div>
+<div class='language-typescript snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-typescript" data-lang="typescript"><span class="c1">// The input PCollection of strings.
+</span><span class="c1"></span><span class="nx">words</span> <span class="o">=</span> <span class="p">...</span>
+<span class="kr">const</span> <span class="nx">result</span> <span class="o">=</span> <span class="nx">words</span><span class="p">.</span><span class="nx">flatMap</span><span class="p">((</span><span class="nx">word</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="p">[</span><span class="nx">word</span><span class="p">.</span><span class="nx">length</span><span class="p">]) [...]
+</code></pre></div>
+</div>
+</div>
<p>If your <code>ParDo</code> performs a one-to-one mapping of input elements to output
elements&ndash;that is, for each input element, it applies a function that produces
<em>exactly one</em> output element, <span class="language-go">you can return that
element directly.</span><span class="language-java language-py">you can use the higher-level
-<span class="language-java"><code>MapElements</code></span><span class="language-py"><code>Map</code></span>
+<span class="language-java"><code>MapElements</code></span><span class="language-py language-py"><code>Map</code></span>
transform.</span><span class="language-java"><code>MapElements</code> can accept an anonymous
Java 8 lambda function for additional brevity.</span></p>
<p>Here&rsquo;s the previous example using <span class="language-java"><code>MapElements</code></span>
-<span class="language-py"><code>Map</code></span><span class="language-go">a direct return</span>:</p>
+<span class="language-py language-typescript"><code>Map</code></span><span class="language-go">a direct return</span>:</p>
<div class='language-java snippet'>
<div class="notebook-skip code-snippet">
<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
@@ -5324,6 +5503,17 @@ Java 8 lambda function for additional brevity.</span></p>
<span class="p">},</span> <span class="nx">words</span><span class="p">)</span></code></pre></div>
</div>
</div>
+<div class='language-typescript snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-typescript" data-lang="typescript"><span class="c1">// The input PCollection of string.
+</span><span class="c1"></span><span class="nx">words</span> <span class="o">=</span> <span class="p">...</span>
+<span class="kr">const</span> <span class="nx">result</span> <span class="o">=</span> <span class="nx">words</span><span class="p">.</span><span class="nx">map</span><span class="p">((</span><span class="nx">word</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="nx">word</span><span class="p">.</span><span class="nx">length</span><span class="p">);</span>
+</code></pre></div>
+</div>
+</div>
<p><span class="language-java" ></p>
<blockquote>
<p><strong>Note:</strong> You can use Java 8 lambda functions with several other Beam
@@ -5390,6 +5580,11 @@ tree, [2]
</code></pre><p>Thus, <code>GroupByKey</code> represents a transform from a multimap (multiple keys to
individual values) to a uni-map (unique keys to collections of values).</p>
<p><span class="language-java">Using <code>GroupByKey</code> is straightforward:</span></p>
+<p class="language-py language-typescript">While all SDKs have a <code>GroupByKey</code> transform, using <code>GroupBy</code> is
+generally more natural.
+The <code>GroupBy</code> transform can be parameterized by the name(s) of properties
+on which to group the elements of the PCollection, or a function taking
+the each element as input that maps to a key on which to do grouping.</p>
<div class='language-java snippet'>
<div class="notebook-skip code-snippet">
<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
@@ -5403,6 +5598,16 @@ individual values) to a uni-map (unique keys to collections of values).</p>
<span class="n">mapped</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">GroupByKey</span><span class="o">.&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span><span class="n">create</span><span class="o">());</span></code></pre></div>
</div>
</div>
+<div class='language-py snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-py" data-lang="py"><span class="c1"># The input PCollection of (`string`, `int`) tuples.</span>
+<span class="n">words_and_counts</span> <span class="o">=</span> <span class="o">...</span>
+<span class="n">grouped_words</span> <span class="o">=</span> <span class="n">words_and_counts</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">GroupByKey</span><span class="p">()</span></code></pre></div>
+</div>
+</div>
<div class='language-go snippet'>
<div class="notebook-skip code-snippet">
<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
@@ -5414,6 +5619,27 @@ individual values) to a uni-map (unique keys to collections of values).</p>
<span class="nx">keyed</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">GroupByKey</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">pairs</span><span class="p">)</span></code></pre></div>
</div>
</div>
+<div class='language-typescript snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-typescript" data-lang="typescript"><span class="c1">// A PCollection of elements like
+</span><span class="c1">// {word: &#34;cat&#34;, score: 1}, {word: &#34;dog&#34;, score: 5}, {word: &#34;cat&#34;, score: 5}, ...
+</span><span class="c1"></span><span class="kr">const</span> <span class="nx">scores</span> : <span class="kt">PCollection</span><span class="o">&lt;</span><span class="p">{</span><span class="nx">word</span>: <span class="kt">string</span><span class="p">,</span> <span class="nx">score</span>: <span class="kt">number</span><span class="p">}</span><span class="o">&gt;</span> <span class="o">=</ [...]
+<span class="c1">// This will produce a PCollection with elements like
+</span><span class="c1">// {key: &#34;cat&#34;, value: [{ word: &#34;cat&#34;, score: 1 },
+</span><span class="c1">// { word: &#34;cat&#34;, score: 5 }, ...]}
+</span><span class="c1">// {key: &#34;dog&#34;, value: [{ word: &#34;dog&#34;, score: 5 }, ...]}
+</span><span class="c1"></span><span class="kr">const</span> <span class="nx">grouped_by_word</span> <span class="o">=</span> <span class="nx">scores</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nx">groupBy</span><span class="p">(</span><span class="s2">&#34;word&#34;</span><span class="p">));</span>
+<span class="c1">// This will produce a PCollection with elements like
+</span><span class="c1">// {key: 3, value: [{ word: &#34;cat&#34;, score: 1 },
+</span><span class="c1">// { word: &#34;dog&#34;, score: 5 },
+</span><span class="c1">// { word: &#34;cat&#34;, score: 5 }, ...]}
+</span><span class="c1"></span><span class="kr">const</span> <span class="nx">by_word_length</span> <span class="o">=</span> <span class="nx">scores</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nx">groupBy</span><span class="p">((</span><span class="nx">x</span><span class="p">)</span> <span class= [...]
+</code></pre></div>
+</div>
+</div>
<h5 id="groupbykey-and-unbounded-pcollections">4.2.2.1 GroupByKey and unbounded PCollections</h5>
<p>If you are using unbounded <code>PCollection</code>s, you must use either <a href="#setting-your-pcollections-windowing-function">non-global
windowing</a> or an
@@ -5557,6 +5783,28 @@ data contains names and phone numbers.</p>
<span class="nx">phones</span> <span class="o">:=</span> <span class="nf">CreateAndSplit</span><span class="p">(</span><span class="nx">s</span><span class="p">.</span><span class="nf">Scope</span><span class="p">(</span><span class="s">&#34;CreatePhones&#34;</span><span class="p">),</span> <span class="nx">phoneSlice</span><span class="p">)</span></code></pre></div>
</div>
</div>
+<div class='language-typescript snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-typescript" data-lang="typescript"><span class="kr">const</span> <span class="nx">emails_list</span> <span class="o">=</span> <span class="p">[</span>
+<span class="p">{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;amy&#34;</span><span class="p">,</span> <span class="nx">email</span><span class="o">:</span> <span class="s2">&#34;amy@example.com&#34;</span> <span class="p">},</span>
+<span class="p">{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;carl&#34;</span><span class="p">,</span> <span class="nx">email</span><span class="o">:</span> <span class="s2">&#34;carl@example.com&#34;</span> <span class="p">},</span>
+<span class="p">{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;julia&#34;</span><span class="p">,</span> <span class="nx">email</span><span class="o">:</span> <span class="s2">&#34;julia@example.com&#34;</span> <span class="p">},</span>
+<span class="p">{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;carl&#34;</span><span class="p">,</span> <span class="nx">email</span><span class="o">:</span> <span class="s2">&#34;carl@email.com&#34;</span> <span class="p">},</span>
+<span class="p">];</span>
+<span class="kr">const</span> <span class="nx">phones_list</span> <span class="o">=</span> <span class="p">[</span>
+<span class="p">{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;amy&#34;</span><span class="p">,</span> <span class="nx">phone</span><span class="o">:</span> <span class="s2">&#34;111-222-3333&#34;</span> <span class="p">},</span>
+<span class="p">{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;james&#34;</span><span class="p">,</span> <span class="nx">phone</span><span class="o">:</span> <span class="s2">&#34;222-333-4444&#34;</span> <span class="p">},</span>
+<span class="p">{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;amy&#34;</span><span class="p">,</span> <span class="nx">phone</span><span class="o">:</span> <span class="s2">&#34;333-444-5555&#34;</span> <span class="p">},</span>
+<span class="p">{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;carl&#34;</span><span class="p">,</span> <span class="nx">phone</span><span class="o">:</span> <span class="s2">&#34;444-555-6666&#34;</span> <span class="p">},</span>
+<span class="p">];</span>
+<span class="kr">const</span> <span class="nx">emails</span> <span class="o">=</span> <span class="nx">root</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nx">create</span><span class="p">(</span><span class="nx">emails_list</span><span class="p">));</span>
+<span class="kr">const</span> <span class="nx">phones</span> <span class="o">=</span> <span class="nx">root</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nx">create</span><span class="p">(</span><span class="nx">phones_list</span><span class="p">));</span>
+</code></pre></div>
+</div>
+</div>
<p>After <code>CoGroupByKey</code>, the resulting data contains all data associated with each
unique key from any of the input collections.</p>
<div class='language-java snippet'>
@@ -5645,7 +5893,51 @@ unique key from any of the input collections.</p>
<span class="p">}</span></code></pre></div>
</div>
</div>
-<p class="language-java language-py">The following code example joins the two <code>PCollection</code>s with <code>CoGroupByKey</code>,
+<div class='language-typescript snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-typescript" data-lang="typescript"><span class="kr">const</span> <span class="nx">results</span> <span class="o">=</span> <span class="p">[</span>
+<span class="p">{</span>
+<span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;amy&#34;</span><span class="p">,</span>
+<span class="nx">values</span><span class="o">:</span> <span class="p">{</span>
+<span class="nx">emails</span><span class="o">:</span> <span class="p">[{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;amy&#34;</span><span class="p">,</span> <span class="nx">email</span><span class="o">:</span> <span class="s2">&#34;amy@example.com&#34;</span> <span class="p">}],</span>
+<span class="nx">phones</span><span class="o">:</span> <span class="p">[</span>
+<span class="p">{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;amy&#34;</span><span class="p">,</span> <span class="nx">phone</span><span class="o">:</span> <span class="s2">&#34;111-222-3333&#34;</span> <span class="p">},</span>
+<span class="p">{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;amy&#34;</span><span class="p">,</span> <span class="nx">phone</span><span class="o">:</span> <span class="s2">&#34;333-444-5555&#34;</span> <span class="p">},</span>
+<span class="p">],</span>
+<span class="p">},</span>
+<span class="p">},</span>
+<span class="p">{</span>
+<span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;carl&#34;</span><span class="p">,</span>
+<span class="nx">values</span><span class="o">:</span> <span class="p">{</span>
+<span class="nx">emails</span><span class="o">:</span> <span class="p">[</span>
+<span class="p">{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;carl&#34;</span><span class="p">,</span> <span class="nx">email</span><span class="o">:</span> <span class="s2">&#34;carl@example.com&#34;</span> <span class="p">},</span>
+<span class="p">{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;carl&#34;</span><span class="p">,</span> <span class="nx">email</span><span class="o">:</span> <span class="s2">&#34;carl@email.com&#34;</span> <span class="p">},</span>
+<span class="p">],</span>
+<span class="nx">phones</span><span class="o">:</span> <span class="p">[{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;carl&#34;</span><span class="p">,</span> <span class="nx">phone</span><span class="o">:</span> <span class="s2">&#34;444-555-6666&#34;</span> <span class="p">}],</span>
+<span class="p">},</span>
+<span class="p">},</span>
+<span class="p">{</span>
+<span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;james&#34;</span><span class="p">,</span>
+<span class="nx">values</span><span class="o">:</span> <span class="p">{</span>
+<span class="nx">emails</span><span class="o">:</span> <span class="p">[],</span>
+<span class="nx">phones</span><span class="o">:</span> <span class="p">[{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;james&#34;</span><span class="p">,</span> <span class="nx">phone</span><span class="o">:</span> <span class="s2">&#34;222-333-4444&#34;</span> <span class="p">}],</span>
+<span class="p">},</span>
+<span class="p">},</span>
+<span class="p">{</span>
+<span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;julia&#34;</span><span class="p">,</span>
+<span class="nx">values</span><span class="o">:</span> <span class="p">{</span>
+<span class="nx">emails</span><span class="o">:</span> <span class="p">[{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;julia&#34;</span><span class="p">,</span> <span class="nx">email</span><span class="o">:</span> <span class="s2">&#34;julia@example.com&#34;</span> <span class="p">}],</span>
+<span class="nx">phones</span><span class="o">:</span> <span class="p">[],</span>
+<span class="p">},</span>
+<span class="p">},</span>
+<span class="p">];</span>
+</code></pre></div>
+</div>
+</div>
+<p class="language-java language-py language-typescript">The following code example joins the two <code>PCollection</code>s with <code>CoGroupByKey</code>,
followed by a <code>ParDo</code> to consume the result. Then, the code uses tags to look up
and format data from each collection.</p>
<p class="language-go">The following code example joins the two <code>PCollection</code>s with <code>CoGroupByKey</code>,
@@ -5744,6 +6036,22 @@ parameters maps to the ordering of the <code>CoGroupByKey</code> inputs.&l
<span class="p">}</span></code></pre></div>
</div>
</div>
+<div class='language-typescript snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-typescript" data-lang="typescript"><span class="kr">const</span> <span class="nx">formatted_results_pcoll</span> <span class="o">=</span> <span class="nx">beam</span>
+<span class="p">.</span><span class="nx">P</span><span class="p">({</span> <span class="nx">emails</span><span class="p">,</span> <span class="nx">phones</span> <span class="p">})</span>
+<span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nx">coGroupBy</span><span class="p">(</span><span class="s2">&#34;name&#34;</span><span class="p">))</span>
+<span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="kd">function</span> <span class="nx">formatResults</span><span class="p">({</span> <span class="nx">key</span><span class="p">,</span> <span class="nx">values</span> <span class="p">})</span> <span class="p">{</span>
+<span class="kr">const</span> <span class="nx">emails</span> <span class="o">=</span> <span class="nx">values</span><span class="p">.</span><span class="nx">emails</span><span class="p">.</span><span class="nx">map</span><span class="p">((</span><span class="nx">x</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="nx">x</span><span class="p">.</span><span class="nx">email</ [...]
+<span class="kr">const</span> <span class="nx">phones</span> <span class="o">=</span> <span class="nx">values</span><span class="p">.</span><span class="nx">phones</span><span class="p">.</span><span class="nx">map</span><span class="p">((</span><span class="nx">x</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="nx">x</span><span class="p">.</span><span class="nx">phone</ [...]
+<span class="k">return</span> <span class="sb">`</span><span class="si">${</span><span class="nx">key</span><span class="si">}</span><span class="sb">; [</span><span class="si">${</span><span class="nx">emails</span><span class="si">}</span><span class="sb">]; [</span><span class="si">${</span><span class="nx">phones</span><span class="si">}</span><span class="sb">]`</span><span class="p">;</span>
+<span class="p">});</span>
+</code></pre></div>
+</div>
+</div>
<p>The formatted data looks like this:</p>
<div class='language-java snippet'>
<div class="notebook-skip code-snippet">
@@ -5784,10 +6092,25 @@ parameters maps to the ordering of the <code>CoGroupByKey</code> inputs.&l
<span class="p">}</span></code></pre></div>
</div>
</div>
+<div class='language-typescript snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-typescript" data-lang="typescript"><span class="kr">const</span> <span class="nx">formatted_results</span> <span class="o">=</span> <span class="p">[</span>
+<span class="s2">&#34;amy; [amy@example.com]; [111-222-3333,333-444-5555]&#34;</span><span class="p">,</span>
+<span class="s2">&#34;carl; [carl@email.com,carl@example.com]; [444-555-6666]&#34;</span><span class="p">,</span>
+<span class="s2">&#34;james; []; [222-333-4444]&#34;</span><span class="p">,</span>
+<span class="s2">&#34;julia; [julia@example.com]; []&#34;</span><span class="p">,</span>
+<span class="p">];</span>
+</code></pre></div>
+</div>
+</div>
<h4 id="combine">4.2.4. Combine</h4>
<p><span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.40.0/index.html?org/apache/beam/sdk/transforms/Combine.html"><code>Combine</code></a></span>
<span class="language-py"><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/core.py"><code>Combine</code></a></span>
<span class="language-go"><a href="https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/combine.go#L27"><code>Combine</code></a></span>
+<span class="language-typescript"><a href="https://github.com/apache/beam/blob/master/sdks/typescript/src/apache_beam/transforms/group_and_combine.ts"><code>Combine</code></a></span>
is a Beam transform for combining collections of elements or values in your
data. <code>Combine</code> has variants that work on entire <code>PCollection</code>s, and some that
combine the values for each key in <code>PCollection</code>s of key/value pairs.</p>
@@ -5818,7 +6141,14 @@ that your streaming computation might be idle. Incremental combining also
reduces the storage of intermediate accumulators.</li>
</ul>
<h5 id="simple-combines">4.2.4.1. Simple combinations using simple functions</h5>
-<p>The following example code shows a simple combine function.</p>
+<p>The following example code shows a simple combine function.
+<span class="language-typescript">
+Combining is done by modifying a grouping transform with the <code>combining</code> method.
+This method takes three parameters: the value to combine (either as a named
+property of the input elements, or a function of the entire input),
+the combining operation (either a binary function or a <code>CombineFn</code>),
+and finally a name for the combined value in the output object.
+</span></p>
<div class='language-java snippet'>
<div class="notebook-skip code-snippet">
<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
@@ -5896,6 +6226,22 @@ the type of the output.</p>
<span class="p">}</span></code></pre></div>
</div>
</div>
+<div class='language-typescript snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-typescript" data-lang="typescript"><span class="kr">const</span> <span class="nx">pcoll</span> <span class="o">=</span> <span class="nx">root</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nx">create</span><span class="p">([</span><span class="mi"> [...]
+<span class="kr">const</span> <span class="nx">result</span> <span class="o">=</span> <span class="nx">pcoll</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span>
+<span class="nx">beam</span>
+<span class="p">.</span><span class="nx">groupGlobally</span><span class="p">()</span>
+<span class="p">.</span><span class="nx">combining</span><span class="p">((</span><span class="nx">c</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="nx">c</span><span class="p">,</span> <span class="p">(</span><span class="nx">x</span><span class="p">,</span> <span class="nx">y</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="nx">x</span> < [...]
+<span class="p">.</span><span class="nx">combining</span><span class="p">((</span><span class="nx">c</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="nx">c</span><span class="p">,</span> <span class="p">(</span><span class="nx">x</span><span class="p">,</span> <span class="nx">y</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="nx">x</span> < [...]
+<span class="p">);</span>
+<span class="kr">const</span> <span class="nx">expected</span> <span class="o">=</span> <span class="p">{</span> <span class="nx">sum</span>: <span class="kt">1111</span><span class="p">,</span> <span class="nx">product</span>: <span class="kt">1000000</span> <span class="p">}</span>
+</code></pre></div>
+</div>
+</div>
<h5 id="advanced-combines">4.2.4.2. Advanced combinations using CombineFn</h5>
<p>For more complex combine functions, you can define a
<span class="language-java language-py">subclass of</span><code>CombineFn</code>.
@@ -6017,6 +6363,27 @@ mean average:</p>
<span class="p">}</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="kd">const</span> <span class="nx">meanCombineFn</span><span class="p">:</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">CombineFn</span><span class="p">&lt;</span><span class="nx">number</span><span class="p">,</span> <span class="p">[</span><span class="nx">number</span><span class=" [...]
+<span class="p">{</span>
+<span class="nx">createAccumulator</span><span class="p">:</span> <span class="p">()</span> <span class="p">=&gt;</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">],</span>
+<span class="nx">addInput</span><span class="p">:</span> <span class="p">([</span><span class="nx">sum</span><span class="p">,</span> <span class="nx">count</span><span class="p">]:</span> <span class="p">[</span><span class="nx">number</span><span class="p">,</span> <span class="nx">number</span><span class="p">],</span> <span class="nx">i</span><span class="p">:</span> <span class="nx">number</spa [...]
+<span class="nx">sum</span> <span class="o">+</span> <span class="nx">i</span><span class="p">,</span>
+<span class="nx">count</span> <span class="o">+</span> <span class="mi">1</span><span class="p">,</span>
+<span class="p">],</span>
+<span class="nx">mergeAccumulators</span><span class="p">:</span> <span class="p">(</span><span class="nx">accumulators</span><span class="p">:</span> <span class="p">[</span><span class="nx">number</span><span class="p">,</span> <span class="nx">number</span><span class="p">][])</span> <span class="p">=&gt;</span>
+<span class="nx">accumulators</span><span class="p">.</span><span class="nf">reduce</span><span class="p">(([</span><span class="nx">sum0</span><span class="p">,</span> <span class="nx">count0</span><span class="p">],</span> <span class="p">[</span><span class="nx">sum1</span><span class="p">,</span> <span class="nx">count1</span><span class="p">])</span> <span class="p">=&gt;</span> <span class="p [...]
+<span class="nx">sum0</span> <span class="o">+</span> <span class="nx">sum1</span><span class="p">,</span>
+<span class="nx">count0</span> <span class="o">+</span> <span class="nx">count1</span><span class="p">,</span>
+<span class="p">]),</span>
+<span class="nx">extractOutput</span><span class="p">:</span> <span class="p">([</span><span class="nx">sum</span><span class="p">,</span> <span class="nx">count</span><span class="p">]:</span> <span class="p">[</span><span class="nx">number</span><span class="p">,</span> <span class="nx">number</span><span class="p">])</span> <span class="p">=&gt;</span> <span class="nx">sum</span> <span class="o" [...]
+<span class="p">};</span></code></pre></div>
+</div>
+</div>
<span class="language-go">
<blockquote>
<p><strong>Note</strong>: Only <code>MergeAccumulators</code> is a required method. The others will have a default interpretation
@@ -6061,6 +6428,17 @@ of integers.</p>
<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="nx">average</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">Combine</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="o">&amp;</span><span class="nx">averageFn</span><span class="p">{},</span> <span class="nx">ints&l [...]
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="kd">const</span> <span class="nx">pcoll</span> <span class="p">=</span> <span class="nx">root</span><span class="p">.</span><span class="nf">apply</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nf">create</span><span class="p">([</span><span class="mi">4</span><s [...]
+<span class="kd">const</span> <span class="nx">result</span> <span class="p">=</span> <span class="nx">pcoll</span><span class="p">.</span><span class="nf">apply</span><span class="p">(</span>
+<span class="nx">beam</span><span class="p">.</span><span class="nf">groupGlobally</span><span class="p">().</span><span class="nf">combining</span><span class="p">((</span><span class="nx">c</span><span class="p">)</span> <span class="p">=&gt;</span> <span class="nx">c</span><span class="p">,</span> <span class="nx">meanCombineFn</span><span class="p">,</span> <span class="s">&#34;mean&#34;</ [...]
+<span class="p">);</span></code></pre></div>
+</div>
+</div>
<h5 id="combine-global-windowing">4.2.4.4. Combine and global windowing</h5>
<p>If your input <code>PCollection</code> uses the default global windowing, the default
behavior is to return a <code>PCollection</code> containing one item. That item&rsquo;s value
@@ -6192,10 +6570,38 @@ PerKey must be an associative reduction function or a
<span class="o">//</span> <span class="nx">avgAccuracyPerPlayer</span> <span class="nx">is</span> <span class="nx">a</span> <span class="nx">PCollection</span><span class="p">&lt;</span><span class="kt">string</span><span class="p">,</span><span class="kt">float64</span><span class="p">&gt;</span></code></pre></div>
</div>
</div>
+<div class='language-typescript snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-typescript" data-lang="typescript"><span class="kr">const</span> <span class="nx">pcoll</span> <span class="o">=</span> <span class="nx">root</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span>
+<span class="nx">beam</span><span class="p">.</span><span class="nx">create</span><span class="p">([</span>
+<span class="p">{</span> <span class="nx">player</span><span class="o">:</span> <span class="s2">&#34;alice&#34;</span><span class="p">,</span> <span class="nx">accuracy</span>: <span class="kt">1.0</span> <span class="p">},</span>
+<span class="p">{</span> <span class="nx">player</span><span class="o">:</span> <span class="s2">&#34;bob&#34;</span><span class="p">,</span> <span class="nx">accuracy</span>: <span class="kt">0.99</span> <span class="p">},</span>
+<span class="p">{</span> <span class="nx">player</span><span class="o">:</span> <span class="s2">&#34;eve&#34;</span><span class="p">,</span> <span class="nx">accuracy</span>: <span class="kt">0.5</span> <span class="p">},</span>
+<span class="p">{</span> <span class="nx">player</span><span class="o">:</span> <span class="s2">&#34;eve&#34;</span><span class="p">,</span> <span class="nx">accuracy</span>: <span class="kt">0.25</span> <span class="p">},</span>
+<span class="p">])</span>
+<span class="p">);</span>
+<span class="kr">const</span> <span class="nx">result</span> <span class="o">=</span> <span class="nx">pcoll</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span>
+<span class="nx">beam</span>
+<span class="p">.</span><span class="nx">groupBy</span><span class="p">(</span><span class="s2">&#34;player&#34;</span><span class="p">)</span>
+<span class="p">.</span><span class="nx">combining</span><span class="p">(</span><span class="s2">&#34;accuracy&#34;</span><span class="p">,</span> <span class="nx">combiners</span><span class="p">.</span><span class="nx">mean</span><span class="p">,</span> <span class="s2">&#34;mean&#34;</span><span class="p">)</span>
+<span class="p">.</span><span class="nx">combining</span><span class="p">(</span><span class="s2">&#34;accuracy&#34;</span><span class="p">,</span> <span class="nx">combiners</span><span class="p">.</span><span class="nx">max</span><span class="p">,</span> <span class="s2">&#34;max&#34;</span><span class="p">)</span>
+<span class="p">);</span>
+<span class="kr">const</span> <span class="nx">expected</span> <span class="o">=</span> <span class="p">[</span>
+<span class="p">{</span> <span class="nx">player</span><span class="o">:</span> <span class="s2">&#34;alice&#34;</span><span class="p">,</span> <span class="nx">mean</span>: <span class="kt">1.0</span><span class="p">,</span> <span class="nx">max</span>: <span class="kt">1.0</span> <span class="p">},</span>
+<span class="p">{</span> <span class="nx">player</span><span class="o">:</span> <span class="s2">&#34;bob&#34;</span><span class="p">,</span> <span class="nx">mean</span>: <span class="kt">0.99</span><span class="p">,</span> <span class="nx">max</span>: <span class="kt">0.99</span> <span class="p">},</span>
+<span class="p">{</span> <span class="nx">player</span><span class="o">:</span> <span class="s2">&#34;eve&#34;</span><span class="p">,</span> <span class="nx">mean</span>: <span class="kt">0.375</span><span class="p">,</span> <span class="nx">max</span>: <span class="kt">0.5</span> <span class="p">},</span>
+<span class="p">];</span>
+</code></pre></div>
+</div>
+</div>
<h4 id="flatten">4.2.5. Flatten</h4>
<p><span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.40.0/index.html?org/apache/beam/sdk/transforms/Flatten.html"><code>Flatten</code></a></span>
<span class="language-py"><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/core.py"><code>Flatten</code></a></span>
<span class="language-go"><a href="https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/flatten.go"><code>Flatten</code></a></span>
+<span class="language-typescript"><code>Flatten</code></span>
is a Beam transform for <code>PCollection</code> objects that store the same data type.
<code>Flatten</code> merges multiple <code>PCollection</code> objects into a single logical
<code>PCollection</code>.</p>
@@ -6239,6 +6645,20 @@ is a Beam transform for <code>PCollection</code> objects that store the sa
<span class="nx">merged</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">Flatten</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">pcol1</span><span class="p">,</span> <span class="nx">pcol2</span><span class="p">,</span> <span class="nx">pcol3</span><span class="p">)</span></code></pre></div>
</div>
</div>
+<div class='language-typescript snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-typescript" data-lang="typescript"><span class="c1">// Flatten takem an array of PCollection objects, wrapped in beam.P(...)
+</span><span class="c1">// Returns a single PCollection that contains a union of all of the elements in all input PCollections.
+</span><span class="c1"></span>
+<span class="kr">const</span> <span class="nx">fib</span> <span class="o">=</span> <span class="nx">root</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nx">create</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <spa [...]
+<span class="kr">const</span> <span class="nx">pow</span> <span class="o">=</span> <span class="nx">root</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nx">create</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <spa [...]
+<span class="kr">const</span> <span class="nx">result</span> <span class="o">=</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">P</span><span class="p">([</span><span class="nx">fib</span><span class="p">,</span> <span class="nx">pow</span><span class="p">]).</span><span class="nx">apply</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span>< [...]
+</code></pre></div>
+</div>
+</div>
<h5 id="data-encoding-merged-collections">4.2.5.1. Data encoding in merged collections</h5>
<p>By default, the coder for the output <code>PCollection</code> is the same as the coder for
the first <code>PCollection</code> in the input <code>PCollectionList</code>. However, the input
@@ -6257,9 +6677,11 @@ pipeline is constructed.</p>
<p><span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.40.0/index.html?org/apache/beam/sdk/transforms/Partition.html"><code>Partition</code></a></span>
<span class="language-py"><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/core.py"><code>Partition</code></a></span>
<span class="language-go"><a href="https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/partition.go"><code>Partition</code></a></span>
+<span class="language-typescript"><code>Partition</code></span>
is a Beam transform for <code>PCollection</code> objects that store the same data
type. <code>Partition</code> splits a single <code>PCollection</code> into a fixed number of smaller
collections.</p>
+<p class="language-typescript">Often in the Typescript SDK the <code>Split</code> transform is more natural to use.</p>
<p><code>Partition</code> divides the elements of a <code>PCollection</code> according to a partitioning
function that you provide. The partitioning function contains the logic that
determines how to split up the elements of the input <code>PCollection</code> into each
@@ -6322,6 +6744,22 @@ for instance).</p>
</span><span class="c1"></span><span class="nx">fortiethPercentile</span> <span class="o">:=</span> <span class="nx">studentsByPercentile</span><span class="p">[</span><span class="mi">4</span><span class="p">]</span></code></pre></div>
</div>
</div>
+<div class='language-typescript snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-typescript" data-lang="typescript"><span class="kr">const</span> <span class="nx">deciles</span>: <span class="kt">PCollection</span><span class="o">&lt;</span><span class="nx">Student</span><span class="o">&gt;</span><span class="p">[]</span> <span class="o">=</span> <span class="nx">students</span><span class="p">.</span><span class="nx">apply [...]
+<span class="nx">beam</span><span class="p">.</span><span class="nx">partition</span><span class="p">(</span>
+<span class="p">(</span><span class="nx">student</span><span class="p">,</span> <span class="nx">numPartitions</span><span class="p">)</span> <span class="o">=&gt;</span>
+<span class="nb">Math</span><span class="p">.</span><span class="nx">floor</span><span class="p">((</span><span class="nx">getPercentile</span><span class="p">(</span><span class="nx">student</span><span class="p">)</span> <span class="o">/</span> <span class="mi">100</span><span class="p">)</span> <span class="o">*</span> <span class="nx">numPartitions</span><span class="p">),</span>
+<span class="mi">10</span>
+<span class="p">)</span>
+<span class="p">);</span>
+<span class="kr">const</span> <span class="nx">topDecile</span>: <span class="kt">PCollection</span><span class="o">&lt;</span><span class="nx">Student</span><span class="o">&gt;</span> <span class="o">=</span> <span class="nx">deciles</span><span class="p">[</span><span class="mi">9</span><span class="p">];</span>
+</code></pre></div>
+</div>
+</div>
<h3 id="requirements-for-writing-user-code-for-beam-transforms">4.3. Requirements for writing user code for Beam transforms</h3>
<p>When you build user code for a Beam transform, you should keep in mind the
distributed nature of execution. For example, there might be many copies of your
@@ -6367,7 +6805,21 @@ however, your subclass must not add any non-serializable members.</span>
they are registered with <code>register.FunctionXxY</code> (for simple functions) or
<code>register.DoFnXxY</code> (for sturctural DoFns), and are not closures. Structural
<code>DoFn</code>s will have all exported fields serialized. Unexported fields are unable to
-be serialized, and will be silently ignored.</span></p>
+be serialized, and will be silently ignored.</span>
+<span class="language-typescript">
+The Typescript SDK use <a href="https://github.com/nokia/ts-serialize-closures">ts-serialize-closures</a>
+to serialize functions (and other objects).
+This works out of the box for functions that are not closures, and also works
+for closures as long as the function in question (and any closures it references)
+are compiled with the
+<a href="https://github.com/apache/beam/blob/master/sdks/typescript/tsconfig.json"><code>ts-closure-transform</code> hooks</a>
+(e.g. by using <code>ttsc</code> in place of <code>tsc</code>).
+One can alternatively call
+<code>requireForSerialization(&quot;importableModuleDefiningFunc&quot;, {func})</code>
+to <a href="https://github.com/apache/beam/blob/master/sdks/typescript/src/apache_beam/serialization.ts">register a function directly</a> by name which can be less error-prone.
+Note that if, as is often the case in Javascript, <code>func</code> returns objects that
+contain closures, it is not sufficient to register <code>func</code> alone&ndash;its return
+value must be registered if used.</span></p>
<p>Some other serializability factors you should keep in mind are:</p>
<ul>
<li><span class="language-java language-py">Transient</span><span class="langauage-go">Unexported</span>
@@ -6549,6 +7001,43 @@ function. This optimizes runtime execution of the iterable.</p>
</span><span class="c1"></span><span class="o">//</span> <span class="nx">on</span> <span class="nx">how</span> <span class="nx">to</span> <span class="nx">contribute</span> <span class="nx">them</span><span class="p">!</span></code></pre></div>
</div>
</div>
+<div class='language-typescript snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-typescript" data-lang="typescript"><span class="c1">// Side inputs are provided by passing an extra context object to
+</span><span class="c1">// `map`, `flatMap`, or `parDo` transforms. This object will get passed as an
+</span><span class="c1">// extra argument to the provided function (or `process` method of the `DoFn`).
+</span><span class="c1">// `SideInputParam` properties (generally created with `pardo.xxxSideInput(...)`)
+</span><span class="c1">// have a `lookup` method that can be invoked from within the process method.
+</span><span class="c1"></span>
+<span class="c1">// Let words be a PCollection of strings.
+</span><span class="c1"></span><span class="kr">const</span> <span class="nx">words</span> : <span class="kt">PCollection</span><span class="o">&lt;</span><span class="kt">string</span><span class="o">&gt;</span> <span class="o">=</span> <span class="p">...</span>
+<span class="c1">// meanLengthPColl will contain a single number whose value is the
+</span><span class="c1">// average length of the words
+</span><span class="c1"></span><span class="kr">const</span> <span class="nx">meanLengthPColl</span>: <span class="kt">PCollection</span><span class="o">&lt;</span><span class="kt">number</span><span class="o">&gt;</span> <span class="o">=</span> <span class="nx">words</span>
+<span class="p">.</span><span class="nx">apply</span><span class="p">(</span>
+<span class="nx">beam</span>
+<span class="p">.</span><span class="nx">groupGlobally</span><span class="o">&lt;</span><span class="kt">string</span><span class="o">&gt;</span><span class="p">()</span>
+<span class="p">.</span><span class="nx">combining</span><span class="p">((</span><span class="nx">word</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="nx">word</span><span class="p">.</span><span class="nx">length</span><span class="p">,</span> <span class="nx">combiners</span><span class="p">.</span><span class="nx">mean</span><span class="p">,</span> <span class="s2">&am [...]
+<span class="p">)</span>
+<span class="p">.</span><span class="nx">map</span><span class="p">(({</span> <span class="nx">mean</span> <span class="p">})</span> <span class="o">=&gt;</span> <span class="nx">mean</span><span class="p">);</span>
+<span class="c1">// Now we use this as a side input to yield only words that are
+</span><span class="c1">// smaller than average.
+</span><span class="c1"></span><span class="kr">const</span> <span class="nx">smallWords</span> <span class="o">=</span> <span class="nx">words</span><span class="p">.</span><span class="nx">flatMap</span><span class="p">(</span>
+<span class="c1">// This is the function, taking context as a second argument.
+</span><span class="c1"></span> <span class="kd">function</span><span class="o">*</span> <span class="nx">keepSmall</span><span class="p">(</span><span class="nx">word</span><span class="p">,</span> <span class="nx">context</span><span class="p">)</span> <span class="p">{</span>
+<span class="k">if</span> <span class="p">(</span><span class="nx">word</span><span class="p">.</span><span class="nx">length</span> <span class="o">&lt;</span> <span class="nx">context</span><span class="p">.</span><span class="nx">meanLength</span><span class="p">.</span><span class="nx">lookup</span><span class="p">())</span> <span class="p">{</span>
+<span class="nx">yield</span> <span class="nx">word</span><span class="p">;</span>
+<span class="p">}</span>
+<span class="p">},</span>
+<span class="c1">// This is the context that will be passed as a second argument.
+</span><span class="c1"></span> <span class="p">{</span> <span class="nx">meanLength</span>: <span class="kt">pardo.singletonSideInput</span><span class="p">(</span><span class="nx">meanLengthPColl</span><span class="p">)</span> <span class="p">}</span>
+<span class="p">);</span>
+</code></pre></div>
+</div>
+</div>
<h4 id="side-inputs-windowing">4.4.2. Side inputs and windowing</h4>
<p>A windowed <code>PCollection</code> may be infinite and thus cannot be compressed into a
single value (or single collection class). When you create a <code>PCollectionView</code>
@@ -6584,7 +7073,17 @@ If you choose to have multiple outputs, your <code>DoFn</code> needs to be
function that matches the number of outputs. <code>beam.ParDo2</code> for two output <code>PCollection</code>s,
<code>beam.ParDo3</code> for three and so on until <code>beam.ParDo7</code>. If you need more, you can
use <code>beam.ParDoN</code> which will return a <code>[]beam.PCollection</code>.</p>
+<p class="language-typescript">While <code>ParDo</code> always produces a main output <code>PCollection</code> (as the return value
+from <code>apply</code>). If you want to have multiple outputs, emit an object with distinct
+properties in your <code>ParDo</code> operation and follow this operation with a <code>Split</code>
+to break it into multiple <code>PCollection</code>s.</p>
<h4 id="output-tags">4.5.1. Tags for multiple outputs</h4>
+<p class="language-typescript">The <code>Split</code> PTransform will take a PCollection of elements of the form
+<code>{tagA?: A, tagB?: B, ...}</code> and return a object
+<code>{tagA: PCollection&lt;A&gt;, tagB: PCollection&lt;B&gt;, ...}</code>.
+The set of expected tags is passed to the operation; how multiple or
+unknown tags are handled can be specified by passing a non-default
+<code>SplitOptions</code> instance.</p>
<p class="language-go">The Go SDK doesn&rsquo;t use output tags, and instead uses positional ordering for
multiple output PCollections.</p>
<div class='language-java snippet'>
@@ -6678,13 +7177,24 @@ multiple output PCollections.</p>
</span><span class="c1"></span><span class="nx">length</span><span class="p">,</span> <span class="nx">mixedMarked</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo2</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">processWordsMixed</span><span class="p">,</span> <span class="nx">words</span><sp [...]
</div>
</div>
+<div class='language-typescript snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-typescript" data-lang="typescript"><span class="err">#</span> <span class="nx">Create</span> <span class="nx">three</span> <span class="nx">PCollections</span> <span class="nx">from</span> <span class="nx">a</span> <span class="nx">single</span> <span class="nx">input</span> <span class="nx">PCollection</span><span class="p">.</span>
+<span class="kr">const</span> <span class="p">{</span> <span class="nx">below</span><span class="p">,</span> <span class="nx">above</span><span class="p">,</span> <span class="nx">marked</span> <span class="p">}</span> <span class="o">=</span> <span class="nx">to_split</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span>
+<span class="nx">beam</span><span class="p">.</span><span class="nx">split</span><span class="p">([</span><span class="s2">&#34;below&#34;</span><span class="p">,</span> <span class="s2">&#34;above&#34;</span><span class="p">,</span> <span class="s2">&#34;marked&#34;</span><span class="p">])</span>
+<span class="p">);</span>
+</code></pre></div>
+</div>
+</div>
<h4 id="multiple-outputs-dofn">4.5.2. Emitting to multiple outputs in your DoFn</h4>
-<p class="language-go"><p>Call emitter functions as needed to produce 0 or more elements for its matching
+<p class="language-go">Call emitter functions as needed to produce 0 or more elements for its matching
<code>PCollection</code>. The same value can be emitted with multiple emitters.
As normal, do not mutate values after emitting them from any emitter.</p>
-<p>All emitters should be registered using a generic <code>register.EmitterX[...]</code>
+<p class="language-go">All emitters should be registered using a generic <code>register.EmitterX[...]</code>
function. This optimizes runtime execution of the emitter.</p>
-</p>
<p class="language-go">DoFns can also return a single element via the standard return.
The standard return is always the first PCollection returned from beam.ParDo.
Other emitters output to their own PCollections in their defined parameter order.</p>
@@ -6781,11 +7291,31 @@ Other emitters output to their own PCollections in their defined parameter order
<span class="p">}</span></code></pre></div>
</div>
</div>
+<div class='language-typescript snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-typescript" data-lang="typescript"><span class="kr">const</span> <span class="nx">to_split</span> <span class="o">=</span> <span class="nx">words</span><span class="p">.</span><span class="nx">flatMap</span><span class="p">(</span><span class="kd">function</span><span class="o">*</span> <span class="p">(</span><span class="nx">word</span><span cla [...]
+<span class="k">if</span> <span class="p">(</span><span class="nx">word</span><span class="p">.</span><span class="nx">length</span> <span class="o">&lt;</span> <span class="mi">5</span><span class="p">)</span> <span class="p">{</span>
+<span class="nx">yield</span> <span class="p">{</span> <span class="nx">below</span>: <span class="kt">word</span> <span class="p">};</span>
+<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
+<span class="nx">yield</span> <span class="p">{</span> <span class="nx">above</span>: <span class="kt">word</span> <span class="p">};</span>
+<span class="p">}</span>
+<span class="k">if</span> <span class="p">(</span><span class="nx">isMarkedWord</span><span class="p">(</span><span class="nx">word</span><span class="p">))</span> <span class="p">{</span>
+<span class="nx">yield</span> <span class="p">{</span> <span class="nx">marked</span>: <span class="kt">word</span> <span class="p">};</span>
+<span class="p">}</span>
+<span class="p">});</span>
+</code></pre></div>
+</div>
+</div>
<h4 id="other-dofn-parameters">4.5.3. Accessing additional parameters in your DoFn</h4>
<p class="language-java">In addition to the element and the <code>OutputReceiver</code>, Beam will populate other parameters to your DoFn&rsquo;s <code>@ProcessElement</code> method.
Any combination of these parameters can be added to your process method in any order.</p>
<p class="language-py">In addition to the element, Beam will populate other parameters to your DoFn&rsquo;s <code>process</code> method.
Any combination of these parameters can be added to your process method in any order.</p>
+<p class="language-typescript">In addition to the element, Beam will populate other parameters to your DoFn&rsquo;s <code>process</code> method.
+These are available by placing accessors in the context argument, just as for side inputs.</p>
<p class="language-go">In addition to the element, Beam will populate other parameters to your DoFn&rsquo;s <code>ProcessElement</code> method.
Any combination of these parameters can be added to your process method in a standard order.</p>
<p class="language-go"><strong>context.Context:</strong>
@@ -6805,6 +7335,8 @@ To access the timestamp of an input element, add a parameter annotated with <
To access the timestamp of an input element, add a keyword parameter default to <code>DoFn.TimestampParam</code>. For example:</p>
<p class="language-go"><strong>Timestamp:</strong>
To access the timestamp of an input element, add a <code>beam.EventTime</code> parameter before the element. For example:</p>
+<p class="language-typescript"><strong>Timestamp:</strong>
+To access the window an input element falls into, add a <code>pardo.windowParam()</code> to the context argument.</p>
<div class='language-java snippet'>
<div class="notebook-skip code-snippet">
<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
@@ -6835,6 +7367,18 @@ To access the timestamp of an input element, add a <code>beam.EventTime</c
<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="kd">func</span> <span class="nf">MyDoFn</span><span class="p">(</span><span class="nx">ts</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">EventTime</span><span class="p">,</span> <span class="nx">word</span> <span class="kt">string</span><span class="p">)</span> <span class="kt">string< [...]
</div>
</div>
+<div class='language-typescript snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-typescript" data-lang="typescript"><span class="kd">function</span> <span class="nx">processFn</span><span class="p">(</span><span class="nx">element</span><span class="p">,</span> <span class="nx">context</span><span class="p">)</span> <span class="p">{</span>
+<span class="k">return</span> <span class="nx">context</span><span class="p">.</span><span class="nx">timestamp</span><span class="p">.</span><span class="nx">lookup</span><span class="p">();</span>
+<span class="p">}</span>
+<span class="nx">pcoll</span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="nx">processFn</span><span class="p">,</span> <span class="p">{</span> <span class="nx">timestamp</span>: <span class="kt">pardo.timestampParam</span><span class="p">()</span> <span class="p">});</span>
+</code></pre></div>
+</div>
+</div>
<p class="language-java"><strong>Window:</strong>
To access the window an input element falls into, add a parameter of the type of the window used for the input <code>PCollection</code>.
If the parameter is a window type (a subclass of <code>BoundedWindow</code>) that does not match the input <code>PCollection</code>, then an error
@@ -6851,6 +7395,10 @@ If an element falls in multiple windows (for example, this will happen when usin
then the <code>ProcessElement</code> method will be invoked multiple time for the element, once for each window.
Since <code>beam.Window</code> is an interface it&rsquo;s possible to type assert to the concrete implementation of the window.
For example, when fixed windows are being used, the window is of type <code>window.IntervalWindow</code>.</p>
+<p class="language-typescript"><strong>Window:</strong>
+To access the window an input element falls into, add a <code>pardo.windowParam()</code> to the context argument.
+If an element falls in multiple windows (for example, this will happen when using <code>SlidingWindows</code>), then the
+function will be invoked multiple time for the element, once for each window.</p>
<div class='language-java snippet'>
<div class="notebook-skip code-snippet">
<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
@@ -6884,6 +7432,15 @@ For example, when fixed windows are being used, the window is of type <code>w
<span class="p">}</span></code></pre></div>
</div>
</div>
+<div class='language-typescript snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-typescript" data-lang="typescript"><span class="nx">pcoll</span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="nx">processFn</span><span class="p">,</span> <span class="p">{</span> <span class="nx">timestamp</span>: <span class="kt">pardo.windowParam</span><span class="p">()</span> <span class="p">});</span>
+</code></pre></div>
+</div>
+</div>
<p class="language-java"><strong>PaneInfo:</strong>
When triggers are used, Beam provides a <code>PaneInfo</code> object that contains information about the current firing. Using <code>PaneInfo</code>
you can determine whether this is an early or a late firing, and how many times this window has already fired for this key.</p>
@@ -6894,6 +7451,10 @@ This feature implementation in Python SDK is not fully completed; see more at &l
<p class="language-go"><strong>PaneInfo:</strong>
When triggers are used, Beam provides <code>beam.PaneInfo</code> object that contains information about the current firing. Using <code>beam.PaneInfo</code>
you can determine whether this is an early or a late firing, and how many times this window has already fired for this key.</p>
+<p class="language-typescript"><strong>Window:</strong>
+To access the window an input element falls into, add a <code>pardo.paneInfoParam()</code> to the context argument.
+Using <code>beam.PaneInfo</code> you can determine whether this is an early or a late firing,
+and how many times this window has already fired for this key.</p>
<div class='language-java snippet'>
<div class="notebook-skip code-snippet">
<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
@@ -6941,6 +7502,15 @@ you can determine whether this is an early or a late firing, and how many times
<span class="p">}</span></code></pre></div>
</div>
</div>
+<div class='language-typescript snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-typescript" data-lang="typescript"><span class="nx">pcoll</span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="nx">processFn</span><span class="p">,</span> <span class="p">{</span> <span class="nx">timestamp</span>: <span class="kt">pardo.paneInfoParam</span><span class="p">()</span> <span class="p">});</span>
+</code></pre></div>
+</div>
+</div>
<p class="language-java"><strong>PipelineOptions:</strong>
The <code>PipelineOptions</code> for the current pipeline can always be accessed in a process method by adding it
as a parameter:</p>
@@ -6965,6 +7535,9 @@ Timers and States are explained in more detail in the
<a href="/blog/2017/08/28/timely-processing.html">Timely (and Stateful) Processing with Apache Beam</a> blog post.</p>
<p class="language-go"><strong>Timer and State:</strong>
This feature isn&rsquo;t implemented in the Go SDK; see more at <a href="https://github.com/apache/beam/issues/20510">Issue 20510</a>. Once implemented, user defined Timer and State parameters can be used in a stateful DoFn.</p>
+<p class="language-typescript"><strong>Timer and State:</strong>
+This feature isn&rsquo;t implemented in the Typescript SDK yet,
+but can be used from a cross-language transform.</p>
<div class='language-py snippet'>
<div class="notebook-skip code-snippet">
<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
@@ -7117,11 +7690,30 @@ appeared in the original data.</li>
<span class="p">}</span></code></pre></div>
</div>
</div>
+<div class='language-typescript snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-typescript" data-lang="typescript"><span class="kd">function</span> <span class="nx">countWords</span><span class="p">(</span><span class="nx">lines</span>: <span class="kt">PCollection</span><span class="o">&lt;</span><span class="kt">string</span><span class="o">&gt;</span><span class="p">)</span> <span class="p">{</span>
+<span class="k">return</span> <span class="nx">lines</span> <span class="c1">//
+</span><span class="c1"></span> <span class="p">.</span><span class="nx">map</span><span class="p">((</span><span class="nx">s</span>: <span class="kt">string</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="nx">s</span><span class="p">.</span><span class="nx">toLowerCase</span><span class="p">())</span>
+<span class="p">.</span><span class="nx">flatMap</span><span class="p">(</span><span class="kd">function</span><span class="o">*</span> <span class="nx">splitWords</span><span class="p">(</span><span class="nx">line</span>: <span class="kt">string</span><span class="p">)</span> <span class="p">{</span>
+<span class="nx">yield</span><span class="o">*</span> <span class="nx">line</span><span class="p">.</span><span class="nx">split</span><span class="p">(</span><span class="sr">/[^a-z]+/</span><span class="p">);</span>
+<span class="p">})</span>
+<span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nx">countPerElement</span><span class="p">());</span>
+<span class="p">}</span>
+<span class="kr">const</span> <span class="nx">counted</span> <span class="o">=</span> <span class="nx">lines</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="nx">countWords</span><span class="p">);</span>
+</code></pre></div>
+</div>
+</div>
<blockquote>
<p><strong>Note:</strong> Because <code>Count</code> is itself a composite transform,
<code>CountWords</code> is also a nested composite transform.</p>
</blockquote>
<h4 id="composite-transform-creation">4.6.2. Creating a composite transform</h4>
+<p class="language-typescript">A PTransform in the Typescript SDK is simply a function that accepts and
+returns <code>PValue</code>s such as <code>PCollection</code>s.</p>
<p class="language-java language-py">To create your own composite transform, create a subclass of the <code>PTransform</code>
class and override the <code>expand</code> method to specify the actual processing logic.
You can then use this transform just as you would a built-in transform from the
diff --git a/website/generated-content/documentation/programming-guide/index.html b/website/generated-content/documentation/programming-guide/index.html
index 81505b2ee6b..5659b0353a3 100644
--- a/website/generated-content/documentation/programming-guide/index.html
+++ b/website/generated-content/documentation/programming-guide/index.html
@@ -28,7 +28,7 @@ guide to programmatically building your Beam pipeline. As the programming guide
is filled out, the text will include code samples in multiple languages to help
illustrate how to implement Beam concepts in your pipelines.</p><p>If you want a brief introduction to Beam’s basic concepts before reading the
programming guide, take a look at the
-<a href=/documentation/basics/>Basics of the Beam model</a> page.</p><nav class=language-switcher><strong>Adapt for:</strong><ul><li data-type=language-java class=active>Java SDK</li><li data-type=language-py>Python SDK</li><li data-type=language-go>Go SDK</li></ul></nav><p class=language-py>The Python SDK supports Python 3.7, 3.8, and 3.9.</p><p class=language-go>The Go SDK supports Go v1.18+. SDK release 2.32.0 is the last experimental version.</p><h2 id=overview>1. Overview</h2><p>To [...]
+<a href=/documentation/basics/>Basics of the Beam model</a> page.</p><nav class=language-switcher><strong>Adapt for:</strong><ul><li data-type=language-java class=active>Java SDK</li><li data-type=language-py>Python SDK</li><li data-type=language-go>Go SDK</li><li data-type=language-typescript>Typescript SDK</li></ul></nav><p class=language-py>The Python SDK supports Python 3.7, 3.8, and 3.9.</p><p class=language-go>The Go SDK supports Go v1.18+. SDK release 2.32.0 is the last experiment [...]
of the Beam SDKs. Your driver program <em>defines</em> your pipeline, including all of
the inputs, transforms, and outputs; it also sets execution options for your
pipeline (typically passed in using command-line options). These include the
@@ -81,7 +81,8 @@ class <code>Pipeline</code> (typically in the <code>main()</code> function). Whe
<code>Pipeline</code>, you’ll also need to set some <strong>configuration options</strong>. You can set
your pipeline’s configuration options programmatically, but it’s often easier to
set the options ahead of time (or read them from the command line) and pass them
-to the <code>Pipeline</code> object when you create the object.</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=c1>// Start by defining the options for the pipeline.
+to the <code>Pipeline</code> object when you create the object.</p><span class=language-typescript>A Pipeline in the Typescript API is simply a function that will be called
+with a single `root` object and is passed to a Runner's `run` method.</span><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=c1>// Start by defining the options for the pipeline.
</span><span class=c1></span><span class=n>PipelineOptions</span> <span class=n>options</span> <span class=o>=</span> <span class=n>PipelineOptionsFactory</span><span class=o>.</span><span class=na>create</span><span class=o>();</span>
<span class=c1>// Then create the pipeline.
@@ -93,7 +94,10 @@ to the <code>Pipeline</code> object when you create the object.</p><div class="l
</span><span class=c1></span><span class=nx>beam</span><span class=p>.</span><span class=nf>Init</span><span class=p>()</span>
<span class=c1>// Create the Pipeline object and root scope.
-</span><span class=c1></span><span class=nx>pipeline</span><span class=p>,</span> <span class=nx>scope</span> <span class=o>:=</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>NewPipelineWithRoot</span><span class=p>()</span></code></pre></div></div></div><h3 id=configuring-pipeline-options>2.1. Configuring pipeline options</h3><p>Use the pipeline options to configure different aspects of your pipeline, such
+</span><span class=c1></span><span class=nx>pipeline</span><span class=p>,</span> <span class=nx>scope</span> <span class=o>:=</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>NewPipelineWithRoot</span><span class=p>()</span></code></pre></div></div></div><div class="language-typescript snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a [...]
+ <span class=c1>// Use root to build a pipeline.
+</span><span class=c1></span><span class=p>});</span>
+</code></pre></div></div></div><h3 id=configuring-pipeline-options>2.1. Configuring pipeline options</h3><p>Use the pipeline options to configure different aspects of your pipeline, such
as the pipeline runner that will execute your pipeline and any runner-specific
configuration required by the chosen runner. Your pipeline options will
potentially include information such as your project ID or a location for
@@ -103,20 +107,29 @@ to a DoFn’s <code>@ProcessElement</code> method, it will be populated by t
setting the fields directly, the Beam SDKs include a command-line parser that
you can use to set fields in <code>PipelineOptions</code> using command-line arguments.</p><p class="language-java language-py">To read options from the command-line, construct your <code>PipelineOptions</code> object
as demonstrated in the following example code:</p><p class=language-go>Use Go flags to parse command line arguments to configure your pipeline. Flags must be parsed
-before <code>beam.Init()</code> is called.</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=n>PipelineOptions</span> <span class=n>options</span> <span class=o>=</span>
+before <code>beam.Init()</code> is called.</p><p class=language-typescript>Any Javascript object can be used as pipeline options.
+One can either construct one manually, but it is also common to pass an object
+created from command line options such as <code>yargs.argv</code>.</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=n>PipelineOptions</span> <span class=n>options</span> <span class=o>=</span>
<span class=n>PipelineOptionsFactory</span><span class=o>.</span><span class=na>fromArgs</span><span class=o>(</span><span class=n>args</span><span class=o>).</span><span class=na>withValidation</span><span class=o>().</span><span class=na>create</span><span class=o>();</span></code></pre></div></div></div><div class="language-py snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img sr [...]
<span class=n>beam_options</span> <span class=o>=</span> <span class=n>PipelineOptions</span><span class=p>()</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-go data-lang=go><span class=c1>// If beamx or Go flags are used, flags must b [...]
</span><span class=c1>// before beam.Init() is called.
-</span><span class=c1></span><span class=nx>flag</span><span class=p>.</span><span class=nf>Parse</span><span class=p>()</span></code></pre></div></div></div><p>This interprets command-line arguments that follow the format:</p><pre><code>--<option>=<value>
+</span><span class=c1></span><span class=nx>flag</span><span class=p>.</span><span class=nf>Parse</span><span class=p>()</span></code></pre></div></div></div><div class="language-typescript snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-typescript data-lang=typescript><span class=kr>const</span [...]
+ <span class=nx>runner</span><span class=o>:</span> <span class=s2>"default"</span><span class=p>,</span>
+ <span class=nx>project</span><span class=o>:</span> <span class=s2>"my_project"</span><span class=p>,</span>
+<span class=p>};</span>
+
+<span class=kr>const</span> <span class=nx>runner</span> <span class=o>=</span> <span class=nx>beam</span><span class=p>.</span><span class=nx>createRunner</span><span class=p>(</span><span class=nx>pipeline_options</span><span class=p>);</span>
+
+<span class=kr>const</span> <span class=nx>runnerFromCommandLineOptions</span> <span class=o>=</span> <span class=nx>beam</span><span class=p>.</span><span class=nx>createRunner</span><span class=p>(</span><span class=nx>yargs</span><span class=p>.</span><span class=nx>argv</span><span class=p>);</span>
+</code></pre></div></div></div><p>This interprets command-line arguments that follow the format:</p><pre><code>--<option>=<value>
</code></pre><span class=language-java><blockquote><p>Appending the method <code>.withValidation</code> will check for required
command-line arguments and validate argument values.</p></blockquote></span><p class="language-java language-py">Building your <code>PipelineOptions</code> this way lets you specify any of the options as
a command-line argument.</p><p class=language-go>Defining flag variables this way lets you specify any of the options as a command-line argument.</p><blockquote><p><strong>Note:</strong> The <a href=/get-started/wordcount-example>WordCount example pipeline</a>
demonstrates how to set pipeline options at runtime by using command-line
options.</p></blockquote><h4 id=creating-custom-options>2.1.2. Creating custom options</h4><p>You can add your own custom options in addition to the standard
-<code>PipelineOptions</code>. To add your own options, define an interface with getter and
-setter methods for each option, as in the following example for
-adding <code>input</code> and <code>output</code> custom options:</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=kd>public</span> <span class=kd>interface</span> <span class=nc>MyOptions</span> <span class=kd>extends</span> <span clas [...]
+<code>PipelineOptions</code>.<p class=language-java>To add your own options, define an interface with getter and
+setter methods for each option.</p>The following example shows how to add <code>input</code> and <code>output</code> custom options:</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=kd>public</span> <span class=kd>interface</span> <span [...]
<span class=n>String</span> <span class=nf>getInput</span><span class=o>();</span>
<span class=kt>void</span> <span class=nf>setInput</span><span class=o>(</span><span class=n>String</span> <span class=n>input</span><span class=o>);</span>
@@ -127,13 +140,16 @@ adding <code>input</code> and <code>output</code> custom options:</p><div class=
<span class=k>class</span> <span class=nc>MyOptions</span><span class=p>(</span><span class=n>PipelineOptions</span><span class=p>):</span>
<span class=nd>@classmethod</span>
<span class=k>def</span> <span class=nf>_add_argparse_args</span><span class=p>(</span><span class=bp>cls</span><span class=p>,</span> <span class=n>parser</span><span class=p>):</span>
- <span class=n>parser</span><span class=o>.</span><span class=n>add_argument</span><span class=p>(</span><span class=s1>'--input-file'</span><span class=p>)</span>
- <span class=n>parser</span><span class=o>.</span><span class=n>add_argument</span><span class=p>(</span><span class=s1>'--output-path'</span><span class=p>)</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-go data-lang=go><s [...]
+ <span class=n>parser</span><span class=o>.</span><span class=n>add_argument</span><span class=p>(</span><span class=s1>'--input'</span><span class=p>)</span>
+ <span class=n>parser</span><span class=o>.</span><span class=n>add_argument</span><span class=p>(</span><span class=s1>'--output'</span><span class=p>)</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-go data-lang=go><span c [...]
</span><span class=c1></span><span class=kd>var</span> <span class=p>(</span>
<span class=nx>input</span> <span class=p>=</span> <span class=nx>flag</span><span class=p>.</span><span class=nf>String</span><span class=p>(</span><span class=s>"input"</span><span class=p>,</span> <span class=s>""</span><span class=p>,</span> <span class=s>""</span><span class=p>)</span>
<span class=nx>output</span> <span class=p>=</span> <span class=nx>flag</span><span class=p>.</span><span class=nf>String</span><span class=p>(</span><span class=s>"output"</span><span class=p>,</span> <span class=s>""</span><span class=p>,</span> <span class=s>""</span><span class=p>)</span>
-<span class=p>)</span></code></pre></div></div></div><p>You can also specify a description, which appears when a user passes <code>--help</code> as
-a command-line argument, and a default value.</p><p>You set the description and default value using annotations, as follows:</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=kd>public</span> <span class=kd>interface</span> <span class=n [...]
+<span class=p>)</span></code></pre></div></div></div><div class="language-typescript snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-typescript data-lang=typescript><span class=kr>const</span> <span class=nx>options</span> <span class=o>=</span> <span class=nx>yargs</span><span class=p>.</span>< [...]
+</span><span class=c1></span>
+<span class=c1>// Use options.input and options.output during pipeline construction.
+</span></code></pre></div></div></div><p>You can also specify a description, which appears when a user passes <code>--help</code> as
+a command-line argument, and a default value.</p><p class="language-java language-py langauge-go">You set the description and default value using annotations, as follows:</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=kd>public</span> [...]
<span class=nd>@Description</span><span class=o>(</span><span class=s>"Input for the pipeline"</span><span class=o>)</span>
<span class=nd>@Default.String</span><span class=o>(</span><span class=s>"gs://my-bucket/input"</span><span class=o>)</span>
<span class=n>String</span> <span class=nf>getInput</span><span class=o>();</span>
@@ -149,16 +165,15 @@ a command-line argument, and a default value.</p><p>You set the description and
<span class=nd>@classmethod</span>
<span class=k>def</span> <span class=nf>_add_argparse_args</span><span class=p>(</span><span class=bp>cls</span><span class=p>,</span> <span class=n>parser</span><span class=p>):</span>
<span class=n>parser</span><span class=o>.</span><span class=n>add_argument</span><span class=p>(</span>
- <span class=s1>'--input-file'</span><span class=p>,</span>
+ <span class=s1>'--input'</span><span class=p>,</span>
<span class=n>default</span><span class=o>=</span><span class=s1>'gs://dataflow-samples/shakespeare/kinglear.txt'</span><span class=p>,</span>
<span class=n>help</span><span class=o>=</span><span class=s1>'The file path for the input text to process.'</span><span class=p>)</span>
<span class=n>parser</span><span class=o>.</span><span class=n>add_argument</span><span class=p>(</span>
- <span class=s1>'--output-path'</span><span class=p>,</span>
- <span class=n>required</span><span class=o>=</span><span class=bp>True</span><span class=p>,</span>
- <span class=n>help</span><span class=o>=</span><span class=s1>'The path prefix for output files.'</span><span class=p>)</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-go data-lang=go><span class=kd>var</span> <span cla [...]
+ <span class=s1>'--output'</span><span class=p>,</span> <span class=n>required</span><span class=o>=</span><span class=bp>True</span><span class=p>,</span> <span class=n>help</span><span class=o>=</span><span class=s1>'The path prefix for output files.'</span><span class=p>)</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Co [...]
<span class=nx>input</span> <span class=p>=</span> <span class=nx>flag</span><span class=p>.</span><span class=nf>String</span><span class=p>(</span><span class=s>"input"</span><span class=p>,</span> <span class=s>"gs://my-bucket/input"</span><span class=p>,</span> <span class=s>"Input for the pipeline"</span><span class=p>)</span>
<span class=nx>output</span> <span class=p>=</span> <span class=nx>flag</span><span class=p>.</span><span class=nf>String</span><span class=p>(</span><span class=s>"output"</span><span class=p>,</span> <span class=s>"gs://my-bucket/output"</span><span class=p>,</span> <span class=s>"Output for the pipeline"</span><span class=p>)</span>
-<span class=p>)</span></code></pre></div></div></div><p class=language-java>It’s recommended that you register your interface with <code>PipelineOptionsFactory</code>
+<span class=p>)</span></code></pre></div></div></div><p class=language-py>For Python, you can also simply parse your custom options with argparse; there
+is no need to create a separate PipelineOptions subclass.</p><p class=language-java>It’s recommended that you register your interface with <code>PipelineOptionsFactory</code>
and then pass the interface when creating the <code>PipelineOptions</code> object. When you
register your interface with <code>PipelineOptionsFactory</code>, the <code>--help</code> can find
your custom options interface and add it to the output of the <code>--help</code> command.
@@ -189,12 +204,16 @@ represent the data records in that source.</p><p>Each data source adapter has a
transform to the <code>Pipeline</code> object itself.
<span class=language-java><code>TextIO.Read</code></span>
<span class=language-py><code>io.TextFileSource</code></span>
-<span class=language-go><code>textio.Read</code></span>, for example, reads from an
+<span class=language-go><code>textio.Read</code></span>
+<span class=language-typescript><code>textio.ReadFromText</code></span>,
+for example, reads from an
external text file and returns a <code>PCollection</code> whose elements are of type
<code>String</code>, each <code>String</code> represents one line from the text file. Here’s how you
would apply <span class=language-java><code>TextIO.Read</code></span>
<span class=language-py><code>io.TextFileSource</code></span>
-<span class=language-go><code>textio.Read</code></span> to your <code>Pipeline</code> to create
+<span class=language-go><code>textio.Read</code></span>
+<span class=language-typescript><code>textio.ReadFromText</code></span>
+to your <code>Pipeline</code> <span class=language-typescript>root</span> to create
a <code>PCollection</code>:</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=kd>public</span> <span class=kd>static</span> <span class=kt>void</span> <span class=nf>main</span><span class=o>(</span><span class=n>String</span><span class [...]
<span class=c1>// Create the pipeline.
</span><span class=c1></span> <span class=n>PipelineOptions</span> <span class=n>options</span> <span class=o>=</span>
@@ -209,7 +228,12 @@ a <code>PCollection</code>:</p><div class="language-java snippet"><div class="no
</span><span class=c1>// the lines as a PCollection<string>.
</span><span class=c1>// Notice the scope as the first variable when calling
</span><span class=c1>// the method as is needed when calling all transforms.
-</span><span class=c1></span><span class=nx>lines</span> <span class=o>:=</span> <span class=nx>textio</span><span class=p>.</span><span class=nf>Read</span><span class=p>(</span><span class=nx>scope</span><span class=p>,</span> <span class=s>"gs://some/inputData.txt"</span><span class=p>)</span></code></pre></div></div></div><p>See the <a href=#pipeline-io>section on I/O</a> to learn more about how to read from the
+</span><span class=c1></span><span class=nx>lines</span> <span class=o>:=</span> <span class=nx>textio</span><span class=p>.</span><span class=nf>Read</span><span class=p>(</span><span class=nx>scope</span><span class=p>,</span> <span class=s>"gs://some/inputData.txt"</span><span class=p>)</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Co [...]
+ <span class=c1>// Note that textio.ReadFromText is an AsyncPTransform.
+</span><span class=c1></span> <span class=kd>const</span> <span class=nx>pcoll</span><span class=p>:</span> <span class=nx>PCollection</span><span class=p><</span><span class=kt>string</span><span class=p>></span> <span class=p>=</span> <span class=nx>await</span> <span class=nx>root</span><span class=p>.</span><span class=nf>asyncApply</span><span class=p>(</span>
+ <span class=nx>textio</span><span class=p>.</span><span class=nf>ReadFromText</span><span class=p>(</span><span class=s>"path/to/text_pattern"</span><span class=p>)</span>
+ <span class=p>);</span>
+<span class=p>}</span></code></pre></div></div></div><p>See the <a href=#pipeline-io>section on I/O</a> to learn more about how to read from the
various data sources supported by the Beam SDK.</p><h4 id=creating-pcollection-in-memory>3.1.2. Creating a PCollection from in-memory data</h4><p class=language-java>To create a <code>PCollection</code> from an in-memory Java <code>Collection</code>, you use the
Beam-provided <code>Create</code> transform. Much like a data adapter’s <code>Read</code>, you apply
<code>Create</code> directly to your <code>Pipeline</code> object itself.</p><p class=language-java>As parameters, <code>Create</code> accepts the Java <code>Collection</code> and a <code>Coder</code> object. The
@@ -217,8 +241,12 @@ Beam-provided <code>Create</code> transform. Much like a data adapter’s <c
<a href=#element-type>encoded</a>.</p><p class=language-py>To create a <code>PCollection</code> from an in-memory <code>list</code>, you use the Beam-provided
<code>Create</code> transform. Apply this transform directly to your <code>Pipeline</code> object
itself.</p><p class=language-go>To create a <code>PCollection</code> from an in-memory <code>slice</code>, you use the Beam-provided
-<code>beam.CreateList</code> transform. Pass the pipeline <code>scope</code>, and the <code>slice</code> to this transform.</p><p>The following example code shows how to create a <code>PCollection</code> from an in-memory
-<span class=language-java><code>List</code></span><span class=language-py><code>list</code></span><span class=language-go><code>slice</code></span>:</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=kd>public</span> <span class=kd>static [...]
+<code>beam.CreateList</code> transform. Pass the pipeline <code>scope</code>, and the <code>slice</code> to this transform.</p><p class=language-typescript>To create a <code>PCollection</code> from an in-memory <code>array</code>, you use the Beam-provided
+<code>Create</code> transform. Apply this transform directly to your <code>Root</code> object.</p><p>The following example code shows how to create a <code>PCollection</code> from an in-memory
+<span class=language-java><code>List</code></span>
+<span class=language-py><code>list</code></span>
+<span class=language-go><code>slice</code></span>
+<span class=language-typescript><code>array</code></span>:</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=kd>public</span> <span class=kd>static</span> <span class=kt>void</span> <span class=nf>main</span><span class=o>(</span><span c [...]
<span class=c1>// Create a Java Collection, in this case a List of Strings.
</span><span class=c1></span> <span class=kd>final</span> <span class=n>List</span><span class=o><</span><span class=n>String</span><span class=o>></span> <span class=n>LINES</span> <span class=o>=</span> <span class=n>Arrays</span><span class=o>.</span><span class=na>asList</span><span class=o>(</span>
<span class=s>"To be, or not to be: that is the question: "</span><span class=o>,</span>
@@ -258,7 +286,17 @@ itself.</p><p class=language-go>To create a <code>PCollection</code> from an in-
<span class=c1>// Pass the slice to beam.CreateList, to create the pcollection.
</span><span class=c1>// The scope variable s is used to add the CreateList transform
</span><span class=c1>// to the pipeline.
-</span><span class=c1></span><span class=nx>linesPCol</span> <span class=o>:=</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>CreateList</span><span class=p>(</span><span class=nx>s</span><span class=p>,</span> <span class=nx>lines</span><span class=p>)</span></code></pre></div></div></div><h3 id=pcollection-characteristics>3.2. PCollection characteristics</h3><p>A <code>PCollection</code> is owned by the specific <code>Pipeline</code> object for which it is
+</span><span class=c1></span><span class=nx>linesPCol</span> <span class=o>:=</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>CreateList</span><span class=p>(</span><span class=nx>s</span><span class=p>,</span> <span class=nx>lines</span><span class=p>)</span></code></pre></div></div></div><div class="language-typescript snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard [...]
+ <span class=kr>const</span> <span class=nx>pcoll</span> <span class=o>=</span> <span class=nx>root</span><span class=p>.</span><span class=nx>apply</span><span class=p>(</span>
+ <span class=nx>beam</span><span class=p>.</span><span class=nx>create</span><span class=p>([</span>
+ <span class=s2>"To be, or not to be: that is the question: "</span><span class=p>,</span>
+ <span class=s2>"Whether 'tis nobler in the mind to suffer "</span><span class=p>,</span>
+ <span class=s2>"The slings and arrows of outrageous fortune, "</span><span class=p>,</span>
+ <span class=s2>"Or to take arms against a sea of troubles, "</span><span class=p>,</span>
+ <span class=p>])</span>
+ <span class=p>);</span>
+<span class=p>}</span>
+</code></pre></div></div></div><h3 id=pcollection-characteristics>3.2. PCollection characteristics</h3><p>A <code>PCollection</code> is owned by the specific <code>Pipeline</code> object for which it is
created; multiple pipelines cannot share a <code>PCollection</code>.
<span class=language-java>In some respects, a <code>PCollection</code> functions like
a <code>Collection</code> class. However, a <code>PCollection</code> can differ in a few key ways:</span></p><h4 id=element-type>3.2.1. Element type</h4><p>The elements of a <code>PCollection</code> may be of any type, but must all be of the same
@@ -332,7 +370,9 @@ transform in the Beam SDKs has a generic <code>apply</code> method <span class=l
Invoking multiple Beam transforms is similar to <em>method chaining</em>, but with one
slight difference: You apply the transform to the input <code>PCollection</code>, passing
the transform itself as an argument, and the operation returns the output
-<code>PCollection</code>. This takes the general form:</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=o>[</span><span class=n>Output</span> <span class=n>PCollection</span><span class=o>]</span> <span class=o>=</span> <span class=o>[< [...]
+<code>PCollection</code>. This takes the general form:</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=o>[</span><span class=n>Output</span> <span class=n>PCollection</span><span class=o>]</span> <span class=o>=</span> <span class=o>[< [...]
+<span class=p>[</span><span class=nx>Output</span> <span class=nx>PCollection</span><span class=p>]</span> <span class=o>=</span> <span class=nx>await</span> <span class=p>[</span><span class=nx>Input</span> <span class=nx>PCollection</span><span class=p>].</span><span class=nx>asyncApply</span><span class=p>([</span><span class=nx>AsyncTransform</span><span class=p>])</span>
+</code></pre></div></div></div><p class="language-java language-py language-typescript">Because Beam uses a generic <code>apply</code> method for <code>PCollection</code>, you can both chain
transforms sequentially and also apply transforms that contain other transforms
nested within (called <a href=#composite-transforms>composite transforms</a> in the Beam
SDKs).</p><p class=language-go>It’s recommended to create a new variable for each new <code>PCollection</code> to
@@ -349,7 +389,10 @@ inputs and emit <code>PCollection</code> nodes as outputs.
<span class=o>|</span> <span class=p>[</span><span class=n>Second</span> <span class=n>Transform</span><span class=p>]</span>
<span class=o>|</span> <span class=p>[</span><span class=n>Third</span> <span class=n>Transform</span><span class=p>])</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-go data-lang=go><span class=p>[</span><span class=nx>S [...]
<span class=p>[</span><span class=nx>Third</span> <span class=nx>PCollection</span><span class=p>]</span> <span class=o>:=</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>ParDo</span><span class=p>(</span><span class=nx>scope</span><span class=p>,</span> <span class=p>[</span><span class=nx>Second</span> <span class=nx>Transform</span><span class=p>],</span> <span class=p>[</span><span class=nx>Second</span> <span class=nx>PCollection</span><span class=p>])</span>
-<span class=p>[</span><span class=nx>Final</span> <span class=nx>Output</span> <span class=nx>PCollection</span><span class=p>]</span> <span class=o>:=</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>ParDo</span><span class=p>(</span><span class=nx>scope</span><span class=p>,</span> <span class=p>[</span><span class=nx>Third</span> <span class=nx>Transform</span><span class=p>],</span> <span class=p>[</span><span class=nx>Third</span> <span class=nx>PCollection</spa [...]
+<span class=p>[</span><span class=nx>Final</span> <span class=nx>Output</span> <span class=nx>PCollection</span><span class=p>]</span> <span class=o>:=</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>ParDo</span><span class=p>(</span><span class=nx>scope</span><span class=p>,</span> <span class=p>[</span><span class=nx>Third</span> <span class=nx>Transform</span><span class=p>],</span> <span class=p>[</span><span class=nx>Third</span> <span class=nx>PCollection</spa [...]
+<span class=p>.</span><span class=nx>apply</span><span class=p>([</span><span class=nx>Second</span> <span class=nx>Transform</span><span class=p>])</span>
+<span class=p>.</span><span class=nx>apply</span><span class=p>([</span><span class=nx>Third</span> <span class=nx>Transform</span><span class=p>])</span>
+</code></pre></div></div></div><p>The graph of this pipeline looks like the following:</p><p><img src=/images/design-your-pipeline-linear.svg alt="This linear pipeline starts with one input collection, sequentially appliesthree transforms, and ends with one output collection."></p><p><em>Figure 1: A linear pipeline with three sequential transforms.</em></p><p>However, note that a transform <em>does not consume or otherwise alter</em> the input
collection — remember that a <code>PCollection</code> is immutable by definition. This means
that you can apply multiple transforms to the same input <code>PCollection</code> to create
a branching pipeline, like so:</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=o>[</span><span class=n>PCollection</span> <span class=n>of</span> <span class=n>database</span> <span class=n>table</span> <span class=n>rows</span><span c [...]
@@ -358,12 +401,25 @@ a branching pipeline, like so:</p><div class="language-java snippet"><div class=
<span class=p>[</span><span class=n>PCollection</span> <span class=n>of</span> <span class=s1>'A'</span> <span class=n>names</span><span class=p>]</span> <span class=o>=</span> <span class=p>[</span><span class=n>PCollection</span> <span class=n>of</span> <span class=n>database</span> <span class=n>table</span> <span class=n>rows</span><span class=p>]</span> <span class=o>|</span> <span class=p>[</span><span class=n>Transform</span> <span class=n>A</span><span class=p>]</span>
<span class=p>[</span><span class=n>PCollection</span> <span class=n>of</span> <span class=s1>'B'</span> <span class=n>names</span><span class=p>]</span> <span class=o>=</span> <span class=p>[</span><span class=n>PCollection</span> <span class=n>of</span> <span class=n>database</span> <span class=n>table</span> <span class=n>rows</span><span class=p>]</span> <span class=o>|</span> <span class=p>[</span><span class=n>Transform</span> <span class=n>B</span><span class=p>]</span></c [...]
<span class=p>[</span><span class=nx>PCollection</span> <span class=nx>of</span> <span class=sc>'A'</span> <span class=nx>names</span><span class=p>]</span> <span class=p>=</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>ParDo</span><span class=p>(</span><span class=nx>scope</span><span class=p>,</span> <span class=p>[</span><span class=nx>Transform</span> <span class=nx>A</span><span class=p>],</span> <span class=p>[</span><span class=nx>PCollection</span> [...]
-<span class=p>[</span><span class=nx>PCollection</span> <span class=nx>of</span> <span class=sc>'B'</span> <span class=nx>names</span><span class=p>]</span> <span class=p>=</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>ParDo</span><span class=p>(</span><span class=nx>scope</span><span class=p>,</span> <span class=p>[</span><span class=nx>Transform</span> <span class=nx>B</span><span class=p>],</span> <span class=p>[</span><span class=nx>PCollection</span> [...]
+<span class=p>[</span><span class=nx>PCollection</span> <span class=nx>of</span> <span class=sc>'B'</span> <span class=nx>names</span><span class=p>]</span> <span class=p>=</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>ParDo</span><span class=p>(</span><span class=nx>scope</span><span class=p>,</span> <span class=p>[</span><span class=nx>Transform</span> <span class=nx>B</span><span class=p>],</span> <span class=p>[</span><span class=nx>PCollection</span> [...]
+<span class=p>[</span><span class=nx>PCollection</span> <span class=nx>of</span> <span class=s1>'A'</span> <span class=nx>names</span><span class=p>]</span> <span class=o>=</span> <span class=p>[</span><span class=nx>PCollection</span> <span class=nx>of</span> <span class=nx>database</span> <span class=nx>table</span> <span class=nx>rows</span><span class=p>].</span><span class=nx>apply</span><span class=p>([</span><span class=nx>Transform</span> <span class=nx>A</span><span clas [...]
+<span class=p>[</span><span class=nx>PCollection</span> <span class=nx>of</span> <span class=s1>'B'</span> <span class=nx>names</span><span class=p>]</span> <span class=o>=</span> <span class=p>[</span><span class=nx>PCollection</span> <span class=nx>of</span> <span class=nx>database</span> <span class=nx>table</span> <span class=nx>rows</span><span class=p>].</span><span class=nx>apply</span><span class=p>([</span><span class=nx>Transform</span> <span class=nx>B</span><span clas [...]
+</code></pre></div></div></div><p>The graph of this branching pipeline looks like the following:</p><p><img src=/images/design-your-pipeline-multiple-pcollections.svg alt="This pipeline applies two transforms to a single input collection. Eachtransform produces an output collection."></p><p><em>Figure 2: A branching pipeline. Two transforms are applied to a single
PCollection of database table rows.</em></p><p>You can also build your own <a href=#composite-transforms>composite transforms</a> that
nest multiple transforms inside a single, larger transform. Composite transforms
are particularly useful for building a reusable sequence of simple steps that
-get used in a lot of different places.</p><h3 id=core-beam-transforms>4.2. Core Beam transforms</h3><p>Beam provides the following core transforms, each of which represents a different
-processing paradigm:</p><ul><li><code>ParDo</code></li><li><code>GroupByKey</code></li><li><code>CoGroupByKey</code></li><li><code>Combine</code></li><li><code>Flatten</code></li><li><code>Partition</code></li></ul><h4 id=pardo>4.2.1. ParDo</h4><p><code>ParDo</code> is a Beam transform for generic parallel processing. The <code>ParDo</code>
+get used in a lot of different places.</p><p class=language-python>The pipe syntax allows one to apply PTransforms to <code>tuple</code>s and <code>dict</code>s of
+PCollections as well for those transforms accepting multiple inputs (such as
+<code>Flatten</code> and <code>CoGroupByKey</code>).</p><p class=language-typescript>PTransforms can also be applied to any <code>PValue</code>, which include the Root object,
+PCollections, arrays of <code>PValue</code>s, and objects with <code>PValue</code> values.
+One can apply transforms to these composite types by wrapping them with
+<code>beam.P</code>, e.g.
+<code>beam.P({left: pcollA, right: pcollB}).apply(transformExpectingTwoPCollections)</code>.</p><p class=language-typescript>PTransforms come in two flavors, synchronous and asynchronous, depending on
+whether their <em>application</em>* involves asynchronous invocations.
+An <code>AsyncTransform</code> must be applied with <code>asyncApply</code> and returns a <code>Promise</code>
+which must be awaited before further pipeline construction.</p><h3 id=core-beam-transforms>4.2. Core Beam transforms</h3><p>Beam provides the following core transforms, each of which represents a different
+processing paradigm:</p><ul><li><code>ParDo</code></li><li><code>GroupByKey</code></li><li><code>CoGroupByKey</code></li><li><code>Combine</code></li><li><code>Flatten</code></li><li><code>Partition</code></li></ul><p class=language-typescript>The Typescript SDK provides some of the most basic of these transforms
+as methods on <code>PCollection</code> itself.</p><h4 id=pardo>4.2.1. ParDo</h4><p><code>ParDo</code> is a Beam transform for generic parallel processing. The <code>ParDo</code>
processing paradigm is similar to the “Map” phase of a
<a href=https://en.wikipedia.org/wiki/MapReduce>Map/Shuffle/Reduce</a>-style
algorithm: a <code>ParDo</code> transform considers each element in the input
@@ -449,7 +505,22 @@ as shown in the following example code:</p><div class="language-java snippet"><d
<span class=c1>// words is an input PCollection of strings
</span><span class=c1></span><span class=kd>var</span> <span class=nx>words</span> <span class=nx>beam</span><span class=p>.</span><span class=nx>PCollection</span> <span class=p>=</span> <span class=o>...</span>
-<span class=nx>wordLengths</span> <span class=o>:=</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>ParDo</span><span class=p>(</span><span class=nx>s</span><span class=p>,</span> <span class=o>&</span><span class=nx>ComputeWordLengthFn</span><span class=p>{},</span> <span class=nx>words</span><span class=p>)</span></code></pre></div></div></div><p>In the example, our input <code>PCollection</code> contains <span class="language-java language-py"><code>String</co [...]
+<span class=nx>wordLengths</span> <span class=o>:=</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>ParDo</span><span class=p>(</span><span class=nx>s</span><span class=p>,</span> <span class=o>&</span><span class=nx>ComputeWordLengthFn</span><span class=p>{},</span> <span class=nx>words</span><span class=p>)</span></code></pre></div></div></div><div class="language-typescript snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-togg [...]
+<span class=kr>const</span> <span class=nx>words</span> : <span class=kt>PCollection</span><span class=o><</span><span class=kt>string</span><span class=o>></span> <span class=o>=</span> <span class=p>...</span>
+
+<span class=err>#</span> <span class=nx>The</span> <span class=nx>DoFn</span> <span class=nx>to</span> <span class=nx>perform</span> <span class=nx>on</span> <span class=nx>each</span> <span class=nx>element</span> <span class=k>in</span> <span class=nx>the</span> <span class=nx>input</span> <span class=nx>PCollection</span><span class=p>.</span>
+
+<span class=kd>function</span> <span class=nx>computeWordLengthFn</span><span class=p>()</span><span class=o>:</span> <span class=nx>beam</span><span class=p>.</span><span class=nx>DoFn</span><span class=o><</span><span class=kt>string</span><span class=p>,</span> <span class=kt>number</span><span class=o>></span> <span class=p>{</span>
+ <span class=k>return</span> <span class=p>{</span>
+ <span class=nx>process</span>: <span class=kt>function</span><span class=o>*</span> <span class=p>(</span><span class=nx>element</span><span class=p>)</span> <span class=p>{</span>
+ <span class=nx>yield</span> <span class=nx>element</span><span class=p>.</span><span class=nx>length</span><span class=p>;</span>
+ <span class=p>},</span>
+ <span class=p>};</span>
+<span class=p>}</span>
+
+
+<span class=kr>const</span> <span class=nx>result</span> <span class=o>=</span> <span class=nx>words</span><span class=p>.</span><span class=nx>apply</span><span class=p>(</span><span class=nx>beam</span><span class=p>.</span><span class=nx>parDo</span><span class=p>(</span><span class=nx>computeWordLengthFn</span><span class=p>()));</span>
+</code></pre></div></div></div><p>In the example, our input <code>PCollection</code> contains <span class="language-java language-py"><code>String</code></span>
<span class=language-go><code>string</code></span> values. We apply a
<code>ParDo</code> transform that specifies a function (<code>ComputeWordLengthFn</code>) to compute
the length of each string, and outputs the result to a new <code>PCollection</code> of
@@ -494,7 +565,7 @@ elements, the method can also take a parameter of type <code>OutputReceiver</cod
provides a method for emitting elements. The parameter types must match the input
and output types of your <code>DoFn</code> or the framework will raise an error. Note: <code>@Element</code> and
<code>OutputReceiver</code> were introduced in Beam 2.5.0; if using an earlier release of Beam, a
-<code>ProcessContext</code> parameter should be used instead.</p><p class=language-py>Inside your <code>DoFn</code> subclass, you’ll write a method <code>process</code> where you provide
+<code>ProcessContext</code> parameter should be used instead.</p><p class="language-py language-typescript">Inside your <code>DoFn</code> subclass, you’ll write a method <code>process</code> where you provide
the actual processing logic. You don’t need to manually extract the elements
from the input collection; the Beam SDKs handle that for you. Your <code>process</code> method
should accept an argument <code>element</code>, which is the input element, and return an
@@ -530,7 +601,14 @@ or the framework will raise an error.</p><div class="language-java snippet"><div
</span><span class=c1></span> <span class=c1>// Input/output types are included in order in the brackets
</span><span class=c1></span> <span class=nx>register</span><span class=p>.</span><span class=nx>DoFn2x0</span><span class=p>[</span><span class=kt>string</span><span class=p>,</span> <span class=kd>func</span><span class=p>(</span><span class=kt>int</span><span class=p>)](</span><span class=o>&</span><span class=nx>ComputeWordLengthFn</span><span class=p>{})</span>
<span class=nx>register</span><span class=p>.</span><span class=nx>Emitter1</span><span class=p>[</span><span class=kt>int</span><span class=p>]()</span>
-<span class=p>}</span></code></pre></div></div></div><p class=language-go>Simple DoFns can also be written as functions.</p><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-go data-lang=go><span class=kd>func</span> <span class=nf>ComputeWordLengthFn</span><span class=p>(< [...]
+<span class=p>}</span></code></pre></div></div></div><div class="language-typescript snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-typescript data-lang=typescript><span class=kd>function</span> <span class=nx>computeWordLengthFn</span><span class=p>()</span><span class=o>:</span> <span class=n [...]
+ <span class=k>return</span> <span class=p>{</span>
+ <span class=nx>process</span>: <span class=kt>function</span><span class=o>*</span> <span class=p>(</span><span class=nx>element</span><span class=p>)</span> <span class=p>{</span>
+ <span class=nx>yield</span> <span class=nx>element</span><span class=p>.</span><span class=nx>length</span><span class=p>;</span>
+ <span class=p>},</span>
+ <span class=p>};</span>
+<span class=p>}</span>
+</code></pre></div></div></div><p class=language-go>Simple DoFns can also be written as functions.</p><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-go data-lang=go><span class=kd>func</span> <span class=nf>ComputeWordLengthFn</span><span class=p>(</span><span class=nx>w [...]
<span class=kd>func</span> <span class=nf>init</span><span class=p>()</span> <span class=p>{</span>
<span class=c1>// 2 inputs and 0 outputs => DoFn2x0
@@ -555,7 +633,7 @@ serialize and cache the values in your pipeline. Your method should meet the
following requirements:</p><span class=language-java><ul><li>You should not in any way modify an element returned by
the <code>@Element</code> annotation or <code>ProcessContext.sideInput()</code> (the incoming
elements from the input collection).</li><li>Once you output a value using <code>OutputReceiver.output()</code> you should not modify
-that value in any way.</li></ul></span><span class=language-py><ul><li>You should not in any way modify the <code>element</code> argument provided to the
+that value in any way.</li></ul></span><span class="language-py language-typescript"><ul><li>You should not in any way modify the <code>element</code> argument provided to the
<code>process</code> method, or any side inputs.</li><li>Once you output a value using <code>yield</code> or <code>return</code>, you should not modify
that value in any way.</li></ul></span><span class=language-go><ul><li>You should not in any way modify the parameters provided to the
<code>ProcessElement</code> method, or any side inputs.</li><li>Once you output a value using an <code>emitter function</code>, you should not modify
@@ -563,11 +641,13 @@ that value in any way.</li></ul></span><h5 id=lightweight-dofns>4.2.1.3. Lightwe
<code>ParDo</code> by providing a lightweight <code>DoFn</code> in-line, as
<span class=language-java>an anonymous inner class instance</span>
<span class=language-py>a lambda function</span>
-<span class=language-go>an anonymous function</span>.</p><p>Here’s the previous example, <code>ParDo</code> with <code>ComputeLengthWordsFn</code>, with the
+<span class=language-go>an anonymous function</span>
+<span class=language-typescript>a function passed to <code>PCollection.map</code> or <code>PCollection.flatMap</code></span>.</p><p>Here’s the previous example, <code>ParDo</code> with <code>ComputeLengthWordsFn</code>, with the
<code>DoFn</code> specified as
<span class=language-java>an anonymous inner class instance</span>
<span class=language-py>a lambda function</span>
-<span class=language-go>an anonymous function</span>:</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=c1>// The input PCollection.
+<span class=language-go>an anonymous function</span>
+<span class=language-typescript>a function</span>:</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=c1>// The input PCollection.
</span><span class=c1></span><span class=n>PCollection</span><span class=o><</span><span class=n>String</span><span class=o>></span> <span class=n>words</span> <span class=o>=</span> <span class=o>...;</span>
<span class=c1>// Apply a ParDo with an anonymous DoFn to the PCollection words.
@@ -590,14 +670,18 @@ that value in any way.</li></ul></span><h5 id=lightweight-dofns>4.2.1.3. Lightwe
<span class=nx>lengths</span> <span class=o>:=</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>ParDo</span><span class=p>(</span><span class=nx>s</span><span class=p>,</span> <span class=kd>func</span> <span class=p>(</span><span class=nx>word</span> <span class=kt>string</span><span class=p>,</span> <span class=nx>emit</span> <span class=kd>func</span><span class=p>(</span><span class=kt>int</span><span class=p>))</span> <span class=p>{</span>
<span class=nf>emit</span><span class=p>(</span><span class=nb>len</span><span class=p>(</span><span class=nx>word</span><span class=p>))</span>
-<span class=p>},</span> <span class=nx>words</span><span class=p>)</span></code></pre></div></div></div><p>If your <code>ParDo</code> performs a one-to-one mapping of input elements to output
+<span class=p>},</span> <span class=nx>words</span><span class=p>)</span></code></pre></div></div></div><div class="language-typescript snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-typescript data-lang=typescript><span class=c1>// The input PCollection of strings.
+</span><span class=c1></span><span class=nx>words</span> <span class=o>=</span> <span class=p>...</span>
+
+<span class=kr>const</span> <span class=nx>result</span> <span class=o>=</span> <span class=nx>words</span><span class=p>.</span><span class=nx>flatMap</span><span class=p>((</span><span class=nx>word</span><span class=p>)</span> <span class=o>=></span> <span class=p>[</span><span class=nx>word</span><span class=p>.</span><span class=nx>length</span><span class=p>]);</span>
+</code></pre></div></div></div><p>If your <code>ParDo</code> performs a one-to-one mapping of input elements to output
elements–that is, for each input element, it applies a function that produces
<em>exactly one</em> output element, <span class=language-go>you can return that
element directly.</span><span class="language-java language-py">you can use the higher-level
-<span class=language-java><code>MapElements</code></span><span class=language-py><code>Map</code></span>
+<span class=language-java><code>MapElements</code></span><span class="language-py language-py"><code>Map</code></span>
transform.</span><span class=language-java><code>MapElements</code> can accept an anonymous
Java 8 lambda function for additional brevity.</span></p><p>Here’s the previous example using <span class=language-java><code>MapElements</code></span>
-<span class=language-py><code>Map</code></span><span class=language-go>a direct return</span>:</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=c1>// The input PCollection.
+<span class="language-py language-typescript"><code>Map</code></span><span class=language-go>a direct return</span>:</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=c1>// The input PCollection.
</span><span class=c1></span><span class=n>PCollection</span><span class=o><</span><span class=n>String</span><span class=o>></span> <span class=n>words</span> <span class=o>=</span> <span class=o>...;</span>
<span class=c1>// Apply a MapElements with an anonymous lambda function to the PCollection words.
@@ -618,7 +702,11 @@ Java 8 lambda function for additional brevity.</span></p><p>Here’s the pre
</span><span class=c1>// Save the result as the PCollection wordLengths.
</span><span class=c1></span><span class=nx>wordLengths</span> <span class=o>:=</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>ParDo</span><span class=p>(</span><span class=nx>s</span><span class=p>,</span> <span class=kd>func</span><span class=p>(</span><span class=nx>word</span> <span class=kt>string</span><span class=p>)</span> <span class=kt>int</span> <span class=p>{</span>
<span class=k>return</span> <span class=nb>len</span><span class=p>(</span><span class=nx>word</span><span class=p>)</span>
-<span class=p>},</span> <span class=nx>words</span><span class=p>)</span></code></pre></div></div></div><p><span class=language-java></p><blockquote><p><strong>Note:</strong> You can use Java 8 lambda functions with several other Beam
+<span class=p>},</span> <span class=nx>words</span><span class=p>)</span></code></pre></div></div></div><div class="language-typescript snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-typescript data-lang=typescript><span class=c1>// The input PCollection of string.
+</span><span class=c1></span><span class=nx>words</span> <span class=o>=</span> <span class=p>...</span>
+
+<span class=kr>const</span> <span class=nx>result</span> <span class=o>=</span> <span class=nx>words</span><span class=p>.</span><span class=nx>map</span><span class=p>((</span><span class=nx>word</span><span class=p>)</span> <span class=o>=></span> <span class=nx>word</span><span class=p>.</span><span class=nx>length</span><span class=p>);</span>
+</code></pre></div></div></div><p><span class=language-java></p><blockquote><p><strong>Note:</strong> You can use Java 8 lambda functions with several other Beam
transforms, including <code>Filter</code>, <code>FlatMapElements</code>, and <code>Partition</code>.</p></blockquote></span><p><span class=language-go></p><blockquote><p><strong>Note:</strong> Anonymous function DoFns may not work on distributed runners.
It’s recommended to use named functions and register them with <code>register.FunctionXxY</code> in
an <code>init()</code> block.</p></blockquote></span><h5 id=dofn>4.2.1.4. DoFn lifecycle</h5><p>Here is a sequence diagram that shows the lifecycle of the DoFn during
@@ -662,16 +750,39 @@ jump, [3]
tree, [2]
...
</code></pre><p>Thus, <code>GroupByKey</code> represents a transform from a multimap (multiple keys to
-individual values) to a uni-map (unique keys to collections of values).</p><p><span class=language-java>Using <code>GroupByKey</code> is straightforward:</span></p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=c1>// The input PCollection.
+individual values) to a uni-map (unique keys to collections of values).</p><p><span class=language-java>Using <code>GroupByKey</code> is straightforward:</span></p><p class="language-py language-typescript">While all SDKs have a <code>GroupByKey</code> transform, using <code>GroupBy</code> is
+generally more natural.
+The <code>GroupBy</code> transform can be parameterized by the name(s) of properties
+on which to group the elements of the PCollection, or a function taking
+the each element as input that maps to a key on which to do grouping.</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=c1>// The input PCollection.
</span><span class=c1></span> <span class=n>PCollection</span><span class=o><</span><span class=n>KV</span><span class=o><</span><span class=n>String</span><span class=o>,</span> <span class=n>String</span><span class=o>>></span> <span class=n>mapped</span> <span class=o>=</span> <span class=o>...;</span>
<span class=c1>// Apply GroupByKey to the PCollection mapped.
</span><span class=c1>// Save the result as the PCollection reduced.
</span><span class=c1></span><span class=n>PCollection</span><span class=o><</span><span class=n>KV</span><span class=o><</span><span class=n>String</span><span class=o>,</span> <span class=n>Iterable</span><span class=o><</span><span class=n>String</span><span class=o>>>></span> <span class=n>reduced</span> <span class=o>=</span>
- <span class=n>mapped</span><span class=o>.</span><span class=na>apply</span><span class=o>(</span><span class=n>GroupByKey</span><span class=o>.<</span><span class=n>String</span><span class=o>,</span> <span class=n>String</span><span class=o>></span><span class=n>create</span><span class=o>());</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom titl [...]
+ <span class=n>mapped</span><span class=o>.</span><span class=na>apply</span><span class=o>(</span><span class=n>GroupByKey</span><span class=o>.<</span><span class=n>String</span><span class=o>,</span> <span class=n>String</span><span class=o>></span><span class=n>create</span><span class=o>());</span></code></pre></div></div></div><div class="language-py snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom titl [...]
+<span class=n>words_and_counts</span> <span class=o>=</span> <span class=o>...</span>
+
+
+<span class=n>grouped_words</span> <span class=o>=</span> <span class=n>words_and_counts</span> <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>GroupByKey</span><span class=p>()</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=c [...]
</span><span class=c1>// from an input slice of stringPair (struct with K, V string fields).
</span><span class=c1></span><span class=nx>pairs</span> <span class=o>:=</span> <span class=nf>CreateAndSplit</span><span class=p>(</span><span class=nx>s</span><span class=p>,</span> <span class=nx>input</span><span class=p>)</span>
-<span class=nx>keyed</span> <span class=o>:=</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>GroupByKey</span><span class=p>(</span><span class=nx>s</span><span class=p>,</span> <span class=nx>pairs</span><span class=p>)</span></code></pre></div></div></div><h5 id=groupbykey-and-unbounded-pcollections>4.2.2.1 GroupByKey and unbounded PCollections</h5><p>If you are using unbounded <code>PCollection</code>s, you must use either <a href=#setting-your-pcollections-windo [...]
+<span class=nx>keyed</span> <span class=o>:=</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>GroupByKey</span><span class=p>(</span><span class=nx>s</span><span class=p>,</span> <span class=nx>pairs</span><span class=p>)</span></code></pre></div></div></div><div class="language-typescript snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg> [...]
+</span><span class=c1>// {word: "cat", score: 1}, {word: "dog", score: 5}, {word: "cat", score: 5}, ...
+</span><span class=c1></span><span class=kr>const</span> <span class=nx>scores</span> : <span class=kt>PCollection</span><span class=o><</span><span class=p>{</span><span class=nx>word</span>: <span class=kt>string</span><span class=p>,</span> <span class=nx>score</span>: <span class=kt>number</span><span class=p>}</span><span class=o>></span> <span class=o>=</span> <span class=p>...</span>
+
+<span class=c1>// This will produce a PCollection with elements like
+</span><span class=c1>// {key: "cat", value: [{ word: "cat", score: 1 },
+</span><span class=c1>// { word: "cat", score: 5 }, ...]}
+</span><span class=c1>// {key: "dog", value: [{ word: "dog", score: 5 }, ...]}
+</span><span class=c1></span><span class=kr>const</span> <span class=nx>grouped_by_word</span> <span class=o>=</span> <span class=nx>scores</span><span class=p>.</span><span class=nx>apply</span><span class=p>(</span><span class=nx>beam</span><span class=p>.</span><span class=nx>groupBy</span><span class=p>(</span><span class=s2>"word"</span><span class=p>));</span>
+
+<span class=c1>// This will produce a PCollection with elements like
+</span><span class=c1>// {key: 3, value: [{ word: "cat", score: 1 },
+</span><span class=c1>// { word: "dog", score: 5 },
+</span><span class=c1>// { word: "cat", score: 5 }, ...]}
+</span><span class=c1></span><span class=kr>const</span> <span class=nx>by_word_length</span> <span class=o>=</span> <span class=nx>scores</span><span class=p>.</span><span class=nx>apply</span><span class=p>(</span><span class=nx>beam</span><span class=p>.</span><span class=nx>groupBy</span><span class=p>((</span><span class=nx>x</span><span class=p>)</span> <span class=o>=></span> <span class=nx>x</span><span class=p>.</span><span class=nx>word</span><span class=p>.</span><span clas [...]
+</code></pre></div></div></div><h5 id=groupbykey-and-unbounded-pcollections>4.2.2.1 GroupByKey and unbounded PCollections</h5><p>If you are using unbounded <code>PCollection</code>s, you must use either <a href=#setting-your-pcollections-windowing-function>non-global
windowing</a> or an
<a href=#triggers>aggregation trigger</a> in order to perform a <code>GroupByKey</code> or
<a href=#cogroupbykey>CoGroupByKey</a>. This is because a bounded <code>GroupByKey</code> or
@@ -785,7 +896,22 @@ data contains names and phone numbers.</p><div class="language-java snippet"><di
<span class=p>{</span><span class=s>"carl"</span><span class=p>,</span> <span class=s>"444-555-6666"</span><span class=p>},</span>
<span class=p>}</span>
<span class=nx>emails</span> <span class=o>:=</span> <span class=nf>CreateAndSplit</span><span class=p>(</span><span class=nx>s</span><span class=p>.</span><span class=nf>Scope</span><span class=p>(</span><span class=s>"CreateEmails"</span><span class=p>),</span> <span class=nx>emailSlice</span><span class=p>)</span>
-<span class=nx>phones</span> <span class=o>:=</span> <span class=nf>CreateAndSplit</span><span class=p>(</span><span class=nx>s</span><span class=p>.</span><span class=nf>Scope</span><span class=p>(</span><span class=s>"CreatePhones"</span><span class=p>),</span> <span class=nx>phoneSlice</span><span class=p>)</span></code></pre></div></div></div><p>After <code>CoGroupByKey</code>, the resulting data contains all data associated with each
+<span class=nx>phones</span> <span class=o>:=</span> <span class=nf>CreateAndSplit</span><span class=p>(</span><span class=nx>s</span><span class=p>.</span><span class=nf>Scope</span><span class=p>(</span><span class=s>"CreatePhones"</span><span class=p>),</span> <span class=nx>phoneSlice</span><span class=p>)</span></code></pre></div></div></div><div class="language-typescript snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data- [...]
+ <span class=p>{</span> <span class=nx>name</span><span class=o>:</span> <span class=s2>"amy"</span><span class=p>,</span> <span class=nx>email</span><span class=o>:</span> <span class=s2>"amy@example.com"</span> <span class=p>},</span>
+ <span class=p>{</span> <span class=nx>name</span><span class=o>:</span> <span class=s2>"carl"</span><span class=p>,</span> <span class=nx>email</span><span class=o>:</span> <span class=s2>"carl@example.com"</span> <span class=p>},</span>
+ <span class=p>{</span> <span class=nx>name</span><span class=o>:</span> <span class=s2>"julia"</span><span class=p>,</span> <span class=nx>email</span><span class=o>:</span> <span class=s2>"julia@example.com"</span> <span class=p>},</span>
+ <span class=p>{</span> <span class=nx>name</span><span class=o>:</span> <span class=s2>"carl"</span><span class=p>,</span> <span class=nx>email</span><span class=o>:</span> <span class=s2>"carl@email.com"</span> <span class=p>},</span>
+<span class=p>];</span>
+<span class=kr>const</span> <span class=nx>phones_list</span> <span class=o>=</span> <span class=p>[</span>
+ <span class=p>{</span> <span class=nx>name</span><span class=o>:</span> <span class=s2>"amy"</span><span class=p>,</span> <span class=nx>phone</span><span class=o>:</span> <span class=s2>"111-222-3333"</span> <span class=p>},</span>
+ <span class=p>{</span> <span class=nx>name</span><span class=o>:</span> <span class=s2>"james"</span><span class=p>,</span> <span class=nx>phone</span><span class=o>:</span> <span class=s2>"222-333-4444"</span> <span class=p>},</span>
+ <span class=p>{</span> <span class=nx>name</span><span class=o>:</span> <span class=s2>"amy"</span><span class=p>,</span> <span class=nx>phone</span><span class=o>:</span> <span class=s2>"333-444-5555"</span> <span class=p>},</span>
+ <span class=p>{</span> <span class=nx>name</span><span class=o>:</span> <span class=s2>"carl"</span><span class=p>,</span> <span class=nx>phone</span><span class=o>:</span> <span class=s2>"444-555-6666"</span> <span class=p>},</span>
+<span class=p>];</span>
+
+<span class=kr>const</span> <span class=nx>emails</span> <span class=o>=</span> <span class=nx>root</span><span class=p>.</span><span class=nx>apply</span><span class=p>(</span><span class=nx>beam</span><span class=p>.</span><span class=nx>create</span><span class=p>(</span><span class=nx>emails_list</span><span class=p>));</span>
+<span class=kr>const</span> <span class=nx>phones</span> <span class=o>=</span> <span class=nx>root</span><span class=p>.</span><span class=nx>apply</span><span class=p>(</span><span class=nx>beam</span><span class=p>.</span><span class=nx>create</span><span class=p>(</span><span class=nx>phones_list</span><span class=p>));</span>
+</code></pre></div></div></div><p>After <code>CoGroupByKey</code>, the resulting data contains all data associated with each
unique key from any of the input collections.</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=kd>final</span> <span class=n>TupleTag</span><span class=o><</span><span class=n>String</span><span class=o>></span> <span class=n>emai [...]
<span class=kd>final</span> <span class=n>TupleTag</span><span class=o><</span><span class=n>String</span><span class=o>></span> <span class=n>phonesTag</span> <span class=o>=</span> <span class=k>new</span> <span class=n>TupleTag</span><span class=o><>();</span>
@@ -852,7 +978,43 @@ unique key from any of the input collections.</p><div class="language-java snipp
<span class=nx>Emails</span><span class=p>:</span> <span class=p>[]</span><span class=kt>string</span><span class=p>{</span><span class=s>"julia@example.com"</span><span class=p>},</span>
<span class=nx>Phones</span><span class=p>:</span> <span class=p>[]</span><span class=kt>string</span><span class=p>{},</span>
<span class=p>},</span>
-<span class=p>}</span></code></pre></div></div></div><p class="language-java language-py">The following code example joins the two <code>PCollection</code>s with <code>CoGroupByKey</code>,
+<span class=p>}</span></code></pre></div></div></div><div class="language-typescript snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-typescript data-lang=typescript><span class=kr>const</span> <span class=nx>results</span> <span class=o>=</span> <span class=p>[</span>
+ <span class=p>{</span>
+ <span class=nx>name</span><span class=o>:</span> <span class=s2>"amy"</span><span class=p>,</span>
+ <span class=nx>values</span><span class=o>:</span> <span class=p>{</span>
+ <span class=nx>emails</span><span class=o>:</span> <span class=p>[{</span> <span class=nx>name</span><span class=o>:</span> <span class=s2>"amy"</span><span class=p>,</span> <span class=nx>email</span><span class=o>:</span> <span class=s2>"amy@example.com"</span> <span class=p>}],</span>
+ <span class=nx>phones</span><span class=o>:</span> <span class=p>[</span>
+ <span class=p>{</span> <span class=nx>name</span><span class=o>:</span> <span class=s2>"amy"</span><span class=p>,</span> <span class=nx>phone</span><span class=o>:</span> <span class=s2>"111-222-3333"</span> <span class=p>},</span>
+ <span class=p>{</span> <span class=nx>name</span><span class=o>:</span> <span class=s2>"amy"</span><span class=p>,</span> <span class=nx>phone</span><span class=o>:</span> <span class=s2>"333-444-5555"</span> <span class=p>},</span>
+ <span class=p>],</span>
+ <span class=p>},</span>
+ <span class=p>},</span>
+ <span class=p>{</span>
+ <span class=nx>name</span><span class=o>:</span> <span class=s2>"carl"</span><span class=p>,</span>
+ <span class=nx>values</span><span class=o>:</span> <span class=p>{</span>
+ <span class=nx>emails</span><span class=o>:</span> <span class=p>[</span>
+ <span class=p>{</span> <span class=nx>name</span><span class=o>:</span> <span class=s2>"carl"</span><span class=p>,</span> <span class=nx>email</span><span class=o>:</span> <span class=s2>"carl@example.com"</span> <span class=p>},</span>
+ <span class=p>{</span> <span class=nx>name</span><span class=o>:</span> <span class=s2>"carl"</span><span class=p>,</span> <span class=nx>email</span><span class=o>:</span> <span class=s2>"carl@email.com"</span> <span class=p>},</span>
+ <span class=p>],</span>
+ <span class=nx>phones</span><span class=o>:</span> <span class=p>[{</span> <span class=nx>name</span><span class=o>:</span> <span class=s2>"carl"</span><span class=p>,</span> <span class=nx>phone</span><span class=o>:</span> <span class=s2>"444-555-6666"</span> <span class=p>}],</span>
+ <span class=p>},</span>
+ <span class=p>},</span>
+ <span class=p>{</span>
+ <span class=nx>name</span><span class=o>:</span> <span class=s2>"james"</span><span class=p>,</span>
+ <span class=nx>values</span><span class=o>:</span> <span class=p>{</span>
+ <span class=nx>emails</span><span class=o>:</span> <span class=p>[],</span>
+ <span class=nx>phones</span><span class=o>:</span> <span class=p>[{</span> <span class=nx>name</span><span class=o>:</span> <span class=s2>"james"</span><span class=p>,</span> <span class=nx>phone</span><span class=o>:</span> <span class=s2>"222-333-4444"</span> <span class=p>}],</span>
+ <span class=p>},</span>
+ <span class=p>},</span>
+ <span class=p>{</span>
+ <span class=nx>name</span><span class=o>:</span> <span class=s2>"julia"</span><span class=p>,</span>
+ <span class=nx>values</span><span class=o>:</span> <span class=p>{</span>
+ <span class=nx>emails</span><span class=o>:</span> <span class=p>[{</span> <span class=nx>name</span><span class=o>:</span> <span class=s2>"julia"</span><span class=p>,</span> <span class=nx>email</span><span class=o>:</span> <span class=s2>"julia@example.com"</span> <span class=p>}],</span>
+ <span class=nx>phones</span><span class=o>:</span> <span class=p>[],</span>
+ <span class=p>},</span>
+ <span class=p>},</span>
+<span class=p>];</span>
+</code></pre></div></div></div><p class="language-java language-py language-typescript">The following code example joins the two <code>PCollection</code>s with <code>CoGroupByKey</code>,
followed by a <code>ParDo</code> to consume the result. Then, the code uses tags to look up
and format data from each collection.</p><p class=language-go>The following code example joins the two <code>PCollection</code>s with <code>CoGroupByKey</code>,
followed by a <code>ParDo</code> to consume the result. The ordering of the <code>DoFn</code> iterator
@@ -932,7 +1094,15 @@ parameters maps to the ordering of the <code>CoGroupByKey</code> inputs.</p><div
<span class=nx>Emails</span><span class=p>:</span> <span class=p>[]</span><span class=kt>string</span><span class=p>{</span><span class=s>"julia@example.com"</span><span class=p>},</span>
<span class=nx>Phones</span><span class=p>:</span> <span class=p>[]</span><span class=kt>string</span><span class=p>{},</span>
<span class=p>},</span>
-<span class=p>}</span></code></pre></div></div></div><p>The formatted data looks like this:</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=kd>final</span> <span class=n>List</span><span class=o><</span><span class=n>String</span><s [...]
+<span class=p>}</span></code></pre></div></div></div><div class="language-typescript snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-typescript data-lang=typescript><span class=kr>const</span> <span class=nx>formatted_results_pcoll</span> <span class=o>=</span> <span class=nx>beam</span>
+ <span class=p>.</span><span class=nx>P</span><span class=p>({</span> <span class=nx>emails</span><span class=p>,</span> <span class=nx>phones</span> <span class=p>})</span>
+ <span class=p>.</span><span class=nx>apply</span><span class=p>(</span><span class=nx>beam</span><span class=p>.</span><span class=nx>coGroupBy</span><span class=p>(</span><span class=s2>"name"</span><span class=p>))</span>
+ <span class=p>.</span><span class=nx>map</span><span class=p>(</span><span class=kd>function</span> <span class=nx>formatResults</span><span class=p>({</span> <span class=nx>key</span><span class=p>,</span> <span class=nx>values</span> <span class=p>})</span> <span class=p>{</span>
+ <span class=kr>const</span> <span class=nx>emails</span> <span class=o>=</span> <span class=nx>values</span><span class=p>.</span><span class=nx>emails</span><span class=p>.</span><span class=nx>map</span><span class=p>((</span><span class=nx>x</span><span class=p>)</span> <span class=o>=></span> <span class=nx>x</span><span class=p>.</span><span class=nx>email</span><span class=p>).</span><span class=nx>sort</span><span class=p>();</span>
+ <span class=kr>const</span> <span class=nx>phones</span> <span class=o>=</span> <span class=nx>values</span><span class=p>.</span><span class=nx>phones</span><span class=p>.</span><span class=nx>map</span><span class=p>((</span><span class=nx>x</span><span class=p>)</span> <span class=o>=></span> <span class=nx>x</span><span class=p>.</span><span class=nx>phone</span><span class=p>).</span><span class=nx>sort</span><span class=p>();</span>
+ <span class=k>return</span> <span class=sb>`</span><span class=si>${</span><span class=nx>key</span><span class=si>}</span><span class=sb>; [</span><span class=si>${</span><span class=nx>emails</span><span class=si>}</span><span class=sb>]; [</span><span class=si>${</span><span class=nx>phones</span><span class=si>}</span><span class=sb>]`</span><span class=p>;</span>
+ <span class=p>});</span>
+</code></pre></div></div></div><p>The formatted data looks like this:</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=kd>final</span> <span class=n>List</span><span class=o><</span><span class=n>String</span><span class=o>></span [...]
<span class=n>Arrays</span><span class=o>.</span><span class=na>asList</span><span class=o>(</span>
<span class=s>"amy; ['amy@example.com']; ['111-222-3333', '333-444-5555']"</span><span class=o>,</span>
<span class=s>"carl; ['carl@email.com', 'carl@example.com']; ['444-555-6666']"</span><span class=o>,</span>
@@ -947,9 +1117,16 @@ parameters maps to the ordering of the <code>CoGroupByKey</code> inputs.</p><div
<span class=s>"carl; ['carl@email.com', 'carl@example.com']; ['444-555-6666']"</span><span class=p>,</span>
<span class=s>"james; []; ['222-333-4444']"</span><span class=p>,</span>
<span class=s>"julia; ['julia@example.com']; []"</span><span class=p>,</span>
-<span class=p>}</span></code></pre></div></div></div><h4 id=combine>4.2.4. Combine</h4><p><span class=language-java><a href=https://beam.apache.org/releases/javadoc/2.40.0/index.html?org/apache/beam/sdk/transforms/Combine.html><code>Combine</code></a></span>
+<span class=p>}</span></code></pre></div></div></div><div class="language-typescript snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-typescript data-lang=typescript><span class=kr>const</span> <span class=nx>formatted_results</span> <span class=o>=</span> <span class=p>[</span>
+ <span class=s2>"amy; [amy@example.com]; [111-222-3333,333-444-5555]"</span><span class=p>,</span>
+ <span class=s2>"carl; [carl@email.com,carl@example.com]; [444-555-6666]"</span><span class=p>,</span>
+ <span class=s2>"james; []; [222-333-4444]"</span><span class=p>,</span>
+ <span class=s2>"julia; [julia@example.com]; []"</span><span class=p>,</span>
+<span class=p>];</span>
+</code></pre></div></div></div><h4 id=combine>4.2.4. Combine</h4><p><span class=language-java><a href=https://beam.apache.org/releases/javadoc/2.40.0/index.html?org/apache/beam/sdk/transforms/Combine.html><code>Combine</code></a></span>
<span class=language-py><a href=https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/core.py><code>Combine</code></a></span>
<span class=language-go><a href=https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/combine.go#L27><code>Combine</code></a></span>
+<span class=language-typescript><a href=https://github.com/apache/beam/blob/master/sdks/typescript/src/apache_beam/transforms/group_and_combine.ts><code>Combine</code></a></span>
is a Beam transform for combining collections of elements or values in your
data. <code>Combine</code> has variants that work on entire <code>PCollection</code>s, and some that
combine the values for each key in <code>PCollection</code>s of key/value pairs.</p><p>When you apply a <code>Combine</code> transform, you must provide the function that
@@ -971,7 +1148,12 @@ term for this optimization is “mapper-side combine.”</li><li><strong
size by a lot, it is useful to combine elements as they emerge from a
streaming shuffle. This spreads out the cost of doing combines over the time
that your streaming computation might be idle. Incremental combining also
-reduces the storage of intermediate accumulators.</li></ul><h5 id=simple-combines>4.2.4.1. Simple combinations using simple functions</h5><p>The following example code shows a simple combine function.</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=jav [...]
+reduces the storage of intermediate accumulators.</li></ul><h5 id=simple-combines>4.2.4.1. Simple combinations using simple functions</h5><p>The following example code shows a simple combine function.
+<span class=language-typescript>Combining is done by modifying a grouping transform with the <code>combining</code> method.
+This method takes three parameters: the value to combine (either as a named
+property of the input elements, or a function of the entire input),
+the combining operation (either a binary function or a <code>CombineFn</code>),
+and finally a name for the combined value in the output object.</span></p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=c1>// Sum a collection of Integer values. The function SumInts implements the interface SerializableFunction.
</span><span class=c1></span><span class=kd>public</span> <span class=kd>static</span> <span class=kd>class</span> <span class=nc>SumInts</span> <span class=kd>implements</span> <span class=n>SerializableFunction</span><span class=o><</span><span class=n>Iterable</span><span class=o><</span><span class=n>Integer</span><span class=o>>,</span> <span class=n>Integer</span><span class=o>></span> <span class=o>{</span>
<span class=nd>@Override</span>
<span class=kd>public</span> <span class=n>Integer</span> <span class=nf>apply</span><span class=o>(</span><span class=n>Iterable</span><span class=o><</span><span class=n>Integer</span><span class=o>></span> <span class=n>input</span><span class=o>)</span> <span class=o>{</span>
@@ -1027,7 +1209,15 @@ the type of the output.</p></p><div class="language-go snippet"><div class="note
<span class=kd>func</span> <span class=nf>globallyBoundedSumInts</span><span class=p>(</span><span class=nx>s</span> <span class=nx>beam</span><span class=p>.</span><span class=nx>Scope</span><span class=p>,</span> <span class=nx>bound</span> <span class=kt>int</span><span class=p>,</span> <span class=nx>ints</span> <span class=nx>beam</span><span class=p>.</span><span class=nx>PCollection</span><span class=p>)</span> <span class=nx>beam</span><span class=p>.</span><span class=nx>PCollec [...]
<span class=k>return</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>Combine</span><span class=p>(</span><span class=nx>s</span><span class=p>,</span> <span class=o>&</span><span class=nx>boundedSum</span><span class=p>{</span><span class=nx>Bound</span><span class=p>:</span> <span class=nx>bound</span><span class=p>},</span> <span class=nx>ints</span><span class=p>)</span>
-<span class=p>}</span></code></pre></div></div></div><h5 id=advanced-combines>4.2.4.2. Advanced combinations using CombineFn</h5><p>For more complex combine functions, you can define a
+<span class=p>}</span></code></pre></div></div></div><div class="language-typescript snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-typescript data-lang=typescript><span class=kr>const</span> <span class=nx>pcoll</span> <span class=o>=</span> <span class=nx>root</span><span class=p>.</span><spa [...]
+<span class=kr>const</span> <span class=nx>result</span> <span class=o>=</span> <span class=nx>pcoll</span><span class=p>.</span><span class=nx>apply</span><span class=p>(</span>
+ <span class=nx>beam</span>
+ <span class=p>.</span><span class=nx>groupGlobally</span><span class=p>()</span>
+ <span class=p>.</span><span class=nx>combining</span><span class=p>((</span><span class=nx>c</span><span class=p>)</span> <span class=o>=></span> <span class=nx>c</span><span class=p>,</span> <span class=p>(</span><span class=nx>x</span><span class=p>,</span> <span class=nx>y</span><span class=p>)</span> <span class=o>=></span> <span class=nx>x</span> <span class=o>+</span> <span class=nx>y</span><span class=p>,</span> <span class=s2>"sum"</span><span class=p>)</span>
+ <span class=p>.</span><span class=nx>combining</span><span class=p>((</span><span class=nx>c</span><span class=p>)</span> <span class=o>=></span> <span class=nx>c</span><span class=p>,</span> <span class=p>(</span><span class=nx>x</span><span class=p>,</span> <span class=nx>y</span><span class=p>)</span> <span class=o>=></span> <span class=nx>x</span> <span class=o>*</span> <span class=nx>y</span><span class=p>,</span> <span class=s2>"product"</span><span class=p>)</span>
+<span class=p>);</span>
+<span class=kr>const</span> <span class=nx>expected</span> <span class=o>=</span> <span class=p>{</span> <span class=nx>sum</span>: <span class=kt>1111</span><span class=p>,</span> <span class=nx>product</span>: <span class=kt>1000000</span> <span class=p>}</span>
+</code></pre></div></div></div><h5 id=advanced-combines>4.2.4.2. Advanced combinations using CombineFn</h5><p>For more complex combine functions, you can define a
<span class="language-java language-py">subclass of</span><code>CombineFn</code>.
You should use a <code>CombineFn</code> if the combine function requires a more sophisticated
accumulator, must perform additional pre- or post-processing, might change the
@@ -1120,7 +1310,20 @@ mean average:</p><div class="language-java snippet"><div class="notebook-skip co
<span class=kd>func</span> <span class=nf>init</span><span class=p>()</span> <span class=p>{</span>
<span class=nx>register</span><span class=p>.</span><span class=nx>Combiner3</span><span class=p>[</span><span class=nx>averageAccum</span><span class=p>,</span> <span class=kt>int</span><span class=p>,</span> <span class=kt>float64</span><span class=p>](</span><span class=o>&</span><span class=nx>averageFn</span><span class=p>{})</span>
-<span class=p>}</span></code></pre></div></div></div><span class=language-go><blockquote><p><strong>Note</strong>: Only <code>MergeAccumulators</code> is a required method. The others will have a default interpretation
+<span class=p>}</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-go data-lang=go><span class=kd>const</span> <span class=nx>meanCombineFn</span><span class=p>:</span> <span class=nx>beam</span><span class=p>.</span><span class=nx>Combin [...]
+ <span class=p>{</span>
+ <span class=nx>createAccumulator</span><span class=p>:</span> <span class=p>()</span> <span class=p>=></span> <span class=p>[</span><span class=mi>0</span><span class=p>,</span> <span class=mi>0</span><span class=p>],</span>
+ <span class=nx>addInput</span><span class=p>:</span> <span class=p>([</span><span class=nx>sum</span><span class=p>,</span> <span class=nx>count</span><span class=p>]:</span> <span class=p>[</span><span class=nx>number</span><span class=p>,</span> <span class=nx>number</span><span class=p>],</span> <span class=nx>i</span><span class=p>:</span> <span class=nx>number</span><span class=p>)</span> <span class=p>=></span> <span class=p>[</span>
+ <span class=nx>sum</span> <span class=o>+</span> <span class=nx>i</span><span class=p>,</span>
+ <span class=nx>count</span> <span class=o>+</span> <span class=mi>1</span><span class=p>,</span>
+ <span class=p>],</span>
+ <span class=nx>mergeAccumulators</span><span class=p>:</span> <span class=p>(</span><span class=nx>accumulators</span><span class=p>:</span> <span class=p>[</span><span class=nx>number</span><span class=p>,</span> <span class=nx>number</span><span class=p>][])</span> <span class=p>=></span>
+ <span class=nx>accumulators</span><span class=p>.</span><span class=nf>reduce</span><span class=p>(([</span><span class=nx>sum0</span><span class=p>,</span> <span class=nx>count0</span><span class=p>],</span> <span class=p>[</span><span class=nx>sum1</span><span class=p>,</span> <span class=nx>count1</span><span class=p>])</span> <span class=p>=></span> <span class=p>[</span>
+ <span class=nx>sum0</span> <span class=o>+</span> <span class=nx>sum1</span><span class=p>,</span>
+ <span class=nx>count0</span> <span class=o>+</span> <span class=nx>count1</span><span class=p>,</span>
+ <span class=p>]),</span>
+ <span class=nx>extractOutput</span><span class=p>:</span> <span class=p>([</span><span class=nx>sum</span><span class=p>,</span> <span class=nx>count</span><span class=p>]:</span> <span class=p>[</span><span class=nx>number</span><span class=p>,</span> <span class=nx>number</span><span class=p>])</span> <span class=p>=></span> <span class=nx>sum</span> <span class=o>/</span> <span class=nx>count</span><span class=p>,</span>
+ <span class=p>};</span></code></pre></div></div></div><span class=language-go><blockquote><p><strong>Note</strong>: Only <code>MergeAccumulators</code> is a required method. The others will have a default interpretation
based on the accumulator type.</p></blockquote></span><h5 id=combining-pcollection>4.2.4.3. Combining a PCollection into a single value</h5><p>Use the global combine to transform all of the elements in a given <code>PCollection</code>
into a single value, represented in your pipeline as a new <code>PCollection</code>
containing one element. The following example code shows how to apply the Beam
@@ -1134,7 +1337,10 @@ of integers.</p><div class="language-java snippet"><div class="notebook-skip cod
<span class=c1># the elements in the input PCollection.</span>
<span class=n>pc</span> <span class=o>=</span> <span class=o>...</span>
-<span class=n>average</span> <span class=o>=</span> <span class=n>pc</span> <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>CombineGlobally</span><span class=p>(</span><span class=n>AverageFn</span><span class=p>())</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg [...]
+<span class=n>average</span> <span class=o>=</span> <span class=n>pc</span> <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>CombineGlobally</span><span class=p>(</span><span class=n>AverageFn</span><span class=p>())</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg [...]
+<span class=kd>const</span> <span class=nx>result</span> <span class=p>=</span> <span class=nx>pcoll</span><span class=p>.</span><span class=nf>apply</span><span class=p>(</span>
+ <span class=nx>beam</span><span class=p>.</span><span class=nf>groupGlobally</span><span class=p>().</span><span class=nf>combining</span><span class=p>((</span><span class=nx>c</span><span class=p>)</span> <span class=p>=></span> <span class=nx>c</span><span class=p>,</span> <span class=nx>meanCombineFn</span><span class=p>,</span> <span class=s>"mean"</span><span class=p>)</span>
+<span class=p>);</span></code></pre></div></div></div><h5 id=combine-global-windowing>4.2.4.4. Combine and global windowing</h5><p>If your input <code>PCollection</code> uses the default global windowing, the default
behavior is to return a <code>PCollection</code> containing one item. That item’s value
comes from the accumulator in the combine function that you specified when
applying <code>Combine</code>. For example, the Beam provided sum combine function returns
@@ -1206,9 +1412,29 @@ PerKey must be an associative reduction function or a
</span><span class=c1></span>
<span class=nx>avgAccuracyPerPlayer</span> <span class=o>:=</span> <span class=nx>stats</span><span class=p>.</span><span class=nf>MeanPerKey</span><span class=p>(</span><span class=nx>s</span><span class=p>,</span> <span class=nx>playerAccuracies</span><span class=p>)</span>
-<span class=o>//</span> <span class=nx>avgAccuracyPerPlayer</span> <span class=nx>is</span> <span class=nx>a</span> <span class=nx>PCollection</span><span class=p><</span><span class=kt>string</span><span class=p>,</span><span class=kt>float64</span><span class=p>></span></code></pre></div></div></div><h4 id=flatten>4.2.5. Flatten</h4><p><span class=language-java><a href=https://beam.apache.org/releases/javadoc/2.40.0/index.html?org/apache/beam/sdk/transforms/Flatten.html><code>Fla [...]
+<span class=o>//</span> <span class=nx>avgAccuracyPerPlayer</span> <span class=nx>is</span> <span class=nx>a</span> <span class=nx>PCollection</span><span class=p><</span><span class=kt>string</span><span class=p>,</span><span class=kt>float64</span><span class=p>></span></code></pre></div></div></div><div class="language-typescript snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><i [...]
+ <span class=nx>beam</span><span class=p>.</span><span class=nx>create</span><span class=p>([</span>
+ <span class=p>{</span> <span class=nx>player</span><span class=o>:</span> <span class=s2>"alice"</span><span class=p>,</span> <span class=nx>accuracy</span>: <span class=kt>1.0</span> <span class=p>},</span>
+ <span class=p>{</span> <span class=nx>player</span><span class=o>:</span> <span class=s2>"bob"</span><span class=p>,</span> <span class=nx>accuracy</span>: <span class=kt>0.99</span> <span class=p>},</span>
+ <span class=p>{</span> <span class=nx>player</span><span class=o>:</span> <span class=s2>"eve"</span><span class=p>,</span> <span class=nx>accuracy</span>: <span class=kt>0.5</span> <span class=p>},</span>
+ <span class=p>{</span> <span class=nx>player</span><span class=o>:</span> <span class=s2>"eve"</span><span class=p>,</span> <span class=nx>accuracy</span>: <span class=kt>0.25</span> <span class=p>},</span>
+ <span class=p>])</span>
+<span class=p>);</span>
+<span class=kr>const</span> <span class=nx>result</span> <span class=o>=</span> <span class=nx>pcoll</span><span class=p>.</span><span class=nx>apply</span><span class=p>(</span>
+ <span class=nx>beam</span>
+ <span class=p>.</span><span class=nx>groupBy</span><span class=p>(</span><span class=s2>"player"</span><span class=p>)</span>
+ <span class=p>.</span><span class=nx>combining</span><span class=p>(</span><span class=s2>"accuracy"</span><span class=p>,</span> <span class=nx>combiners</span><span class=p>.</span><span class=nx>mean</span><span class=p>,</span> <span class=s2>"mean"</span><span class=p>)</span>
+ <span class=p>.</span><span class=nx>combining</span><span class=p>(</span><span class=s2>"accuracy"</span><span class=p>,</span> <span class=nx>combiners</span><span class=p>.</span><span class=nx>max</span><span class=p>,</span> <span class=s2>"max"</span><span class=p>)</span>
+<span class=p>);</span>
+<span class=kr>const</span> <span class=nx>expected</span> <span class=o>=</span> <span class=p>[</span>
+ <span class=p>{</span> <span class=nx>player</span><span class=o>:</span> <span class=s2>"alice"</span><span class=p>,</span> <span class=nx>mean</span>: <span class=kt>1.0</span><span class=p>,</span> <span class=nx>max</span>: <span class=kt>1.0</span> <span class=p>},</span>
+ <span class=p>{</span> <span class=nx>player</span><span class=o>:</span> <span class=s2>"bob"</span><span class=p>,</span> <span class=nx>mean</span>: <span class=kt>0.99</span><span class=p>,</span> <span class=nx>max</span>: <span class=kt>0.99</span> <span class=p>},</span>
+ <span class=p>{</span> <span class=nx>player</span><span class=o>:</span> <span class=s2>"eve"</span><span class=p>,</span> <span class=nx>mean</span>: <span class=kt>0.375</span><span class=p>,</span> <span class=nx>max</span>: <span class=kt>0.5</span> <span class=p>},</span>
+<span class=p>];</span>
+</code></pre></div></div></div><h4 id=flatten>4.2.5. Flatten</h4><p><span class=language-java><a href=https://beam.apache.org/releases/javadoc/2.40.0/index.html?org/apache/beam/sdk/transforms/Flatten.html><code>Flatten</code></a></span>
<span class=language-py><a href=https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/core.py><code>Flatten</code></a></span>
<span class=language-go><a href=https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/flatten.go><code>Flatten</code></a></span>
+<span class=language-typescript><code>Flatten</code></span>
is a Beam transform for <code>PCollection</code> objects that store the same data type.
<code>Flatten</code> merges multiple <code>PCollection</code> objects into a single logical
<code>PCollection</code>.</p><p>The following example shows how to apply a <code>Flatten</code> transform to merge multiple
@@ -1228,7 +1454,13 @@ is a Beam transform for <code>PCollection</code> objects that store the same dat
<span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Flatten</span><span class=p>())</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-go data-lang=go><span class=c1>// Flatten accepts any number of PCo [...]
</span><span class=c1>// Returns a single PCollection that contains all of the elements in input PCollections.
</span><span class=c1></span>
-<span class=nx>merged</span> <span class=o>:=</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>Flatten</span><span class=p>(</span><span class=nx>s</span><span class=p>,</span> <span class=nx>pcol1</span><span class=p>,</span> <span class=nx>pcol2</span><span class=p>,</span> <span class=nx>pcol3</span><span class=p>)</span></code></pre></div></div></div><h5 id=data-encoding-merged-collections>4.2.5.1. Data encoding in merged collections</h5><p>By default, the coder [...]
+<span class=nx>merged</span> <span class=o>:=</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>Flatten</span><span class=p>(</span><span class=nx>s</span><span class=p>,</span> <span class=nx>pcol1</span><span class=p>,</span> <span class=nx>pcol2</span><span class=p>,</span> <span class=nx>pcol3</span><span class=p>)</span></code></pre></div></div></div><div class="language-typescript snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs [...]
+</span><span class=c1>// Returns a single PCollection that contains a union of all of the elements in all input PCollections.
+</span><span class=c1></span>
+<span class=kr>const</span> <span class=nx>fib</span> <span class=o>=</span> <span class=nx>root</span><span class=p>.</span><span class=nx>apply</span><span class=p>(</span><span class=nx>beam</span><span class=p>.</span><span class=nx>create</span><span class=p>([</span><span class=mi>1</span><span class=p>,</span> <span class=mi>1</span><span class=p>,</span> <span class=mi>2</span><span class=p>,</span> <span class=mi>3</span><span class=p>,</span> <span class=mi>5</span><span class= [...]
+<span class=kr>const</span> <span class=nx>pow</span> <span class=o>=</span> <span class=nx>root</span><span class=p>.</span><span class=nx>apply</span><span class=p>(</span><span class=nx>beam</span><span class=p>.</span><span class=nx>create</span><span class=p>([</span><span class=mi>1</span><span class=p>,</span> <span class=mi>2</span><span class=p>,</span> <span class=mi>4</span><span class=p>,</span> <span class=mi>8</span><span class=p>,</span> <span class=mi>16</span><span class [...]
+<span class=kr>const</span> <span class=nx>result</span> <span class=o>=</span> <span class=nx>beam</span><span class=p>.</span><span class=nx>P</span><span class=p>([</span><span class=nx>fib</span><span class=p>,</span> <span class=nx>pow</span><span class=p>]).</span><span class=nx>apply</span><span class=p>(</span><span class=nx>beam</span><span class=p>.</span><span class=nx>flatten</span><span class=p>());</span>
+</code></pre></div></div></div><h5 id=data-encoding-merged-collections>4.2.5.1. Data encoding in merged collections</h5><p>By default, the coder for the output <code>PCollection</code> is the same as the coder for
the first <code>PCollection</code> in the input <code>PCollectionList</code>. However, the input
<code>PCollection</code> objects can each use different coders, as long as they all contain
the same data type in your chosen language.</p><h5 id=merging-windowed-collections>4.2.5.2. Merging windowed collections</h5><p>When using <code>Flatten</code> to merge <code>PCollection</code> objects that have a windowing
@@ -1240,9 +1472,10 @@ incompatible windows, Beam generates an <code>IllegalStateException</code> error
pipeline is constructed.</p><h4 id=partition>4.2.6. Partition</h4><p><span class=language-java><a href=https://beam.apache.org/releases/javadoc/2.40.0/index.html?org/apache/beam/sdk/transforms/Partition.html><code>Partition</code></a></span>
<span class=language-py><a href=https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/core.py><code>Partition</code></a></span>
<span class=language-go><a href=https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/partition.go><code>Partition</code></a></span>
+<span class=language-typescript><code>Partition</code></span>
is a Beam transform for <code>PCollection</code> objects that store the same data
type. <code>Partition</code> splits a single <code>PCollection</code> into a fixed number of smaller
-collections.</p><p><code>Partition</code> divides the elements of a <code>PCollection</code> according to a partitioning
+collections.</p><p class=language-typescript>Often in the Typescript SDK the <code>Split</code> transform is more natural to use.</p><p><code>Partition</code> divides the elements of a <code>PCollection</code> according to a partitioning
function that you provide. The partitioning function contains the logic that
determines how to split up the elements of the input <code>PCollection</code> into each
resulting partition <code>PCollection</code>. The number of partitions must be determined
@@ -1288,7 +1521,15 @@ for instance).</p><p>The following example divides a <code>PCollection</code> in
<span class=c1>// Partition returns a slice of PCollections
</span><span class=c1></span><span class=nx>studentsByPercentile</span> <span class=o>:=</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>Partition</span><span class=p>(</span><span class=nx>s</span><span class=p>,</span> <span class=mi>10</span><span class=p>,</span> <span class=nx>decileFn</span><span class=p>,</span> <span class=nx>students</span><span class=p>)</span>
<span class=c1>// Each partition can be extracted by indexing into the slice.
-</span><span class=c1></span><span class=nx>fortiethPercentile</span> <span class=o>:=</span> <span class=nx>studentsByPercentile</span><span class=p>[</span><span class=mi>4</span><span class=p>]</span></code></pre></div></div></div><h3 id=requirements-for-writing-user-code-for-beam-transforms>4.3. Requirements for writing user code for Beam transforms</h3><p>When you build user code for a Beam transform, you should keep in mind the
+</span><span class=c1></span><span class=nx>fortiethPercentile</span> <span class=o>:=</span> <span class=nx>studentsByPercentile</span><span class=p>[</span><span class=mi>4</span><span class=p>]</span></code></pre></div></div></div><div class="language-typescript snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><co [...]
+ <span class=nx>beam</span><span class=p>.</span><span class=nx>partition</span><span class=p>(</span>
+ <span class=p>(</span><span class=nx>student</span><span class=p>,</span> <span class=nx>numPartitions</span><span class=p>)</span> <span class=o>=></span>
+ <span class=nb>Math</span><span class=p>.</span><span class=nx>floor</span><span class=p>((</span><span class=nx>getPercentile</span><span class=p>(</span><span class=nx>student</span><span class=p>)</span> <span class=o>/</span> <span class=mi>100</span><span class=p>)</span> <span class=o>*</span> <span class=nx>numPartitions</span><span class=p>),</span>
+ <span class=mi>10</span>
+ <span class=p>)</span>
+<span class=p>);</span>
+<span class=kr>const</span> <span class=nx>topDecile</span>: <span class=kt>PCollection</span><span class=o><</span><span class=nx>Student</span><span class=o>></span> <span class=o>=</span> <span class=nx>deciles</span><span class=p>[</span><span class=mi>9</span><span class=p>];</span>
+</code></pre></div></div></div><h3 id=requirements-for-writing-user-code-for-beam-transforms>4.3. Requirements for writing user code for Beam transforms</h3><p>When you build user code for a Beam transform, you should keep in mind the
distributed nature of execution. For example, there might be many copies of your
function running on a lot of different machines in parallel, and those copies
function independently, without communicating or sharing state with any of the
@@ -1314,7 +1555,20 @@ however, your subclass must not add any non-serializable members.</span>
they are registered with <code>register.FunctionXxY</code> (for simple functions) or
<code>register.DoFnXxY</code> (for sturctural DoFns), and are not closures. Structural
<code>DoFn</code>s will have all exported fields serialized. Unexported fields are unable to
-be serialized, and will be silently ignored.</span></p><p>Some other serializability factors you should keep in mind are:</p><ul><li><span class="language-java language-py">Transient</span><span class=langauage-go>Unexported</span>
+be serialized, and will be silently ignored.</span>
+<span class=language-typescript>The Typescript SDK use <a href=https://github.com/nokia/ts-serialize-closures>ts-serialize-closures</a>
+to serialize functions (and other objects).
+This works out of the box for functions that are not closures, and also works
+for closures as long as the function in question (and any closures it references)
+are compiled with the
+<a href=https://github.com/apache/beam/blob/master/sdks/typescript/tsconfig.json><code>ts-closure-transform</code> hooks</a>
+(e.g. by using <code>ttsc</code> in place of <code>tsc</code>).
+One can alternatively call
+<code>requireForSerialization("importableModuleDefiningFunc", {func})</code>
+to <a href=https://github.com/apache/beam/blob/master/sdks/typescript/src/apache_beam/serialization.ts>register a function directly</a> by name which can be less error-prone.
+Note that if, as is often the case in Javascript, <code>func</code> returns objects that
+contain closures, it is not sufficient to register <code>func</code> alone–its return
+value must be registered if used.</span></p><p>Some other serializability factors you should keep in mind are:</p><ul><li><span class="language-java language-py">Transient</span><span class=langauage-go>Unexported</span>
fields in your function object are <em>not</em> transmitted to worker
instances, because they are not automatically serialized.</li><li>Avoid loading a field with a large amount of data before serialization.</li><li>Individual instances of your function object cannot share data.</li><li>Mutating a function object after it gets applied will have no effect.</li></ul><span class=language-java><blockquote><p><strong>Note:</strong> Take care when declaring your function object inline by using an anonymous
inner class instance. In a non-static context, your inner class instance will
@@ -1469,7 +1723,38 @@ function. This optimizes runtime execution of the iterable.</p><h4 id=side-input
<span class=c1>// The Go SDK doesn't support custom ViewFns.
</span><span class=c1>// See https://github.com/apache/beam/issues/18602 for details
-</span><span class=c1></span><span class=o>//</span> <span class=nx>on</span> <span class=nx>how</span> <span class=nx>to</span> <span class=nx>contribute</span> <span class=nx>them</span><span class=p>!</span></code></pre></div></div></div><h4 id=side-inputs-windowing>4.4.2. Side inputs and windowing</h4><p>A windowed <code>PCollection</code> may be infinite and thus cannot be compressed into a
+</span><span class=c1></span><span class=o>//</span> <span class=nx>on</span> <span class=nx>how</span> <span class=nx>to</span> <span class=nx>contribute</span> <span class=nx>them</span><span class=p>!</span></code></pre></div></div></div><div class="language-typescript snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chr [...]
+</span><span class=c1>// `map`, `flatMap`, or `parDo` transforms. This object will get passed as an
+</span><span class=c1>// extra argument to the provided function (or `process` method of the `DoFn`).
+</span><span class=c1>// `SideInputParam` properties (generally created with `pardo.xxxSideInput(...)`)
+</span><span class=c1>// have a `lookup` method that can be invoked from within the process method.
+</span><span class=c1></span>
+<span class=c1>// Let words be a PCollection of strings.
+</span><span class=c1></span><span class=kr>const</span> <span class=nx>words</span> : <span class=kt>PCollection</span><span class=o><</span><span class=kt>string</span><span class=o>></span> <span class=o>=</span> <span class=p>...</span>
+
+<span class=c1>// meanLengthPColl will contain a single number whose value is the
+</span><span class=c1>// average length of the words
+</span><span class=c1></span><span class=kr>const</span> <span class=nx>meanLengthPColl</span>: <span class=kt>PCollection</span><span class=o><</span><span class=kt>number</span><span class=o>></span> <span class=o>=</span> <span class=nx>words</span>
+ <span class=p>.</span><span class=nx>apply</span><span class=p>(</span>
+ <span class=nx>beam</span>
+ <span class=p>.</span><span class=nx>groupGlobally</span><span class=o><</span><span class=kt>string</span><span class=o>></span><span class=p>()</span>
+ <span class=p>.</span><span class=nx>combining</span><span class=p>((</span><span class=nx>word</span><span class=p>)</span> <span class=o>=></span> <span class=nx>word</span><span class=p>.</span><span class=nx>length</span><span class=p>,</span> <span class=nx>combiners</span><span class=p>.</span><span class=nx>mean</span><span class=p>,</span> <span class=s2>"mean"</span><span class=p>)</span>
+ <span class=p>)</span>
+ <span class=p>.</span><span class=nx>map</span><span class=p>(({</span> <span class=nx>mean</span> <span class=p>})</span> <span class=o>=></span> <span class=nx>mean</span><span class=p>);</span>
+
+<span class=c1>// Now we use this as a side input to yield only words that are
+</span><span class=c1>// smaller than average.
+</span><span class=c1></span><span class=kr>const</span> <span class=nx>smallWords</span> <span class=o>=</span> <span class=nx>words</span><span class=p>.</span><span class=nx>flatMap</span><span class=p>(</span>
+ <span class=c1>// This is the function, taking context as a second argument.
+</span><span class=c1></span> <span class=kd>function</span><span class=o>*</span> <span class=nx>keepSmall</span><span class=p>(</span><span class=nx>word</span><span class=p>,</span> <span class=nx>context</span><span class=p>)</span> <span class=p>{</span>
+ <span class=k>if</span> <span class=p>(</span><span class=nx>word</span><span class=p>.</span><span class=nx>length</span> <span class=o><</span> <span class=nx>context</span><span class=p>.</span><span class=nx>meanLength</span><span class=p>.</span><span class=nx>lookup</span><span class=p>())</span> <span class=p>{</span>
+ <span class=nx>yield</span> <span class=nx>word</span><span class=p>;</span>
+ <span class=p>}</span>
+ <span class=p>},</span>
+ <span class=c1>// This is the context that will be passed as a second argument.
+</span><span class=c1></span> <span class=p>{</span> <span class=nx>meanLength</span>: <span class=kt>pardo.singletonSideInput</span><span class=p>(</span><span class=nx>meanLengthPColl</span><span class=p>)</span> <span class=p>}</span>
+<span class=p>);</span>
+</code></pre></div></div></div><h4 id=side-inputs-windowing>4.4.2. Side inputs and windowing</h4><p>A windowed <code>PCollection</code> may be infinite and thus cannot be compressed into a
single value (or single collection class). When you create a <code>PCollectionView</code>
of a windowed <code>PCollection</code>, the <code>PCollectionView</code> represents a single entity
per window (one singleton per window, one list per window, etc.).</p><p>Beam uses the window(s) for the main input element to look up the appropriate
@@ -1495,7 +1780,15 @@ number of additional output <code>PCollections</code>s, or even none at all.
If you choose to have multiple outputs, your <code>DoFn</code> needs to be called with the <code>ParDo</code>
function that matches the number of outputs. <code>beam.ParDo2</code> for two output <code>PCollection</code>s,
<code>beam.ParDo3</code> for three and so on until <code>beam.ParDo7</code>. If you need more, you can
-use <code>beam.ParDoN</code> which will return a <code>[]beam.PCollection</code>.</p><h4 id=output-tags>4.5.1. Tags for multiple outputs</h4><p class=language-go>The Go SDK doesn’t use output tags, and instead uses positional ordering for
+use <code>beam.ParDoN</code> which will return a <code>[]beam.PCollection</code>.</p><p class=language-typescript>While <code>ParDo</code> always produces a main output <code>PCollection</code> (as the return value
+from <code>apply</code>). If you want to have multiple outputs, emit an object with distinct
+properties in your <code>ParDo</code> operation and follow this operation with a <code>Split</code>
+to break it into multiple <code>PCollection</code>s.</p><h4 id=output-tags>4.5.1. Tags for multiple outputs</h4><p class=language-typescript>The <code>Split</code> PTransform will take a PCollection of elements of the form
+<code>{tagA?: A, tagB?: B, ...}</code> and return a object
+<code>{tagA: PCollection<A>, tagB: PCollection<B>, ...}</code>.
+The set of expected tags is passed to the operation; how multiple or
+unknown tags are handled can be specified by passing a non-default
+<code>SplitOptions</code> instance.</p><p class=language-go>The Go SDK doesn’t use output tags, and instead uses positional ordering for
multiple output PCollections.</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=c1>// To emit elements to multiple output PCollections, create a TupleTag object to identify each collection
</span><span class=c1>// that your ParDo produces. For example, if your ParDo produces three output PCollections (the main output
</span><span class=c1>// and two additional outputs), you must create three TupleTags. The following example code shows how to
@@ -1573,10 +1866,15 @@ multiple output PCollections.</p><div class="language-java snippet"><div class="
<span class=c1>// processWordsMixed uses both a standard return and an emitter function.
</span><span class=c1>// The standard return produces the first PCollection from beam.ParDo2,
</span><span class=c1>// and the emitter produces the second PCollection.
-</span><span class=c1></span><span class=nx>length</span><span class=p>,</span> <span class=nx>mixedMarked</span> <span class=o>:=</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>ParDo2</span><span class=p>(</span><span class=nx>s</span><span class=p>,</span> <span class=nx>processWordsMixed</span><span class=p>,</span> <span class=nx>words</span><span class=p>)</span></code></pre></div></div></div><h4 id=multiple-outputs-dofn>4.5.2. Emitting to multiple outputs in [...]
+</span><span class=c1></span><span class=nx>length</span><span class=p>,</span> <span class=nx>mixedMarked</span> <span class=o>:=</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>ParDo2</span><span class=p>(</span><span class=nx>s</span><span class=p>,</span> <span class=nx>processWordsMixed</span><span class=p>,</span> <span class=nx>words</span><span class=p>)</span></code></pre></div></div></div><div class="language-typescript snippet"><div class="notebook-skip c [...]
+
+<span class=kr>const</span> <span class=p>{</span> <span class=nx>below</span><span class=p>,</span> <span class=nx>above</span><span class=p>,</span> <span class=nx>marked</span> <span class=p>}</span> <span class=o>=</span> <span class=nx>to_split</span><span class=p>.</span><span class=nx>apply</span><span class=p>(</span>
+ <span class=nx>beam</span><span class=p>.</span><span class=nx>split</span><span class=p>([</span><span class=s2>"below"</span><span class=p>,</span> <span class=s2>"above"</span><span class=p>,</span> <span class=s2>"marked"</span><span class=p>])</span>
+<span class=p>);</span>
+</code></pre></div></div></div><h4 id=multiple-outputs-dofn>4.5.2. Emitting to multiple outputs in your DoFn</h4><p class=language-go>Call emitter functions as needed to produce 0 or more elements for its matching
<code>PCollection</code>. The same value can be emitted with multiple emitters.
-As normal, do not mutate values after emitting them from any emitter.</p><p>All emitters should be registered using a generic <code>register.EmitterX[...]</code>
-function. This optimizes runtime execution of the emitter.</p></p><p class=language-go>DoFns can also return a single element via the standard return.
+As normal, do not mutate values after emitting them from any emitter.</p><p class=language-go>All emitters should be registered using a generic <code>register.EmitterX[...]</code>
+function. This optimizes runtime execution of the emitter.</p><p class=language-go>DoFns can also return a single element via the standard return.
The standard return is always the first PCollection returned from beam.ParDo.
Other emitters output to their own PCollections in their defined parameter order.</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=c1>// Inside your ParDo's DoFn, you can emit an element to a specific output PCollection by providing a
</span><span class=c1>// MultiOutputReceiver to your process method, and passing in the appropriate TupleTag to obtain an OutputReceiver.
@@ -1658,15 +1956,27 @@ Other emitters output to their own PCollections in their defined parameter order
<span class=nx>register</span><span class=p>.</span><span class=nf>Function2x1</span><span class=p>(</span><span class=nx>processWordsMixed</span><span class=p>)</span>
<span class=c1>// 1 input of type string => Emitter1[string]
</span><span class=c1></span> <span class=nx>register</span><span class=p>.</span><span class=nx>Emitter1</span><span class=p>[</span><span class=kt>string</span><span class=p>]()</span>
-<span class=p>}</span></code></pre></div></div></div><h4 id=other-dofn-parameters>4.5.3. Accessing additional parameters in your DoFn</h4><p class=language-java>In addition to the element and the <code>OutputReceiver</code>, Beam will populate other parameters to your DoFn’s <code>@ProcessElement</code> method.
+<span class=p>}</span></code></pre></div></div></div><div class="language-typescript snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-typescript data-lang=typescript><span class=kr>const</span> <span class=nx>to_split</span> <span class=o>=</span> <span class=nx>words</span><span class=p>.</span> [...]
+ <span class=k>if</span> <span class=p>(</span><span class=nx>word</span><span class=p>.</span><span class=nx>length</span> <span class=o><</span> <span class=mi>5</span><span class=p>)</span> <span class=p>{</span>
+ <span class=nx>yield</span> <span class=p>{</span> <span class=nx>below</span>: <span class=kt>word</span> <span class=p>};</span>
+ <span class=p>}</span> <span class=k>else</span> <span class=p>{</span>
+ <span class=nx>yield</span> <span class=p>{</span> <span class=nx>above</span>: <span class=kt>word</span> <span class=p>};</span>
+ <span class=p>}</span>
+ <span class=k>if</span> <span class=p>(</span><span class=nx>isMarkedWord</span><span class=p>(</span><span class=nx>word</span><span class=p>))</span> <span class=p>{</span>
+ <span class=nx>yield</span> <span class=p>{</span> <span class=nx>marked</span>: <span class=kt>word</span> <span class=p>};</span>
+ <span class=p>}</span>
+<span class=p>});</span>
+</code></pre></div></div></div><h4 id=other-dofn-parameters>4.5.3. Accessing additional parameters in your DoFn</h4><p class=language-java>In addition to the element and the <code>OutputReceiver</code>, Beam will populate other parameters to your DoFn’s <code>@ProcessElement</code> method.
Any combination of these parameters can be added to your process method in any order.</p><p class=language-py>In addition to the element, Beam will populate other parameters to your DoFn’s <code>process</code> method.
-Any combination of these parameters can be added to your process method in any order.</p><p class=language-go>In addition to the element, Beam will populate other parameters to your DoFn’s <code>ProcessElement</code> method.
+Any combination of these parameters can be added to your process method in any order.</p><p class=language-typescript>In addition to the element, Beam will populate other parameters to your DoFn’s <code>process</code> method.
+These are available by placing accessors in the context argument, just as for side inputs.</p><p class=language-go>In addition to the element, Beam will populate other parameters to your DoFn’s <code>ProcessElement</code> method.
Any combination of these parameters can be added to your process method in a standard order.</p><p class=language-go><strong>context.Context:</strong>
To support consolidated logging and user defined metrics, a <code>context.Context</code> parameter can be requested.
Per Go conventions, if present it’s required to be the first parameter of the <code>DoFn</code> method.</p><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-go data-lang=go><span class=kd>func</span> <span class=nf>MyDoFn</span><span class=p>(</span><span class=nx>ctx [...]
To access the timestamp of an input element, add a parameter annotated with <code>@Timestamp</code> of type <code>Instant</code>. For example:</p><p class=language-py><strong>Timestamp:</strong>
To access the timestamp of an input element, add a keyword parameter default to <code>DoFn.TimestampParam</code>. For example:</p><p class=language-go><strong>Timestamp:</strong>
-To access the timestamp of an input element, add a <code>beam.EventTime</code> parameter before the element. For example:</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=o>.</span><span class=na>of</span><span class=o>(</span><span cla [...]
+To access the timestamp of an input element, add a <code>beam.EventTime</code> parameter before the element. For example:</p><p class=language-typescript><strong>Timestamp:</strong>
+To access the window an input element falls into, add a <code>pardo.windowParam()</code> to the context argument.</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=o>.</span><span class=na>of</span><span class=o>(</span><span class=k>new [...]
<span class=kd>public</span> <span class=kt>void</span> <span class=nf>processElement</span><span class=o>(</span><span class=nd>@Element</span> <span class=n>String</span> <span class=n>word</span><span class=o>,</span> <span class=nd>@Timestamp</span> <span class=n>Instant</span> <span class=n>timestamp</span><span class=o>)</span> <span class=o>{</span>
<span class=o>}})</span></code></pre></div></div></div><div class="language-py snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-py data-lang=py><span class=kn>import</span> <span class=nn>apache_beam</span> <span class=kn>as</span> <span class=nn>beam</span>
@@ -1674,7 +1984,12 @@ To access the timestamp of an input element, add a <code>beam.EventTime</code> p
<span class=k>def</span> <span class=nf>process</span><span class=p>(</span><span class=bp>self</span><span class=p>,</span> <span class=n>element</span><span class=p>,</span> <span class=n>timestamp</span><span class=o>=</span><span class=n>beam</span><span class=o>.</span><span class=n>DoFn</span><span class=o>.</span><span class=n>TimestampParam</span><span class=p>):</span>
<span class=c1># access timestamp of element.</span>
- <span class=k>pass</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-go data-lang=go><span class=kd>func</span> <span class=nf>MyDoFn</span><span class=p>(</span><span class=nx>ts</span> <span class=nx>beam</span><span class=p>.</sp [...]
+ <span class=k>pass</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-go data-lang=go><span class=kd>func</span> <span class=nf>MyDoFn</span><span class=p>(</span><span class=nx>ts</span> <span class=nx>beam</span><span class=p>.</sp [...]
+ <span class=k>return</span> <span class=nx>context</span><span class=p>.</span><span class=nx>timestamp</span><span class=p>.</span><span class=nx>lookup</span><span class=p>();</span>
+<span class=p>}</span>
+
+<span class=nx>pcoll</span><span class=p>.</span><span class=nx>map</span><span class=p>(</span><span class=nx>processFn</span><span class=p>,</span> <span class=p>{</span> <span class=nx>timestamp</span>: <span class=kt>pardo.timestampParam</span><span class=p>()</span> <span class=p>});</span>
+</code></pre></div></div></div><p class=language-java><strong>Window:</strong>
To access the window an input element falls into, add a parameter of the type of the window used for the input <code>PCollection</code>.
If the parameter is a window type (a subclass of <code>BoundedWindow</code>) that does not match the input <code>PCollection</code>, then an error
will be raised. If an element falls in multiple windows (for example, this will happen when using <code>SlidingWindows</code>), then the
@@ -1687,7 +2002,10 @@ To access the window an input element falls into, add a <code>beam.Window</code>
If an element falls in multiple windows (for example, this will happen when using SlidingWindows),
then the <code>ProcessElement</code> method will be invoked multiple time for the element, once for each window.
Since <code>beam.Window</code> is an interface it’s possible to type assert to the concrete implementation of the window.
-For example, when fixed windows are being used, the window is of type <code>window.IntervalWindow</code>.</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=o>.</span><span class=na>of</span><span class=o>(</span><span class=k>new</span> [...]
+For example, when fixed windows are being used, the window is of type <code>window.IntervalWindow</code>.</p><p class=language-typescript><strong>Window:</strong>
+To access the window an input element falls into, add a <code>pardo.windowParam()</code> to the context argument.
+If an element falls in multiple windows (for example, this will happen when using <code>SlidingWindows</code>), then the
+function will be invoked multiple time for the element, once for each window.</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=o>.</span><span class=na>of</span><span class=o>(</span><span class=k>new</span> <span class=n>DoFn</span><sp [...]
<span class=kd>public</span> <span class=kt>void</span> <span class=nf>processElement</span><span class=o>(</span><span class=nd>@Element</span> <span class=n>String</span> <span class=n>word</span><span class=o>,</span> <span class=n>IntervalWindow</span> <span class=n>window</span><span class=o>)</span> <span class=o>{</span>
<span class=o>}})</span></code></pre></div></div></div><div class="language-py snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-py data-lang=py><span class=kn>import</span> <span class=nn>apache_beam</span> <span class=kn>as</span> <span class=nn>beam</span>
@@ -1698,14 +2016,18 @@ For example, when fixed windows are being used, the window is of type <code>wind
<span class=k>pass</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-go data-lang=go><span class=kd>func</span> <span class=nf>MyDoFn</span><span class=p>(</span><span class=nx>w</span> <span class=nx>beam</span><span class=p>.</spa [...]
<span class=nx>iw</span> <span class=o>:=</span> <span class=nx>w</span><span class=p>.(</span><span class=nx>window</span><span class=p>.</span><span class=nx>IntervalWindow</span><span class=p>)</span>
<span class=o>...</span>
-<span class=p>}</span></code></pre></div></div></div><p class=language-java><strong>PaneInfo:</strong>
+<span class=p>}</span></code></pre></div></div></div><div class="language-typescript snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-typescript data-lang=typescript><span class=nx>pcoll</span><span class=p>.</span><span class=nx>map</span><span class=p>(</span><span class=nx>processFn</span><spa [...]
+</code></pre></div></div></div><p class=language-java><strong>PaneInfo:</strong>
When triggers are used, Beam provides a <code>PaneInfo</code> object that contains information about the current firing. Using <code>PaneInfo</code>
you can determine whether this is an early or a late firing, and how many times this window has already fired for this key.</p><p class=language-py><strong>PaneInfo:</strong>
When triggers are used, Beam provides a <code>DoFn.PaneInfoParam</code> object that contains information about the current firing. Using <code>DoFn.PaneInfoParam</code>
you can determine whether this is an early or a late firing, and how many times this window has already fired for this key.
This feature implementation in Python SDK is not fully completed; see more at <a href=https://github.com/apache/beam/issues/18721>Issue 17821</a>.</p><p class=language-go><strong>PaneInfo:</strong>
When triggers are used, Beam provides <code>beam.PaneInfo</code> object that contains information about the current firing. Using <code>beam.PaneInfo</code>
-you can determine whether this is an early or a late firing, and how many times this window has already fired for this key.</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=o>.</span><span class=na>of</span><span class=o>(</span><span c [...]
+you can determine whether this is an early or a late firing, and how many times this window has already fired for this key.</p><p class=language-typescript><strong>Window:</strong>
+To access the window an input element falls into, add a <code>pardo.paneInfoParam()</code> to the context argument.
+Using <code>beam.PaneInfo</code> you can determine whether this is an early or a late firing,
+and how many times this window has already fired for this key.</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=o>.</span><span class=na>of</span><span class=o>(</span><span class=k>new</span> <span class=n>DoFn</span><span class=o>< [...]
<span class=kd>public</span> <span class=kt>void</span> <span class=nf>processElement</span><span class=o>(</span><span class=nd>@Element</span> <span class=n>String</span> <span class=n>word</span><span class=o>,</span> <span class=n>PaneInfo</span> <span class=n>paneInfo</span><span class=o>)</span> <span class=o>{</span>
<span class=o>}})</span></code></pre></div></div></div><div class="language-py snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-py data-lang=py><span class=kn>import</span> <span class=nn>apache_beam</span> <span class=kn>as</span> <span class=nn>beam</span>
@@ -1731,7 +2053,8 @@ you can determine whether this is an early or a late firing, and how many times
<span class=k>for</span> <span class=nx>_</span><span class=p>,</span> <span class=nx>w</span> <span class=o>:=</span> <span class=k>range</span> <span class=nx>words</span> <span class=p>{</span>
<span class=nf>emitWords</span><span class=p>(</span><span class=nx>w</span><span class=p>)</span>
<span class=p>}</span>
-<span class=p>}</span></code></pre></div></div></div><p class=language-java><strong>PipelineOptions:</strong>
+<span class=p>}</span></code></pre></div></div></div><div class="language-typescript snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-typescript data-lang=typescript><span class=nx>pcoll</span><span class=p>.</span><span class=nx>map</span><span class=p>(</span><span class=nx>processFn</span><spa [...]
+</code></pre></div></div></div><p class=language-java><strong>PipelineOptions:</strong>
The <code>PipelineOptions</code> for the current pipeline can always be accessed in a process method by adding it
as a parameter:</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=o>.</span><span class=na>of</span><span class=o>(</span><span class=k>new</span> <span class=n>DoFn</span><span class=o><</span><span class=n>String</span><span class=o [...]
<span class=kd>public</span> <span class=kt>void</span> <span class=nf>processElement</span><span class=o>(</span><span class=nd>@Element</span> <span class=n>String</span> <span class=n>word</span><span class=o>,</span> <span class=n>PipelineOptions</span> <span class=n>options</span><span class=o>)</span> <span class=o>{</span>
@@ -1743,7 +2066,9 @@ Timers are explained in more detail in the
In addition to aforementioned parameters, user defined Timer and State parameters can be used in a stateful DoFn.
Timers and States are explained in more detail in the
<a href=/blog/2017/08/28/timely-processing.html>Timely (and Stateful) Processing with Apache Beam</a> blog post.</p><p class=language-go><strong>Timer and State:</strong>
-This feature isn’t implemented in the Go SDK; see more at <a href=https://github.com/apache/beam/issues/20510>Issue 20510</a>. Once implemented, user defined Timer and State parameters can be used in a stateful DoFn.</p><div class="language-py snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=languag [...]
+This feature isn’t implemented in the Go SDK; see more at <a href=https://github.com/apache/beam/issues/20510>Issue 20510</a>. Once implemented, user defined Timer and State parameters can be used in a stateful DoFn.</p><p class=language-typescript><strong>Timer and State:</strong>
+This feature isn’t implemented in the Typescript SDK yet,
+but can be used from a cross-language transform.</p><div class="language-py snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-py data-lang=py><span class=k>class</span> <span class=nc>StatefulDoFn</span><span class=p>(</span><span class=n>beam</span><span class=o>.</span><span class=n>DoFn</span>< [...]
<span class=s2>"""An example stateful DoFn with state and timer"""</span>
<span class=n>BUFFER_STATE_1</span> <span class=o>=</span> <span class=n>BagStateSpec</span><span class=p>(</span><span class=s1>'buffer1'</span><span class=p>,</span> <span class=n>beam</span><span class=o>.</span><span class=n>BytesCoder</span><span class=p>())</span>
@@ -1856,8 +2181,19 @@ appeared in the original data.</li></ol><div class="language-java snippet"><div
<span class=c1>// Return any PCollections that should be available after
</span><span class=c1></span> <span class=c1>// the composite transform.
</span><span class=c1></span> <span class=k>return</span> <span class=nx>wordCounts</span>
-<span class=p>}</span></code></pre></div></div></div><blockquote><p><strong>Note:</strong> Because <code>Count</code> is itself a composite transform,
-<code>CountWords</code> is also a nested composite transform.</p></blockquote><h4 id=composite-transform-creation>4.6.2. Creating a composite transform</h4><p class="language-java language-py">To create your own composite transform, create a subclass of the <code>PTransform</code>
+<span class=p>}</span></code></pre></div></div></div><div class="language-typescript snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-typescript data-lang=typescript><span class=kd>function</span> <span class=nx>countWords</span><span class=p>(</span><span class=nx>lines</span>: <span class=kt>PC [...]
+ <span class=k>return</span> <span class=nx>lines</span> <span class=c1>//
+</span><span class=c1></span> <span class=p>.</span><span class=nx>map</span><span class=p>((</span><span class=nx>s</span>: <span class=kt>string</span><span class=p>)</span> <span class=o>=></span> <span class=nx>s</span><span class=p>.</span><span class=nx>toLowerCase</span><span class=p>())</span>
+ <span class=p>.</span><span class=nx>flatMap</span><span class=p>(</span><span class=kd>function</span><span class=o>*</span> <span class=nx>splitWords</span><span class=p>(</span><span class=nx>line</span>: <span class=kt>string</span><span class=p>)</span> <span class=p>{</span>
+ <span class=nx>yield</span><span class=o>*</span> <span class=nx>line</span><span class=p>.</span><span class=nx>split</span><span class=p>(</span><span class=sr>/[^a-z]+/</span><span class=p>);</span>
+ <span class=p>})</span>
+ <span class=p>.</span><span class=nx>apply</span><span class=p>(</span><span class=nx>beam</span><span class=p>.</span><span class=nx>countPerElement</span><span class=p>());</span>
+<span class=p>}</span>
+
+<span class=kr>const</span> <span class=nx>counted</span> <span class=o>=</span> <span class=nx>lines</span><span class=p>.</span><span class=nx>apply</span><span class=p>(</span><span class=nx>countWords</span><span class=p>);</span>
+</code></pre></div></div></div><blockquote><p><strong>Note:</strong> Because <code>Count</code> is itself a composite transform,
+<code>CountWords</code> is also a nested composite transform.</p></blockquote><h4 id=composite-transform-creation>4.6.2. Creating a composite transform</h4><p class=language-typescript>A PTransform in the Typescript SDK is simply a function that accepts and
+returns <code>PValue</code>s such as <code>PCollection</code>s.</p><p class="language-java language-py">To create your own composite transform, create a subclass of the <code>PTransform</code>
class and override the <code>expand</code> method to specify the actual processing logic.
You can then use this transform just as you would a built-in transform from the
Beam SDK.</p><p class=language-java>For the <code>PTransform</code> class type parameters, you pass the <code>PCollection</code> types
@@ -4488,7 +4824,7 @@ functions as shortcuts for single, unnamed inputs/outputs or define a map for na
<span class=nx>outT</span> <span class=o>:=</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>UnnamedOutput</span><span class=p>(</span><span class=nx>typex</span><span class=p>.</span><span class=nf>New</span><span class=p>(</span><span class=nx>reflectx</span><span class=p>.</span><span class=nx>String</span><span class=p>))</span>
<span class=nx>res</span> <span class=o>:=</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>CrossLanguage</span><span class=p>(</span><span class=nx>s</span><span class=p>,</span> <span class=nx>urn</span><span class=p>,</span> <span class=nx>payload</span><span class=p>,</span> <span class=nx>expansionAddr</span><span class=p>,</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>UnnamedInput</span><span class=p>(</span><span class=nx>inputPCol</spa [...]
</code></pre></div></li><li><p>After the job has been submitted to the Beam runner, shutdown the expansion service by
-terminating the expansion service process.</p></li></ol><h3 id=x-lang-transform-runner-support>13.3. Runner Support</h3><p>Currently, portable runners such as Flink, Spark, and the direct runner can be used with multi-language pipelines.</p><p>Dataflow supports multi-language pipelines through the Dataflow Runner v2 backend architecture.</p><h3 id=x-lang-transform-tips-troubleshooting>13.4 Tips and Troubleshooting</h3><p>For additional tips and troubleshooting information, see <a href=ht [...]
+terminating the expansion service process.</p></li></ol><h3 id=x-lang-transform-runner-support>13.3. Runner Support</h3><p>Currently, portable runners such as Flink, Spark, and the direct runner can be used with multi-language pipelines.</p><p>Dataflow supports multi-language pipelines through the Dataflow Runner v2 backend architecture.</p><h3 id=x-lang-transform-tips-troubleshooting>13.4 Tips and Troubleshooting</h3><p>For additional tips and troubleshooting information, see <a href=ht [...]
<a href=http://www.apache.org>The Apache Software Foundation</a>
| <a href=/privacy_policy>Privacy Policy</a>
| <a href=/feed.xml>RSS Feed</a><br><br>Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation. All other products or name brands are trademarks of their respective holders, including The Apache Software Foundation.</div></div><div class="footer__cols__col footer__cols__col__logos"><div class=footer__cols__col--group><div class=footer__cols__col__logo><a href=https://github.com/apache/beam><im [...]
\ No newline at end of file
diff --git a/website/generated-content/sitemap.xml b/website/generated-content/sitemap.xml
index a63abbc22dd..08f13c12777 100644
--- a/website/generated-content/sitemap.xml
+++ b/website/generated-content/sitemap.xml
@@ -1 +1 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/go-2.40/</loc><lastmod>2022-07-06T14:03:32-04:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2022-07-06T14:03:32-04:00</lastmod></url><url><loc>/blog/</loc><lastmod>2022-07-06T14:03:32-04:00</lastmod></url><url><loc>/categories/</loc><lastmod>2022-07-06T14:03:32-04:00</lastmod></url><url><loc>/categories [...]
\ No newline at end of file
+<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/go-2.40/</loc><lastmod>2022-07-06T14:03:32-04:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2022-07-06T14:03:32-04:00</lastmod></url><url><loc>/blog/</loc><lastmod>2022-07-06T14:03:32-04:00</lastmod></url><url><loc>/categories/</loc><lastmod>2022-07-06T14:03:32-04:00</lastmod></url><url><loc>/categories [...]
\ No newline at end of file