You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2020/10/24 00:03:07 UTC
[beam] branch asf-site updated: Publishing website 2020/10/24
00:02:43 at commit 56e9ce2
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 9bfa8b7 Publishing website 2020/10/24 00:02:43 at commit 56e9ce2
9bfa8b7 is described below
commit 9bfa8b7fe3063731d0466646dd8152a15be45d10
Author: jenkins <bu...@apache.org>
AuthorDate: Sat Oct 24 00:02:43 2020 +0000
Publishing website 2020/10/24 00:02:43 at commit 56e9ce2
---
website/generated-content/beam-2.25.0.md | 85 ++++++++++++++++++++++
website/generated-content/documentation/index.xml | 34 ++++-----
.../documentation/programming-guide/index.html | 36 ++++-----
website/generated-content/sitemap.xml | 2 +-
4 files changed, 121 insertions(+), 36 deletions(-)
diff --git a/website/generated-content/beam-2.25.0.md b/website/generated-content/beam-2.25.0.md
new file mode 100644
index 0000000..d757147
--- /dev/null
+++ b/website/generated-content/beam-2.25.0.md
@@ -0,0 +1,85 @@
+---
+title: "Apache Beam 2.25.0"
+date: 2020-10-23 14:00:00 -0800
+categories:
+ - blog
+authors:
+ - Robin Qiu
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+We are happy to present the new 2.25.0 release of Apache Beam. This release includes both improvements and new functionality.
+See the [download page](/get-started/downloads/#2250-2020-10-23) for this release.
+For more information on changes in 2.25.0, check out the
+[detailed release notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12347147).
+
+## Highlights
+
+* Splittable DoFn is now the default for executing the Read transform for Java based runners (Direct, Flink, Jet, Samza, Twister2). The expected output of the Read transform is unchanged. Users can opt-out using `--experiments=use_deprecated_read`. The Apache Beam community is looking for feedback for this change as the community is planning to make this change permanent with no opt-out. If you run into an issue requiring the opt-out, please send an e-mail to [user@beam.apache.org](mailt [...]
+
+## I/Os
+
+* Added cross-language support to Java's KinesisIO, now available in the Python module `apache_beam.io.kinesis` ([BEAM-10138](https://issues.apache.org/jira/browse/BEAM-10138), [BEAM-10137](https://issues.apache.org/jira/browse/BEAM-10137)).
+* Update Snowflake JDBC dependency for SnowflakeIO ([BEAM-10864](https://issues.apache.org/jira/browse/BEAM-10864))
+* Added cross-language support to Java's SnowflakeIO.Write, now available in the Python module `apache_beam.io.snowflake` ([BEAM-9898](https://issues.apache.org/jira/browse/BEAM-9898)).
+* Added delete function to Java's `ElasticsearchIO#Write`. Now, Java's ElasticsearchIO can be used to selectively delete documents using `withIsDeleteFn` function ([BEAM-5757](https://issues.apache.org/jira/browse/BEAM-5757)).
+* Java SDK: Added new IO connector for InfluxDB - InfluxDbIO ([BEAM-2546](https://issues.apache.org/jira/browse/BEAM-2546)).
+
+## New Features / Improvements
+
+* Support for repeatable fields in JSON decoder for `ReadFromBigQuery` added. (Python) ([BEAM-10524](https://issues.apache.org/jira/browse/BEAM-10524))
+* Added an opt-in, performance-driven runtime type checking system for the Python SDK ([BEAM-10549](https://issues.apache.org/jira/browse/BEAM-10549)).
+ More details will be in an upcoming [blog post](https://beam.apache.org/blog/python-performance-runtime-type-checking/index.html).
+* Added support for Python 3 type annotations on PTransforms using typed PCollections ([BEAM-10258](https://issues.apache.org/jira/browse/BEAM-10258)).
+ More details will be in an upcoming [blog post](https://beam.apache.org/blog/python-improved-annotations/index.html).
+* Improved the Interactive Beam API where recording streaming jobs now start a long running background recording job. Running ib.show() or ib.collect() samples from the recording ([BEAM-10603](https://issues.apache.org/jira/browse/BEAM-10603)).
+* In Interactive Beam, ib.show() and ib.collect() now have "n" and "duration" as parameters. These mean read only up to "n" elements and up to "duration" seconds of data read from the recording ([BEAM-10603](https://issues.apache.org/jira/browse/BEAM-10603)).
+* Initial preview of [Dataframes](https://s.apache.org/simpler-python-pipelines-2020#slide=id.g905ac9257b_1_21) support.
+ See also example at apache_beam/examples/wordcount_dataframe.py
+* Fixed support for type hints on `@ptransform_fn` decorators in the Python SDK.
+ ([BEAM-4091](https://issues.apache.org/jira/browse/BEAM-4091))
+ This has not enabled by default to preserve backwards compatibility; use the
+ `--type_check_additional=ptransform_fn` flag to enable. It may be enabled by
+ default in future versions of Beam.
+
+## Breaking Changes
+
+* Python 2 and Python 3.5 support dropped ([BEAM-10644](https://issues.apache.org/jira/browse/BEAM-10644), [BEAM-9372](https://issues.apache.org/jira/browse/BEAM-9372)).
+* Pandas 1.x allowed. Older version of Pandas may still be used, but may not be as well tested.
+
+## Deprecations
+
+* Python transform ReadFromSnowflake has been moved from `apache_beam.io.external.snowflake` to `apache_beam.io.snowflake`. The previous path will be removed in the future versions.
+
+## Known Issues
+
+* Dataflow streaming timers once against not strictly time ordered when set earlier mid-bundle, as the fix for [BEAM-8543](https://issues.apache.org/jira/browse/BEAM-8543) introduced more severe bugs and has been rolled back.
+
+
+## List of Contributors
+
+According to git shortlog, the following people contributed to the 2.25.0 release. Thank you to all contributors!
+
+Ahmet Altay, Alan Myrvold, Aldair Coronel Ruiz, Alexey Romanenko, Andrew Pilloud, Ankur Goenka,
+Ayoub ENNASSIRI, Bipin Upadhyaya, Boyuan Zhang, Brian Hulette, Brian Michalski, Chad Dombrova,
+Chamikara Jayalath, Damon Douglas, Daniel Oliveira, David Cavazos, David Janicek, Doug Roeper, Eric
+Roshan-Eisner, Etta Rapp, Eugene Kirpichov, Filipe Regadas, Heejong Lee, Ihor Indyk, Irvi Firqotul
+Aini, Ismaël Mejía, Jan Lukavský, Jayendra, Jiadai Xia, Jithin Sukumar, Jozsef Bartok, Kamil
+Gałuszka, Kamil Wasilewski, Kasia Kucharczyk, Kenneth Jung, Kenneth Knowles, Kevin Puthusseri, Kevin
+Sijo Puthusseri, KevinGG, Kyle Weaver, Leiyi Zhang, Lourens Naudé, Luke Cwik, Matthew Ouyang,
+Maximilian Michels, Michal Walenia, Milan Cermak, Monica Song, Nelson Osacky, Neville Li, Ning Kang,
+Pablo Estrada, Piotr Szuberski, Qihang, Rehman, Reuven Lax, Robert Bradshaw, Robert Burke, Rui Wang,
+Saavan Nanavati, Sam Bourne, Sam Rohde, Sam Whittle, Sergiy Kolesnikov, Sindy Li, Siyuan Chen, Steve
+Niemitz, Terry Xian, Thomas Weise, Tobiasz Kędzierski, Truc Le, Tyson Hamilton, Udi Meiri, Valentyn
+Tymofieiev, Yichi Zhang, Yifan Mai, Yueyang Qiu, annaqin418, danielxjd, dennis, dp, fuyuwei,
+lostluck, nehsyc, odeshpande, odidev, pulasthi, purbanow, rworley-monster, sclukas77, terryxian78,
+tvalentyn, yoshiki.obata
diff --git a/website/generated-content/documentation/index.xml b/website/generated-content/documentation/index.xml
index 0ffa63e..837871e 100644
--- a/website/generated-content/documentation/index.xml
+++ b/website/generated-content/documentation/index.xml
@@ -6293,7 +6293,7 @@ potentially while the pipeline is running. There could be different reasons for
<li>Retrieve an accurate count of the number of elements that have been processed;</li>
<li>&hellip;and so on.</li>
</ul>
-<h3 id="101-the-main-concepts-of-beam-metrics">10.1 The main concepts of Beam metrics</h3>
+<h3 id="101-the-main-concepts-of-beam-metrics">10.1. The main concepts of Beam metrics</h3>
<ul>
<li><strong>Named</strong>. Each metric has a name which consists of a namespace and an actual name. The
namespace can be used to differentiate between multiple metrics with the same name and also
@@ -6314,7 +6314,7 @@ transform reported, as well as aggregating the metric across the entire pipeline
<p><strong>Note:</strong> It is runner-dependent whether metrics are accessible during pipeline execution or only
after jobs have completed.</p>
</blockquote>
-<h3 id="types-of-metrics">10.2 Types of metrics</h3>
+<h3 id="types-of-metrics">10.2. Types of metrics</h3>
<p>There are three types of metrics that are supported for the moment: <code>Counter</code>, <code>Distribution</code> and
<code>Gauge</code>.</p>
<p><strong>Counter</strong>: A metric that reports a single long value and can be incremented or decremented.</p>
@@ -6350,7 +6350,7 @@ collected from many workers the value may not be the absolute last, but one of t
<span class="o">...</span>
<span class="o">}</span></code></pre></div>
</div>
-<h3 id="querying-metrics">10.3 Querying metrics</h3>
+<h3 id="querying-metrics">10.3. Querying metrics</h3>
<p><code>PipelineResult</code> has a method <code>metrics()</code> which returns a <code>MetricResults</code> object that allows
accessing metrics. The main method available in <code>MetricResults</code> allows querying for all metrics
matching a given filter.</p>
@@ -6373,7 +6373,7 @@ matching a given filter.</p>
<span class="n">T</span> <span class="nf">getAttempted</span><span class="o">();</span>
<span class="o">}</span></code></pre></div>
</div>
-<h3 id="using-metrics">10.4 Using metrics in pipeline</h3>
+<h3 id="using-metrics">10.4. Using metrics in pipeline</h3>
<p>Below, there is a simple example of how to use a <code>Counter</code> metric in a user pipeline.</p>
<div class=language-java>
<div class="highlight"><pre class="chroma"><code class="language-java" data-lang="java"><span class="c1">// creating a pipeline with custom metrics DoFn
@@ -6404,7 +6404,7 @@ matching a given filter.</p>
<span class="o">}</span>
<span class="o">}</span></code></pre></div>
</div>
-<h3 id="export-metrics">10.5 Export metrics</h3>
+<h3 id="export-metrics">10.5. Export metrics</h3>
<p>Beam metrics can be exported to external sinks. If a metrics sink is set up in the configuration, the runner will push metrics to it at a default 5s period.
The configuration is held in the <a href="https://beam.apache.org/releases/javadoc/2.19.0/org/apache/beam/sdk/metrics/MetricsOptions.html">MetricsOptions</a> class.
It contains push period configuration and also sink specific options such as type and URL. As for now only the REST HTTP and the Graphite sinks are supported and only
@@ -6434,7 +6434,7 @@ to other nodes in the graph. A <code>DoFn</code> can declare multiple stat
<p class="language-py">In Python DoFn declares states to be accessed by creating <code>StateSpec</code> class member variables representing each state. Each
<code>StateSpec</code> is initialized with a name, this name is unique to a ParDo in the graph and has no relation
to other nodes in the graph. A <code>DoFn</code> can declare multiple state variables.</p>
-<h3 id="types-of-state">11.1 Types of state</h3>
+<h3 id="types-of-state">11.1. Types of state</h3>
<p>Beam provides several types of state:</p>
<h4 id="valuestate">ValueState</h4>
<p>A ValueState is a scalar state value. For each key in the input, a ValueState will store a typed value that can be
@@ -6519,7 +6519,7 @@ bags larger than available memory.</p>
<span class="n">_</span> <span class="o">=</span> <span class="p">(</span><span class="n">p</span> <span class="o">|</span> <span class="s1">&#39;Read per user&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">ReadPerUser</span><span class="p">()</span>
<span class="o">|</span> <span class="s1">&#39;Bag state pardo&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">BagStateDoFn</span><span class="p">()))</span></code></pre></div>
</div>
-<h3 id="deferred-state-reads">11.2 Deferred state reads</h3>
+<h3 id="deferred-state-reads">11.2. Deferred state reads</h3>
<p>When a <code>DoFn</code> contains multiple state specifications, reading each one in order can be slow. Calling the <code>read()</code> function
on a state can cause the runner to perform a blocking read. Performing multiple blocking reads in sequence adds latency
to element processing. If you know that a state will always be read, you can annotate it as @AlwaysFetched, and then the
@@ -6565,12 +6565,12 @@ be read in the future, allowing multiple state reads to be batched together.<
<span class="o">}</span>
<span class="o">}));</span></code></pre></div>
</div>
-<h3 id="timers">11.3 Timers</h3>
+<h3 id="timers">11.3. Timers</h3>
<p>Beam provides a per-key timer callback API. This allows for delayed processing of data stored using the state API.
Timers can be set to callback at either an event-time or a processing-time timestamp. Every timer is identified with a
TimerId. A given timer for a key can only be set for a single timestamp. Calling set on a timer overwrites the previous
firing time for that key&rsquo;s timer.</p>
-<h4 id="event-time-timers">11.3.1 Event-time timers</h4>
+<h4 id="event-time-timers">11.3.1. Event-time timers</h4>
<p>Event-time timers fire when the input watermark for the DoFn passes the time at which the timer is set, meaning that
the runner believes that there are no more elements to be processed with timestamps before the timer timestamp. This
allows for event-time aggregations.</p>
@@ -6611,7 +6611,7 @@ allows for event-time aggregations.</p>
<span class="n">_</span> <span class="o">=</span> <span class="p">(</span><span class="n">p</span> <span class="o">|</span> <span class="s1">&#39;Read per user&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">ReadPerUser</span><span class="p">()</span>
<span class="o">|</span> <span class="s1">&#39;EventTime timer pardo&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">EventTimerDoFn</span><span class="p">()))</span></code></pre></div>
</div>
-<h4 id="processing-time-timers">11.3.2 Processing-time timers</h4>
+<h4 id="processing-time-timers">11.3.2. Processing-time timers</h4>
<p>Processing-time timers fire when the real wall-clock time passes. This is often used to create larger batches of data
before processing. It can also be used to schedule events that should occur at a specific time. Just like with
event-time timers, processing-time timers are per key - each key has a separate copy of the timer.</p>
@@ -6649,7 +6649,7 @@ to the current time. In Java, the <code>Timer.offset</code> and <code>T
<span class="n">_</span> <span class="o">=</span> <span class="p">(</span><span class="n">p</span> <span class="o">|</span> <span class="s1">&#39;Read per user&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">ReadPerUser</span><span class="p">()</span>
<span class="o">|</span> <span class="s1">&#39;ProcessingTime timer pardo&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">ProcessingTimerDoFn</span><span class="p">()))</span></code></pre></div>
</div>
-<h4 id="dynamic-timer-tags">11.3.3 Dynamic timer tags</h4>
+<h4 id="dynamic-timer-tags">11.3.3. Dynamic timer tags</h4>
<p>Beam also supports dynamically setting a timer tag using <code>TimerMap</code>. This allows for setting multiple different timers
in a <code>DoFn</code> and allowing for the timer tags to be dynamically chosen - e.g. based on data in the input elements. A
timer with a specific tag can only be set to a single timestamp, so setting the timer again has the effect of
@@ -6674,7 +6674,7 @@ id, and timers in different timer families are independent.</p>
<div class=language-python>
<div class="highlight"><pre class="chroma"><code class="language-python" data-lang="python"><span class="n">To</span> <span class="n">be</span> <span class="n">supported</span><span class="p">,</span> <span class="n">See</span> <span class="n">BEAM</span><span class="o">-</span><span class="mi">9602</span></code></pre></div>
</div>
-<h4 id="timer-output-timestamps">11.3.4 Timer output timestamps</h4>
+<h4 id="timer-output-timestamps">11.3.4. Timer output timestamps</h4>
<p>By default, event-time timers will hold the output watermark of the <code>ParDo</code> to the timestamp of the timer. This means
that if a timer is set to 12pm, any windowed aggregations or event-time timers later in the pipeline graph that finish
after 12pm will not expire. The timestamp of the timer is also the default output timestamp for the timer callback. This
@@ -6765,10 +6765,10 @@ past the timestamp of the minimum element. The following code demonstrates this.
<span class="o">}</span>
<span class="o">}));</span></code></pre></div>
</div>
-<h3 id="garbage-collecting-state">11.4 Garbage collecting state</h3>
+<h3 id="garbage-collecting-state">11.4. Garbage collecting state</h3>
<p>Per-key state needs to be garbage collected, or eventually the increasing size of state may negatively impact
performance. There are two common strategies for garbage collecting state.</p>
-<h5 id="using-windows-for-garbage-collection">11.4.1 <strong>Using windows for garbage collection</strong></h5>
+<h5 id="using-windows-for-garbage-collection">11.4.1. <strong>Using windows for garbage collection</strong></h5>
<p>All state and timers for a key is scoped to the window it is in. This means that depending on the timestamp of the
input element the ParDo will see different values for the state depending on the window that element falls into. In
addition, once the input watermark passes the end of the window, the runner should garbage collect all state for that
@@ -6802,7 +6802,7 @@ garbage-collection strategy.</p>
</div>
<p>This <code>ParDo</code> stores state per day. Once the pipeline is done processing data for a given day, all the state for that
day is garbage collected.</p>
-<h5 id="using-timers-for-garbage-collection">11.4.1 <strong>Using timers For garbage collection</strong></h5>
+<h5 id="using-timers-for-garbage-collection">11.4.1. <strong>Using timers For garbage collection</strong></h5>
<p>In some cases, it is difficult to find a windowing strategy that models the desired garbage-collection strategy. For
example, a common desire is to garbage collect state for a key once no activity has been seen on the key for some time.
This can be done by updating a timer that garbage collects state. For example</p>
@@ -6865,7 +6865,7 @@ This can be done by updating a timer that garbage collects state. For example<
<span class="n">_</span> <span class="o">=</span> <span class="p">(</span><span class="n">p</span> <span class="o">|</span> <span class="s1">&#39;Read per user&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">ReadPerUser</span><span class="p">()</span>
<span class="o">|</span> <span class="s1">&#39;User DoFn&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">UserDoFn</span><span class="p">()))</span></code></pre></div>
</div>
-<h3 id="state-timers-examples">11.5 State and timers examples</h3>
+<h3 id="state-timers-examples">11.5. State and timers examples</h3>
<p>Following are some example uses of state and timers</p>
<h4 id="joining-clicks-and-views">11.5.1. Joining clicks and views</h4>
<p>In this example, the pipeline is processing data from an e-commerce site&rsquo;s home page. There are two input streams:
@@ -6938,7 +6938,7 @@ event before the view event. The one hour join timeout should be based on event
<span class="o">}</span>
<span class="o">}));</span></code></pre></div>
</div>
-<h4 id="batching-rpcs">11.5.2 Batching RPCs</h4>
+<h4 id="batching-rpcs">11.5.2. Batching RPCs</h4>
<p>In this example, input elements are being forwarded to an external RPC service. The RPC accepts batch requests -
multiple events for the same user can be batched in a single RPC call. Since this RPC service also imposes rate limits,
we want to batch ten seconds worth of events together in order to reduce the number of calls.</p>
diff --git a/website/generated-content/documentation/programming-guide/index.html b/website/generated-content/documentation/programming-guide/index.html
index 58831b2..fceb292 100644
--- a/website/generated-content/documentation/programming-guide/index.html
+++ b/website/generated-content/documentation/programming-guide/index.html
@@ -1,7 +1,7 @@
<!doctype html><html lang=en class=no-js><head><meta charset=utf-8><meta http-equiv=x-ua-compatible content="IE=edge"><meta name=viewport content="width=device-width,initial-scale=1"><title>Beam Programming Guide</title><meta name=description content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain [...]
<span class=sr-only>Toggle navigation</span>
<span class=icon-bar></span><span class=icon-bar></span><span class=icon-bar></span></button>
-<a href=/ class=navbar-brand><img alt=Brand style=height:25px src=/images/beam_logo_navbar.png></a></div><div class="navbar-mask closed"></div><div id=navbar class="navbar-container closed"><ul class="nav navbar-nav"><li><a href=/get-started/beam-overview/>Get Started</a></li><li><a href=/documentation/>Documentation</a></li><li><a href=/documentation/sdks/java/>Languages</a></li><li><a href=/documentation/runners/capability-matrix/>RUNNERS</a></li><li><a href=/roadmap/>Roadmap</a></li>< [...]
+<a href=/ class=navbar-brand><img alt=Brand style=height:25px src=/images/beam_logo_navbar.png></a></div><div class="navbar-mask closed"></div><div id=navbar class="navbar-container closed"><ul class="nav navbar-nav"><li><a href=/get-started/beam-overview/>Get Started</a></li><li><a href=/documentation/>Documentation</a></li><li><a href=/documentation/sdks/java/>Languages</a></li><li><a href=/documentation/runners/capability-matrix/>RUNNERS</a></li><li><a href=/roadmap/>Roadmap</a></li>< [...]
Beam SDKs to create data processing pipelines. It provides guidance for using
the Beam SDK classes to build and test your pipeline. It is not intended as an
exhaustive reference, but as a language-agnostic, high-level guide to
@@ -2166,7 +2166,7 @@ elements, or after a minute.</p><div class=language-java><div class=highlight><p
<span class=n>trigger</span><span class=o>=</span><span class=n>Repeatedly</span><span class=p>(</span>
<span class=n>AfterAny</span><span class=p>(</span><span class=n>AfterCount</span><span class=p>(</span><span class=mi>100</span><span class=p>),</span> <span class=n>AfterProcessingTime</span><span class=p>(</span><span class=mi>1</span> <span class=o>*</span> <span class=mi>60</span><span class=p>))),</span>
<span class=n>accumulation_mode</span><span class=o>=</span><span class=n>AccumulationMode</span><span class=o>.</span><span class=n>DISCARDING</span><span class=p>)</span></code></pre></div></div><h2 id=metrics>10. Metrics</h2><p>In the Beam model, metrics provide some insight into the current state of a user pipeline,
-potentially while the pipeline is running. There could be different reasons for that, for instance:</p><ul><li>Check the number of errors encountered while running a specific step in the pipeline;</li><li>Monitor the number of RPCs made to backend service;</li><li>Retrieve an accurate count of the number of elements that have been processed;</li><li>…and so on.</li></ul><h3 id=101-the-main-concepts-of-beam-metrics>10.1 The main concepts of Beam metrics</h3><ul><li><strong>Named</str [...]
+potentially while the pipeline is running. There could be different reasons for that, for instance:</p><ul><li>Check the number of errors encountered while running a specific step in the pipeline;</li><li>Monitor the number of RPCs made to backend service;</li><li>Retrieve an accurate count of the number of elements that have been processed;</li><li>…and so on.</li></ul><h3 id=101-the-main-concepts-of-beam-metrics>10.1. The main concepts of Beam metrics</h3><ul><li><strong>Named</st [...]
namespace can be used to differentiate between multiple metrics with the same name and also
allows querying for all metrics within a specific namespace.</li><li><strong>Scoped</strong>. Each metric is reported against a specific step in the pipeline, indicating what
code was running when the metric was incremented.</li><li><strong>Dynamically Created</strong>. Metrics may be created during runtime without pre-declaring them, in
@@ -2176,7 +2176,7 @@ fallback behavior is to drop the metric updates rather than failing the pipeline
doesn’t support some part of querying metrics, the runner will not return the associated data.</li></ul><p>Reported metrics are implicitly scoped to the transform within the pipeline that reported them.
This allows reporting the same metric name in multiple places and identifying the value each
transform reported, as well as aggregating the metric across the entire pipeline.</p><blockquote><p><strong>Note:</strong> It is runner-dependent whether metrics are accessible during pipeline execution or only
-after jobs have completed.</p></blockquote><h3 id=types-of-metrics>10.2 Types of metrics</h3><p>There are three types of metrics that are supported for the moment: <code>Counter</code>, <code>Distribution</code> and
+after jobs have completed.</p></blockquote><h3 id=types-of-metrics>10.2. Types of metrics</h3><p>There are three types of metrics that are supported for the moment: <code>Counter</code>, <code>Distribution</code> and
<code>Gauge</code>.</p><p><strong>Counter</strong>: A metric that reports a single long value and can be incremented or decremented.</p><div class=language-java><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=n>Counter</span> <span class=n>counter</span> <span class=o>=</span> <span class=n>Metrics</span><span class=o>.</span><span class=na>counter</span><span class=o>(</span> <span class=s>"namespace"</span><span class=o>,</span> <span [...]
<span class=nd>@ProcessElement</span>
@@ -2201,7 +2201,7 @@ collected from many workers the value may not be the absolute last, but one of t
<span class=c1>// create a gauge (latest value received) of the values
</span><span class=c1></span> <span class=n>gauge</span><span class=o>.</span><span class=na>set</span><span class=o>(</span><span class=n>element</span><span class=o>);</span>
<span class=o>...</span>
-<span class=o>}</span></code></pre></div></div><h3 id=querying-metrics>10.3 Querying metrics</h3><p><code>PipelineResult</code> has a method <code>metrics()</code> which returns a <code>MetricResults</code> object that allows
+<span class=o>}</span></code></pre></div></div><h3 id=querying-metrics>10.3. Querying metrics</h3><p><code>PipelineResult</code> has a method <code>metrics()</code> which returns a <code>MetricResults</code> object that allows
accessing metrics. The main method available in <code>MetricResults</code> allows querying for all metrics
matching a given filter.</p><div class=language-java><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=kd>public</span> <span class=kd>interface</span> <span class=nc>PipelineResult</span> <span class=o>{</span>
<span class=n>MetricResults</span> <span class=nf>metrics</span><span class=o>();</span>
@@ -2222,7 +2222,7 @@ matching a given filter.</p><div class=language-java><div class=highlight><pre c
<span class=n>String</span> <span class=nf>getStep</span><span class=o>();</span>
<span class=n>T</span> <span class=nf>getCommitted</span><span class=o>();</span>
<span class=n>T</span> <span class=nf>getAttempted</span><span class=o>();</span>
-<span class=o>}</span></code></pre></div></div><h3 id=using-metrics>10.4 Using metrics in pipeline</h3><p>Below, there is a simple example of how to use a <code>Counter</code> metric in a user pipeline.</p><div class=language-java><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=c1>// creating a pipeline with custom metrics DoFn
+<span class=o>}</span></code></pre></div></div><h3 id=using-metrics>10.4. Using metrics in pipeline</h3><p>Below, there is a simple example of how to use a <code>Counter</code> metric in a user pipeline.</p><div class=language-java><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=c1>// creating a pipeline with custom metrics DoFn
</span><span class=c1></span><span class=n>pipeline</span>
<span class=o>.</span><span class=na>apply</span><span class=o>(...)</span>
<span class=o>.</span><span class=na>apply</span><span class=o>(</span><span class=n>ParDo</span><span class=o>.</span><span class=na>of</span><span class=o>(</span><span class=k>new</span> <span class=n>MyMetricsDoFn</span><span class=o>()));</span>
@@ -2253,7 +2253,7 @@ matching a given filter.</p><div class=language-java><div class=highlight><pre c
</span><span class=c1></span> <span class=n>counter</span><span class=o>.</span><span class=na>inc</span><span class=o>();</span>
<span class=n>context</span><span class=o>.</span><span class=na>output</span><span class=o>(</span><span class=n>context</span><span class=o>.</span><span class=na>element</span><span class=o>());</span>
<span class=o>}</span>
-<span class=o>}</span></code></pre></div></div><h3 id=export-metrics>10.5 Export metrics</h3><p>Beam metrics can be exported to external sinks. If a metrics sink is set up in the configuration, the runner will push metrics to it at a default 5s period.
+<span class=o>}</span></code></pre></div></div><h3 id=export-metrics>10.5. Export metrics</h3><p>Beam metrics can be exported to external sinks. If a metrics sink is set up in the configuration, the runner will push metrics to it at a default 5s period.
The configuration is held in the <a href=https://beam.apache.org/releases/javadoc/2.19.0/org/apache/beam/sdk/metrics/MetricsOptions.html>MetricsOptions</a> class.
It contains push period configuration and also sink specific options such as type and URL. As for now only the REST HTTP and the Graphite sinks are supported and only
Flink and Spark runners support metrics export.</p><p>Also Beam metrics are exported to inner Spark and Flink dashboards to be consulted in their respective UI.</p><h2 id=state-and-timers>11. State and Timers</h2><p>Beam’s windowing and triggering facilities provide a powerful abstraction for grouping and aggregating unbounded input
@@ -2273,7 +2273,7 @@ processing, and are recommended to test for correctness.</p><p class=language-ja
state must be named using the <code>StateId</code> annotation; this name is unique to a ParDo in the graph and has no relation
to other nodes in the graph. A <code>DoFn</code> can declare multiple state variables.</p><p class=language-py>In Python DoFn declares states to be accessed by creating <code>StateSpec</code> class member variables representing each state. Each
<code>StateSpec</code> is initialized with a name, this name is unique to a ParDo in the graph and has no relation
-to other nodes in the graph. A <code>DoFn</code> can declare multiple state variables.</p><h3 id=types-of-state>11.1 Types of state</h3><p>Beam provides several types of state:</p><h4 id=valuestate>ValueState</h4><p>A ValueState is a scalar state value. For each key in the input, a ValueState will store a typed value that can be
+to other nodes in the graph. A <code>DoFn</code> can declare multiple state variables.</p><h3 id=types-of-state>11.1. Types of state</h3><p>Beam provides several types of state:</p><h4 id=valuestate>ValueState</h4><p>A ValueState is a scalar state value. For each key in the input, a ValueState will store a typed value that can be
read and modified inside the DoFn’s <code>@ProcessElement</code> or <code>@OnTimer</code> methods. If the type of the ValueState has a coder
registered, then Beam will automatically infer the coder for the state value. Otherwise, a coder can be explicitly
specified when creating the ValueState. For example, the following ParDo creates a single state variable that
@@ -2338,7 +2338,7 @@ bags larger than available memory.</p><div class=language-java><div class=highli
<span class=n>state</span><span class=o>.</span><span class=n>clear</span><span class=p>()</span>
<span class=n>_</span> <span class=o>=</span> <span class=p>(</span><span class=n>p</span> <span class=o>|</span> <span class=s1>'Read per user'</span> <span class=o>>></span> <span class=n>ReadPerUser</span><span class=p>()</span>
- <span class=o>|</span> <span class=s1>'Bag state pardo'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span class=n>BagStateDoFn</span><span class=p>()))</span></code></pre></div></div><h3 id=deferred-state-reads>11.2 Deferred state reads</h3><p>When a <code>DoFn</code> contains multiple state specifications, reading each one in order can be slow. Calling the <code>read()</code> function
+ <span class=o>|</span> <span class=s1>'Bag state pardo'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span class=n>BagStateDoFn</span><span class=p>()))</span></code></pre></div></div><h3 id=deferred-state-reads>11.2. Deferred state reads</h3><p>When a <code>DoFn</code> contains multiple state specifications, reading each one in order can be slow. Calling the <code>read()</code> function
on a state can cause the runner to perform a blocking read. Performing multiple blocking reads in sequence adds latency
to element processing. If you know that a state will always be read, you can annotate it as @AlwaysFetched, and then the
runner can prefetch all of the states necessary. For example:</p><div class=language-java><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=n>PCollection</span><span class=o><</span><span class=n>KV</span><span class=o><</span><span class=n>String</span><span class=o>,</span> <span class=n>ValueT</span><span class=o>>></span> <span class=n>perUser</span> <span class=o>=</span> <span class=n>readPerUser</span><span class=o>();</span>
@@ -2378,10 +2378,10 @@ be read in the future, allowing multiple state reads to be batched together.</p>
<span class=n>processState2</span><span class=o>(</span><span class=n>state2</span><span class=o>.</span><span class=na>read</span><span class=o>());</span>
<span class=n>processState3</span><span class=o>(</span><span class=n>state3</span><span class=o>.</span><span class=na>read</span><span class=o>());</span>
<span class=o>}</span>
-<span class=o>}));</span></code></pre></div></div><h3 id=timers>11.3 Timers</h3><p>Beam provides a per-key timer callback API. This allows for delayed processing of data stored using the state API.
+<span class=o>}));</span></code></pre></div></div><h3 id=timers>11.3. Timers</h3><p>Beam provides a per-key timer callback API. This allows for delayed processing of data stored using the state API.
Timers can be set to callback at either an event-time or a processing-time timestamp. Every timer is identified with a
TimerId. A given timer for a key can only be set for a single timestamp. Calling set on a timer overwrites the previous
-firing time for that key’s timer.</p><h4 id=event-time-timers>11.3.1 Event-time timers</h4><p>Event-time timers fire when the input watermark for the DoFn passes the time at which the timer is set, meaning that
+firing time for that key’s timer.</p><h4 id=event-time-timers>11.3.1. Event-time timers</h4><p>Event-time timers fire when the input watermark for the DoFn passes the time at which the timer is set, meaning that
the runner believes that there are no more elements to be processed with timestamps before the timer timestamp. This
allows for event-time aggregations.</p><div class=language-java><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=n>PCollection</span><span class=o><</span><span class=n>KV</span><span class=o><</span><span class=n>String</span><span class=o>,</span> <span class=n>ValueT</span><span class=o>>></span> <span class=n>perUser</span> <span class=o>=</span> <span class=n>readPerUser</span><span class=o>();</span>
<span class=n>perUser</span><span class=o>.</span><span class=na>apply</span><span class=o>(</span><span class=n>ParDo</span><span class=o>.</span><span class=na>of</span><span class=o>(</span><span class=k>new</span> <span class=n>DoFn</span><span class=o><</span><span class=n>KV</span><span class=o><</span><span class=n>String</span><span class=o>,</span> <span class=n>ValueT</span><span class=o>>,</span> <span class=n>OutputT</span><span class=o>>()</span> <span class=o>{</span>
@@ -2419,7 +2419,7 @@ allows for event-time aggregations.</p><div class=language-java><div class=highl
<span class=n>state</span><span class=o>.</span><span class=n>clear</span><span class=p>()</span>
<span class=n>_</span> <span class=o>=</span> <span class=p>(</span><span class=n>p</span> <span class=o>|</span> <span class=s1>'Read per user'</span> <span class=o>>></span> <span class=n>ReadPerUser</span><span class=p>()</span>
- <span class=o>|</span> <span class=s1>'EventTime timer pardo'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span class=n>EventTimerDoFn</span><span class=p>()))</span></code></pre></div></div><h4 id=processing-time-timers>11.3.2 Processing-time timers</h4><p>Processing-time timers fire when the real wall-clock time passes. This is often used to create larger batches of data
+ <span class=o>|</span> <span class=s1>'EventTime timer pardo'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span class=n>EventTimerDoFn</span><span class=p>()))</span></code></pre></div></div><h4 id=processing-time-timers>11.3.2. Processing-time timers</h4><p>Processing-time timers fire when the real wall-clock time passes. This is often used to create larger batches of data
before processing. It can also be used to schedule events that should occur at a specific time. Just like with
event-time timers, processing-time timers are per key - each key has a separate copy of the timer.</p><p>While processing-time timers can be set to an absolute timestamp, it is very common to set them to an offset relative
to the current time. In Java, the <code>Timer.offset</code> and <code>Timer.setRelative</code> methods can be used to accomplish this.</p><div class=language-java><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=n>PCollection</span><span class=o><</span><span class=n>KV</span><span class=o><</span><span class=n>String</span><span class=o>,</span> <span class=n>ValueT</span><span class=o>>></span> <span class=n>perUser</span> <span cl [...]
@@ -2453,7 +2453,7 @@ to the current time. In Java, the <code>Timer.offset</code> and <code>Timer.setR
<span class=n>state</span><span class=o>.</span><span class=n>clear</span><span class=p>()</span>
<span class=n>_</span> <span class=o>=</span> <span class=p>(</span><span class=n>p</span> <span class=o>|</span> <span class=s1>'Read per user'</span> <span class=o>>></span> <span class=n>ReadPerUser</span><span class=p>()</span>
- <span class=o>|</span> <span class=s1>'ProcessingTime timer pardo'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span class=n>ProcessingTimerDoFn</span><span class=p>()))</span></code></pre></div></div><h4 id=dynamic-timer-tags>11.3.3 Dynamic timer tags</h4><p>Beam also supports dynamically setting a timer tag using <code>TimerMap</code>. This allows for setting multiple different timers
+ <span class=o>|</span> <span class=s1>'ProcessingTime timer pardo'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span class=n>ProcessingTimerDoFn</span><span class=p>()))</span></code></pre></div></div><h4 id=dynamic-timer-tags>11.3.3. Dynamic timer tags</h4><p>Beam also supports dynamically setting a timer tag using <code>TimerMap</code>. This allows for setting multiple different timers
in a <code>DoFn</code> and allowing for the timer tags to be dynamically chosen - e.g. based on data in the input elements. A
timer with a specific tag can only be set to a single timestamp, so setting the timer again has the effect of
overwriting the previous expiration time for the timer with that tag. Each <code>TimerMap</code> is identified with a timer family
@@ -2472,7 +2472,7 @@ id, and timers in different timer families are independent.</p><div class=langua
<span class=nd>@OnTimerFamily</span><span class=o>(</span><span class=s>"actionTimers"</span><span class=o>)</span> <span class=kd>public</span> <span class=kt>void</span> <span class=nf>onTimer</span><span class=o>(</span><span class=nd>@TimerId</span> <span class=n>String</span> <span class=n>timerId</span><span class=o>)</span> <span class=o>{</span>
<span class=n>LOG</span><span class=o>.</span><span class=na>info</span><span class=o>(</span><span class=s>"Timer fired with id "</span> <span class=o>+</span> <span class=n>timerId</span><span class=o>);</span>
<span class=o>}</span>
-<span class=o>}));</span></code></pre></div></div><div class=language-python><div class=highlight><pre class=chroma><code class=language-python data-lang=python><span class=n>To</span> <span class=n>be</span> <span class=n>supported</span><span class=p>,</span> <span class=n>See</span> <span class=n>BEAM</span><span class=o>-</span><span class=mi>9602</span></code></pre></div></div><h4 id=timer-output-timestamps>11.3.4 Timer output timestamps</h4><p>By default, event-time timers will hol [...]
+<span class=o>}));</span></code></pre></div></div><div class=language-python><div class=highlight><pre class=chroma><code class=language-python data-lang=python><span class=n>To</span> <span class=n>be</span> <span class=n>supported</span><span class=p>,</span> <span class=n>See</span> <span class=n>BEAM</span><span class=o>-</span><span class=mi>9602</span></code></pre></div></div><h4 id=timer-output-timestamps>11.3.4. Timer output timestamps</h4><p>By default, event-time timers will ho [...]
that if a timer is set to 12pm, any windowed aggregations or event-time timers later in the pipeline graph that finish
after 12pm will not expire. The timestamp of the timer is also the default output timestamp for the timer callback. This
means that any elements output from the onTimer method will have a timestamp equal to the timestamp of the timer firing.
@@ -2559,8 +2559,8 @@ past the timestamp of the minimum element. The following code demonstrates this.
<span class=c1>// Note that the timer has now fired.
</span><span class=c1></span> <span class=n>timerTimestamp</span><span class=o>.</span><span class=na>clear</span><span class=o>();</span>
<span class=o>}</span>
-<span class=o>}));</span></code></pre></div></div><h3 id=garbage-collecting-state>11.4 Garbage collecting state</h3><p>Per-key state needs to be garbage collected, or eventually the increasing size of state may negatively impact
-performance. There are two common strategies for garbage collecting state.</p><h5 id=using-windows-for-garbage-collection>11.4.1 <strong>Using windows for garbage collection</strong></h5><p>All state and timers for a key is scoped to the window it is in. This means that depending on the timestamp of the
+<span class=o>}));</span></code></pre></div></div><h3 id=garbage-collecting-state>11.4. Garbage collecting state</h3><p>Per-key state needs to be garbage collected, or eventually the increasing size of state may negatively impact
+performance. There are two common strategies for garbage collecting state.</p><h5 id=using-windows-for-garbage-collection>11.4.1. <strong>Using windows for garbage collection</strong></h5><p>All state and timers for a key is scoped to the window it is in. This means that depending on the timestamp of the
input element the ParDo will see different values for the state depending on the window that element falls into. In
addition, once the input watermark passes the end of the window, the runner should garbage collect all state for that
window. (note: if allowed lateness is set to a positive value for the window, the runner must wait for the watermark to
@@ -2586,7 +2586,7 @@ garbage-collection strategy.</p><p>For example, given the following:</p><div cla
<span class=n>_</span> <span class=o>=</span> <span class=p>(</span><span class=n>p</span> <span class=o>|</span> <span class=s1>'Read per user'</span> <span class=o>>></span> <span class=n>ReadPerUser</span><span class=p>()</span>
<span class=o>|</span> <span class=s1>'Windowing'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>WindowInto</span><span class=p>(</span><span class=n>FixedWindows</span><span class=p>(</span><span class=mi>60</span> <span class=o>*</span> <span class=mi>60</span> <span class=o>*</span> <span class=mi>24</span><span class=p>))</span>
<span class=o>|</span> <span class=s1>'DoFn'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span class=n>StateDoFn</span><span class=p>()))</span></code></pre></div></div><p>This <code>ParDo</code> stores state per day. Once the pipeline is done processing data for a given day, all the state for that
-day is garbage collected.</p><h5 id=using-timers-for-garbage-collection>11.4.1 <strong>Using timers For garbage collection</strong></h5><p>In some cases, it is difficult to find a windowing strategy that models the desired garbage-collection strategy. For
+day is garbage collected.</p><h5 id=using-timers-for-garbage-collection>11.4.1. <strong>Using timers For garbage collection</strong></h5><p>In some cases, it is difficult to find a windowing strategy that models the desired garbage-collection strategy. For
example, a common desire is to garbage collect state for a key once no activity has been seen on the key for some time.
This can be done by updating a timer that garbage collects state. For example</p><div class=language-java><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=n>PCollection</span><span class=o><</span><span class=n>KV</span><span class=o><</span><span class=n>String</span><span class=o>,</span> <span class=n>ValueT</span><span class=o>>></span> <span class=n>perUser</span> <span class=o>=</span> <span class=n>readPerUser</span><span clas [...]
<span class=n>perUser</span><span class=o>.</span><span class=na>apply</span><span class=o>(</span><span class=n>ParDo</span><span class=o>.</span><span class=na>of</span><span class=o>(</span><span class=k>new</span> <span class=n>DoFn</span><span class=o><</span><span class=n>KV</span><span class=o><</span><span class=n>String</span><span class=o>,</span> <span class=n>ValueT</span><span class=o>>,</span> <span class=n>OutputT</span><span class=o>>()</span> <span class=o>{</span>
@@ -2651,7 +2651,7 @@ This can be done by updating a timer that garbage collects state. For example</p
<span class=n>_</span> <span class=o>=</span> <span class=p>(</span><span class=n>p</span> <span class=o>|</span> <span class=s1>'Read per user'</span> <span class=o>>></span> <span class=n>ReadPerUser</span><span class=p>()</span>
- <span class=o>|</span> <span class=s1>'User DoFn'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span class=n>UserDoFn</span><span class=p>()))</span></code></pre></div></div><h3 id=state-timers-examples>11.5 State and timers examples</h3><p>Following are some example uses of state and timers</p><h4 id=joining-clicks-and-views>11.5.1. Joining clicks and views</h4><p>In this example, the p [...]
+ <span class=o>|</span> <span class=s1>'User DoFn'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span class=n>UserDoFn</span><span class=p>()))</span></code></pre></div></div><h3 id=state-timers-examples>11.5. State and timers examples</h3><p>Following are some example uses of state and timers</p><h4 id=joining-clicks-and-views>11.5.1. Joining clicks and views</h4><p>In this example, the [...]
a stream of views, representing suggested product links displayed to the user on the home page, and a stream of
clicks, representing actual user clicks on these links. The goal of the pipeline is to join click events with view
events, outputting a new joined event that contains information from both events. Each link has a unique identifier
@@ -2723,7 +2723,7 @@ event before the view event. The one hour join timeout should be based on event
<span class=n>clickState</span><span class=o>.</span><span class=na>clear</span><span class=o>();</span>
<span class=n>maxTimestampState</span><span class=o>.</span><span class=na>clear</span><span class=o>();</span>
<span class=o>}</span>
- <span class=o>}));</span></code></pre></div></div><h4 id=batching-rpcs>11.5.2 Batching RPCs</h4><p>In this example, input elements are being forwarded to an external RPC service. The RPC accepts batch requests -
+ <span class=o>}));</span></code></pre></div></div><h4 id=batching-rpcs>11.5.2. Batching RPCs</h4><p>In this example, input elements are being forwarded to an external RPC service. The RPC accepts batch requests -
multiple events for the same user can be batched in a single RPC call. Since this RPC service also imposes rate limits,
we want to batch ten seconds worth of events together in order to reduce the number of calls.</p><div class=language-java><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=n>PCollection</span><span class=o><</span><span class=n>KV</span><span class=o><</span><span class=n>String</span><span class=o>,</span> <span class=n>ValueT</span><span class=o>>></span> <span class=n>perUser</span> <span class=o>=</span> <span class=n>readPerUser< [...]
<span class=n>perUser</span><span class=o>.</span><span class=na>apply</span><span class=o>(</span><span class=n>ParDo</span><span class=o>.</span><span class=na>of</span><span class=o>(</span><span class=k>new</span> <span class=n>DoFn</span><span class=o><</span><span class=n>KV</span><span class=o><</span><span class=n>String</span><span class=o>,</span> <span class=n>ValueT</span><span class=o>>,</span> <span class=n>OutputT</span><span class=o>>()</span> <span class=o>{</span>
diff --git a/website/generated-content/sitemap.xml b/website/generated-content/sitemap.xml
index 9a3febc..cd183e8 100644
--- a/website/generated-content/sitemap.xml
+++ b/website/generated-content/sitemap.xml
@@ -1 +1 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/beam-2.24.0/</loc><lastmod>2020-09-18T12:38:38-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2020-09-18T12:38:38-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2020-09-18T12:38:38-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2020-09-18T12:38:38-07:00</lastmod></url><url><loc>/blog/p [...]
\ No newline at end of file
+<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/beam-2.24.0/</loc><lastmod>2020-09-18T12:38:38-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2020-09-18T12:38:38-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2020-09-18T12:38:38-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2020-09-18T12:38:38-07:00</lastmod></url><url><loc>/blog/p [...]
\ No newline at end of file