You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2020/06/17 00:02:05 UTC
[beam] branch asf-site updated: Publishing website 2020/06/17
00:01:54 at commit ca0611a
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 36c2c72 Publishing website 2020/06/17 00:01:54 at commit ca0611a
36c2c72 is described below
commit 36c2c72f7a6355134fcfd6d64dbe27a5ef57111e
Author: jenkins <us...@infra.apache.org>
AuthorDate: Wed Jun 17 00:01:55 2020 +0000
Publishing website 2020/06/17 00:01:54 at commit ca0611a
---
.../blog/2020/06/01/beam-katas-kotlin-release.html | 1 +
.../blog/beam-katas-kotlin-release/index.html | 21 +
website/generated-content/blog/index.html | 4 +-
website/generated-content/blog/index.xml | 1184 +-------------------
.../generated-content/categories/blog/index.xml | 40 +-
website/generated-content/documentation/index.xml | 71 +-
.../documentation/programming-guide/index.html | 68 +-
website/generated-content/feed.xml | 103 +-
.../beam-katas-kotlin-release/beam-and-kotlin.png | Bin 0 -> 81087 bytes
.../beam-katas-in-edutools.png | Bin 0 -> 46768 bytes
website/generated-content/index.html | 2 +-
website/generated-content/sitemap.xml | 2 +-
12 files changed, 225 insertions(+), 1271 deletions(-)
diff --git a/website/generated-content/blog/2020/06/01/beam-katas-kotlin-release.html b/website/generated-content/blog/2020/06/01/beam-katas-kotlin-release.html
new file mode 100644
index 0000000..4a868aa
--- /dev/null
+++ b/website/generated-content/blog/2020/06/01/beam-katas-kotlin-release.html
@@ -0,0 +1 @@
+<!doctype html><html><head><title>/blog/beam-katas-kotlin-release/</title><link rel=canonical href=/blog/beam-katas-kotlin-release/><meta name=robots content="noindex"><meta charset=utf-8><meta http-equiv=refresh content="0; url=/blog/beam-katas-kotlin-release/"></head></html>
\ No newline at end of file
diff --git a/website/generated-content/blog/beam-katas-kotlin-release/index.html b/website/generated-content/blog/beam-katas-kotlin-release/index.html
new file mode 100644
index 0000000..b7ba8f4
--- /dev/null
+++ b/website/generated-content/blog/beam-katas-kotlin-release/index.html
@@ -0,0 +1,21 @@
+<!doctype html><html lang=en class=no-js><head><meta charset=utf-8><meta http-equiv=x-ua-compatible content="IE=edge"><meta name=viewport content="width=device-width,initial-scale=1"><title>Announcing Beam Katas for Kotlin</title><meta name=description content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) [...]
+<span class=sr-only>Toggle navigation</span>
+<span class=icon-bar></span><span class=icon-bar></span><span class=icon-bar></span></button>
+<a href=/ class=navbar-brand><img alt=Brand style=height:25px src=/images/beam_logo_navbar.png></a></div><div class="navbar-mask closed"></div><div id=navbar class="navbar-container closed"><ul class="nav navbar-nav"><li><a href=/get-started/beam-overview/>Get Started</a></li><li><a href=/documentation/>Documentation</a></li><li><a href=/documentation/sdks/java/>Languages</a></li><li><a href=/documentation/runners/capability-matrix/>RUNNERS</a></li><li><a href=/roadmap/>Roadmap</a></li>< [...]
+•
+Rion Williams [<a href=https://twitter.com/rionmonster>@rionmonster</a>]</p></header><div class=post-content itemprop=articleBody><p>Today, we are happy to announce a new addition to the Beam Katas family: Kotlin!</p><p><img src=/images/blog/beam-katas-kotlin-release/beam-and-kotlin.png alt="Apache Beam and Kotlin Shaking Hands" height=330 width=800></p><p>You may remember <a href=https://beam.apache.org/blog/beam-kata-release>a post from last year</a> that informed everyone of the wonde [...]
+for learning more about writing Apache Beam applications, working with its various APIs and programming model
+hands-on, all from the comfort of your favorite IDEs. As of today, you can now work through all of the progressive
+exercises to learn about the fundamentals of Beam in Kotlin.</p><p><a href=https://kotlinlang.org>Kotlin</a> is a modern, open-source, statically typed language that targets the JVM. It is most commonly used by
+Android developers, however it has recently risen in popularity due to its extensive feature set that enables
+more concise and cleaner code than Java, without sacrificing performance or type safety. It recently was <a href=https://insights.stackoverflow.com/survey/2020#technology-most-loved-dreaded-and-wanted-languages-loved>ranked
+as one of the most beloved programming languages in the annual Stack Overflow Developer Survey</a>, so don’t take
+just our word for it.</p><p>The relationship between Apache Beam and Kotlin isn’t a new one. You can find examples scattered across the web
+of engineering teams embracing the two technologies including <a href=https://beam.apache.org/blog/beam-kotlin/>a series of samples announced on this very blog</a>.
+If you are new to Beam or are an experienced veteran looking for a change of pace, we’d encourage you to give
+Kotlin a try.</p><p>You can find the Kotlin and the other excellent Beam Katas below (or by just searching for “Beam Katas” within
+<a href="https://www.jetbrains.com/education/download/#section=idea">IntelliJ</a> or <a href="https://www.jetbrains.com/education/download/#section=pycharm-edu">PyCharm</a> through <a href=https://plugins.jetbrains.com/plugin/10081-edutools>the EduTools plugin</a>):</p><ul><li><a href=https://stepik.org/course/72488><strong>Kotlin</strong></a></li><li><a href=https://stepik.org/course/54530><strong>Java</strong></a></li><li><a href=https://stepik.org/course/54532><strong>Python</strong>< [...]
+and his support during the review process and making this effort a reality.</p><p><br></p><p><img src=/images/blog/beam-katas-kotlin-release/beam-katas-in-edutools.png alt="Access Beam Katas Kotlin through a JetBrains Educational Product" height=252 width=800></p></div></article></div><footer class=footer><div class=footer__contained><div class=footer__cols><div class=footer__cols__col><div class=footer__cols__col__logo><img src=/images/beam_logo_circle.svg class=footer__logo alt="Beam l [...]
+<a href=http://www.apache.org>The Apache Software Foundation</a>
+| <a href=/privacy_policy>Privacy Policy</a>
+| <a href=/feed.xml>RSS Feed</a><br><br>Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation. All other products or name brands are trademarks of their respective holders, including The Apache Software Foundation.</div></footer></body></html>
\ No newline at end of file
diff --git a/website/generated-content/blog/index.html b/website/generated-content/blog/index.html
index 88bcb33..2605f6c 100644
--- a/website/generated-content/blog/index.html
+++ b/website/generated-content/blog/index.html
@@ -3,7 +3,9 @@
<span class=icon-bar></span><span class=icon-bar></span><span class=icon-bar></span></button>
<a href=/ class=navbar-brand><img alt=Brand style=height:25px src=/images/beam_logo_navbar.png></a></div><div class="navbar-mask closed"></div><div id=navbar class="navbar-container closed"><ul class="nav navbar-nav"><li><a href=/get-started/beam-overview/>Get Started</a></li><li><a href=/documentation/>Documentation</a></li><li><a href=/documentation/sdks/java/>Languages</a></li><li><a href=/documentation/runners/capability-matrix/>RUNNERS</a></li><li><a href=/roadmap/>Roadmap</a></li>< [...]
•
-bhulette</i></p>We are happy to present the new 2.<p><a class="btn btn-default btn-sm" href=/blog/beam-2.22.0/ role=button>Read more <span class="glyphicon glyphicon-menu-right" aria-hidden=true></span></a></p><hr><h3><a class=post-link href=/blog/python-typing/>Python SDK Typing Changes</a></h3><p><i>May 28, 2020
+bhulette</i></p>We are happy to present the new 2.<p><a class="btn btn-default btn-sm" href=/blog/beam-2.22.0/ role=button>Read more <span class="glyphicon glyphicon-menu-right" aria-hidden=true></span></a></p><hr><h3><a class=post-link href=/blog/beam-katas-kotlin-release/>Announcing Beam Katas for Kotlin</a></h3><p><i>Jun 1, 2020
+•
+Rion Williams [<a href=https://twitter.com/rionmonster>@rionmonster</a>]</i></p><p>Today, we are happy to announce a new addition to the Beam Katas family: Kotlin!<p><a class="btn btn-default btn-sm" href=/blog/beam-katas-kotlin-release/ role=button>Read more <span class="glyphicon glyphicon-menu-right" aria-hidden=true></span></a></p><hr><h3><a class=post-link href=/blog/python-typing/>Python SDK Typing Changes</a></h3><p><i>May 28, 2020
•
Chad Dombrova
&
diff --git a/website/generated-content/blog/index.xml b/website/generated-content/blog/index.xml
index 76049fe..49bcc11 100644
--- a/website/generated-content/blog/index.xml
+++ b/website/generated-content/blog/index.xml
@@ -43,7 +43,45 @@ For more information on changes in 2.22.0, check out the
</ul>
<h2 id="list-of-contributors">List of Contributors</h2>
<p>According to git shortlog, the following people contributed to the 2.22.0 release. Thank you to all contributors!</p>
-<p>Ahmet Altay, aiyangar, Ajo Thomas, Akshay-Iyangar, Alan Pryor, Alexey Romanenko, Allen Pradeep Xavier, amaliujia, Andrew Pilloud, Ankur Goenka, Ashwin Ramaswami, bntnam, Borzoo Esmailloo, Boyuan Zhang, Brian Hulette, Chamikara Jayalath, Colm O hEigeartaigh, Craig Chambers, Damon Douglas, Daniel Oliveira, David Cavazos, David Moravek, Esun Kim, Etienne Chauchot, Filipe Regadas, Graeme Morgan, Hannah Jiang, Hannah-Jiang, Harch Vardhan, Heejong Lee, Henry Suryawirawan, Ismaël Mejía, I [...]
+<p>Ahmet Altay, aiyangar, Ajo Thomas, Akshay-Iyangar, Alan Pryor, Alexey Romanenko, Allen Pradeep Xavier, amaliujia, Andrew Pilloud, Ankur Goenka, Ashwin Ramaswami, bntnam, Borzoo Esmailloo, Boyuan Zhang, Brian Hulette, Chamikara Jayalath, Colm O hEigeartaigh, Craig Chambers, Damon Douglas, Daniel Oliveira, David Cavazos, David Moravek, Esun Kim, Etienne Chauchot, Filipe Regadas, Graeme Morgan, Hannah Jiang, Hannah-Jiang, Harch Vardhan, Heejong Lee, Henry Suryawirawan, Ismaël Mejía, I [...]
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<p>Today, we are happy to announce a new addition to the Beam Katas family: Kotlin!</p>
+<p><img src="/images/blog/beam-katas-kotlin-release/beam-and-kotlin.png" alt="Apache Beam and Kotlin Shaking Hands" height="330" width="800" ></p>
+<p>You may remember <a href="https://beam.apache.org/blog/beam-kata-release">a post from last year</a> that informed everyone of the wonderful Beam Katas available on <a href="https://stepik.org">Stepik</a>
+for learning more about writing Apache Beam applications, working with its various APIs and programming model
+hands-on, all from the comfort of your favorite IDEs. As of today, you can now work through all of the progressive
+exercises to learn about the fundamentals of Beam in Kotlin.</p>
+<p><a href="https://kotlinlang.org">Kotlin</a> is a modern, open-source, statically typed language that targets the JVM. It is most commonly used by
+Android developers, however it has recently risen in popularity due to its extensive feature set that enables
+more concise and cleaner code than Java, without sacrificing performance or type safety. It recently was <a href="https://insights.stackoverflow.com/survey/2020#technology-most-loved-dreaded-and-wanted-languages-loved">ranked
+as one of the most beloved programming languages in the annual Stack Overflow Developer Survey</a>, so don&rsquo;t take
+just our word for it.</p>
+<p>The relationship between Apache Beam and Kotlin isn&rsquo;t a new one. You can find examples scattered across the web
+of engineering teams embracing the two technologies including <a href="https://beam.apache.org/blog/beam-kotlin/">a series of samples announced on this very blog</a>.
+If you are new to Beam or are an experienced veteran looking for a change of pace, we&rsquo;d encourage you to give
+Kotlin a try.</p>
+<p>You can find the Kotlin and the other excellent Beam Katas below (or by just searching for &ldquo;Beam Katas&rdquo; within
+<a href="https://www.jetbrains.com/education/download/#section=idea">IntelliJ</a> or <a href="https://www.jetbrains.com/education/download/#section=pycharm-edu">PyCharm</a> through <a href="https://plugins.jetbrains.com/plugin/10081-edutools">the EduTools plugin</a>):</p>
+<ul>
+<li><a href="https://stepik.org/course/72488"><strong>Kotlin</strong></a></li>
+<li><a href="https://stepik.org/course/54530"><strong>Java</strong></a></li>
+<li><a href="https://stepik.org/course/54532"><strong>Python</strong></a></li>
+<li><a href="https://stepik.org/course/70387"><strong>Go (in development)</strong></a></li>
+</ul>
+<p>I&rsquo;d like to extend a very special thanks to <a href="https://twitter.com/henry_ken">Henry Suryawirawan</a> for his creation of the original series of Katas
+and his support during the review process and making this effort a reality.</p>
+<p><br /></p>
+<p><img src="/images/blog/beam-katas-kotlin-release/beam-katas-in-edutools.png" alt="Access Beam Katas Kotlin through a JetBrains Educational Product" height="252" width="800" ></p></description></item><item><title>Blog: Python SDK Typing Changes</title><link>/blog/python-typing/</link><pubDate>Thu, 28 May 2020 00:00:01 -0800</pubDate><guid>/blog/python-typing/</guid><description>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -4963,1146 +5001,4 @@ limitations under the License.
</ul>
</li>
</ul>
-<p>As Apache Beam grows, so will this repository of presentation materials. We are excited to add new materials as the Apache Beam ecosystem grows with new runners, SDKs, and so on. If you are interested in contributing content or have a request, please see the <a href="/contribute/presentation-materials/">Apache Beam presentation materials</a> page or email the <a href="mailto:user@beam.apache.org"><code>user@beam.apache.org</code></a> mailing list with your ideas o [...]
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-http://www.apache.org/licenses/LICENSE-2.0
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-<p>With initial code drops complete (<a href="https://github.com/apache/beam/pull/1">Dataflow SDK and Runner</a>, <a href="https://github.com/apache/beam/pull/12">Flink Runner</a>, <a href="https://github.com/apache/beam/pull/42">Spark Runner</a>) and expressed interest in runner implementations for <a href="https://issues.apache.org/jira/browse/BEAM-9">Storm</a>, <a href="https://issues.apache.org/jira/browse/BEAM-19">Hadoop</a>, and <a href="https:// [...]
-<p>While we’d love to have a world where all runners support the full suite of semantics included in the Beam Model (formerly referred to as the <a href="https://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf">Dataflow Model</a>), practically speaking, there will always be certain features that some runners can’t provide. For example, a Hadoop-based runner would be inherently batch-based and may be unable to (easily) implement support for unbounded collections. However, that doesn’t p [...]
-<p>To help clarify things, we’ve been working on enumerating the key features of the Beam model in a <a href="/documentation/runners/capability-matrix/">capability matrix</a> for all existing runners, categorized around the four key questions addressed by the model: <span class="wwwh-what-dark">What</span> / <span class="wwwh-where-dark">Where</span> / <span class="wwwh-when-dark">When</span> / <span class="wwwh-how-dark">How</span> (if you’re not familia [...]
-<p>Included below is a summary snapshot of our current understanding of the capabilities of the existing runners (see the <a href="/documentation/runners/capability-matrix/">live version</a> for full details, descriptions, and Jira links); since integration is still under way, the system as whole isn’t yet in a completely stable, usable state. But that should be changing in the near future, and we’ll be updating loud and clear on this blog when the first supported Beam 1.0 relea [...]
-<p>In the meantime, these tables should help clarify where we expect to be in the very near term, and help guide expectations about what existing runners are capable of, and what features runner implementers will be tackling next.</p>
-<p>
-<script type="text/javascript">
-function ToggleTables(showDetails, anchor) {
-document.getElementById("cap-summary").style.display = showDetails ? "none" : "block";
-document.getElementById("cap-full").style.display = showDetails ? "block" : "none";
-location.hash = anchor;
-}
-</script>
-<div id='cap-blog' style='display:block'>
-<table class='cap-summary'>
-<tr class='cap-summary' id='cap-blog-what'>
-<th class='cap-summary color-metadata format-category' colspan='8' style='color:#ca1'>
-What is being computed?
-</th>
-</tr>
-<tr class='cap-summary'>
-<th class='cap-summary color-capability'></th>
-<th class='cap-summary color-platform format-platform' style='color:#ec3'>Beam Model</th>
-<th class='cap-summary color-platform format-platform' style='color:#ec3'>Google Cloud Dataflow</th>
-<th class='cap-summary color-platform format-platform' style='color:#ec3'>Apache Flink</th>
-<th class='cap-summary color-platform format-platform' style='color:#ec3'>Apache Spark</th>
-</tr>
-<tr class='cap-summary'>
-<th class='cap-summary color-capability format-capability' style='color:#ec3'>ParDo</th>
-<td width='25%' class='cap-summary' style='background-color:#ec3;border-color:#ca1'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ec3;border-color:#ca1'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ec3;border-color:#ca1'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ec3;border-color:#ca1'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-</tr>
-<tr class='cap-summary'>
-<th class='cap-summary color-capability format-capability' style='color:#ec3'>GroupByKey</th>
-<td width='25%' class='cap-summary' style='background-color:#ec3;border-color:#ca1'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ec3;border-color:#ca1'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ec3;border-color:#ca1'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#fe5;border-color:#ca1'>
-<center>
-<b>
-~
-</b>
-<b>
-</b>
-</center>
-</td>
-</tr>
-<tr class='cap-summary'>
-<th class='cap-summary color-capability format-capability' style='color:#ec3'>Flatten</th>
-<td width='25%' class='cap-summary' style='background-color:#ec3;border-color:#ca1'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ec3;border-color:#ca1'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ec3;border-color:#ca1'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ec3;border-color:#ca1'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-</tr>
-<tr class='cap-summary'>
-<th class='cap-summary color-capability format-capability' style='color:#ec3'>Combine</th>
-<td width='25%' class='cap-summary' style='background-color:#ec3;border-color:#ca1'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ec3;border-color:#ca1'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ec3;border-color:#ca1'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ec3;border-color:#ca1'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-</tr>
-<tr class='cap-summary'>
-<th class='cap-summary color-capability format-capability' style='color:#ec3'>Composite Transforms</th>
-<td width='25%' class='cap-summary' style='background-color:#ec3;border-color:#ca1'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#fe5;border-color:#ca1'>
-<center>
-<b>
-~
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#fe5;border-color:#ca1'>
-<center>
-<b>
-~
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#fe5;border-color:#ca1'>
-<center>
-<b>
-~
-</b>
-<b>
-</b>
-</center>
-</td>
-</tr>
-<tr class='cap-summary'>
-<th class='cap-summary color-capability format-capability' style='color:#ec3'>Side Inputs</th>
-<td width='25%' class='cap-summary' style='background-color:#ec3;border-color:#ca1'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ec3;border-color:#ca1'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#fe5;border-color:#ca1'>
-<center>
-<b>
-~
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#fe5;border-color:#ca1'>
-<center>
-<b>
-~
-</b>
-<b>
-</b>
-</center>
-</td>
-</tr>
-<tr class='cap-summary'>
-<th class='cap-summary color-capability format-capability' style='color:#ec3'>Source API</th>
-<td width='25%' class='cap-summary' style='background-color:#ec3;border-color:#ca1'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ec3;border-color:#ca1'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#fe5;border-color:#ca1'>
-<center>
-<b>
-~
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ec3;border-color:#ca1'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-</tr>
-<tr class='cap-summary'>
-<th class='cap-summary color-capability format-capability' style='color:#ec3'>Aggregators</th>
-<td width='25%' class='cap-summary' style='background-color:#fe5;border-color:#ca1'>
-<center>
-<b>
-~
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#fe5;border-color:#ca1'>
-<center>
-<b>
-~
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#fe5;border-color:#ca1'>
-<center>
-<b>
-~
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#fe5;border-color:#ca1'>
-<center>
-<b>
-~
-</b>
-<b>
-</b>
-</center>
-</td>
-</tr>
-<tr class='cap-summary'>
-<th class='cap-summary color-capability format-capability' style='color:#ec3'>Keyed State</th>
-<td width='25%' class='cap-summary' style='background-color:#ddd;border-color:#ca1'>
-<center>
-<b>
-&#x2715;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ddd;border-color:#ca1'>
-<center>
-<b>
-&#x2715;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ddd;border-color:#ca1'>
-<center>
-<b>
-&#x2715;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ddd;border-color:#ca1'>
-<center>
-<b>
-&#x2715;
-</b>
-<b>
-</b>
-</center>
-</td>
-</tr>
-<tr class='cap-summary'>
-<td class='cap-summary color-blank cap-blank' colspan='5'></td>
-</tr>
-<tr class='cap-summary' id='cap-blog-where'>
-<th class='cap-summary color-metadata format-category' colspan='8' style='color:#37d'>
-Where in event time?
-</th>
-</tr>
-<tr class='cap-summary'>
-<th class='cap-summary color-capability'></th>
-<th class='cap-summary color-platform format-platform' style='color:#59f'>Beam Model</th>
-<th class='cap-summary color-platform format-platform' style='color:#59f'>Google Cloud Dataflow</th>
-<th class='cap-summary color-platform format-platform' style='color:#59f'>Apache Flink</th>
-<th class='cap-summary color-platform format-platform' style='color:#59f'>Apache Spark</th>
-</tr>
-<tr class='cap-summary'>
-<th class='cap-summary color-capability format-capability' style='color:#59f'>Global windows</th>
-<td width='25%' class='cap-summary' style='background-color:#59f;border-color:#37d'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#59f;border-color:#37d'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#59f;border-color:#37d'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#59f;border-color:#37d'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-</tr>
-<tr class='cap-summary'>
-<th class='cap-summary color-capability format-capability' style='color:#59f'>Fixed windows</th>
-<td width='25%' class='cap-summary' style='background-color:#59f;border-color:#37d'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#59f;border-color:#37d'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#59f;border-color:#37d'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#8cf;border-color:#37d'>
-<center>
-<b>
-~
-</b>
-<b>
-</b>
-</center>
-</td>
-</tr>
-<tr class='cap-summary'>
-<th class='cap-summary color-capability format-capability' style='color:#59f'>Sliding windows</th>
-<td width='25%' class='cap-summary' style='background-color:#59f;border-color:#37d'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#59f;border-color:#37d'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#59f;border-color:#37d'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ddd;border-color:#37d'>
-<center>
-<b>
-&#x2715;
-</b>
-<b>
-</b>
-</center>
-</td>
-</tr>
-<tr class='cap-summary'>
-<th class='cap-summary color-capability format-capability' style='color:#59f'>Session windows</th>
-<td width='25%' class='cap-summary' style='background-color:#59f;border-color:#37d'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#59f;border-color:#37d'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#59f;border-color:#37d'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ddd;border-color:#37d'>
-<center>
-<b>
-&#x2715;
-</b>
-<b>
-</b>
-</center>
-</td>
-</tr>
-<tr class='cap-summary'>
-<th class='cap-summary color-capability format-capability' style='color:#59f'>Custom windows</th>
-<td width='25%' class='cap-summary' style='background-color:#59f;border-color:#37d'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#59f;border-color:#37d'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#59f;border-color:#37d'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ddd;border-color:#37d'>
-<center>
-<b>
-&#x2715;
-</b>
-<b>
-</b>
-</center>
-</td>
-</tr>
-<tr class='cap-summary'>
-<th class='cap-summary color-capability format-capability' style='color:#59f'>Custom merging windows</th>
-<td width='25%' class='cap-summary' style='background-color:#59f;border-color:#37d'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#59f;border-color:#37d'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#59f;border-color:#37d'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ddd;border-color:#37d'>
-<center>
-<b>
-&#x2715;
-</b>
-<b>
-</b>
-</center>
-</td>
-</tr>
-<tr class='cap-summary'>
-<th class='cap-summary color-capability format-capability' style='color:#59f'>Timestamp control</th>
-<td width='25%' class='cap-summary' style='background-color:#59f;border-color:#37d'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#59f;border-color:#37d'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#59f;border-color:#37d'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ddd;border-color:#37d'>
-<center>
-<b>
-&#x2715;
-</b>
-<b>
-</b>
-</center>
-</td>
-</tr>
-<tr class='cap-summary'>
-<td class='cap-summary color-blank cap-blank' colspan='5'></td>
-</tr>
-<tr class='cap-summary' id='cap-blog-when'>
-<th class='cap-summary color-metadata format-category' colspan='8' style='color:#6a4'>
-When in processing time?
-</th>
-</tr>
-<tr class='cap-summary'>
-<th class='cap-summary color-capability'></th>
-<th class='cap-summary color-platform format-platform' style='color:#8c6'>Beam Model</th>
-<th class='cap-summary color-platform format-platform' style='color:#8c6'>Google Cloud Dataflow</th>
-<th class='cap-summary color-platform format-platform' style='color:#8c6'>Apache Flink</th>
-<th class='cap-summary color-platform format-platform' style='color:#8c6'>Apache Spark</th>
-</tr>
-<tr class='cap-summary'>
-<th class='cap-summary color-capability format-capability' style='color:#8c6'>Configurable triggering</th>
-<td width='25%' class='cap-summary' style='background-color:#8c6;border-color:#6a4'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#8c6;border-color:#6a4'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#8c6;border-color:#6a4'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ddd;border-color:#6a4'>
-<center>
-<b>
-&#x2715;
-</b>
-<b>
-</b>
-</center>
-</td>
-</tr>
-<tr class='cap-summary'>
-<th class='cap-summary color-capability format-capability' style='color:#8c6'>Event-time triggers</th>
-<td width='25%' class='cap-summary' style='background-color:#8c6;border-color:#6a4'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#8c6;border-color:#6a4'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#8c6;border-color:#6a4'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ddd;border-color:#6a4'>
-<center>
-<b>
-&#x2715;
-</b>
-<b>
-</b>
-</center>
-</td>
-</tr>
-<tr class='cap-summary'>
-<th class='cap-summary color-capability format-capability' style='color:#8c6'>Processing-time triggers</th>
-<td width='25%' class='cap-summary' style='background-color:#8c6;border-color:#6a4'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#8c6;border-color:#6a4'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#8c6;border-color:#6a4'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#8c6;border-color:#6a4'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-</tr>
-<tr class='cap-summary'>
-<th class='cap-summary color-capability format-capability' style='color:#8c6'>Count triggers</th>
-<td width='25%' class='cap-summary' style='background-color:#8c6;border-color:#6a4'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#8c6;border-color:#6a4'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#8c6;border-color:#6a4'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ddd;border-color:#6a4'>
-<center>
-<b>
-&#x2715;
-</b>
-<b>
-</b>
-</center>
-</td>
-</tr>
-<tr class='cap-summary'>
-<th class='cap-summary color-capability format-capability' style='color:#8c6'>[Meta]data driven triggers</th>
-<td width='25%' class='cap-summary' style='background-color:#ddd;border-color:#6a4'>
-<center>
-<b>
-&#x2715;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ddd;border-color:#6a4'>
-<center>
-<b>
-&#x2715;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ddd;border-color:#6a4'>
-<center>
-<b>
-&#x2715;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ddd;border-color:#6a4'>
-<center>
-<b>
-&#x2715;
-</b>
-<b>
-</b>
-</center>
-</td>
-</tr>
-<tr class='cap-summary'>
-<th class='cap-summary color-capability format-capability' style='color:#8c6'>Composite triggers</th>
-<td width='25%' class='cap-summary' style='background-color:#8c6;border-color:#6a4'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#8c6;border-color:#6a4'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#8c6;border-color:#6a4'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ddd;border-color:#6a4'>
-<center>
-<b>
-&#x2715;
-</b>
-<b>
-</b>
-</center>
-</td>
-</tr>
-<tr class='cap-summary'>
-<th class='cap-summary color-capability format-capability' style='color:#8c6'>Allowed lateness</th>
-<td width='25%' class='cap-summary' style='background-color:#8c6;border-color:#6a4'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#8c6;border-color:#6a4'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#8c6;border-color:#6a4'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ddd;border-color:#6a4'>
-<center>
-<b>
-&#x2715;
-</b>
-<b>
-</b>
-</center>
-</td>
-</tr>
-<tr class='cap-summary'>
-<th class='cap-summary color-capability format-capability' style='color:#8c6'>Timers</th>
-<td width='25%' class='cap-summary' style='background-color:#ddd;border-color:#6a4'>
-<center>
-<b>
-&#x2715;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ddd;border-color:#6a4'>
-<center>
-<b>
-&#x2715;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ddd;border-color:#6a4'>
-<center>
-<b>
-&#x2715;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ddd;border-color:#6a4'>
-<center>
-<b>
-&#x2715;
-</b>
-<b>
-</b>
-</center>
-</td>
-</tr>
-<tr class='cap-summary'>
-<td class='cap-summary color-blank cap-blank' colspan='5'></td>
-</tr>
-<tr class='cap-summary' id='cap-blog-how'>
-<th class='cap-summary color-metadata format-category' colspan='8' style='color:#b55'>
-How do refinements relate?
-</th>
-</tr>
-<tr class='cap-summary'>
-<th class='cap-summary color-capability'></th>
-<th class='cap-summary color-platform format-platform' style='color:#d77'>Beam Model</th>
-<th class='cap-summary color-platform format-platform' style='color:#d77'>Google Cloud Dataflow</th>
-<th class='cap-summary color-platform format-platform' style='color:#d77'>Apache Flink</th>
-<th class='cap-summary color-platform format-platform' style='color:#d77'>Apache Spark</th>
-</tr>
-<tr class='cap-summary'>
-<th class='cap-summary color-capability format-capability' style='color:#d77'>Discarding</th>
-<td width='25%' class='cap-summary' style='background-color:#d77;border-color:#b55'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#d77;border-color:#b55'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#d77;border-color:#b55'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#d77;border-color:#b55'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-</tr>
-<tr class='cap-summary'>
-<th class='cap-summary color-capability format-capability' style='color:#d77'>Accumulating</th>
-<td width='25%' class='cap-summary' style='background-color:#d77;border-color:#b55'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#d77;border-color:#b55'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#d77;border-color:#b55'>
-<center>
-<b>
-&#x2713;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ddd;border-color:#b55'>
-<center>
-<b>
-&#x2715;
-</b>
-<b>
-</b>
-</center>
-</td>
-</tr>
-<tr class='cap-summary'>
-<th class='cap-summary color-capability format-capability' style='color:#d77'>Accumulating &amp; Retracting</th>
-<td width='25%' class='cap-summary' style='background-color:#ddd;border-color:#b55'>
-<center>
-<b>
-&#x2715;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ddd;border-color:#b55'>
-<center>
-<b>
-&#x2715;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ddd;border-color:#b55'>
-<center>
-<b>
-&#x2715;
-</b>
-<b>
-</b>
-</center>
-</td>
-<td width='25%' class='cap-summary' style='background-color:#ddd;border-color:#b55'>
-<center>
-<b>
-&#x2715;
-</b>
-<b>
-</b>
-</center>
-</td>
-</tr>
-<tr class='cap-summary'>
-<td class='cap-summary color-blank cap-blank' colspan='5'></td>
-</tr>
-</table>
-</div>
-</p></description></item></channel></rss>
\ No newline at end of file
+<p>As Apache Beam grows, so will this repository of presentation materials. We are excited to add new materials as the Apache Beam ecosystem grows with new runners, SDKs, and so on. If you are interested in contributing content or have a request, please see the <a href="/contribute/presentation-materials/">Apache Beam presentation materials</a> page or email the <a href="mailto:user@beam.apache.org"><code>user@beam.apache.org</code></a> mailing list with your ideas o [...]
\ No newline at end of file
diff --git a/website/generated-content/categories/blog/index.xml b/website/generated-content/categories/blog/index.xml
index 85bef1d..9c1d270 100644
--- a/website/generated-content/categories/blog/index.xml
+++ b/website/generated-content/categories/blog/index.xml
@@ -43,7 +43,45 @@ For more information on changes in 2.22.0, check out the
</ul>
<h2 id="list-of-contributors">List of Contributors</h2>
<p>According to git shortlog, the following people contributed to the 2.22.0 release. Thank you to all contributors!</p>
-<p>Ahmet Altay, aiyangar, Ajo Thomas, Akshay-Iyangar, Alan Pryor, Alexey Romanenko, Allen Pradeep Xavier, amaliujia, Andrew Pilloud, Ankur Goenka, Ashwin Ramaswami, bntnam, Borzoo Esmailloo, Boyuan Zhang, Brian Hulette, Chamikara Jayalath, Colm O hEigeartaigh, Craig Chambers, Damon Douglas, Daniel Oliveira, David Cavazos, David Moravek, Esun Kim, Etienne Chauchot, Filipe Regadas, Graeme Morgan, Hannah Jiang, Hannah-Jiang, Harch Vardhan, Heejong Lee, Henry Suryawirawan, Ismaël Mejía, I [...]
+<p>Ahmet Altay, aiyangar, Ajo Thomas, Akshay-Iyangar, Alan Pryor, Alexey Romanenko, Allen Pradeep Xavier, amaliujia, Andrew Pilloud, Ankur Goenka, Ashwin Ramaswami, bntnam, Borzoo Esmailloo, Boyuan Zhang, Brian Hulette, Chamikara Jayalath, Colm O hEigeartaigh, Craig Chambers, Damon Douglas, Daniel Oliveira, David Cavazos, David Moravek, Esun Kim, Etienne Chauchot, Filipe Regadas, Graeme Morgan, Hannah Jiang, Hannah-Jiang, Harch Vardhan, Heejong Lee, Henry Suryawirawan, Ismaël Mejía, I [...]
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<p>Today, we are happy to announce a new addition to the Beam Katas family: Kotlin!</p>
+<p><img src="/images/blog/beam-katas-kotlin-release/beam-and-kotlin.png" alt="Apache Beam and Kotlin Shaking Hands" height="330" width="800" ></p>
+<p>You may remember <a href="https://beam.apache.org/blog/beam-kata-release">a post from last year</a> that informed everyone of the wonderful Beam Katas available on <a href="https://stepik.org">Stepik</a>
+for learning more about writing Apache Beam applications, working with its various APIs and programming model
+hands-on, all from the comfort of your favorite IDEs. As of today, you can now work through all of the progressive
+exercises to learn about the fundamentals of Beam in Kotlin.</p>
+<p><a href="https://kotlinlang.org">Kotlin</a> is a modern, open-source, statically typed language that targets the JVM. It is most commonly used by
+Android developers, however it has recently risen in popularity due to its extensive feature set that enables
+more concise and cleaner code than Java, without sacrificing performance or type safety. It recently was <a href="https://insights.stackoverflow.com/survey/2020#technology-most-loved-dreaded-and-wanted-languages-loved">ranked
+as one of the most beloved programming languages in the annual Stack Overflow Developer Survey</a>, so don&rsquo;t take
+just our word for it.</p>
+<p>The relationship between Apache Beam and Kotlin isn&rsquo;t a new one. You can find examples scattered across the web
+of engineering teams embracing the two technologies including <a href="https://beam.apache.org/blog/beam-kotlin/">a series of samples announced on this very blog</a>.
+If you are new to Beam or are an experienced veteran looking for a change of pace, we&rsquo;d encourage you to give
+Kotlin a try.</p>
+<p>You can find the Kotlin and the other excellent Beam Katas below (or by just searching for &ldquo;Beam Katas&rdquo; within
+<a href="https://www.jetbrains.com/education/download/#section=idea">IntelliJ</a> or <a href="https://www.jetbrains.com/education/download/#section=pycharm-edu">PyCharm</a> through <a href="https://plugins.jetbrains.com/plugin/10081-edutools">the EduTools plugin</a>):</p>
+<ul>
+<li><a href="https://stepik.org/course/72488"><strong>Kotlin</strong></a></li>
+<li><a href="https://stepik.org/course/54530"><strong>Java</strong></a></li>
+<li><a href="https://stepik.org/course/54532"><strong>Python</strong></a></li>
+<li><a href="https://stepik.org/course/70387"><strong>Go (in development)</strong></a></li>
+</ul>
+<p>I&rsquo;d like to extend a very special thanks to <a href="https://twitter.com/henry_ken">Henry Suryawirawan</a> for his creation of the original series of Katas
+and his support during the review process and making this effort a reality.</p>
+<p><br /></p>
+<p><img src="/images/blog/beam-katas-kotlin-release/beam-katas-in-edutools.png" alt="Access Beam Katas Kotlin through a JetBrains Educational Product" height="252" width="800" ></p></description></item><item><title>Blog: Python SDK Typing Changes</title><link>/blog/python-typing/</link><pubDate>Thu, 28 May 2020 00:00:01 -0800</pubDate><guid>/blog/python-typing/</guid><description>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
diff --git a/website/generated-content/documentation/index.xml b/website/generated-content/documentation/index.xml
index 40937ee..ddcfc0b 100644
--- a/website/generated-content/documentation/index.xml
+++ b/website/generated-content/documentation/index.xml
@@ -1623,7 +1623,7 @@ sets as <code>PCollection</code>s and its operations as <code>Transform
<p>To use Beam, your driver program must first create an instance of the Beam SDK
class <code>Pipeline</code> (typically in the <code>main()</code> function). When you create your
<code>Pipeline</code>, you&rsquo;ll also need to set some <strong>configuration options</strong>. You can set
-your pipeline&rsquo;s configuration options programatically, but it&rsquo;s often easier to
+your pipeline&rsquo;s configuration options programmatically, but it&rsquo;s often easier to
set the options ahead of time (or read them from the command line) and pass them
to the <code>Pipeline</code> object when you create the object.</p>
<div class=language-java>
@@ -1858,9 +1858,9 @@ itself.</p>
</div>
<h3 id="pcollection-characteristics">3.2. PCollection characteristics</h3>
<p>A <code>PCollection</code> is owned by the specific <code>Pipeline</code> object for which it is
-created; multiple pipelines cannot share a <code>PCollection</code>. In some respects, a
-<code>PCollection</code> functions like a collection class. However, a <code>PCollection</code> can
-differ in a few key ways:</p>
+created; multiple pipelines cannot share a <code>PCollection</code>. <span language="java">
+In some respects, a <code>PCollection</code> functions like a <code>Collection</code> class. However,
+a <code>PCollection</code> can differ in a few key ways:</span></p>
<h4 id="element-type">3.2.1. Element type</h4>
<p>The elements of a <code>PCollection</code> may be of any type, but must all be of the same
type. However, to support distributed processing, Beam needs to be able to
@@ -1962,7 +1962,10 @@ transforms sequentially and also apply transforms that contain other transforms
nested within (called <a href="#composite-transforms">composite transforms</a> in the Beam
SDKs).</p>
<p>How you apply your pipeline&rsquo;s transforms determines the structure of your
-pipeline. The best way to think of your pipeline is as a directed acyclic graph, where <code>PTransform</code> nodes are subroutines that accept <code>PCollection</code> nodes as inputs and emit <code>PCollection</code> nodes as outputs. For example, you can chain together transforms to create a pipeline that successively modifies input data:</p>
+pipeline. The best way to think of your pipeline is as a directed acyclic graph,
+where <code>PTransform</code> nodes are subroutines that accept <code>PCollection</code> nodes as
+inputs and emit <code>PCollection</code> nodes as outputs. For example, you can chain
+together transforms to create a pipeline that successively modifies input data:</p>
<div class=language-java>
<div class="highlight"><pre class="chroma"><code class="language-java" data-lang="java"><span class="o">[</span><span class="n">Final</span> <span class="n">Output</span> <span class="n">PCollection</span><span class="o">]</span> <span class="o">=</span> <span class="o">[</span><span class="n">Initial</span> <span class="n">Input</span> <span class="n">PCollection</span><span class="o">].</span><span class="na" [...]
<span class="o">.</span><span class="na">apply</span><span class="o">([</span><span class="n">Second</span> <span class="n">Transform</span><span class="o">])</span>
@@ -1977,7 +1980,7 @@ pipeline. The best way to think of your pipeline is as a directed acyclic graph,
<p><img src="/images/design-your-pipeline-linear.svg" alt="This linear pipeline starts with one input collection, sequentially appliesthree transforms, and ends with one output collection."></p>
<p><em>Figure 1: A linear pipeline with three sequential transforms.</em></p>
<p>However, note that a transform <em>does not consume or otherwise alter</em> the input
-collection&ndash;remember that a <code>PCollection</code> is immutable by definition. This means
+collection — remember that a <code>PCollection</code> is immutable by definition. This means
that you can apply multiple transforms to the same input <code>PCollection</code> to create
a branching pipeline, like so:</p>
<div class=language-java>
@@ -1995,7 +1998,7 @@ a branching pipeline, like so:</p>
<p><em>Figure 2: A branching pipeline. Two transforms are applied to a single
PCollection of database table rows.</em></p>
<p>You can also build your own <a href="#composite-transforms">composite transforms</a> that
-nest multiple sub-steps inside a single, larger transform. Composite transforms
+nest multiple transforms inside a single, larger transform. Composite transforms
are particularly useful for building a reusable sequence of simple steps that
get used in a lot of different places.</p>
<h3 id="core-beam-transforms">4.2. Core Beam transforms</h3>
@@ -2011,14 +2014,15 @@ processing paradigm:</p>
</ul>
<h4 id="pardo">4.2.1. ParDo</h4>
<p><code>ParDo</code> is a Beam transform for generic parallel processing. The <code>ParDo</code>
-processing paradigm is similar to the &ldquo;Map&rdquo; phase of a Map/Shuffle/Reduce-style
+processing paradigm is similar to the &ldquo;Map&rdquo; phase of a
+<a href="https://en.wikipedia.org/wiki/MapReduce">Map/Shuffle/Reduce</a>-style
algorithm: a <code>ParDo</code> transform considers each element in the input
<code>PCollection</code>, performs some processing function (your user code) on that
element, and emits zero, one, or multiple elements to an output <code>PCollection</code>.</p>
<p><code>ParDo</code> is useful for a variety of common data processing operations, including:</p>
<ul>
<li><strong>Filtering a data set.</strong> You can use <code>ParDo</code> to consider each element in a
-<code>PCollection</code> and either output that element to a new collection, or discard
+<code>PCollection</code> and either output that element to a new collection or discard
it.</li>
<li><strong>Formatting or type-converting each element in a data set.</strong> If your input
<code>PCollection</code> contains elements that are of a different type or format than
@@ -2084,7 +2088,7 @@ the length of each string, and outputs the result to a new <code>PCollection&
<h5 id="4212-creating-a-dofn">4.2.1.2. Creating a DoFn</h5>
<p>The <code>DoFn</code> object that you pass to <code>ParDo</code> contains the processing logic that
gets applied to the elements in the input collection. When you use Beam, often
-the most important pieces of code you&rsquo;ll write are these <code>DoFn</code>s&ndash;they&rsquo;re what
+the most important pieces of code you&rsquo;ll write are these <code>DoFn</code>s - they&rsquo;re what
define your pipeline&rsquo;s exact data processing tasks.</p>
<blockquote>
<p><strong>Note:</strong> When you create your <code>DoFn</code>, be mindful of the <a href="#requirements-for-writing-user-code-for-beam-transforms">Requirements
@@ -2107,15 +2111,16 @@ that for you. Your <code>@ProcessElement</code> method should accept a par
<code>@Element</code>, which will be populated with the input element. In order to output
elements, the method can also take a parameter of type <code>OutputReceiver</code> which
provides a method for emitting elements. The parameter types must match the input
-and output types of your <code>DoFn</code> or the framework will raise an error. Note: @Element and
-OutputReceiver were introduced in Beam 2.5.0; if using an earlier release of Beam, a
-ProcessContext parameter should be used instead.</p>
+and output types of your <code>DoFn</code> or the framework will raise an error. Note: <code>@Element</code> and
+<code>OutputReceiver</code> were introduced in Beam 2.5.0; if using an earlier release of Beam, a
+<code>ProcessContext</code> parameter should be used instead.</p>
<p class="language-py">Inside your <code>DoFn</code> subclass, you&rsquo;ll write a method <code>process</code> where you provide
the actual processing logic. You don&rsquo;t need to manually extract the elements
-from the input collection; the Beam SDKs handle that for you. Your <code>process</code>
-method should accept an object of type <code>element</code>. This is the input element and
-output is emitted by using <code>yield</code> or <code>return</code> statement inside <code>process</code>
-method.</p>
+from the input collection; the Beam SDKs handle that for you. Your <code>process</code> method
+should accept an argument <code>element</code>, which is the input element, and return an
+iterable with its output values. You can accomplish this by emitting individual
+elements with <code>yield</code> statements. You can also use a <code>return</code> statement
+with an iterable, like a list or a generator.</p>
<div class=language-java>
<div class="highlight"><pre class="chroma"><code class="language-java" data-lang="java"><span class="kd">static</span> <span class="kd">class</span> <span class="nc">ComputeWordLengthFn</span> <span class="kd">extends</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">&gt;</span> <span class="o">{</span>
<span class="nd">@ProcessElement</span>
@@ -2152,6 +2157,12 @@ elements from the input collection).</li>
<li>Once you output a value using <code>OutputReceiver.output()</code> you should not modify
that value in any way.</li>
</ul></p>
+<p class="language-python"><ul>
+<li>You should not in any way modify the <code>element</code> argument provided to the
+<code>process</code> method, or any side inputs.</li>
+<li>Once you output a value using <code>yield</code> or <code>return</code>, you should not modify
+that value in any way.</li>
+</ul></p>
<h5 id="lightweight-dofns">4.2.1.3. Lightweight DoFns and other abstractions</h5>
<p>If your function is relatively straightforward, you can simplify your use of
<code>ParDo</code> by providing a lightweight <code>DoFn</code> in-line, as
@@ -3083,7 +3094,7 @@ If an element falls in multiple windows (for example, this will happen when usin
<div class="highlight"><pre class="chroma"><code class="language-py" data-lang="py"><span class="kn">import</span> <span class="nn">apache_beam</span> <span class="kn">as</span> <span class="nn">beam</span>
<span class="k">class</span> <span class="nc">ProcessRecord</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">element</span><span class="p">,</span> <span class="n">window</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="o">.</span><span class="n">WindowParam</span><span class="p">):</span>
-<span class="c1"># access window e.g window.end.micros</span>
+<span class="c1"># access window e.g. window.end.micros</span>
<span class="k">pass</span>
</code></pre></div>
</div>
@@ -3093,7 +3104,7 @@ you can determine whether this is an early or a late firing, and how many times
<p class="language-py"><strong>PaneInfo:</strong>
When triggers are used, Beam provides a <code>DoFn.PaneInfoParam</code> object that contains information about the current firing. Using <code>DoFn.PaneInfoParam</code>
you can determine whether this is an early or a late firing, and how many times this window has already fired for this key.
-This feature implementation in python sdk is not fully completed, see more at <a href="https://issues.apache.org/jira/browse/BEAM-3759">BEAM-3759</a>.</p>
+This feature implementation in Python SDK is not fully completed; see more at <a href="https://issues.apache.org/jira/browse/BEAM-3759">BEAM-3759</a>.</p>
<div class=language-java>
<div class="highlight"><pre class="chroma"><code class="language-java" data-lang="java"><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;()</span> <span class="o">{</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span><span class="nd">@Element</span> <span class="n">String</span> <span class="n">word</span><span class="o">,</span> <span class="n">PaneInfo</span> <span class="n">paneInfo</span><span class="o">)</span> <span class="o">{</span>
@@ -3103,7 +3114,7 @@ This feature implementation in python sdk is not fully completed, see more at &l
<div class="highlight"><pre class="chroma"><code class="language-py" data-lang="py"><span class="kn">import</span> <span class="nn">apache_beam</span> <span class="kn">as</span> <span class="nn">beam</span>
<span class="k">class</span> <span class="nc">ProcessRecord</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">element</span><span class="p">,</span> <span class="n">pane_info</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="o">.</span><span class="n">PaneInfoParam</span><span class="p">):</span>
-<span class="c1"># access pane info e.g pane_info.is_first, pane_info.is_last, pane_info.timing</span>
+<span class="c1"># access pane info, e.g. pane_info.is_first, pane_info.is_last, pane_info.timing</span>
<span class="k">pass</span>
</code></pre></div>
</div>
@@ -3121,7 +3132,7 @@ a parameter of type <code>TimeDomain</code> which tells whether the timer
Timers are explained in more detail in the
<a href="/blog/2017/08/28/timely-processing.html">Timely (and Stateful) Processing with Apache Beam</a> blog post.</p>
<p class="language-py"><strong>Timer and State:</strong>
-In addition to aforementioned parameters, user defined Timer and State parameters can be used in a Stateful DoFn.
+In addition to aforementioned parameters, user defined Timer and State parameters can be used in a stateful DoFn.
Timers and States are explained in more detail in the
<a href="/blog/2017/08/28/timely-processing.html">Timely (and Stateful) Processing with Apache Beam</a> blog post.</p>
<div class=language-py>
@@ -3137,7 +3148,7 @@ Timers and States are explained in more detail in the
<span class="n">buffer_1</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="o">.</span><span class="n">StateParam</span><span class="p">(</span><span class="n">BUFFER_STATE_1</span><span class="p">),</span>
<span class="n">buffer_2</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="o">.</span><span class="n">StateParam</span><span class="p">(</span><span class="n">BUFFER_STATE_2</span><span class="p">),</span>
<span class="n">watermark_timer</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="o">.</span><span class="n">TimerParam</span><span class="p">(</span><span class="n">WATERMARK_TIMER</span><span class="p">)):</span>
-<span class="c1"># Do you processing here</span>
+<span class="c1"># Do your processing here</span>
<span class="n">key</span><span class="p">,</span> <span class="n">value</span> <span class="o">=</span> <span class="n">element</span>
<span class="c1"># Read all the data from buffer1</span>
<span class="n">all_values_in_buffer_1</span> <span class="o">=</span> <span class="p">[</span><span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">buffer_1</span><span class="o">.</span><span class="n">read</span><span class="p">()]</span>
@@ -3195,11 +3206,6 @@ words, producing a <code>PCollection</code> of key/value pairs. Each key r
word in the text, and each value represents the number of times that word
appeared in the original data.</li>
</ol>
-<p>Note that this is also an example of nested composite transforms, as <code>Count</code>
-is, by itself, a composite transform.</p>
-<p>Your composite transform&rsquo;s parameters and return value must match the initial
-input type and final return type for the entire transform, even if the
-transform&rsquo;s intermediate data changes type multiple times.</p>
<div class=language-java>
<div class="highlight"><pre class="chroma"><code class="language-java" data-lang="java"> <span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">CountWords</span> <span class="kd">extends</span> <span class="n">PTransform</span><span class="o">&lt;</span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o [...]
<span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">&gt;&gt;&gt;</span> <span class="o">{</span>
@@ -3228,6 +3234,10 @@ transform&rsquo;s intermediate data changes type multiple times.</p>
<span class="c1"># Format each word and count into a printable string.</span>
<span class="o">|</span> <span class="s1">&#39;FormatCounts&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">FormatCountsFn</span><span class="p">()))</span></code></pre></div>
</div>
+<blockquote>
+<p><strong>Note:</strong> Because <code>Count</code> is itself a composite transform,
+<code>CountWords</code> is also a nested composite transform.</p>
+</blockquote>
<h4 id="composite-transform-creation">4.6.2. Creating a composite transform</h4>
<p>To create your own composite transform, create a subclass of the <code>PTransform</code>
class and override the <code>expand</code> method to specify the actual processing logic.
@@ -3279,6 +3289,9 @@ accept the appropriate input <code>PCollection</code>(s) and return the co
output <code>PCollection</code>(s), you can include as many transforms as you want. These
transforms can include core transforms, composite transforms, or the transforms
included in the Beam SDK libraries.</p>
+<p>Your composite transform&rsquo;s parameters and return value must match the initial
+input type and final return type for the entire transform, even if the
+transform&rsquo;s intermediate data changes type multiple times.</p>
<p><strong>Note:</strong> The <code>expand</code> method of a <code>PTransform</code> is not meant to be invoked
directly by the user of a transform. Instead, you should call the <code>apply</code> method
on the <code>PCollection</code> itself, with the transform as an argument. This allows
@@ -3324,7 +3337,7 @@ a <code>PCollection</code>'s data at any point in your pipeline.</p>
<p>Many read transforms support reading from multiple input files matching a glob
operator you provide. Note that glob operators are filesystem-specific and obey
filesystem-specific consistency models. The following TextIO example uses a glob
-operator (*) to read all matching input files that have prefix &ldquo;input-&rdquo; and the
+operator (<code>*</code>) to read all matching input files that have prefix &ldquo;input-&rdquo; and the
suffix &ldquo;.csv&rdquo; in the given location:</p>
<div class=language-java>
<div class="highlight"><pre class="chroma"><code class="language-java" data-lang="java"><span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;ReadFromText&#34;</span><span class="o">,</span>
@@ -3363,7 +3376,7 @@ structures that can often be determined by examining the type. Even within a SDK
(or equivalent structures in other languages) are often used as intermediate types, and these also have a
clear structure that can be inferred by inspecting the class. By understanding the structure of a pipeline’s
records, we can provide much more concise APIs for data processing.</p>
-<h3 id="what-is-a-schema">6.1. What is a schema</h3>
+<h3 id="what-is-a-schema">6.1. What is a schema?</h3>
<p>Most structured records share some common characteristics:</p>
<ul>
<li>They can be subdivided into separate named fields. Fields usually have string names, but sometimes - as in the case of indexed
diff --git a/website/generated-content/documentation/programming-guide/index.html b/website/generated-content/documentation/programming-guide/index.html
index 2a91194..5df6f90 100644
--- a/website/generated-content/documentation/programming-guide/index.html
+++ b/website/generated-content/documentation/programming-guide/index.html
@@ -1,7 +1,7 @@
<!doctype html><html lang=en class=no-js><head><meta charset=utf-8><meta http-equiv=x-ua-compatible content="IE=edge"><meta name=viewport content="width=device-width,initial-scale=1"><title>Beam Programming Guide</title><meta name=description content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain [...]
<span class=sr-only>Toggle navigation</span>
<span class=icon-bar></span><span class=icon-bar></span><span class=icon-bar></span></button>
-<a href=/ class=navbar-brand><img alt=Brand style=height:25px src=/images/beam_logo_navbar.png></a></div><div class="navbar-mask closed"></div><div id=navbar class="navbar-container closed"><ul class="nav navbar-nav"><li><a href=/get-started/beam-overview/>Get Started</a></li><li><a href=/documentation/>Documentation</a></li><li><a href=/documentation/sdks/java/>Languages</a></li><li><a href=/documentation/runners/capability-matrix/>RUNNERS</a></li><li><a href=/roadmap/>Roadmap</a></li>< [...]
+<a href=/ class=navbar-brand><img alt=Brand style=height:25px src=/images/beam_logo_navbar.png></a></div><div class="navbar-mask closed"></div><div id=navbar class="navbar-container closed"><ul class="nav navbar-nav"><li><a href=/get-started/beam-overview/>Get Started</a></li><li><a href=/documentation/>Documentation</a></li><li><a href=/documentation/sdks/java/>Languages</a></li><li><a href=/documentation/runners/capability-matrix/>RUNNERS</a></li><li><a href=/roadmap/>Roadmap</a></li>< [...]
Beam SDKs to create data processing pipelines. It provides guidance for using
the Beam SDK classes to build and test your pipeline. It is not intended as an
exhaustive reference, but as a language-agnostic, high-level guide to
@@ -54,7 +54,7 @@ object, and then using that object as the basis for creating the pipeline’
sets as <code>PCollection</code>s and its operations as <code>Transform</code>s.</p><p>To use Beam, your driver program must first create an instance of the Beam SDK
class <code>Pipeline</code> (typically in the <code>main()</code> function). When you create your
<code>Pipeline</code>, you’ll also need to set some <strong>configuration options</strong>. You can set
-your pipeline’s configuration options programatically, but it’s often easier to
+your pipeline’s configuration options programmatically, but it’s often easier to
set the options ahead of time (or read them from the command line) and pass them
to the <code>Pipeline</code> object when you create the object.</p><div class=language-java><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=c1>// Start by defining the options for the pipeline.
</span><span class=c1></span><span class=n>PipelineOptions</span> <span class=n>options</span> <span class=o>=</span> <span class=n>PipelineOptionsFactory</span><span class=o>.</span><span class=na>create</span><span class=o>();</span>
@@ -209,9 +209,8 @@ itself.</p><p>The following example code shows how to create a <code>PCollection
<span class=s1>'The slings and arrows of outrageous fortune, '</span><span class=p>,</span>
<span class=s1>'Or to take arms against a sea of troubles, '</span><span class=p>,</span>
<span class=p>]))</span></code></pre></div></div><h3 id=pcollection-characteristics>3.2. PCollection characteristics</h3><p>A <code>PCollection</code> is owned by the specific <code>Pipeline</code> object for which it is
-created; multiple pipelines cannot share a <code>PCollection</code>. In some respects, a
-<code>PCollection</code> functions like a collection class. However, a <code>PCollection</code> can
-differ in a few key ways:</p><h4 id=element-type>3.2.1. Element type</h4><p>The elements of a <code>PCollection</code> may be of any type, but must all be of the same
+created; multiple pipelines cannot share a <code>PCollection</code>. <span language=java>In some respects, a <code>PCollection</code> functions like a <code>Collection</code> class. However,
+a <code>PCollection</code> can differ in a few key ways:</span></p><h4 id=element-type>3.2.1. Element type</h4><p>The elements of a <code>PCollection</code> may be of any type, but must all be of the same
type. However, to support distributed processing, Beam needs to be able to
encode each individual element as a byte string (so elements can be passed
around to distributed workers). The Beam SDKs provide a data encoding mechanism
@@ -280,12 +279,15 @@ the transform itself as an argument, and the operation returns the output
transforms sequentially and also apply transforms that contain other transforms
nested within (called <a href=#composite-transforms>composite transforms</a> in the Beam
SDKs).</p><p>How you apply your pipeline’s transforms determines the structure of your
-pipeline. The best way to think of your pipeline is as a directed acyclic graph, where <code>PTransform</code> nodes are subroutines that accept <code>PCollection</code> nodes as inputs and emit <code>PCollection</code> nodes as outputs. For example, you can chain together transforms to create a pipeline that successively modifies input data:</p><div class=language-java><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=o>[</span><span class=n>Fin [...]
+pipeline. The best way to think of your pipeline is as a directed acyclic graph,
+where <code>PTransform</code> nodes are subroutines that accept <code>PCollection</code> nodes as
+inputs and emit <code>PCollection</code> nodes as outputs. For example, you can chain
+together transforms to create a pipeline that successively modifies input data:</p><div class=language-java><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=o>[</span><span class=n>Final</span> <span class=n>Output</span> <span class=n>PCollection</span><span class=o>]</span> <span class=o>=</span> <span class=o>[</span><span class=n>Initial</span> <span class=n>Input</span> <span class=n>PCollection</span><span class=o>].</span><span class=na>a [...]
<span class=o>.</span><span class=na>apply</span><span class=o>([</span><span class=n>Second</span> <span class=n>Transform</span><span class=o>])</span>
<span class=o>.</span><span class=na>apply</span><span class=o>([</span><span class=n>Third</span> <span class=n>Transform</span><span class=o>])</span></code></pre></div></div><div class=language-py><div class=highlight><pre class=chroma><code class=language-py data-lang=py><span class=p>[</span><span class=n>Final</span> <span class=n>Output</span> <span class=n>PCollection</span><span class=p>]</span> <span class=o>=</span> <span class=p>([</span><span class=n>Initial</span> <span cla [...]
<span class=o>|</span> <span class=p>[</span><span class=n>Second</span> <span class=n>Transform</span><span class=p>]</span>
<span class=o>|</span> <span class=p>[</span><span class=n>Third</span> <span class=n>Transform</span><span class=p>])</span></code></pre></div></div><p>The graph of this pipeline looks like the following:</p><p><img src=/images/design-your-pipeline-linear.svg alt="This linear pipeline starts with one input collection, sequentially appliesthree transforms, and ends with one output collection."></p><p><em>Figure 1: A linear pipeline with three sequential transforms.</em></p> [...]
-collection–remember that a <code>PCollection</code> is immutable by definition. This means
+collection — remember that a <code>PCollection</code> is immutable by definition. This means
that you can apply multiple transforms to the same input <code>PCollection</code> to create
a branching pipeline, like so:</p><div class=language-java><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=o>[</span><span class=n>PCollection</span> <span class=n>of</span> <span class=n>database</span> <span class=n>table</span> <span class=n>rows</span><span class=o>]</span> <span class=o>=</span> <span class=o>[</span><span class=n>Database</span> <span class=n>Table</span> <span class=n>Reader</span><span class=o>].</span><span class=na>ap [...]
<span class=o>[</span><span class=n>PCollection</span> <span class=n>of</span> <span class=sc>'A'</span> <span class=n>names</span><span class=o>]</span> <span class=o>=</span> <span class=o>[</span><span class=n>PCollection</span> <span class=n>of</span> <span class=n>database</span> <span class=n>table</span> <span class=n>rows</span><span class=o>].</span><span class=na>apply</span><span class=o>([</span><span class=n>Transform</span> <span class=n>A</span><span class=o>])</span>
@@ -293,15 +295,16 @@ a branching pipeline, like so:</p><div class=language-java><div class=highlight>
<span class=p>[</span><span class=n>PCollection</span> <span class=n>of</span> <span class=s1>'A'</span> <span class=n>names</span><span class=p>]</span> <span class=o>=</span> <span class=p>[</span><span class=n>PCollection</span> <span class=n>of</span> <span class=n>database</span> <span class=n>table</span> <span class=n>rows</span><span class=p>]</span> <span class=o>|</span> <span class=p>[</span><span class=n>Transform</span> <span class=n>A</span><span class=p>]</span>
<span class=p>[</span><span class=n>PCollection</span> <span class=n>of</span> <span class=s1>'B'</span> <span class=n>names</span><span class=p>]</span> <span class=o>=</span> <span class=p>[</span><span class=n>PCollection</span> <span class=n>of</span> <span class=n>database</span> <span class=n>table</span> <span class=n>rows</span><span class=p>]</span> <span class=o>|</span> <span class=p>[</span><span class=n>Transform</span> <span class=n>B</span><span class=p>]</span></c [...]
PCollection of database table rows.</em></p><p>You can also build your own <a href=#composite-transforms>composite transforms</a> that
-nest multiple sub-steps inside a single, larger transform. Composite transforms
+nest multiple transforms inside a single, larger transform. Composite transforms
are particularly useful for building a reusable sequence of simple steps that
get used in a lot of different places.</p><h3 id=core-beam-transforms>4.2. Core Beam transforms</h3><p>Beam provides the following core transforms, each of which represents a different
processing paradigm:</p><ul><li><code>ParDo</code></li><li><code>GroupByKey</code></li><li><code>CoGroupByKey</code></li><li><code>Combine</code></li><li><code>Flatten</code></li><li><code>Partition</code></li></ul><h4 id=pardo>4.2.1. ParDo</h4><p><code>ParDo</code> is a Beam transform for generic parallel processing. The <code>ParDo</code>
-processing paradigm is similar to the “Map” phase of a Map/Shuffle/Reduce-style
+processing paradigm is similar to the “Map” phase of a
+<a href=https://en.wikipedia.org/wiki/MapReduce>Map/Shuffle/Reduce</a>-style
algorithm: a <code>ParDo</code> transform considers each element in the input
<code>PCollection</code>, performs some processing function (your user code) on that
element, and emits zero, one, or multiple elements to an output <code>PCollection</code>.</p><p><code>ParDo</code> is useful for a variety of common data processing operations, including:</p><ul><li><strong>Filtering a data set.</strong> You can use <code>ParDo</code> to consider each element in a
-<code>PCollection</code> and either output that element to a new collection, or discard
+<code>PCollection</code> and either output that element to a new collection or discard
it.</li><li><strong>Formatting or type-converting each element in a data set.</strong> If your input
<code>PCollection</code> contains elements that are of a different type or format than
you want, you can use <code>ParDo</code> to perform a conversion on each element and
@@ -353,7 +356,7 @@ following example code:</p><div class=language-java><div class=highlight><pre cl
the length of each string, and outputs the result to a new <code>PCollection</code> of
<code>Integer</code> values that stores the length of each word.</p><h5 id=4212-creating-a-dofn>4.2.1.2. Creating a DoFn</h5><p>The <code>DoFn</code> object that you pass to <code>ParDo</code> contains the processing logic that
gets applied to the elements in the input collection. When you use Beam, often
-the most important pieces of code you’ll write are these <code>DoFn</code>s–they’re what
+the most important pieces of code you’ll write are these <code>DoFn</code>s - they’re what
define your pipeline’s exact data processing tasks.</p><blockquote><p><strong>Note:</strong> When you create your <code>DoFn</code>, be mindful of the <a href=#requirements-for-writing-user-code-for-beam-transforms>Requirements
for writing user code for Beam transforms</a>
and ensure that your code follows them.</p></blockquote><p class=language-java>A <code>DoFn</code> processes one element at a time from the input <code>PCollection</code>. When you
@@ -368,14 +371,15 @@ that for you. Your <code>@ProcessElement</code> method should accept a parameter
<code>@Element</code>, which will be populated with the input element. In order to output
elements, the method can also take a parameter of type <code>OutputReceiver</code> which
provides a method for emitting elements. The parameter types must match the input
-and output types of your <code>DoFn</code> or the framework will raise an error. Note: @Element and
-OutputReceiver were introduced in Beam 2.5.0; if using an earlier release of Beam, a
-ProcessContext parameter should be used instead.</p><p class=language-py>Inside your <code>DoFn</code> subclass, you’ll write a method <code>process</code> where you provide
+and output types of your <code>DoFn</code> or the framework will raise an error. Note: <code>@Element</code> and
+<code>OutputReceiver</code> were introduced in Beam 2.5.0; if using an earlier release of Beam, a
+<code>ProcessContext</code> parameter should be used instead.</p><p class=language-py>Inside your <code>DoFn</code> subclass, you’ll write a method <code>process</code> where you provide
the actual processing logic. You don’t need to manually extract the elements
-from the input collection; the Beam SDKs handle that for you. Your <code>process</code>
-method should accept an object of type <code>element</code>. This is the input element and
-output is emitted by using <code>yield</code> or <code>return</code> statement inside <code>process</code>
-method.</p><div class=language-java><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=kd>static</span> <span class=kd>class</span> <span class=nc>ComputeWordLengthFn</span> <span class=kd>extends</span> <span class=n>DoFn</span><span class=o><</span><span class=n>String</span><span class=o>,</span> <span class=n>Integer</span><span class=o>></span> <span class=o>{</span>
+from the input collection; the Beam SDKs handle that for you. Your <code>process</code> method
+should accept an argument <code>element</code>, which is the input element, and return an
+iterable with its output values. You can accomplish this by emitting individual
+elements with <code>yield</code> statements. You can also use a <code>return</code> statement
+with an iterable, like a list or a generator.</p><div class=language-java><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=kd>static</span> <span class=kd>class</span> <span class=nc>ComputeWordLengthFn</span> <span class=kd>extends</span> <span class=n>DoFn</span><span class=o><</span><span class=n>String</span><span class=o>,</span> <span class=n>Integer</span><span class=o>></span> <span class=o>{</span>
<span class=nd>@ProcessElement</span>
<span class=kd>public</span> <span class=kt>void</span> <span class=nf>processElement</span><span class=o>(</span><span class=nd>@Element</span> <span class=n>String</span> <span class=n>word</span><span class=o>,</span> <span class=n>OutputReceiver</span><span class=o><</span><span class=n>Integer</span><span class=o>></span> <span class=n>out</span><span class=o>)</span> <span class=o>{</span>
<span class=c1>// Use OutputReceiver.output to emit the output element.
@@ -396,6 +400,8 @@ serialize and cache the values in your pipeline. Your method should meet the
following requirements:</p><p class=language-java><ul><li>You should not in any way modify an element returned by
the <code>@Element</code> annotation or <code>ProcessContext.sideInput()</code> (the incoming
elements from the input collection).</li><li>Once you output a value using <code>OutputReceiver.output()</code> you should not modify
+that value in any way.</li></ul></p><p class=language-python><ul><li>You should not in any way modify the <code>element</code> argument provided to the
+<code>process</code> method, or any side inputs.</li><li>Once you output a value using <code>yield</code> or <code>return</code>, you should not modify
that value in any way.</li></ul></p><h5 id=lightweight-dofns>4.2.1.3. Lightweight DoFns and other abstractions</h5><p>If your function is relatively straightforward, you can simplify your use of
<code>ParDo</code> by providing a lightweight <code>DoFn</code> in-line, as
<span class=language-java>an anonymous inner class instance</span>
@@ -1151,21 +1157,21 @@ If an element falls in multiple windows (for example, this will happen when usin
<span class=k>class</span> <span class=nc>ProcessRecord</span><span class=p>(</span><span class=n>beam</span><span class=o>.</span><span class=n>DoFn</span><span class=p>):</span>
<span class=k>def</span> <span class=nf>process</span><span class=p>(</span><span class=bp>self</span><span class=p>,</span> <span class=n>element</span><span class=p>,</span> <span class=n>window</span><span class=o>=</span><span class=n>beam</span><span class=o>.</span><span class=n>DoFn</span><span class=o>.</span><span class=n>WindowParam</span><span class=p>):</span>
- <span class=c1># access window e.g window.end.micros</span>
+ <span class=c1># access window e.g. window.end.micros</span>
<span class=k>pass</span>
</code></pre></div></div><p class=language-java><strong>PaneInfo:</strong>
When triggers are used, Beam provides a <code>PaneInfo</code> object that contains information about the current firing. Using <code>PaneInfo</code>
you can determine whether this is an early or a late firing, and how many times this window has already fired for this key.</p><p class=language-py><strong>PaneInfo:</strong>
When triggers are used, Beam provides a <code>DoFn.PaneInfoParam</code> object that contains information about the current firing. Using <code>DoFn.PaneInfoParam</code>
you can determine whether this is an early or a late firing, and how many times this window has already fired for this key.
-This feature implementation in python sdk is not fully completed, see more at <a href=https://issues.apache.org/jira/browse/BEAM-3759>BEAM-3759</a>.</p><div class=language-java><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=o>.</span><span class=na>of</span><span class=o>(</span><span class=k>new</span> <span class=n>DoFn</span><span class=o><</span><span class=n>String</span><span class=o>,</span> <span class=n>String</span><span class=o>& [...]
+This feature implementation in Python SDK is not fully completed; see more at <a href=https://issues.apache.org/jira/browse/BEAM-3759>BEAM-3759</a>.</p><div class=language-java><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=o>.</span><span class=na>of</span><span class=o>(</span><span class=k>new</span> <span class=n>DoFn</span><span class=o><</span><span class=n>String</span><span class=o>,</span> <span class=n>String</span><span class=o>& [...]
<span class=kd>public</span> <span class=kt>void</span> <span class=nf>processElement</span><span class=o>(</span><span class=nd>@Element</span> <span class=n>String</span> <span class=n>word</span><span class=o>,</span> <span class=n>PaneInfo</span> <span class=n>paneInfo</span><span class=o>)</span> <span class=o>{</span>
<span class=o>}})</span></code></pre></div></div><div class=language-py><div class=highlight><pre class=chroma><code class=language-py data-lang=py><span class=kn>import</span> <span class=nn>apache_beam</span> <span class=kn>as</span> <span class=nn>beam</span>
<span class=k>class</span> <span class=nc>ProcessRecord</span><span class=p>(</span><span class=n>beam</span><span class=o>.</span><span class=n>DoFn</span><span class=p>):</span>
<span class=k>def</span> <span class=nf>process</span><span class=p>(</span><span class=bp>self</span><span class=p>,</span> <span class=n>element</span><span class=p>,</span> <span class=n>pane_info</span><span class=o>=</span><span class=n>beam</span><span class=o>.</span><span class=n>DoFn</span><span class=o>.</span><span class=n>PaneInfoParam</span><span class=p>):</span>
- <span class=c1># access pane info e.g pane_info.is_first, pane_info.is_last, pane_info.timing</span>
+ <span class=c1># access pane info, e.g. pane_info.is_first, pane_info.is_last, pane_info.timing</span>
<span class=k>pass</span>
</code></pre></div></div><p class=language-java><strong>PipelineOptions:</strong>
The <code>PipelineOptions</code> for the current pipeline can always be accessed in a process method by adding it
@@ -1176,7 +1182,7 @@ as a parameter:</p><div class=language-java><div class=highlight><pre class=chro
a parameter of type <code>TimeDomain</code> which tells whether the timer is based on event time or processing time.
Timers are explained in more detail in the
<a href=/blog/2017/08/28/timely-processing.html>Timely (and Stateful) Processing with Apache Beam</a> blog post.</p><p class=language-py><strong>Timer and State:</strong>
-In addition to aforementioned parameters, user defined Timer and State parameters can be used in a Stateful DoFn.
+In addition to aforementioned parameters, user defined Timer and State parameters can be used in a stateful DoFn.
Timers and States are explained in more detail in the
<a href=/blog/2017/08/28/timely-processing.html>Timely (and Stateful) Processing with Apache Beam</a> blog post.</p><div class=language-py><div class=highlight><pre class=chroma><code class=language-py data-lang=py><span class=k>class</span> <span class=nc>StatefulDoFn</span><span class=p>(</span><span class=n>beam</span><span class=o>.</span><span class=n>DoFn</span><span class=p>):</span>
<span class=s2>"""An example stateful DoFn with state and timer"""</span>
@@ -1193,7 +1199,7 @@ Timers and States are explained in more detail in the
<span class=n>buffer_2</span><span class=o>=</span><span class=n>beam</span><span class=o>.</span><span class=n>DoFn</span><span class=o>.</span><span class=n>StateParam</span><span class=p>(</span><span class=n>BUFFER_STATE_2</span><span class=p>),</span>
<span class=n>watermark_timer</span><span class=o>=</span><span class=n>beam</span><span class=o>.</span><span class=n>DoFn</span><span class=o>.</span><span class=n>TimerParam</span><span class=p>(</span><span class=n>WATERMARK_TIMER</span><span class=p>)):</span>
- <span class=c1># Do you processing here</span>
+ <span class=c1># Do your processing here</span>
<span class=n>key</span><span class=p>,</span> <span class=n>value</span> <span class=o>=</span> <span class=n>element</span>
<span class=c1># Read all the data from buffer1</span>
<span class=n>all_values_in_buffer_1</span> <span class=o>=</span> <span class=p>[</span><span class=n>x</span> <span class=k>for</span> <span class=n>x</span> <span class=ow>in</span> <span class=n>buffer_1</span><span class=o>.</span><span class=n>read</span><span class=p>()]</span>
@@ -1242,10 +1248,7 @@ transform operations:</p><ol><li>It applies a <code>ParDo</code> on the input <c
an output <code>PCollection</code> of individual words.</li><li>It applies the Beam SDK library transform <code>Count</code> on the <code>PCollection</code> of
words, producing a <code>PCollection</code> of key/value pairs. Each key represents a
word in the text, and each value represents the number of times that word
-appeared in the original data.</li></ol><p>Note that this is also an example of nested composite transforms, as <code>Count</code>
-is, by itself, a composite transform.</p><p>Your composite transform’s parameters and return value must match the initial
-input type and final return type for the entire transform, even if the
-transform’s intermediate data changes type multiple times.</p><div class=language-java><div class=highlight><pre class=chroma><code class=language-java data-lang=java> <span class=kd>public</span> <span class=kd>static</span> <span class=kd>class</span> <span class=nc>CountWords</span> <span class=kd>extends</span> <span class=n>PTransform</span><span class=o><</span><span class=n>PCollection</span><span class=o><</span><span class=n>String</span><span class=o>>,</span>
+appeared in the original data.</li></ol><div class=language-java><div class=highlight><pre class=chroma><code class=language-java data-lang=java> <span class=kd>public</span> <span class=kd>static</span> <span class=kd>class</span> <span class=nc>CountWords</span> <span class=kd>extends</span> <span class=n>PTransform</span><span class=o><</span><span class=n>PCollection</span><span class=o><</span><span class=n>String</span><span class=o>>,</span>
<span class=n>PCollection</span><span class=o><</span><span class=n>KV</span><span class=o><</span><span class=n>String</span><span class=o>,</span> <span class=n>Long</span><span class=o>>>></span> <span class=o>{</span>
<span class=nd>@Override</span>
<span class=kd>public</span> <span class=n>PCollection</span><span class=o><</span><span class=n>KV</span><span class=o><</span><span class=n>String</span><span class=o>,</span> <span class=n>Long</span><span class=o>>></span> <span class=nf>expand</span><span class=o>(</span><span class=n>PCollection</span><span class=o><</span><span class=n>String</span><span class=o>></span> <span class=n>lines</span><span class=o>)</span> <span class=o>{</span>
@@ -1270,7 +1273,8 @@ transform’s intermediate data changes type multiple times.</p><div class=l
<span class=c1># Count the number of times each word occurs.</span>
<span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>combiners</span><span class=o>.</span><span class=n>Count</span><span class=o>.</span><span class=n>PerElement</span><span class=p>()</span>
<span class=c1># Format each word and count into a printable string.</span>
- <span class=o>|</span> <span class=s1>'FormatCounts'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span class=n>FormatCountsFn</span><span class=p>()))</span></code></pre></div></div><h4 id=composite-transform-creation>4.6.2. Creating a composite transform</h4><p>To create your own composite transform, create a subclass of the <code>PTransform</code>
+ <span class=o>|</span> <span class=s1>'FormatCounts'</span> <span class=o>>></span> <span class=n>beam</span><span class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span class=n>FormatCountsFn</span><span class=p>()))</span></code></pre></div></div><blockquote><p><strong>Note:</strong> Because <code>Count</code> is itself a composite transform,
+<code>CountWords</code> is also a nested composite transform.</p></blockquote><h4 id=composite-transform-creation>4.6.2. Creating a composite transform</h4><p>To create your own composite transform, create a subclass of the <code>PTransform</code>
class and override the <code>expand</code> method to specify the actual processing logic.
You can then use this transform just as you would a built-in transform from the
Beam SDK.</p><p class=language-java>For the <code>PTransform</code> class type parameters, you pass the <code>PCollection</code> types
@@ -1302,7 +1306,9 @@ value.</p><p>The following code sample shows how to override <code>expand</code>
accept the appropriate input <code>PCollection</code>(s) and return the corresponding
output <code>PCollection</code>(s), you can include as many transforms as you want. These
transforms can include core transforms, composite transforms, or the transforms
-included in the Beam SDK libraries.</p><p><strong>Note:</strong> The <code>expand</code> method of a <code>PTransform</code> is not meant to be invoked
+included in the Beam SDK libraries.</p><p>Your composite transform’s parameters and return value must match the initial
+input type and final return type for the entire transform, even if the
+transform’s intermediate data changes type multiple times.</p><p><strong>Note:</strong> The <code>expand</code> method of a <code>PTransform</code> is not meant to be invoked
directly by the user of a transform. Instead, you should call the <code>apply</code> method
on the <code>PCollection</code> itself, with the transform as an argument. This allows
transforms to be nested within the structure of your pipeline.</p><h4 id=ptransform-style-guide>4.6.3. PTransform Style Guide</h4><p>The <a href=/contribute/ptransform-style-guide/>PTransform Style Guide</a>
@@ -1324,7 +1330,7 @@ your pipeline’s final results. However, you can use a write transform to o
a <code>PCollection</code>'s data at any point in your pipeline.</p><div class=language-java><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=n>output</span><span class=o>.</span><span class=na>apply</span><span class=o>(</span><span class=n>TextIO</span><span class=o>.</span><span class=na>write</span><span class=o>().</span><span class=na>to</span><span class=o>(</span><span class=s>"gs://some/outputData"</span><span class=o>));</span> [...]
operator you provide. Note that glob operators are filesystem-specific and obey
filesystem-specific consistency models. The following TextIO example uses a glob
-operator (*) to read all matching input files that have prefix “input-” and the
+operator (<code>*</code>) to read all matching input files that have prefix “input-” and the
suffix “.csv” in the given location:</p><div class=language-java><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=n>p</span><span class=o>.</span><span class=na>apply</span><span class=o>(</span><span class=s>"ReadFromText"</span><span class=o>,</span>
<span class=n>TextIO</span><span class=o>.</span><span class=na>read</span><span class=o>().</span><span class=na>from</span><span class=o>(</span><span class=s>"protocol://my_bucket/path/to/input-*.csv"</span><span class=o>));</span></code></pre></div></div><div class=language-py><div class=highlight><pre class=chroma><code class=language-py data-lang=py><span class=n>lines</span> <span class=o>=</span> <span class=n>p</span> <span class=o>|</span> <span class=s1>'ReadFr [...]
independently and then use the <a href=#flatten>Flatten</a> transform to create a single
@@ -1342,7 +1348,7 @@ JSON, Avro, Protocol Buffer, or database row objects; all of these types have we
structures that can often be determined by examining the type. Even within a SDK pipeline, Simple Java POJOs
(or equivalent structures in other languages) are often used as intermediate types, and these also have a
clear structure that can be inferred by inspecting the class. By understanding the structure of a pipeline’s
-records, we can provide much more concise APIs for data processing.</p><h3 id=what-is-a-schema>6.1. What is a schema</h3><p>Most structured records share some common characteristics:</p><ul><li>They can be subdivided into separate named fields. Fields usually have string names, but sometimes - as in the case of indexed
+records, we can provide much more concise APIs for data processing.</p><h3 id=what-is-a-schema>6.1. What is a schema?</h3><p>Most structured records share some common characteristics:</p><ul><li>They can be subdivided into separate named fields. Fields usually have string names, but sometimes - as in the case of indexed
tuples - have numerical indices instead.</li><li>There is a confined list of primitive types that a field can have. These often match primitive types in most programming
languages: int, long, string, etc.</li><li>Often a field type can be marked as optional (sometimes referred to as nullable) or required.</li></ul><p>Often records have a nested structure. A nested structure occurs when a field itself has subfields so the
type of the field itself has a schema. Fields that are array or map types is also a common feature of these structured
diff --git a/website/generated-content/feed.xml b/website/generated-content/feed.xml
index b8ef234..cc29f67 100644
--- a/website/generated-content/feed.xml
+++ b/website/generated-content/feed.xml
@@ -43,7 +43,45 @@ For more information on changes in 2.22.0, check out the
</ul>
<h2 id="list-of-contributors">List of Contributors</h2>
<p>According to git shortlog, the following people contributed to the 2.22.0 release. Thank you to all contributors!</p>
-<p>Ahmet Altay, aiyangar, Ajo Thomas, Akshay-Iyangar, Alan Pryor, Alexey Romanenko, Allen Pradeep Xavier, amaliujia, Andrew Pilloud, Ankur Goenka, Ashwin Ramaswami, bntnam, Borzoo Esmailloo, Boyuan Zhang, Brian Hulette, Chamikara Jayalath, Colm O hEigeartaigh, Craig Chambers, Damon Douglas, Daniel Oliveira, David Cavazos, David Moravek, Esun Kim, Etienne Chauchot, Filipe Regadas, Graeme Morgan, Hannah Jiang, Hannah-Jiang, Harch Vardhan, Heejong Lee, Henry Suryawirawan, Ismaël Mejía, I [...]
+<p>Ahmet Altay, aiyangar, Ajo Thomas, Akshay-Iyangar, Alan Pryor, Alexey Romanenko, Allen Pradeep Xavier, amaliujia, Andrew Pilloud, Ankur Goenka, Ashwin Ramaswami, bntnam, Borzoo Esmailloo, Boyuan Zhang, Brian Hulette, Chamikara Jayalath, Colm O hEigeartaigh, Craig Chambers, Damon Douglas, Daniel Oliveira, David Cavazos, David Moravek, Esun Kim, Etienne Chauchot, Filipe Regadas, Graeme Morgan, Hannah Jiang, Hannah-Jiang, Harch Vardhan, Heejong Lee, Henry Suryawirawan, Ismaël Mejía, I [...]
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<p>Today, we are happy to announce a new addition to the Beam Katas family: Kotlin!</p>
+<p><img src="/images/blog/beam-katas-kotlin-release/beam-and-kotlin.png" alt="Apache Beam and Kotlin Shaking Hands" height="330" width="800" ></p>
+<p>You may remember <a href="https://beam.apache.org/blog/beam-kata-release">a post from last year</a> that informed everyone of the wonderful Beam Katas available on <a href="https://stepik.org">Stepik</a>
+for learning more about writing Apache Beam applications, working with its various APIs and programming model
+hands-on, all from the comfort of your favorite IDEs. As of today, you can now work through all of the progressive
+exercises to learn about the fundamentals of Beam in Kotlin.</p>
+<p><a href="https://kotlinlang.org">Kotlin</a> is a modern, open-source, statically typed language that targets the JVM. It is most commonly used by
+Android developers, however it has recently risen in popularity due to its extensive feature set that enables
+more concise and cleaner code than Java, without sacrificing performance or type safety. It recently was <a href="https://insights.stackoverflow.com/survey/2020#technology-most-loved-dreaded-and-wanted-languages-loved">ranked
+as one of the most beloved programming languages in the annual Stack Overflow Developer Survey</a>, so don&rsquo;t take
+just our word for it.</p>
+<p>The relationship between Apache Beam and Kotlin isn&rsquo;t a new one. You can find examples scattered across the web
+of engineering teams embracing the two technologies including <a href="https://beam.apache.org/blog/beam-kotlin/">a series of samples announced on this very blog</a>.
+If you are new to Beam or are an experienced veteran looking for a change of pace, we&rsquo;d encourage you to give
+Kotlin a try.</p>
+<p>You can find the Kotlin and the other excellent Beam Katas below (or by just searching for &ldquo;Beam Katas&rdquo; within
+<a href="https://www.jetbrains.com/education/download/#section=idea">IntelliJ</a> or <a href="https://www.jetbrains.com/education/download/#section=pycharm-edu">PyCharm</a> through <a href="https://plugins.jetbrains.com/plugin/10081-edutools">the EduTools plugin</a>):</p>
+<ul>
+<li><a href="https://stepik.org/course/72488"><strong>Kotlin</strong></a></li>
+<li><a href="https://stepik.org/course/54530"><strong>Java</strong></a></li>
+<li><a href="https://stepik.org/course/54532"><strong>Python</strong></a></li>
+<li><a href="https://stepik.org/course/70387"><strong>Go (in development)</strong></a></li>
+</ul>
+<p>I&rsquo;d like to extend a very special thanks to <a href="https://twitter.com/henry_ken">Henry Suryawirawan</a> for his creation of the original series of Katas
+and his support during the review process and making this effort a reality.</p>
+<p><br /></p>
+<p><img src="/images/blog/beam-katas-kotlin-release/beam-katas-in-edutools.png" alt="Access Beam Katas Kotlin through a JetBrains Educational Product" height="252" width="800" ></p></description><link>/blog/beam-katas-kotlin-release/</link><pubDate>Mon, 01 Jun 2020 00:00:01 -0800</pubDate><guid>/blog/beam-katas-kotlin-release/</guid><category>blog</category></item><item><title>Python SDK Typing Changes</title><description>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -684,65 +722,4 @@ Salman Raza, Sam Rohde, Saul Chavez, Shoaib, Shoaib Zafar, Slava Chernyak, Tanay
Thinh Ha, Thomas Weise, Tianzi Cai, Tim van der Lippe, Tomer Zeltzer, Tudor Marian,
Udi Meiri, Valentyn Tymofieiev, Yichi Zhang, Yifan Zou, Yueyang Qiu, gxercavins,
jesusrv1103, lostluck, matt-darwin, mrociorg, ostrokach, parahul, rahul8383, rosetn,
-sunjincheng121, the1plummie, ttanay, tvalentyn, venn001, yoshiki.obata, Łukasz Gajowy</p></description><link>/blog/beam-2.16.0/</link><pubDate>Mon, 07 Oct 2019 00:00:01 -0800</pubDate><guid>/blog/beam-2.16.0/</guid><category>blog</category></item><item><title>Google Summer of Code '19</title><description>
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-http://www.apache.org/licenses/LICENSE-2.0
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-<p>Google Summer of Code was an amazing learning experience for me.
-I contributed to open source, learned about Apache Beam’s internals and worked with the best engineers in the world.</p>
-<h2 id="motivation">Motivation</h2>
-<p>Two of my friends had participated in GSoC in 2018. I was intrigued by their experience.
-The idea of working on open-source software that could potentially be used by developers across the world, while being mentored by the best people in a field was exciting!
-So, I decided to give Google Summer of Code a shot this year.</p>
-<h2 id="what-is-google-summer-of-code">What is Google Summer of Code?</h2>
-<p><a href="https://summerofcode.withgoogle.com/">Google Summer of Code</a> is a global program hosted by Google focused on introducing students to open source software development.
-Students work on a 3 month programming project with an open source organization during their break from university.</p>
-<h2 id="why-apache-beam">Why Apache Beam?</h2>
-<p>While interning at <a href="https://atlan.com/">Atlan</a>, I discovered the field of Data Engineering. I found the challenges and the discussions of the engineers there interesting. While researching for my internship project, I came across the Streaming Systems book. It introduced me to the unified model of Apache Beam for Batch and Streaming Systems, which I was fascinated by.
-I wanted to explore Data Engineering, so for GSoC, I wanted to work on a project in that field. Towards the end of my internship, I started contributing to Apache Airflow(very cool project) and Apache Beam, hoping one of them would participate in GSoC. I got lucky!</p>
-<p><a href="https://youtu.be/U2eWLb-LD44">Also, Spotify’s Discover Weekly uses Apache Beam!</a></p>
-<h2 id="preparation">Preparation</h2>
-<p>I had already read the <a href="http://streamingsystems.net/">Streaming Systems book</a>. So, I had an idea of the concepts that Beam is built on, but had never actually used Beam.
-Before actually submitting a proposal, I went through a bunch of resources to make sure I had a concrete understanding of Beam.
-I read the <a href="https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101">Streaming 101</a> and <a href="https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102">Streaming 102</a> blogs by Tyler Akidau. They are the perfect introduction to Beam’s unified model for Batch and Streaming.
-In addition, I watched all Beam talks on YouTube. You can find them on the <a href="https://beam.apache.org/documentation/resources/videos-and-podcasts/">Beam Website</a>.
-Beam has really good documentation. The <a href="https://beam.apache.org/documentation/programming-guide/">Programming Guide</a> lays out all of Beam’s concepts really well. <a href="https://beam.apache.org/documentation/runtime/model">Beam’s execution model</a> is also documented well and is a must-read to understand how Beam processes data.
-<a href="https://www.waitingforcode.com/apache-beam">waitingforcode.com</a> also has good blog posts about Beam concepts.
-To get a better sense of the Beam codebase, I played around with it and worked on some PRs to understand Beam better and got familiar with the test suite and workflows.</p>
-<h2 id="gsoc-journey">GSoC Journey</h2>
-<p>GSoC has 2 phases. The first is the Community Bonding period in which students get familiar with the project and the community. The other is the actual Coding Period in which students work on their projects. Since the Coding Period has three evaluations spaced out by a month, I divided my project into three parts focusing on the implementation, tests, and documentation or improvements.</p>
-<h3 id="project">Project</h3>
-<p>My project(<a href="https://issues.apache.org/jira/browse/BEAM-6611">BEAM-6611</a>) added support for File Loads method of inserting data into BigQuery for streaming pipelines. It builds on PR - <a href="https://github.com/apache/beam/pull/7655">#7655</a> for <a href="https://issues.apache.org/jira/browse/BEAM-6553">BEAM-6553</a> that added support in the Python SDK for writing to BigQuery using File Loads method for Batch pipelines. Streaming pipelines with non-d [...]
-You can find my proposal <a href="https://docs.google.com/document/d/15Peyd3Z_wu5rvGWw8lMLpZuTyyreM_JOAEFFWvF97YY/edit?usp=sharing">here</a>.</p>
-<h3 id="community-bonding">Community Bonding</h3>
-<p>When GSoC started, my semester end exams had not yet finished. As a result, I couldn’t get much done. I worked on three PTransforms for the Python SDK - Latest, WithKeys and Reify.</p>
-<h3 id="coding-period-i">Coding Period I</h3>
-<p>In this period, I wrote some Integration Tests for the BigQuery sink using Streaming Inserts in streaming mode. I worked on a failing integration test for my project. I also finished the implementation of my project. But, one PostCommit test didn’t pass. I realized that the matcher for the Integration Test that queried BigQuery for the results was intended to be used in Batch mode. So, I wrote a version of the matcher to work in streaming mode.</p>
-<h3 id="coding-period-ii">Coding Period II</h3>
-<p>Even after I had added the matcher for streaming mode, the PostComit tests did not pass. A test was being run even though it was not specified. I isolated the failure to a <a href="https://nose.readthedocs.io/en/latest/doc_tests/test_multiprocess/multiprocess.html#other-differences-in-test-running">limitation</a> of the multiprocess plugin for <a href="https://nose.readthedocs.io/en/latest/">nose(a Python test framework)</a> due to which it found more tests than had bee [...]
-I also worked on small issues related to testing.</p>
-<p>This period was marked by a few exciting events:</p>
-<ul>
-<li>Ending up in the top #100 contributors to apache/beam.</li>
-<li>My first ever PR Review on an open source project.</li>
-</ul>
-<p><img src="https://pbs.twimg.com/media/D_XNSC-UIAUmswG?format=png&name=small" alt="Weird flex but ok" /></p>
-<h3 id="coding-period-iii">Coding Period III</h3>
-<p>This was the final coding period before the program ended. Since my project was merged earlier than expected, my mentor suggested another issue(<a href="https://issues.apache.org/jira/browse/BEAM-7742">BEAM-7742</a>) in the same area - BigQueryIO, that I found interesting. So, I worked on partitioning written files in BigQuery to ensure that all load jobs triggered adhere to the load job size limitations specified for BigQuery.
-While working on my project, I was using a pipeline that uses PubSub as a source and BigQuery as a sink to validate my changes. My mentor suggested we add them to the Beam test suite as it would be the ultimate test for BigQueryIO. I also worked on adding this test to Beam.</p>
-<p>You can find the list of PRs I worked on <a href="https://github.com/apache/beam/pulls?utf8=%E2%9C%93&amp;q=is%3Apr+author%3Attanay">here</a>.</p>
-<h2 id="conclusion">Conclusion</h2>
-<p>GSoC has been a lesson in discipline and goal-setting for me. Deciding what I wanted to work on and how much I wanted to get done each week was an important lesson.
-I had never worked remotely, so this was a new experience. Although I struggled with it initially, I appreciate the flexibility that it comes with.
-I also had a lot of fun learning about Apache Beam’s internals, and other tools in the same ecosystem.
-This was also the first time I had written code with a test-first approach.</p>
-<p>I thank my mentor - Pablo Estrada, Apache Beam, The Apache Software Foundation and Google Summer of Code for this opportunity. I am also grateful to my mentor for helping me with everything I needed and more, and the Apache Beam community for being supportive and encouraging.</p>
-<p>With the right effort, perseverance, conviction, and a plan, anything is possible. Anything.</p></description><link>/blog/gsoc-19/</link><pubDate>Wed, 04 Sep 2019 00:00:01 -0800</pubDate><guid>/blog/gsoc-19/</guid><category>blog</category><category>gsoc</category></item></channel></rss>
\ No newline at end of file
+sunjincheng121, the1plummie, ttanay, tvalentyn, venn001, yoshiki.obata, Łukasz Gajowy</p></description><link>/blog/beam-2.16.0/</link><pubDate>Mon, 07 Oct 2019 00:00:01 -0800</pubDate><guid>/blog/beam-2.16.0/</guid><category>blog</category></item></channel></rss>
\ No newline at end of file
diff --git a/website/generated-content/images/blog/beam-katas-kotlin-release/beam-and-kotlin.png b/website/generated-content/images/blog/beam-katas-kotlin-release/beam-and-kotlin.png
new file mode 100644
index 0000000..13bf0f9
Binary files /dev/null and b/website/generated-content/images/blog/beam-katas-kotlin-release/beam-and-kotlin.png differ
diff --git a/website/generated-content/images/blog/beam-katas-kotlin-release/beam-katas-in-edutools.png b/website/generated-content/images/blog/beam-katas-kotlin-release/beam-katas-in-edutools.png
new file mode 100644
index 0000000..a3e59e7
Binary files /dev/null and b/website/generated-content/images/blog/beam-katas-kotlin-release/beam-katas-in-edutools.png differ
diff --git a/website/generated-content/index.html b/website/generated-content/index.html
index f30018c..b066ea5 100644
--- a/website/generated-content/index.html
+++ b/website/generated-content/index.html
@@ -5,7 +5,7 @@
<a class="button button--primary" href=/get-started/try-apache-beam/>Try Beam</a>
<a class="button button--primary" href=/get-started/downloads/>Download Beam SDK 2.22.0</a></div><div class=hero__ctas><a class=button href=/get-started/quickstart-java/>Java Quickstart</a>
<a class=button href=/get-started/quickstart-py/>Python Quickstart</a>
-<a class=button href=/get-started/quickstart-go/>Go Quickstart</a></div></div></div><div class=hero__cols__col><div class=hero__blog><div class=hero__blog__title>The latest from the blog</div><div class=hero__blog__cards><a class=hero__blog__cards__card href=/blog/beam-2.22.0/><div class=hero__blog__cards__card__title>Apache Beam 2.22.0</div><div class=hero__blog__cards__card__date>Jun 8, 2020</div></a><a class=hero__blog__cards__card href=/blog/python-typing/><div class=hero__blog__card [...]
+<a class=button href=/get-started/quickstart-go/>Go Quickstart</a></div></div></div><div class=hero__cols__col><div class=hero__blog><div class=hero__blog__title>The latest from the blog</div><div class=hero__blog__cards><a class=hero__blog__cards__card href=/blog/beam-2.22.0/><div class=hero__blog__cards__card__title>Apache Beam 2.22.0</div><div class=hero__blog__cards__card__date>Jun 8, 2020</div></a><a class=hero__blog__cards__card href=/blog/beam-katas-kotlin-release/><div class=hero [...]
<a class="button button--primary" href=/get-started/downloads/>Download Beam SDK 2.22.0</a></div><div class=ctas__ctas><a class=button href=/get-started/quickstart-java/>Java Quickstart</a>
<a class=button href=/get-started/quickstart-py/>Python Quickstart</a>
<a class=button href=/get-started/quickstart-go/>Go Quickstart</a></div></div></div><footer class=footer><div class=footer__contained><div class=footer__cols><div class=footer__cols__col><div class=footer__cols__col__logo><img src=/images/beam_logo_circle.svg class=footer__logo alt="Beam logo"></div><div class=footer__cols__col__logo><img src=/images/apache_logo_circle.svg class=footer__logo alt="Apache logo"></div></div><div class="footer__cols__col footer__cols__col--md"><div class=foo [...]
diff --git a/website/generated-content/sitemap.xml b/website/generated-content/sitemap.xml
index e40e8b9..d8b49cd 100644
--- a/website/generated-content/sitemap.xml
+++ b/website/generated-content/sitemap.xml
@@ -1 +1 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/beam-2.22.0/</loc><lastmod>2020-06-08T14:13:37-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2020-06-08T14:13:37-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2020-06-08T14:13:37-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2020-06-08T14:13:37-07:00</lastmod></url><url><loc>/catego [...]
\ No newline at end of file
+<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/beam-2.22.0/</loc><lastmod>2020-06-08T14:13:37-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2020-06-08T14:13:37-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2020-06-08T14:13:37-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2020-06-08T14:13:37-07:00</lastmod></url><url><loc>/blog/b [...]
\ No newline at end of file