You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by sc...@apache.org on 2018/10/11 16:39:47 UTC
[beam] branch asf-site updated (8feb4da -> 56bfddf)
This is an automated email from the ASF dual-hosted git repository.
scott pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git.
discard 8feb4da Publishing website 2018/10/11 01:03:11 at commit 4e0c9a1
discard b57ef3e Publishing website 2018/10/11 00:00:38 at commit 223372c
discard 22801b7 Publishing website 2018/10/10 23:28:05 at commit 223372c
discard 5eb0960 Publishing website 2018/10/10 22:56:15 at commit 738f796
discard 81b34c2 Publishing website 2018/10/10 20:11:44 at commit 685e79d
discard e7bf9d0 Publishing website 2018/10/10 19:58:02 at commit 9be4fb9
discard e6863c3 Publishing website 2018/10/10 19:32:08 at commit 31a3e3e
discard 433cbee Publishing website 2018/10/10 18:22:25 at commit dc96860
discard d06f3c3 Publishing website 2018/10/10 18:00:37 at commit 61223f4
discard 30df68b Publishing website 2018/10/10 16:41:21 at commit 61223f4
discard deaab34 Publishing website 2018/10/10 16:09:01 at commit afa062a
discard c0a5528 Publishing website 2018/10/10 15:56:48 at commit d96d596
discard 56405d9 Publishing website 2018/10/10 14:55:43 at commit 18d2168
discard 082eb2c Publishing website 2018/10/10 13:55:38 at commit 3f803f1
discard f3a9383 Publishing website 2018/10/10 12:00:35 at commit 1dca49f
discard e69de04 Publishing website 2018/10/10 11:52:16 at commit 1dca49f
discard c831636 Publishing website 2018/10/10 11:48:41 at commit b436c31
discard 234dce3 Publishing website 2018/10/10 09:06:16 at commit a432a4f
discard 028ef3f Publishing website 2018/10/10 06:00:35 at commit 88c6845
discard 7cd37ac Publishing website 2018/10/10 05:35:38 at commit 88c6845
discard 63c9eec Publishing website 2018/10/10 02:57:47 at commit d32364c
discard 906a922 Publishing website 2018/10/10 00:57:24 at commit ae3d9ca
discard 863bdbe Publishing website 2018/10/10 00:47:15 at commit 82c1e6e
discard 7919a9c Publishing website 2018/10/10 00:21:04 at commit b9af715
discard c7a5019 Publishing website 2018/10/09 23:34:17 at commit 399c09f
discard 5fb2c22 Publishing website 2018/10/09 22:59:23 at commit 1fce865
discard 910e066 Publishing website 2018/10/09 22:47:33 at commit c356c57
discard ca713d0 Publishing website 2018/10/09 21:28:46 at commit 4e132f3
discard e6f6495 Publishing website 2018/10/09 20:09:55 at commit 4afd5ec
discard fe92be1 Publishing website 2018/10/09 19:58:54 at commit 0e8f28a
discard b2579c5 Publishing website 2018/10/09 19:55:23 at commit 092d496
discard 66cf000 Publishing website 2018/10/09 19:54:38 at commit 36da831
discard c3ecd83 Publishing website 2018/10/09 18:41:21 at commit ddbca9d
discard 3ba8c4a Publishing website 2018/10/09 18:38:19 at commit 6578254
discard 1df1402 Publishing website 2018/10/09 18:37:33 at commit f58f6e6
discard 9f8bb5e Publishing website 2018/10/09 18:28:48 at commit ac21fff
discard 4fd6bc1 Publishing website 2018/10/09 18:00:36 at commit f6945d5
discard 0b32751 Publishing website 2018/10/09 16:01:28 at commit f6945d5
discard da7cb61 Publishing website 2018/10/07 06:00:37 at commit b2ad757
discard b84b825 Publishing website 2018/10/07 00:03:21 at commit b2ad757
discard abd4390 Publishing website 2018/10/07 00:02:34 at commit 324f0b3
discard e2d5f41 Publishing website 2018/10/06 18:00:46 at commit 324f0b3
discard 1006a8e Publishing website 2018/10/06 12:00:44 at commit 324f0b3
discard 5791c01 Publishing website 2018/10/06 06:00:46 at commit 324f0b3
discard 38e9657 Publishing website 2018/10/06 00:00:36 at commit 324f0b3
discard cf3ccb8 Publishing website 2018/10/05 23:12:59 at commit 324f0b3
discard a3a8775 Publishing website 2018/10/05 23:02:55 at commit 9ac7ad4
discard 6c9bf8f Publishing website 2018/10/05 22:39:21 at commit a19183b
discard ca42eb4 Merge pull request #6591: [BEAM-5669] Stop empty commits being pushed by PostCommit_Website_Publish jenkins job
omit 7e4f2d0 [BEAM-5669] Stop empty commits being pushed by PostCommit_Website_Publish jenkins job
omit 15fb58a Publishing website 2018/10/05 06:01:07 at commit fed2d1d
omit ae0e876 Publishing website 2018/10/05 05:38:08 at commit fed2d1d
omit 7a89eea Publishing website 2018/10/05 01:15:27 at commit dae9a52
omit 55a9321 Publishing website 2018/10/05 00:04:26 at commit b57283b
omit 815d88b Publishing website 2018/10/05 00:01:05 at commit 845f8d0
omit 0ba00ad Publishing website 2018/10/04 22:55:07 at commit 845f8d0
omit 5d79e0e Publishing website 2018/10/04 22:34:23 at commit 0bef2fc
omit a949eab Publishing website 2018/10/04 22:23:47 at commit 3de49ec
omit 47dc8c9 Publishing website 2018/10/04 20:54:56 at commit 87dae1e
omit ce5129e Publishing website 2018/10/04 20:26:23 at commit ef02c39
omit 5d91e4d Publishing website 2018/10/04 20:21:56 at commit 4506080
omit 77be92a Publishing website 2018/10/04 18:01:09 at commit f9da7d5
omit 6efbf14 Publishing website 2018/10/04 16:56:02 at commit f9da7d5
omit 97d5835 Publishing website 2018/10/04 15:35:16 at commit 59fd44f
omit b61a2d0 Publishing website 2018/10/04 12:01:16 at commit 302a842
omit b3a1f92 Publishing website 2018/10/04 11:17:20 at commit 302a842
omit bf0e6a3 Publishing website 2018/10/04 10:41:23 at commit 0ce9eac
omit de43c0d Publishing website 2018/10/04 06:01:05 at commit 63102f8
omit 903aa3a Publishing website 2018/10/04 04:11:17 at commit 63102f8
omit 7798b01 Publishing website 2018/10/04 00:01:05 at commit 66d7c86
omit 87d4d5e Publishing website 2018/10/03 23:29:04 at commit 66d7c86
omit 66ae268 Publishing website 2018/10/03 22:46:11 at commit cc90583
omit 2e822ea Publishing website 2018/10/03 22:42:46 at commit 6ef5087
omit e8180a4 Publishing website 2018/10/03 22:15:41 at commit b34b180
omit 00c706d Publishing website 2018/10/03 22:07:46 at commit f58a812
omit e7aa46a Publishing website 2018/10/03 21:15:19 at commit 3e153f9
omit 0bfcc77 Publishing website 2018/10/03 19:46:50 at commit 004c6cd
omit a17d69e Publishing website 2018/10/03 18:01:09 at commit e4b7f09
omit dfabdbb Publishing website 2018/10/03 16:22:18 at commit e4b7f09
omit 1ec1b31 Publishing website 2018/10/03 16:01:42 at commit 3560300
omit 9be4073 Publishing website 2018/10/03 12:01:09 at commit 872a368
omit d3eab85 Publishing website 2018/10/03 06:01:09 at commit 872a368
omit ce9ab38 Publishing website 2018/10/03 05:03:09 at commit 872a368
omit 5a67f50 Publishing website 2018/10/03 00:01:08 at commit c48f429
omit 318ff2f Publishing website 2018/10/02 23:10:57 at commit c48f429
omit 4f9af11 Publishing website 2018/10/02 22:51:51 at commit be8d469
omit 0f379ff Publishing website 2018/10/02 22:00:07 at commit ed45509
omit 637a9d4 Publishing website 2018/10/01 20:34:04 at commit 1ccbbe5
omit c88162a Publishing website 2018/10/01 20:26:56 at commit 4331fc5
omit 5c2db15 Publishing website 2018/09/27 21:57:29 at commit 430fb03
omit b71c78d Publishing website 2018/09/27 21:52:18 at commit 901588e
omit dccd0bd Publishing website 2018/09/27 21:33:04 at commit 24de10e
omit 8a7532d Publishing website 2018/09/26 14:37:37 at commit da3fc69
omit 6382293 Publishing website 2018/09/26 03:27:28 at commit b50c0ae
omit 12b0ceb Publishing website 2018/09/26 03:22:29 at commit 4edeecc
omit b3c8fc8 Publishing website 2018/09/26 03:14:38 at commit b16bd7d
omit ac3df3e Publishing website 2018/09/26 03:07:13 at commit 86f7f05
omit 20cc797 Publishing website 2018/09/26 03:03:34 at commit 7576a9b
omit be93908 Publishing website 2018/09/26 03:00:36 at commit ba06d1f
omit 42e0dec Publishing website 2018/09/26 02:51:58 at commit 6fb19ce
omit 827cc9a Publishing website 2018/09/26 02:39:15 at commit 7875d2c
omit b2c6acd Publishing website 2018/09/26 02:31:39 at commit 076f73c
omit bbb83a4 Publishing website 2018/09/26 02:26:05 at commit fdcaa7c
omit a156066 Publishing website 2018/09/26 02:20:18 at commit 8cf8566
omit 615322e Publishing website 2018/09/26 02:14:31 at commit d87b42f
new 774749f Publishing website 2018/09/26 14:37:37 at commit da3fc69
new 3baba93 Publishing website 2018/09/27 21:33:04 at commit 24de10e
new 0442b47 Publishing website 2018/10/01 20:26:56 at commit 4331fc5
new 2c342fc Publishing website 2018/10/04 20:21:56 at commit 4506080
new 316b5ca Publishing website 2018/10/04 22:23:47 at commit 3de49ec
new a605e3e Publishing website 2018/10/04 22:34:23 at commit 0bef2fc
new 938c022 [BEAM-5669] Stop empty commits being pushed by PostCommit_Website_Publish jenkins job
new 444a308 Publishing website 2018/10/05 22:39:21 at commit a19183b
new 7e2bb0d Publishing website 2018/10/05 23:02:55 at commit 9ac7ad4
new 167811e Publishing website 2018/10/09 18:41:21 at commit ddbca9d
new 56bfddf Publishing website 2018/10/10 19:58:02 at commit 9be4fb9
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (8feb4da)
\
N -- N -- N refs/heads/asf-site (56bfddf)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 11 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
[beam] 03/11: Publishing website 2018/10/01 20:26:56 at commit
4331fc5
Posted by sc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 0442b47240ab522196b5cc644eb579c1ecca4179
Author: jenkins <bu...@apache.org>
AuthorDate: Mon Oct 1 20:26:57 2018 +0000
Publishing website 2018/10/01 20:26:56 at commit 4331fc5
---
website/generated-content/get-started/downloads/index.html | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/website/generated-content/get-started/downloads/index.html b/website/generated-content/get-started/downloads/index.html
index 7a1ca7d..ef1648f 100644
--- a/website/generated-content/get-started/downloads/index.html
+++ b/website/generated-content/get-started/downloads/index.html
@@ -276,9 +276,9 @@ versions denoted <code class="highlighter-rouge">0.x.y</code>.</p>
<h2 id="releases">Releases</h2>
<h3 id="260-2018-08-08">2.6.0 (2018-08-08)</h3>
-<p>Official <a href="https://dist.apache.org/repos/dist/release/beam/2.6.0/apache-beam-2.6.0-source-release.zip">source code download</a>
-<a href="https://dist.apache.org/repos/dist/release/beam/2.6.0/apache-beam-2.6.0-source-release.zip.sha512">SHA-512</a>
-<a href="https://dist.apache.org/repos/dist/release/beam/2.6.0/apache-beam-2.6.0-source-release.zip.asc">signature</a>.</p>
+<p>Official <a href="https://archive.apache.org/dist/beam/2.6.0/apache-beam-2.6.0-source-release.zip">source code download</a>
+<a href="https://archive.apache.org/dist/beam/2.6.0/apache-beam-2.6.0-source-release.zip.sha512">SHA-512</a>
+<a href="https://archive.apache.org/dist/beam/2.6.0/apache-beam-2.6.0-source-release.zip.asc">signature</a>.</p>
<p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343392">Release notes</a>.</p>
[beam] 06/11: Publishing website 2018/10/04 22:34:23 at commit
0bef2fc
Posted by sc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
commit a605e3e0740a988cea69bd7fbbef9db177de2422
Author: jenkins <bu...@apache.org>
AuthorDate: Thu Oct 4 22:34:23 2018 +0000
Publishing website 2018/10/04 22:34:23 at commit 0bef2fc
---
.../contribute/become-a-committer/index.html | 4 +-
.../contribute/committer-guide/index.html | 4 +-
.../contribute/dependencies/index.html | 4 +-
.../contribute/design-documents/index.html | 4 +-
.../contribute/docker-images/index.html | 4 +-
.../contribute/eclipse/index.html | 387 --------------------
website/generated-content/contribute/index.html | 4 +-
.../contribute/intellij/index.html | 389 ---------------------
.../contribute/portability/index.html | 4 +-
.../contribute/postcommits-guides/index.html | 4 +-
.../postcommits-policies-details/index.html | 4 +-
.../contribute/postcommits-policies/index.html | 4 +-
.../contribute/precommit-policies/index.html | 4 +-
.../contribute/precommit-triage-guide/index.html | 4 +-
.../contribute/ptransform-style-guide/index.html | 4 +-
.../contribute/release-guide/index.html | 4 +-
.../contribute/runner-guide/index.html | 4 +-
.../contribute/testing/index.html | 4 +-
.../contribute/website-contributions/index.html | 4 +-
19 files changed, 34 insertions(+), 810 deletions(-)
diff --git a/website/generated-content/contribute/become-a-committer/index.html b/website/generated-content/contribute/become-a-committer/index.html
index 0c910c5..20ebfac 100644
--- a/website/generated-content/contribute/become-a-committer/index.html
+++ b/website/generated-content/contribute/become-a-committer/index.html
@@ -142,8 +142,8 @@
<span class="section-nav-list-title">IDE tips</span>
<ul class="section-nav-list">
- <li><a href="/contribute/intellij/">IntelliJ Tips</a></li>
- <li><a href="/contribute/eclipse/">Eclipse Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/IntelliJ+Tips">IntelliJ Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/Eclipse+Tips">Eclipse Tips</a></li>
</ul>
</li>
<li>
diff --git a/website/generated-content/contribute/committer-guide/index.html b/website/generated-content/contribute/committer-guide/index.html
index 3ad43d6..fa29d8d 100644
--- a/website/generated-content/contribute/committer-guide/index.html
+++ b/website/generated-content/contribute/committer-guide/index.html
@@ -142,8 +142,8 @@
<span class="section-nav-list-title">IDE tips</span>
<ul class="section-nav-list">
- <li><a href="/contribute/intellij/">IntelliJ Tips</a></li>
- <li><a href="/contribute/eclipse/">Eclipse Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/IntelliJ+Tips">IntelliJ Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/Eclipse+Tips">Eclipse Tips</a></li>
</ul>
</li>
<li>
diff --git a/website/generated-content/contribute/dependencies/index.html b/website/generated-content/contribute/dependencies/index.html
index dd97540..a1f750c 100644
--- a/website/generated-content/contribute/dependencies/index.html
+++ b/website/generated-content/contribute/dependencies/index.html
@@ -142,8 +142,8 @@
<span class="section-nav-list-title">IDE tips</span>
<ul class="section-nav-list">
- <li><a href="/contribute/intellij/">IntelliJ Tips</a></li>
- <li><a href="/contribute/eclipse/">Eclipse Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/IntelliJ+Tips">IntelliJ Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/Eclipse+Tips">Eclipse Tips</a></li>
</ul>
</li>
<li>
diff --git a/website/generated-content/contribute/design-documents/index.html b/website/generated-content/contribute/design-documents/index.html
index fb2559e..4448a5e 100644
--- a/website/generated-content/contribute/design-documents/index.html
+++ b/website/generated-content/contribute/design-documents/index.html
@@ -142,8 +142,8 @@
<span class="section-nav-list-title">IDE tips</span>
<ul class="section-nav-list">
- <li><a href="/contribute/intellij/">IntelliJ Tips</a></li>
- <li><a href="/contribute/eclipse/">Eclipse Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/IntelliJ+Tips">IntelliJ Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/Eclipse+Tips">Eclipse Tips</a></li>
</ul>
</li>
<li>
diff --git a/website/generated-content/contribute/docker-images/index.html b/website/generated-content/contribute/docker-images/index.html
index b19985f..6311a18 100644
--- a/website/generated-content/contribute/docker-images/index.html
+++ b/website/generated-content/contribute/docker-images/index.html
@@ -142,8 +142,8 @@
<span class="section-nav-list-title">IDE tips</span>
<ul class="section-nav-list">
- <li><a href="/contribute/intellij/">IntelliJ Tips</a></li>
- <li><a href="/contribute/eclipse/">Eclipse Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/IntelliJ+Tips">IntelliJ Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/Eclipse+Tips">Eclipse Tips</a></li>
</ul>
</li>
<li>
diff --git a/website/generated-content/contribute/eclipse/index.html b/website/generated-content/contribute/eclipse/index.html
deleted file mode 100644
index 480e7f2..0000000
--- a/website/generated-content/contribute/eclipse/index.html
+++ /dev/null
@@ -1,387 +0,0 @@
-<!DOCTYPE html>
-<!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<html lang="en">
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <title>Beam Eclipse Tips</title>
- <meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow [...]
-">
- <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
- <link rel="stylesheet" href="/css/site.css">
- <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
- <script src="/js/bootstrap.min.js"></script>
- <script src="/js/language-switch.js"></script>
- <script src="/js/fix-menu.js"></script>
- <script src="/js/section-nav.js"></script>
- <script src="/js/page-nav.js"></script>
- <link rel="canonical" href="https://beam.apache.org/contribute/eclipse/" data-proofer-ignore>
- <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
- <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
- <script>
- (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
- (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
- m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
- })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
- ga('create', 'UA-73650088-1', 'auto');
- ga('send', 'pageview');
- </script>
-</head>
-
- <body class="body" data-spy="scroll" data-target=".page-nav" data-offset="0">
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<nav class="header navbar navbar-fixed-top">
- <div class="navbar-header">
- <button type="button" class="navbar-toggle" aria-expanded="false" aria-controls="navbar">
- <span class="sr-only">Toggle navigation</span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- </button>
-
- <a href="/" class="navbar-brand" >
- <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
- </a>
- </div>
-
- <div class="navbar-mask closed"></div>
-
- <div id="navbar" class="navbar-container closed">
- <ul class="nav navbar-nav">
- <li>
- <a href="/get-started/beam-overview/">Get Started</a>
- </li>
- <li>
- <a href="/documentation/">Documentation</a>
- </li>
- <li>
- <a href="/documentation/sdks/java/">SDKS</a>
- </li>
- <li>
- <a href="/documentation/runners/capability-matrix/">RUNNERS</a>
- </li>
- <li>
- <a href="/contribute/">Contribute</a>
- </li>
- <li>
- <a href="/community/contact-us/">Community</a>
- </li>
- <li><a href="/blog">Blog</a></li>
- </ul>
- <ul class="nav navbar-nav navbar-right">
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:20px;"><span class="caret"></span></a>
- <ul class="dropdown-menu dropdown-menu-right">
- <li><a href="http://www.apache.org/">ASF Homepage</a></li>
- <li><a href="http://www.apache.org/licenses/">License</a></li>
- <li><a href="http://www.apache.org/security/">Security</a></li>
- <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
- <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
- <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
- </ul>
- </li>
- </ul>
- </div>
-</nav>
-
- <div class="clearfix container-main-content">
- <div class="section-nav closed" data-offset-top="90" data-offset-bottom="500">
- <span class="section-nav-back glyphicon glyphicon-menu-left"></span>
- <nav>
- <ul class="section-nav-list" data-section-nav>
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<li><span class="section-nav-list-main-title">Contribute</span></li>
-<li><a href="/contribute/">Get started contributing</a></li>
-<li>
- <span class="section-nav-list-title">IDE tips</span>
-
- <ul class="section-nav-list">
- <li><a href="/contribute/intellij/">IntelliJ Tips</a></li>
- <li><a href="/contribute/eclipse/">Eclipse Tips</a></li>
- </ul>
-</li>
-<li>
- <span class="section-nav-list-title">Technical Docs</span>
-
- <ul class="section-nav-list">
- <li><a href="/contribute/testing/">Testing guide</a></li>
- <ul>
- <li><a href="/contribute/precommit-triage-guide/">Pre-commit Slowness Triage Guide</a></li>
- </ul>
- <li><a href="/contribute/ptransform-style-guide/">PTransform style guide</a></li>
- <li><a href="/contribute/runner-guide/">Runner authoring guide</a></li>
- <li><a href="/contribute/portability/">Portability Framework</a></li>
- <li><a href="/contribute/docker-images/">Docker images</a></li>
- <li><a href="/contribute/design-documents/">Design documents</a></li>
- <li><a href="/contribute/dependencies/">Dependencies guide</a></li>
- </ul>
-</li>
-<li>
- <span class="section-nav-list-title">Policies</span>
- <ul class="section-nav-list">
- <li><a href="/contribute/precommit-policies/">Pre-commit test policies</a></li>
- <li><a href="/contribute/postcommits-policies/">Post-commit test policies</a></li>
- </ul>
-</li>
-<li>
- <span class="section-nav-list-title">Committers</span>
-
- <ul class="section-nav-list">
- <li><a href="/contribute/become-a-committer/">Become a committer</a></li>
- <li><a href="/contribute/committer-guide/">Committer guide</a></li>
- <li><a href="/contribute/release-guide/">Release guide</a></li>
- </ul>
-</li>
-
- </ul>
- </nav>
- </div>
-
- <nav class="page-nav clearfix" data-offset-top="90" data-offset-bottom="500">
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-
-
-<ul class="nav">
- <li><a href="#eclipse-version">Eclipse version</a></li>
- <li><a href="#initial-setup">Initial setup</a></li>
- <li><a href="#building">Building</a></li>
- <li><a href="#checkstyle">Checkstyle</a></li>
- <li><a href="#code-style">Code Style</a></li>
-</ul>
-
-
- </nav>
-
- <div class="body__contained body__section-nav">
- <!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-<h1 id="eclipse-tips">Eclipse Tips</h1>
-
-<blockquote>
- <p>These are best-effort community-contributed tips, and are not guaranteed to work with any particular Eclipse setup.</p>
-</blockquote>
-
-<h2 id="eclipse-version">Eclipse version</h2>
-
-<p>Use a recent Eclipse version that includes
-<a href="https://projects.eclipse.org/projects/tools.buildship">Buildship</a> for Gradle
-integration. Currently we recommend Eclipse Oxygen. Start Eclipse with a fresh
-workspace in a separate directory from your checkout.</p>
-
-<h2 id="initial-setup">Initial setup</h2>
-
-<p>Before setting up Eclipse integration, verify that you can successfully build
-from the commandline by building a sample package:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>./gradlew :beam-examples-java:build
-</code></pre>
-</div>
-
-<p>If you receive any errors, first verify your environment setup:</p>
-
-<ol>
- <li>If running on Mac or Linux, launch Eclipse from a terminal. This is necessary
-to ensure that Eclipse has proper environment setup from user profile
-scripts, i.e. .bashrc.</li>
- <li>Install <a href="https://marketplace.eclipse.org/content/buildship-gradle-integration">Buildship Gradle
-Integration</a>.
-This will allow importing and interacting with the Gradle build.</li>
- <li>Open the project import wizard in Eclipse via “File” > “Import”.</li>
- <li>From the “Import” screen, select “Gradle” > “Existing Gradle Project”, and click
-Next.</li>
- <li>From the “Import Gradle Project” screen, fill in the Project root directory
-with your local git path, and click Finish.</li>
-</ol>
-
-<p>Eclipse will scan the project tree and import each as a separate Package.</p>
-
-<p>Verify that your workspace is correctly configured by invoking
-‘beam-runners-direct-java:build’ from the “Gradle Tasks” pane. The build should
-succeed with no errors.</p>
-
-<h2 id="building">Building</h2>
-
-<p>After your Eclipse workspace is properly setup, you will have a “Gradle
-Tasks” window with a set of operations. If you don’t see the pane, open it
-from “Window” > “Show View” > Other.. > “Gradle” > “Gradle Tasks”.</p>
-
-<p>From the “Gradle Tasks” window, you can build any task registered with Gradle.
-For example, if you are working on Kinesis IO, select ‘beam-sdks-java-io-kinesis:build’.</p>
-
-<h2 id="checkstyle">Checkstyle</h2>
-
-<p>Eclipse supports checkstyle within the IDE using the Checkstyle plugin.</p>
-
-<ol>
- <li>Install the <a href="https://marketplace.eclipse.org/content/checkstyle-plug">Checkstyle
-plugin</a>.</li>
- <li>Configure the Checkstyle plugin by going to Preferences -> Checkstyle.
- <ol>
- <li>Click “New…”</li>
- <li>Select “External Configuration File” for type</li>
- <li>Click “Browse…” and select
-<code class="highlighter-rouge">sdks/java/build-tools/src/main/resources/beam/checkstyle.xml</code></li>
- <li>Enter “Beam Checks” under “Name:”</li>
- <li>Click “OK”, then “OK”</li>
- </ol>
- </li>
-</ol>
-
-<h2 id="code-style">Code Style</h2>
-
-<p>Eclipse supports code styles within the IDE. Use one or both of the following
-to ensure your code style matches the project’s checkstyle enforcement.</p>
-
-<ol>
- <li>The simplest way to have uniform code style is to use the <a href="https://github.com/google/google-java-format#eclipse">Google
-Java Format plugin</a></li>
- <li>You can also configure Eclipse to use <code class="highlighter-rouge">beam-codestyle.xml</code>
- <ol>
- <li>Go to Preferences -> Java -> Code Style -> Formatter</li>
- <li>Click “Import…” and select
-<code class="highlighter-rouge">sdks/java/build-tools/src/main/resources/beam/beam-codestyle.xml</code></li>
- <li>Click “Apply” and “OK”</li>
- </ol>
- </li>
-</ol>
-
-
- </div>
- </div>
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<footer class="footer">
- <div class="footer__contained">
- <div class="footer__cols">
- <div class="footer__cols__col">
- <div class="footer__cols__col__logo">
- <img src="/images/beam_logo_circle.svg" class="footer__logo" alt="Beam logo">
- </div>
- <div class="footer__cols__col__logo">
- <img src="/images/apache_logo_circle.svg" class="footer__logo" alt="Apache logo">
- </div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Start</div>
- <div class="footer__cols__col__link"><a href="/get-started/beam-overview/">Overview</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/quickstart-java/">Quickstart (Java)</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/quickstart-py/">Quickstart (Python)</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/quickstart-go/">Quickstart (Go)</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/downloads/">Downloads</a></div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Docs</div>
- <div class="footer__cols__col__link"><a href="/documentation/programming-guide/">Concepts</a></div>
- <div class="footer__cols__col__link"><a href="/documentation/pipelines/design-your-pipeline/">Pipelines</a></div>
- <div class="footer__cols__col__link"><a href="/documentation/runners/capability-matrix/">Runners</a></div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Community</div>
- <div class="footer__cols__col__link"><a href="/contribute/">Contribute</a></div>
- <div class="footer__cols__col__link"><a href="https://projects.apache.org/committee.html?beam" target="_blank">Team<img src="/images/external-link-icon.png"
- width="14" height="14"
- alt="External link."></a></div>
- <div class="footer__cols__col__link"><a href="/contribute/presentation-materials/">Media</a></div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Resources</div>
- <div class="footer__cols__col__link"><a href="/blog/">Blog</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/support/">Support</a></div>
- <div class="footer__cols__col__link"><a href="https://github.com/apache/beam">GitHub</a></div>
- </div>
- </div>
- </div>
- <div class="footer__bottom">
- ©
- <a href="http://www.apache.org">The Apache Software Foundation</a>
- | <a href="/privacy_policy">Privacy Policy</a>
- | <a href="/feed.xml">RSS Feed</a>
- <br><br>
- Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are
- either registered trademarks or trademarks of The Apache Software
- Foundation. All other products or name brands are trademarks of their
- respective holders, including The Apache Software Foundation.
- </div>
-</footer>
-
- </body>
-</html>
diff --git a/website/generated-content/contribute/index.html b/website/generated-content/contribute/index.html
index 099c100..9c8cb3c 100644
--- a/website/generated-content/contribute/index.html
+++ b/website/generated-content/contribute/index.html
@@ -142,8 +142,8 @@
<span class="section-nav-list-title">IDE tips</span>
<ul class="section-nav-list">
- <li><a href="/contribute/intellij/">IntelliJ Tips</a></li>
- <li><a href="/contribute/eclipse/">Eclipse Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/IntelliJ+Tips">IntelliJ Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/Eclipse+Tips">Eclipse Tips</a></li>
</ul>
</li>
<li>
diff --git a/website/generated-content/contribute/intellij/index.html b/website/generated-content/contribute/intellij/index.html
deleted file mode 100644
index c4de113..0000000
--- a/website/generated-content/contribute/intellij/index.html
+++ /dev/null
@@ -1,389 +0,0 @@
-<!DOCTYPE html>
-<!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<html lang="en">
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <title>Beam IntelliJ Tips</title>
- <meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow [...]
-">
- <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
- <link rel="stylesheet" href="/css/site.css">
- <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
- <script src="/js/bootstrap.min.js"></script>
- <script src="/js/language-switch.js"></script>
- <script src="/js/fix-menu.js"></script>
- <script src="/js/section-nav.js"></script>
- <script src="/js/page-nav.js"></script>
- <link rel="canonical" href="https://beam.apache.org/contribute/intellij/" data-proofer-ignore>
- <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
- <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
- <script>
- (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
- (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
- m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
- })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
- ga('create', 'UA-73650088-1', 'auto');
- ga('send', 'pageview');
- </script>
-</head>
-
- <body class="body" data-spy="scroll" data-target=".page-nav" data-offset="0">
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<nav class="header navbar navbar-fixed-top">
- <div class="navbar-header">
- <button type="button" class="navbar-toggle" aria-expanded="false" aria-controls="navbar">
- <span class="sr-only">Toggle navigation</span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- </button>
-
- <a href="/" class="navbar-brand" >
- <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
- </a>
- </div>
-
- <div class="navbar-mask closed"></div>
-
- <div id="navbar" class="navbar-container closed">
- <ul class="nav navbar-nav">
- <li>
- <a href="/get-started/beam-overview/">Get Started</a>
- </li>
- <li>
- <a href="/documentation/">Documentation</a>
- </li>
- <li>
- <a href="/documentation/sdks/java/">SDKS</a>
- </li>
- <li>
- <a href="/documentation/runners/capability-matrix/">RUNNERS</a>
- </li>
- <li>
- <a href="/contribute/">Contribute</a>
- </li>
- <li>
- <a href="/community/contact-us/">Community</a>
- </li>
- <li><a href="/blog">Blog</a></li>
- </ul>
- <ul class="nav navbar-nav navbar-right">
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:20px;"><span class="caret"></span></a>
- <ul class="dropdown-menu dropdown-menu-right">
- <li><a href="http://www.apache.org/">ASF Homepage</a></li>
- <li><a href="http://www.apache.org/licenses/">License</a></li>
- <li><a href="http://www.apache.org/security/">Security</a></li>
- <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
- <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
- <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
- </ul>
- </li>
- </ul>
- </div>
-</nav>
-
- <div class="clearfix container-main-content">
- <div class="section-nav closed" data-offset-top="90" data-offset-bottom="500">
- <span class="section-nav-back glyphicon glyphicon-menu-left"></span>
- <nav>
- <ul class="section-nav-list" data-section-nav>
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<li><span class="section-nav-list-main-title">Contribute</span></li>
-<li><a href="/contribute/">Get started contributing</a></li>
-<li>
- <span class="section-nav-list-title">IDE tips</span>
-
- <ul class="section-nav-list">
- <li><a href="/contribute/intellij/">IntelliJ Tips</a></li>
- <li><a href="/contribute/eclipse/">Eclipse Tips</a></li>
- </ul>
-</li>
-<li>
- <span class="section-nav-list-title">Technical Docs</span>
-
- <ul class="section-nav-list">
- <li><a href="/contribute/testing/">Testing guide</a></li>
- <ul>
- <li><a href="/contribute/precommit-triage-guide/">Pre-commit Slowness Triage Guide</a></li>
- </ul>
- <li><a href="/contribute/ptransform-style-guide/">PTransform style guide</a></li>
- <li><a href="/contribute/runner-guide/">Runner authoring guide</a></li>
- <li><a href="/contribute/portability/">Portability Framework</a></li>
- <li><a href="/contribute/docker-images/">Docker images</a></li>
- <li><a href="/contribute/design-documents/">Design documents</a></li>
- <li><a href="/contribute/dependencies/">Dependencies guide</a></li>
- </ul>
-</li>
-<li>
- <span class="section-nav-list-title">Policies</span>
- <ul class="section-nav-list">
- <li><a href="/contribute/precommit-policies/">Pre-commit test policies</a></li>
- <li><a href="/contribute/postcommits-policies/">Post-commit test policies</a></li>
- </ul>
-</li>
-<li>
- <span class="section-nav-list-title">Committers</span>
-
- <ul class="section-nav-list">
- <li><a href="/contribute/become-a-committer/">Become a committer</a></li>
- <li><a href="/contribute/committer-guide/">Committer guide</a></li>
- <li><a href="/contribute/release-guide/">Release guide</a></li>
- </ul>
-</li>
-
- </ul>
- </nav>
- </div>
-
- <nav class="page-nav clearfix" data-offset-top="90" data-offset-bottom="500">
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-
-
-<ul class="nav">
- <li><a href="#create-a-working-gradle-module">Create a working Gradle module</a></li>
- <li><a href="#checkstyle">Checkstyle</a></li>
- <li><a href="#code-style">Code Style</a></li>
-</ul>
-
-
- </nav>
-
- <div class="body__contained body__section-nav">
- <!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-<h1 id="intellij-tips">IntelliJ Tips</h1>
-
-<blockquote>
- <p>These are best-effort community-contributed tips, and are not guaranteed to work with any particular IntelliJ setup.</p>
-</blockquote>
-
-<h2 id="create-a-working-gradle-module">Create a working Gradle module</h2>
-
-<p>(as of Intellij 2018.1.6)</p>
-
-<ol>
- <li>Create an empty IntelliJ project outside of the Beam source tree.</li>
- <li>Under Project Structure > Project, select a Project SDK.</li>
- <li>Under Project Structure > Modules, click the + sign to add a module and
-select “Import Module”.
- <ol>
- <li>Select the directory containing the Beam source tree.</li>
- <li>Tick the “Import module from external model” button and select Gradle
-from the list.</li>
- <li>Tick the following boxes.
- <ul>
- <li>Create separate module per source set</li>
- <li>Use default gradle wrapper</li>
- </ul>
- </li>
- </ol>
- </li>
- <li>Delegate build actions to Gradle by going to Preferences/Settings > Build, Execution,
-Deployment > Build Tools > Gradle > Runner and checking “Delegate IDE build/run
-actions to gradle”.</li>
-</ol>
-
-<p>This should result in a working Gradle project. Build the project by executing
-the “build” task in the root Gradle module.</p>
-
-<h2 id="checkstyle">Checkstyle</h2>
-
-<p>IntelliJ supports checkstyle within the IDE using the Checkstyle-IDEA plugin.</p>
-
-<p>Note: Older versions of IntelliJ may not support the Checkstyle file used by Beam.</p>
-
-<ol>
- <li>Install the “Checkstyle-IDEA” plugin from the IntelliJ plugin repository</li>
- <li>Configure the plugin by going to Settings -> Other Settings -> Checkstyle</li>
- <li>Set Checkstyle version to the same as in <code class="highlighter-rouge">/build_rules.gradle</code> (e.g. 8.7)</li>
- <li>Set the “Scan Scope” to “Only Java sources (including tests)”</li>
- <li>In the “Configuration File” pane, add a new configuration using the plus icon:
- <ol>
- <li>Set the “Description” to “Beam”</li>
- <li>Select “Use a local Checkstyle file”, and point it to
- <code class="highlighter-rouge">sdks/java/build-tools/src/main/resources/beam/checkstyle.xml</code> within
- your repository</li>
- <li>Check the box for “Store relative to project location”, and click
- “Next”</li>
- <li>Configure the <code class="highlighter-rouge">checkstyle.suppressions.file</code> property value to
- <code class="highlighter-rouge">suppressions.xml</code>, and click “Next”, then “Finish”</li>
- </ol>
- </li>
- <li>Select “Beam” as the only active configuration file, and click “Apply” and
-“OK”</li>
- <li>Checkstyle will now give warnings in the editor for any Checkstyle
-violations</li>
-</ol>
-
-<p>You can also scan an entire module by opening the Checkstyle tools window and
-clicking the “Check Module” button. The scan should report no errors.</p>
-
-<p>Note: Selecting “Check Project” may report some errors from the archetype
-modules as they are not configured for Checkstyle validation.</p>
-
-<h2 id="code-style">Code Style</h2>
-
-<p>Note: As of release 2.6.0 uniform formatting for Java and Groovy code is automated by the build
-through the <a href="https://github.com/diffplug/spotless/tree/master/plugin-gradle">Spotless Gradle plugin</a>.
-Instead of relying on the IDE, now you can run <code class="highlighter-rouge">./gradlew spotlessApply</code>
-to reformat changes prior to commit.</p>
-
-<p>IntelliJ supports code styles within the IDE. Use one or both of the following
-to ensure your code style matches the project’s checkstyle enforcements.</p>
-
-<ol>
- <li>The simplest way to have uniform code style is to use the
-<a href="https://plugins.jetbrains.com/plugin/8527-google-java-format">Google Java Format
-plugin</a></li>
- <li>You can also configure IntelliJ to use <code class="highlighter-rouge">beam-codestyle.xml</code>
- <ol>
- <li>Go to Settings -> Code Style -> Java</li>
- <li>Click the cogwheel icon next to ‘Scheme’ and select Import Scheme -> Eclipse XML Profile</li>
- <li>Select <code class="highlighter-rouge">sdks/java/build-tools/src/main/resources/beam/beam-codestyle.xml</code></li>
- <li>Click “OK”</li>
- <li>Click “Apply” and “OK”</li>
- </ol>
- </li>
-</ol>
-
- </div>
- </div>
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<footer class="footer">
- <div class="footer__contained">
- <div class="footer__cols">
- <div class="footer__cols__col">
- <div class="footer__cols__col__logo">
- <img src="/images/beam_logo_circle.svg" class="footer__logo" alt="Beam logo">
- </div>
- <div class="footer__cols__col__logo">
- <img src="/images/apache_logo_circle.svg" class="footer__logo" alt="Apache logo">
- </div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Start</div>
- <div class="footer__cols__col__link"><a href="/get-started/beam-overview/">Overview</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/quickstart-java/">Quickstart (Java)</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/quickstart-py/">Quickstart (Python)</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/quickstart-go/">Quickstart (Go)</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/downloads/">Downloads</a></div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Docs</div>
- <div class="footer__cols__col__link"><a href="/documentation/programming-guide/">Concepts</a></div>
- <div class="footer__cols__col__link"><a href="/documentation/pipelines/design-your-pipeline/">Pipelines</a></div>
- <div class="footer__cols__col__link"><a href="/documentation/runners/capability-matrix/">Runners</a></div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Community</div>
- <div class="footer__cols__col__link"><a href="/contribute/">Contribute</a></div>
- <div class="footer__cols__col__link"><a href="https://projects.apache.org/committee.html?beam" target="_blank">Team<img src="/images/external-link-icon.png"
- width="14" height="14"
- alt="External link."></a></div>
- <div class="footer__cols__col__link"><a href="/contribute/presentation-materials/">Media</a></div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Resources</div>
- <div class="footer__cols__col__link"><a href="/blog/">Blog</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/support/">Support</a></div>
- <div class="footer__cols__col__link"><a href="https://github.com/apache/beam">GitHub</a></div>
- </div>
- </div>
- </div>
- <div class="footer__bottom">
- ©
- <a href="http://www.apache.org">The Apache Software Foundation</a>
- | <a href="/privacy_policy">Privacy Policy</a>
- | <a href="/feed.xml">RSS Feed</a>
- <br><br>
- Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are
- either registered trademarks or trademarks of The Apache Software
- Foundation. All other products or name brands are trademarks of their
- respective holders, including The Apache Software Foundation.
- </div>
-</footer>
-
- </body>
-</html>
diff --git a/website/generated-content/contribute/portability/index.html b/website/generated-content/contribute/portability/index.html
index a39d239..767fdd4 100644
--- a/website/generated-content/contribute/portability/index.html
+++ b/website/generated-content/contribute/portability/index.html
@@ -142,8 +142,8 @@
<span class="section-nav-list-title">IDE tips</span>
<ul class="section-nav-list">
- <li><a href="/contribute/intellij/">IntelliJ Tips</a></li>
- <li><a href="/contribute/eclipse/">Eclipse Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/IntelliJ+Tips">IntelliJ Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/Eclipse+Tips">Eclipse Tips</a></li>
</ul>
</li>
<li>
diff --git a/website/generated-content/contribute/postcommits-guides/index.html b/website/generated-content/contribute/postcommits-guides/index.html
index c93c868..50a6a4d 100644
--- a/website/generated-content/contribute/postcommits-guides/index.html
+++ b/website/generated-content/contribute/postcommits-guides/index.html
@@ -142,8 +142,8 @@
<span class="section-nav-list-title">IDE tips</span>
<ul class="section-nav-list">
- <li><a href="/contribute/intellij/">IntelliJ Tips</a></li>
- <li><a href="/contribute/eclipse/">Eclipse Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/IntelliJ+Tips">IntelliJ Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/Eclipse+Tips">Eclipse Tips</a></li>
</ul>
</li>
<li>
diff --git a/website/generated-content/contribute/postcommits-policies-details/index.html b/website/generated-content/contribute/postcommits-policies-details/index.html
index 779bee3..fbcdb82 100644
--- a/website/generated-content/contribute/postcommits-policies-details/index.html
+++ b/website/generated-content/contribute/postcommits-policies-details/index.html
@@ -142,8 +142,8 @@
<span class="section-nav-list-title">IDE tips</span>
<ul class="section-nav-list">
- <li><a href="/contribute/intellij/">IntelliJ Tips</a></li>
- <li><a href="/contribute/eclipse/">Eclipse Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/IntelliJ+Tips">IntelliJ Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/Eclipse+Tips">Eclipse Tips</a></li>
</ul>
</li>
<li>
diff --git a/website/generated-content/contribute/postcommits-policies/index.html b/website/generated-content/contribute/postcommits-policies/index.html
index 275e837..af94c5b 100644
--- a/website/generated-content/contribute/postcommits-policies/index.html
+++ b/website/generated-content/contribute/postcommits-policies/index.html
@@ -142,8 +142,8 @@
<span class="section-nav-list-title">IDE tips</span>
<ul class="section-nav-list">
- <li><a href="/contribute/intellij/">IntelliJ Tips</a></li>
- <li><a href="/contribute/eclipse/">Eclipse Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/IntelliJ+Tips">IntelliJ Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/Eclipse+Tips">Eclipse Tips</a></li>
</ul>
</li>
<li>
diff --git a/website/generated-content/contribute/precommit-policies/index.html b/website/generated-content/contribute/precommit-policies/index.html
index 96b28d3..54b1fd0 100644
--- a/website/generated-content/contribute/precommit-policies/index.html
+++ b/website/generated-content/contribute/precommit-policies/index.html
@@ -142,8 +142,8 @@
<span class="section-nav-list-title">IDE tips</span>
<ul class="section-nav-list">
- <li><a href="/contribute/intellij/">IntelliJ Tips</a></li>
- <li><a href="/contribute/eclipse/">Eclipse Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/IntelliJ+Tips">IntelliJ Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/Eclipse+Tips">Eclipse Tips</a></li>
</ul>
</li>
<li>
diff --git a/website/generated-content/contribute/precommit-triage-guide/index.html b/website/generated-content/contribute/precommit-triage-guide/index.html
index 093a1e8..5407c2c 100644
--- a/website/generated-content/contribute/precommit-triage-guide/index.html
+++ b/website/generated-content/contribute/precommit-triage-guide/index.html
@@ -142,8 +142,8 @@
<span class="section-nav-list-title">IDE tips</span>
<ul class="section-nav-list">
- <li><a href="/contribute/intellij/">IntelliJ Tips</a></li>
- <li><a href="/contribute/eclipse/">Eclipse Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/IntelliJ+Tips">IntelliJ Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/Eclipse+Tips">Eclipse Tips</a></li>
</ul>
</li>
<li>
diff --git a/website/generated-content/contribute/ptransform-style-guide/index.html b/website/generated-content/contribute/ptransform-style-guide/index.html
index f577420..0474d13 100644
--- a/website/generated-content/contribute/ptransform-style-guide/index.html
+++ b/website/generated-content/contribute/ptransform-style-guide/index.html
@@ -142,8 +142,8 @@
<span class="section-nav-list-title">IDE tips</span>
<ul class="section-nav-list">
- <li><a href="/contribute/intellij/">IntelliJ Tips</a></li>
- <li><a href="/contribute/eclipse/">Eclipse Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/IntelliJ+Tips">IntelliJ Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/Eclipse+Tips">Eclipse Tips</a></li>
</ul>
</li>
<li>
diff --git a/website/generated-content/contribute/release-guide/index.html b/website/generated-content/contribute/release-guide/index.html
index 5a53b27..6486976 100644
--- a/website/generated-content/contribute/release-guide/index.html
+++ b/website/generated-content/contribute/release-guide/index.html
@@ -142,8 +142,8 @@
<span class="section-nav-list-title">IDE tips</span>
<ul class="section-nav-list">
- <li><a href="/contribute/intellij/">IntelliJ Tips</a></li>
- <li><a href="/contribute/eclipse/">Eclipse Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/IntelliJ+Tips">IntelliJ Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/Eclipse+Tips">Eclipse Tips</a></li>
</ul>
</li>
<li>
diff --git a/website/generated-content/contribute/runner-guide/index.html b/website/generated-content/contribute/runner-guide/index.html
index aff1de0..03f55e3 100644
--- a/website/generated-content/contribute/runner-guide/index.html
+++ b/website/generated-content/contribute/runner-guide/index.html
@@ -142,8 +142,8 @@
<span class="section-nav-list-title">IDE tips</span>
<ul class="section-nav-list">
- <li><a href="/contribute/intellij/">IntelliJ Tips</a></li>
- <li><a href="/contribute/eclipse/">Eclipse Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/IntelliJ+Tips">IntelliJ Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/Eclipse+Tips">Eclipse Tips</a></li>
</ul>
</li>
<li>
diff --git a/website/generated-content/contribute/testing/index.html b/website/generated-content/contribute/testing/index.html
index 2c27a71..c3761cf 100644
--- a/website/generated-content/contribute/testing/index.html
+++ b/website/generated-content/contribute/testing/index.html
@@ -142,8 +142,8 @@
<span class="section-nav-list-title">IDE tips</span>
<ul class="section-nav-list">
- <li><a href="/contribute/intellij/">IntelliJ Tips</a></li>
- <li><a href="/contribute/eclipse/">Eclipse Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/IntelliJ+Tips">IntelliJ Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/Eclipse+Tips">Eclipse Tips</a></li>
</ul>
</li>
<li>
diff --git a/website/generated-content/contribute/website-contributions/index.html b/website/generated-content/contribute/website-contributions/index.html
index faa184c..6c1a0fa 100644
--- a/website/generated-content/contribute/website-contributions/index.html
+++ b/website/generated-content/contribute/website-contributions/index.html
@@ -142,8 +142,8 @@
<span class="section-nav-list-title">IDE tips</span>
<ul class="section-nav-list">
- <li><a href="/contribute/intellij/">IntelliJ Tips</a></li>
- <li><a href="/contribute/eclipse/">Eclipse Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/IntelliJ+Tips">IntelliJ Tips</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/BEAM/Eclipse+Tips">Eclipse Tips</a></li>
</ul>
</li>
<li>
[beam] 02/11: Publishing website 2018/09/27 21:33:04 at commit
24de10e
Posted by sc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 3baba93ad531097344c467e400cdeb0637c2852f
Author: jenkins <bu...@apache.org>
AuthorDate: Thu Sep 27 21:33:05 2018 +0000
Publishing website 2018/09/27 21:33:04 at commit 24de10e
---
website/generated-content/contribute/index.html | 2 +-
.../generated-content/documentation/programming-guide/index.html | 2 +-
website/generated-content/get-started/quickstart-go/index.html | 3 +++
.../generated-content/get-started/wordcount-example/index.html | 9 +++++++++
4 files changed, 14 insertions(+), 2 deletions(-)
diff --git a/website/generated-content/contribute/index.html b/website/generated-content/contribute/index.html
index f4a3af3..099c100 100644
--- a/website/generated-content/contribute/index.html
+++ b/website/generated-content/contribute/index.html
@@ -618,7 +618,7 @@ When submitting a new PR, please tag <a href="https://github.com/robbesneyders">
<p>Work to support the next LTS release of Java is in progress. For more details about the scope and info on the various tasks please see the JIRA ticket.</p>
<ul>
- <li>JIRA: <a href="https://issues.apache.org/jira/issues/BEAM-2530">BEAM-2530</a></li>
+ <li>JIRA: <a href="https://issues.apache.org/jira/browse/BEAM-2530">BEAM-2530</a></li>
<li>Contact: <a href="mailto:iemejia@gmail.com">Ismaël Mejía</a></li>
</ul>
diff --git a/website/generated-content/documentation/programming-guide/index.html b/website/generated-content/documentation/programming-guide/index.html
index 2f19f9c..4836731 100644
--- a/website/generated-content/documentation/programming-guide/index.html
+++ b/website/generated-content/documentation/programming-guide/index.html
@@ -2902,7 +2902,7 @@ least 10 minutes:</p>
</div>
<div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">apache_beam</span> <span class="kn">import</span> <span class="n">window</span>
<span class="n">session_windowed_items</span> <span class="o">=</span> <span class="p">(</span>
- <span class="n">items</span> <span class="o">|</span> <span class="s">'window'</span> <span class="o">>></span> <span class="n">beam</span><span class="o">.</span><span class="n">WindowInto</span><span class="p">(</span><span class="n">window</span><span class="o">.</span><span class="n">Sessions</span><span class="p">(</span><span class="mi">10</span><span class="p">)))</span>
+ <span class="n">items</span> <span class="o">|</span> <span class="s">'window'</span> <span class="o">>></span> <span class="n">beam</span><span class="o">.</span><span class="n">WindowInto</span><span class="p">(</span><span class="n">window</span><span class="o">.</span><span class="n">Sessions</span><span class="p">(</span><span class="mi">10</span> <span class="o">*</span> <span class="mi">60</span><span class="p">)))</span>
</code></pre>
</div>
diff --git a/website/generated-content/get-started/quickstart-go/index.html b/website/generated-content/get-started/quickstart-go/index.html
index 8a2435a..dbfae68 100644
--- a/website/generated-content/get-started/quickstart-go/index.html
+++ b/website/generated-content/get-started/quickstart-go/index.html
@@ -245,11 +245,14 @@ $ wordcount --input <PATH_TO_INPUT_FILE> --output counts
</div>
<div class="runner-dataflow highlighter-rouge"><pre class="highlight"><code>$ go install github.com/apache/beam/sdks/go/examples/wordcount
+# As part of the initial setup, for non linux users - install package unix before run
+$ go get -u golang.org/x/sys/unix
$ wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \
--output gs://<your-gcs-bucket>/counts \
--runner dataflow \
--project your-gcp-project \
--temp_location gs://<your-gcs-bucket>/tmp/ \
+ --staging_location gs://<your-gcs-bucket>/binaries/ \
--worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515
</code></pre>
</div>
diff --git a/website/generated-content/get-started/wordcount-example/index.html b/website/generated-content/get-started/wordcount-example/index.html
index 0b8d502..7878541 100644
--- a/website/generated-content/get-started/wordcount-example/index.html
+++ b/website/generated-content/get-started/wordcount-example/index.html
@@ -689,11 +689,14 @@ $ wordcount --input <PATH_TO_INPUT_FILE> --output counts
</div>
<div class="runner-dataflow highlighter-rouge"><pre class="highlight"><code>$ go install github.com/apache/beam/sdks/go/examples/wordcount
+# As part of the initial setup, for non linux users - install package unix before run
+$ go get -u golang.org/x/sys/unix
$ wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \
--output gs://<your-gcs-bucket>/counts \
--runner dataflow \
--project your-gcp-project \
--temp_location gs://<your-gcs-bucket>/tmp/ \
+ --staging_location gs://<your-gcs-bucket>/binaries/ \
--worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515
</code></pre>
</div>
@@ -1009,11 +1012,14 @@ $ debugging_wordcount --input <PATH_TO_INPUT_FILE> --output counts
</div>
<div class="runner-dataflow highlighter-rouge"><pre class="highlight"><code>$ go install github.com/apache/beam/sdks/go/examples/debugging_wordcount
+# As part of the initial setup, for non linux users - install package unix before run
+$ go get -u golang.org/x/sys/unix
$ debugging_wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \
--output gs://<your-gcs-bucket>/counts \
--runner dataflow \
--project your-gcp-project \
--temp_location gs://<your-gcs-bucket>/tmp/ \
+ --staging_location gs://<your-gcs-bucket>/binaries/ \
--worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515
</code></pre>
</div>
@@ -1343,11 +1349,14 @@ $ windowed_wordcount --input <PATH_TO_INPUT_FILE> --output counts
</div>
<div class="runner-dataflow highlighter-rouge"><pre class="highlight"><code>$ go install github.com/apache/beam/sdks/go/examples/windowed_wordcount
+# As part of the initial setup, for non linux users - install package unix before run
+$ go get -u golang.org/x/sys/unix
$ windowed_wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \
--output gs://<your-gcs-bucket>/counts \
--runner dataflow \
--project your-gcp-project \
--temp_location gs://<your-gcs-bucket>/tmp/ \
+ --staging_location gs://<your-gcs-bucket>/binaries/ \
--worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515
</code></pre>
</div>
[beam] 08/11: Publishing website 2018/10/05 22:39:21 at commit
a19183b
Posted by sc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 444a308b8d368eb1e30b4d6031df6797a22f75dc
Author: jenkins <bu...@apache.org>
AuthorDate: Fri Oct 5 22:39:21 2018 +0000
Publishing website 2018/10/05 22:39:21 at commit a19183b
---
website/generated-content/.htaccess | 2 +-
website/generated-content/contribute/index.html | 10 +-
.../contribute/release-guide/index.html | 92 +++--
.../contribute/website-contributions/index.html | 369 ---------------------
4 files changed, 65 insertions(+), 408 deletions(-)
diff --git a/website/generated-content/.htaccess b/website/generated-content/.htaccess
index b36da9e..82e9c99 100644
--- a/website/generated-content/.htaccess
+++ b/website/generated-content/.htaccess
@@ -19,6 +19,6 @@ RewriteRule ^(.*)$ https://beam.apache.org/$1 [L,R=301]
# They were previously hosted within this repository, and published at the URL
# path /documentation/sdks/(javadoc|pydoc)/..
# The following redirect maintains the previously supported URLs.
-RedirectMatch permanent "/documentation/sdks/(javadoc|pydoc)(.*)" "https://beam.apache.org/documentation/releases/$1$2"
+RedirectMatch permanent "/documentation/sdks/(javadoc|pydoc)(.*)" "https://beam.apache.org/releases/$1$2"
# Keep this updated to point to the current release.
RedirectMatch "/releases/([^/]+)/current(.*)" "https://beam.apache.org/releases/$1/2.6.0$2"
diff --git a/website/generated-content/contribute/index.html b/website/generated-content/contribute/index.html
index 9c8cb3c..094b4c0 100644
--- a/website/generated-content/contribute/index.html
+++ b/website/generated-content/contribute/index.html
@@ -530,12 +530,10 @@ a read-only collaborator to apache/beam by opening an INFRA ticket. -->
<h2 id="contributing-to-the-website">Contributing to the website</h2>
-<p>The Beam website is in the <a href="https://github.com/apache/beam-site">Beam Site GitHub
-mirror</a> repository in the <code class="highlighter-rouge">asf-site</code>
-branch (<em>not</em> <code class="highlighter-rouge">master</code>). The
-<a href="https://github.com/apache/beam-site/blob/asf-site/README.md">README</a> there
-explains how to modify different parts of the site. The GitHub workflow is the
-same - make your change and open a pull request.</p>
+<p>The Beam website is in the <code class="highlighter-rouge">/website</code> directory of the repo. The
+<a href="https://github.com/apache/beam/blob/master/website">README</a> there explains how
+to modify different parts of the site. The GitHub workflow is the same - make
+your change and open a pull request.</p>
<p>Issues are tracked in the
<a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20component%20%3D%20website">website</a>
diff --git a/website/generated-content/contribute/release-guide/index.html b/website/generated-content/contribute/release-guide/index.html
index 6486976..965fdb4 100644
--- a/website/generated-content/contribute/release-guide/index.html
+++ b/website/generated-content/contribute/release-guide/index.html
@@ -327,10 +327,7 @@ limitations under the License.
<li><a href="#stage-source-release-on-distapacheorg" id="markdown-toc-stage-source-release-on-distapacheorg">Stage source release on dist.apache.org</a></li>
<li><a href="#stage-python-binaries-on-distapacheorg" id="markdown-toc-stage-python-binaries-on-distapacheorg">Stage python binaries on dist.apache.org</a></li>
<li><a href="#build-the-pydoc-api-reference" id="markdown-toc-build-the-pydoc-api-reference">Build the Pydoc API reference</a></li>
- <li><a href="#propose-a-pull-request-for-website-updates" id="markdown-toc-propose-a-pull-request-for-website-updates">Propose a pull request for website updates</a> <ul>
- <li><a href="#create-pydoc" id="markdown-toc-create-pydoc">Create Pydoc</a></li>
- </ul>
- </li>
+ <li><a href="#propose-pull-requests-for-website-updates" id="markdown-toc-propose-pull-requests-for-website-updates">Propose pull requests for website updates</a></li>
<li><a href="#build-and-stage-python-wheels" id="markdown-toc-build-and-stage-python-wheels">Build and stage python wheels</a></li>
<li><a href="#checklist-to-proceed-to-the-next-step-2" id="markdown-toc-checklist-to-proceed-to-the-next-step-2">Checklist to proceed to the next step</a></li>
</ul>
@@ -556,8 +553,26 @@ please submit your GPG public key into <a href="http://pgp.mit.edu:11371/">MIT P
<h4 id="website-development-setup">Website development setup</h4>
-<p>Prepare to update the Beam website by following the <a href="/contribute/website-contributions/">website development
-instructions</a>.</p>
+<p>Updating the Beam website requires submitting PRs to both the main <code class="highlighter-rouge">apache/beam</code>
+repo and the <code class="highlighter-rouge">apache/beam-site</code> repo. The first contains reference manuals
+generated from SDK code, while the second updates the current release version
+number.</p>
+
+<p>You should already have setup a local clone of <code class="highlighter-rouge">apache/beam</code>. Setting up a clone
+of <code class="highlighter-rouge">apache/beam-site</code> is similar:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>$ git clone -b release-docs https://github.com/apache/beam-site.git
+$ cd beam-site
+$ git remote add <GitHub_user> git@github.com:<GitHub_user>/beam-site.git
+$ git fetch --all
+$ git checkout -b <my-branch> origin/release-docs
+</code></pre>
+</div>
+
+<p>Further instructions on website development on <code class="highlighter-rouge">apache/beam</code> is
+<a href="https://github.com/apache/beam/blob/master/website">here</a>. Background
+information about how the website is updated can be found in <a href="https://s.apache.org/beam-site-automation">Beam-Site
+Automation Reliability</a>.</p>
<h4 id="register-to-pypi">Register to PyPI</h4>
@@ -932,20 +947,20 @@ When prompted for a description, enter “Apache Beam, version X, release candid
</li>
<li>Stage source release into dist.apache.org dev <a href="https://dist.apache.org/repos/dist/dev/beam/">repo</a>.</li>
<li>Stage,sign and hash python binaries into dist.apache.ord dev repo python dir</li>
- <li>Create a PR to update beam-site, changes includes:
+ <li>Create a PR to update beam and beam-site, changes includes:
<ul>
<li>Copy python doc into beam-site</li>
<li>Copy java doc into beam-site</li>
- <li>Update release version into <a href="https://github.com/apache/beam-site/blob/asf-site/_config.yml">_config.yml</a>.</li>
+ <li>Update release version into <a href="https://github.com/apache/beam/blob/master/website/_config.yml">_config.yml</a>.</li>
</ul>
</li>
</ol>
</li>
<li>Tasks you need to do manually
<ol>
- <li>Add new release into src/get-started/downloads.md</li>
- <li>Update last release download links in src/get-started/downloads.md</li>
- <li>Update the Pydoc link on this page to point to the new version (in src/documentation/sdks/pydoc/current.md.</li>
+ <li>Add new release into <code class="highlighter-rouge">website/src/get-started/downloads.md</code>.</li>
+ <li>Update last release download links in <code class="highlighter-rouge">website/src/get-started/downloads.md</code>.</li>
+ <li>Update <code class="highlighter-rouge">website/src/.htaccess</code> to redirect to the new version.</li>
</ol>
</li>
</ul>
@@ -1068,7 +1083,6 @@ svn commit
<p>Make sure you have <code class="highlighter-rouge">tox</code> installed:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>pip install tox
-
</code></pre>
</div>
<p>Create the Python SDK documentation using sphinx by running a helper script.</p>
@@ -1077,38 +1091,52 @@ svn commit
</div>
<p>By default the Pydoc is generated in <code class="highlighter-rouge">sdks/python/target/docs/_build</code>. Let <code class="highlighter-rouge">${PYDOC_ROOT}</code> be the absolute path to <code class="highlighter-rouge">_build</code>.</p>
-<h4 id="propose-a-pull-request-for-website-updates">Propose a pull request for website updates</h4>
-
-<p>The final step of building the candidate is to propose a website pull request.</p>
+<h4 id="propose-pull-requests-for-website-updates">Propose pull requests for website updates</h4>
-<p>Start by updating <code class="highlighter-rouge">release_latest</code> version flag in the top-level <code class="highlighter-rouge">_config.yml</code>, and list the new release in the <a href="/get-started/downloads/">Apache Beam Downloads</a>, linking to the source code download and the Release Notes in JIRA.</p>
+<p>Beam publishes API reference manuals for each release on the website. For Java
+and Python SDKs, that’s Javadoc and PyDoc, respectively. The final step of
+building the candidate is to propose website pull requests that update these
+manuals.</p>
-<p>Beam publishes API reference manual for each release on the website. For Java SDK, that’s Javadoc.</p>
+<p>Merge the pull requests only after finalizing the release. To avoid invalid
+redirects for the ‘current’ version, merge these PRs in the order listed. Once
+the PR is merged, the new contents will get picked up automatically and served
+to the Beam website, usually within an hour.</p>
-<p>One of the artifacts created in the release contains the Javadoc for the
-website. To update the website, you must unpack this jar file from the release
-candidate into the source tree of the website.</p>
+<p><strong>PR 1: apache/beam-site</strong></p>
-<p>Add the new Javadoc to <a href="https://beam.apache.org/releases/javadoc/">SDK API Reference page</a> page, as follows:</p>
+<p>This pull request is against the <code class="highlighter-rouge">apache/beam-site</code> repo, on the <code class="highlighter-rouge">release-docs</code>
+branch.</p>
<ul>
- <li>Unpack the Maven artifact <code class="highlighter-rouge">org.apache.beam:beam-sdks-java-javadoc</code> into some temporary location. Call this <code class="highlighter-rouge">${JAVADOC_TMP}</code>.</li>
- <li>Copy the generated Javadoc into the website repository: <code class="highlighter-rouge">cp -r ${JAVADOC_TMP} src/documentation/sdks/javadoc/${RELEASE}</code>.</li>
- <li>Set up the necessary git commands to account for the new and deleted files from the javadoc.</li>
- <li>Update the Javadoc link on this page to point to the new version (in <code class="highlighter-rouge">src/documentation/sdks/javadoc/current.md</code>).</li>
+ <li>Add the new Javadoc to <a href="https://beam.apache.org/releases/javadoc/">SDK API Reference page</a> page, as follows:
+ <ul>
+ <li>Unpack the Maven artifact <code class="highlighter-rouge">org.apache.beam:beam-sdks-java-javadoc</code> into some temporary location. Call this <code class="highlighter-rouge">${JAVADOC_TMP}</code>.</li>
+ <li>Copy the generated Javadoc into the website repository: <code class="highlighter-rouge">cp -r ${JAVADOC_TMP} javadoc/${RELEASE}</code>.</li>
+ </ul>
+ </li>
+ <li>Add the new Pydoc to <a href="https://beam.apache.org/releases/pydoc/">SDK API Reference page</a> page, as follows:
+ <ul>
+ <li>Copy the generated Pydoc into the website repository: <code class="highlighter-rouge">cp -r ${PYDOC_ROOT} pydoc/${RELEASE}</code>.</li>
+ <li>Remove <code class="highlighter-rouge">.doctrees</code> directory.</li>
+ </ul>
+ </li>
+ <li>Stage files using: <code class="highlighter-rouge">git add --all javadoc/ pydoc/</code>.</li>
</ul>
-<h5 id="create-pydoc">Create Pydoc</h5>
-<p>Add the new Pydoc to <a href="https://beam.apache.org/releases/pydoc/">SDK API Reference page</a> page, as follows:</p>
+<p><strong>PR 2: apache/beam</strong></p>
+
+<p>This pull request is against the <code class="highlighter-rouge">apache/beam</code> repo, on the <code class="highlighter-rouge">master</code> branch.</p>
<ul>
- <li>Copy the generated Pydoc into the website repository: <code class="highlighter-rouge">cp -r ${PYDOC_ROOT} src/documentation/sdks/pydoc/${RELEASE}</code>.</li>
- <li>Remove <code class="highlighter-rouge">.doctrees</code> directory.</li>
- <li>Update the Pydoc link on this page to point to the new version (in <code class="highlighter-rouge">src/documentation/sdks/pydoc/current.md</code>).</li>
+ <li>Update the <code class="highlighter-rouge">release_latest</code> version flag in <code class="highlighter-rouge">/website/_config.yml</code>, and list
+the new release in <code class="highlighter-rouge">/website/src/get-started/downloads.md</code>, linking to the
+source code download and the Release Notes in JIRA.</li>
+ <li>Update the <code class="highlighter-rouge">RedirectMatch</code> rule in
+<a href="https://github.com/apache/beam/blob/master/website/src/.htaccess">/website/src/.htaccess</a>
+to point to the new release. See file history for examples.</li>
</ul>
-<p>Finally, propose a pull request with these changes. (Don’t merge before finalizing the release.)</p>
-
<h4 id="build-and-stage-python-wheels">Build and stage python wheels</h4>
<p>There is a wrapper repo <a href="https://github.com/apache/beam-wheels">beam-wheels</a> to help build python wheels.</p>
diff --git a/website/generated-content/contribute/website-contributions/index.html b/website/generated-content/contribute/website-contributions/index.html
deleted file mode 100644
index 6c1a0fa..0000000
--- a/website/generated-content/contribute/website-contributions/index.html
+++ /dev/null
@@ -1,369 +0,0 @@
-<!DOCTYPE html>
-<!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<html lang="en">
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <title>Beam Website Contributions</title>
- <meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow [...]
-">
- <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
- <link rel="stylesheet" href="/css/site.css">
- <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
- <script src="/js/bootstrap.min.js"></script>
- <script src="/js/language-switch.js"></script>
- <script src="/js/fix-menu.js"></script>
- <script src="/js/section-nav.js"></script>
- <script src="/js/page-nav.js"></script>
- <link rel="canonical" href="https://beam.apache.org/contribute/website-contributions/" data-proofer-ignore>
- <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
- <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
- <script>
- (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
- (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
- m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
- })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
- ga('create', 'UA-73650088-1', 'auto');
- ga('send', 'pageview');
- </script>
-</head>
-
- <body class="body" data-spy="scroll" data-target=".page-nav" data-offset="0">
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<nav class="header navbar navbar-fixed-top">
- <div class="navbar-header">
- <button type="button" class="navbar-toggle" aria-expanded="false" aria-controls="navbar">
- <span class="sr-only">Toggle navigation</span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- </button>
-
- <a href="/" class="navbar-brand" >
- <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
- </a>
- </div>
-
- <div class="navbar-mask closed"></div>
-
- <div id="navbar" class="navbar-container closed">
- <ul class="nav navbar-nav">
- <li>
- <a href="/get-started/beam-overview/">Get Started</a>
- </li>
- <li>
- <a href="/documentation/">Documentation</a>
- </li>
- <li>
- <a href="/documentation/sdks/java/">SDKS</a>
- </li>
- <li>
- <a href="/documentation/runners/capability-matrix/">RUNNERS</a>
- </li>
- <li>
- <a href="/contribute/">Contribute</a>
- </li>
- <li>
- <a href="/community/contact-us/">Community</a>
- </li>
- <li><a href="/blog">Blog</a></li>
- </ul>
- <ul class="nav navbar-nav navbar-right">
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:20px;"><span class="caret"></span></a>
- <ul class="dropdown-menu dropdown-menu-right">
- <li><a href="http://www.apache.org/">ASF Homepage</a></li>
- <li><a href="http://www.apache.org/licenses/">License</a></li>
- <li><a href="http://www.apache.org/security/">Security</a></li>
- <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
- <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
- <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
- </ul>
- </li>
- </ul>
- </div>
-</nav>
-
- <div class="clearfix container-main-content">
- <div class="section-nav closed" data-offset-top="90" data-offset-bottom="500">
- <span class="section-nav-back glyphicon glyphicon-menu-left"></span>
- <nav>
- <ul class="section-nav-list" data-section-nav>
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<li><span class="section-nav-list-main-title">Contribute</span></li>
-<li><a href="/contribute/">Get started contributing</a></li>
-<li>
- <span class="section-nav-list-title">IDE tips</span>
-
- <ul class="section-nav-list">
- <li><a href="https://cwiki.apache.org/confluence/display/BEAM/IntelliJ+Tips">IntelliJ Tips</a></li>
- <li><a href="https://cwiki.apache.org/confluence/display/BEAM/Eclipse+Tips">Eclipse Tips</a></li>
- </ul>
-</li>
-<li>
- <span class="section-nav-list-title">Technical Docs</span>
-
- <ul class="section-nav-list">
- <li><a href="/contribute/testing/">Testing guide</a></li>
- <ul>
- <li><a href="/contribute/precommit-triage-guide/">Pre-commit Slowness Triage Guide</a></li>
- </ul>
- <li><a href="/contribute/ptransform-style-guide/">PTransform style guide</a></li>
- <li><a href="/contribute/runner-guide/">Runner authoring guide</a></li>
- <li><a href="/contribute/portability/">Portability Framework</a></li>
- <li><a href="/contribute/docker-images/">Docker images</a></li>
- <li><a href="/contribute/design-documents/">Design documents</a></li>
- <li><a href="/contribute/dependencies/">Dependencies guide</a></li>
- </ul>
-</li>
-<li>
- <span class="section-nav-list-title">Policies</span>
- <ul class="section-nav-list">
- <li><a href="/contribute/precommit-policies/">Pre-commit test policies</a></li>
- <li><a href="/contribute/postcommits-policies/">Post-commit test policies</a></li>
- </ul>
-</li>
-<li>
- <span class="section-nav-list-title">Committers</span>
-
- <ul class="section-nav-list">
- <li><a href="/contribute/become-a-committer/">Become a committer</a></li>
- <li><a href="/contribute/committer-guide/">Committer guide</a></li>
- <li><a href="/contribute/release-guide/">Release guide</a></li>
- </ul>
-</li>
-
- </ul>
- </nav>
- </div>
-
- <nav class="page-nav clearfix" data-offset-top="90" data-offset-bottom="500">
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-
-
-<ul class="nav">
- <li><a href="#one-time-setup">One-time Setup</a></li>
- <li><a href="#working-on-your-change">Working on your change</a></li>
- <li><a href="#committing-website-changes-committers-only">Committing website changes (committers only)</a></li>
-</ul>
-
-
- </nav>
-
- <div class="body__contained body__section-nav">
- <!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-<h1 id="website-contributions">Website Contributions</h1>
-
-<p>Contributions to the website require the same steps as contributions to the
-main repository, so start with the instructions for <a href="/contribute/contribution-guide">Contributing
-Code</a>.</p>
-
-<p>The Beam website is in the <a href="https://github.com/apache/beam-site">Beam Site GitHub
-mirror</a> repository in the <code class="highlighter-rouge">asf-site</code>
-branch (<em>not</em> <code class="highlighter-rouge">master</code>).</p>
-
-<p>Issues are tracked in the
-<a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20component%20%3D%20website">website</a>
-component in JIRA.</p>
-
-<h2 id="one-time-setup">One-time Setup</h2>
-
-<p>The <a href="https://github.com/apache/beam-site/blob/asf-site/README.md">README file</a>
-in the website repository has more information on how to set up the required
-dependencies for your development environment.</p>
-
-<p>The general guidelines for cloning a repository can be adjusted to use the
-<code class="highlighter-rouge">asf-site</code> branch of <code class="highlighter-rouge">beam-site</code>:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ git clone -b asf-site https://github.com/apache/beam-site.git
-$ cd beam-site
-$ git remote add <GitHub_user> git@github.com:<GitHub_user>/beam-site.git
-$ git fetch --all
-$ git checkout -b <my-branch> origin/asf-site
-</code></pre>
-</div>
-
-<h2 id="working-on-your-change">Working on your change</h2>
-
-<p>While you are working on your pull request, you can test and develop live by
-running the following command in the root folder of the website:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ bundle exec jekyll serve --incremental
-</code></pre>
-</div>
-
-<p>Jekyll will start a webserver on port 4000. As you make changes to the content,
-Jekyll will rebuild it automatically.</p>
-
-<p>In addition, you can run the tests to valid your links using:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ bundle exec rake test
-</code></pre>
-</div>
-
-<p>Both of these commands will cause the <code class="highlighter-rouge">content/</code> directory to be generated.
-Merging autogenerated content can get tricky, so please leave this directory
-out of your commits and pull request by doing:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ git checkout -- content
-</code></pre>
-</div>
-
-<p>When you are ready, submit a pull request using the <a href="https://github.com/apache/beam-site">Beam Site GitHub
-mirror</a>, including the JIRA issue as
-usual.</p>
-
-<p>During review, committers will patch in your PR, generate the static
-<code class="highlighter-rouge">content/</code>, and review the changes.</p>
-
-<h2 id="committing-website-changes-committers-only">Committing website changes (committers only)</h2>
-
-<p>We have a bot that runs tests and merges changes to the website.
-When the pull request is fully ready to be merged, a committer
-can comment</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>@asfgit merge
-</code></pre>
-</div>
-
-
- </div>
- </div>
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<footer class="footer">
- <div class="footer__contained">
- <div class="footer__cols">
- <div class="footer__cols__col">
- <div class="footer__cols__col__logo">
- <img src="/images/beam_logo_circle.svg" class="footer__logo" alt="Beam logo">
- </div>
- <div class="footer__cols__col__logo">
- <img src="/images/apache_logo_circle.svg" class="footer__logo" alt="Apache logo">
- </div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Start</div>
- <div class="footer__cols__col__link"><a href="/get-started/beam-overview/">Overview</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/quickstart-java/">Quickstart (Java)</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/quickstart-py/">Quickstart (Python)</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/quickstart-go/">Quickstart (Go)</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/downloads/">Downloads</a></div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Docs</div>
- <div class="footer__cols__col__link"><a href="/documentation/programming-guide/">Concepts</a></div>
- <div class="footer__cols__col__link"><a href="/documentation/pipelines/design-your-pipeline/">Pipelines</a></div>
- <div class="footer__cols__col__link"><a href="/documentation/runners/capability-matrix/">Runners</a></div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Community</div>
- <div class="footer__cols__col__link"><a href="/contribute/">Contribute</a></div>
- <div class="footer__cols__col__link"><a href="https://projects.apache.org/committee.html?beam" target="_blank">Team<img src="/images/external-link-icon.png"
- width="14" height="14"
- alt="External link."></a></div>
- <div class="footer__cols__col__link"><a href="/contribute/presentation-materials/">Media</a></div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Resources</div>
- <div class="footer__cols__col__link"><a href="/blog/">Blog</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/support/">Support</a></div>
- <div class="footer__cols__col__link"><a href="https://github.com/apache/beam">GitHub</a></div>
- </div>
- </div>
- </div>
- <div class="footer__bottom">
- ©
- <a href="http://www.apache.org">The Apache Software Foundation</a>
- | <a href="/privacy_policy">Privacy Policy</a>
- | <a href="/feed.xml">RSS Feed</a>
- <br><br>
- Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are
- either registered trademarks or trademarks of The Apache Software
- Foundation. All other products or name brands are trademarks of their
- respective holders, including The Apache Software Foundation.
- </div>
-</footer>
-
- </body>
-</html>
[beam] 01/11: Publishing website 2018/09/26 14:37:37 at commit
da3fc69
Posted by sc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 774749febcea69ccc2ffb8471619a26522f3c702
Author: jenkins <bu...@apache.org>
AuthorDate: Wed Sep 26 14:37:37 2018 +0000
Publishing website 2018/09/26 14:37:37 at commit da3fc69
---
website/generated-content/.htaccess | 15 +
.../capability/2016/03/17/capability-matrix.html | 2447 +++++
.../2016/04/03/presentation-materials.html | 255 +
.../sdk/2016/02/25/python-sdk-now-public.html | 246 +
.../beam/release/2016/06/15/first-release.html | 263 +
.../2016/10/11/strata-hadoop-world-and-beam.html | 256 +
.../update/website/2016/02/22/beam-has-a-logo.html | 252 +
.../blog/2016/05/18/splitAtFraction-method.html | 241 +
.../05/27/where-is-my-pcollection-dot-map.html | 316 +
.../2016/06/13/flink-batch-runner-milestone.html | 256 +
.../blog/2016/08/03/six-months.html | 273 +
.../blog/2016/10/20/test-stream.html | 535 +
.../blog/2017/01/09/added-apex-runner.html | 263 +
.../blog/2017/01/10/beam-graduates.html | 289 +
.../blog/2017/02/01/graduation-media-recap.html | 280 +
.../blog/2017/02/13/stateful-processing.html | 803 ++
.../blog/2017/03/16/python-sdk-release.html | 296 +
.../blog/2017/05/17/beam-first-stable-release.html | 335 +
.../blog/2017/08/16/splittable-do-fn.html | 779 ++
.../blog/2017/08/28/timely-processing.html | 736 ++
.../blog/2018/01/09/beam-a-look-back.html | 362 +
.../blog/2018/02/19/beam-2.3.0.html | 319 +
.../blog/2018/06/26/beam-2.5.0.html | 323 +
.../blog/2018/08/10/beam-2.6.0.html | 308 +
.../08/20/review-input-streaming-connectors.html | 450 +
.../blog/2018/08/21/beam-summit-europe.html | 266 +
website/generated-content/blog/index.html | 1048 ++
.../generated-content/capability-matrix/index.html | 10 +
website/generated-content/coming-soon.html | 211 +
.../community/contact-us/index.html | 305 +
.../generated-content/community/logos/index.html | 577 +
.../community/policies/index.html | 280 +
.../community/presentation-materials/index.html | 293 +
.../community/youtube-channel/index.html | 303 +
.../contribute/become-a-committer/index.html | 393 +
.../contribute/committer-guide/index.html | 383 +
.../contribute/contribution-guide/index.html | 10 +
.../contribute/dependencies/index.html | 374 +
.../contribute/design-documents/index.html | 455 +
.../contribute/design-principles/index.html | 10 +
.../contribute/docker-images/index.html | 482 +
.../contribute/eclipse/index.html | 387 +
.../contribute/feature-branches/index.html | 10 +
website/generated-content/contribute/index.html | 763 ++
.../contribute/intellij/index.html | 389 +
.../generated-content/contribute/logos/index.html | 10 +
.../contribute/portability/index.html | 516 +
.../contribute/postcommits-guides/index.html | 360 +
.../postcommits-policies-details/index.html | 367 +
.../contribute/postcommits-policies/index.html | 376 +
.../contribute/precommit-policies/index.html | 359 +
.../contribute/precommit-triage-guide/index.html | 436 +
.../contribute/presentation-materials/index.html | 10 +
.../contribute/ptransform-style-guide/index.html | 965 ++
.../contribute/release-guide/index.html | 1695 +++
.../contribute/runner-guide/index.html | 1513 +++
.../contribute/source-repository/index.html | 10 +
.../generated-content/contribute/team/index.html | 10 +
.../contribute/testing/index.html | 736 ++
.../contribute/website-contributions/index.html | 369 +
.../contribute/work-in-progress/index.html | 10 +
.../contribution-guide/index.html | 10 +
website/generated-content/css/site.css | 6967 ++++++++++++
.../generated-content/docs/contribute/index.html | 10 +
website/generated-content/docs/learn/index.html | 10 +
.../docs/learn/programming-guide/index.html | 10 +
.../docs/use/beam-overview/index.html | 10 +
website/generated-content/docs/use/index.html | 10 +
.../dsls/sql/aggregate-functions/index.html | 350 +
.../documentation/dsls/sql/create-table/index.html | 737 ++
.../documentation/dsls/sql/data-types/index.html | 396 +
.../documentation/dsls/sql/joins/index.html | 385 +
.../documentation/dsls/sql/lexical/index.html | 1364 +++
.../documentation/dsls/sql/overview/index.html | 338 +
.../dsls/sql/scalar-functions/index.html | 687 ++
.../documentation/dsls/sql/select/index.html | 385 +
.../documentation/dsls/sql/set/index.html | 359 +
.../documentation/dsls/sql/shell/index.html | 464 +
.../dsls/sql/statements/create-table/index.html | 10 +
.../dsls/sql/statements/select/index.html | 10 +
.../dsls/sql/user-defined-functions/index.html | 422 +
.../documentation/dsls/sql/walkthrough/index.html | 464 +
.../dsls/sql/windowing-and-triggering/index.html | 367 +
.../documentation/execution-model/index.html | 546 +
website/generated-content/documentation/index.html | 407 +
.../documentation/io/authoring-java/index.html | 366 +
.../documentation/io/authoring-overview/index.html | 463 +
.../documentation/io/authoring-python/index.html | 356 +
.../documentation/io/built-in/hadoop/index.html | 684 ++
.../documentation/io/built-in/hcatalog/index.html | 488 +
.../documentation/io/built-in/index.html | 468 +
.../documentation/io/contributing/index.html | 356 +
.../documentation/io/io-toc/index.html | 372 +
.../documentation/io/testing/index.html | 1177 ++
.../pipelines/create-your-pipeline/index.html | 446 +
.../pipelines/design-your-pipeline/index.html | 570 +
.../pipelines/test-your-pipeline/index.html | 665 ++
.../documentation/programming-guide/index.html | 3478 ++++++
.../documentation/resources/index.html | 490 +
.../documentation/runners/apex/index.html | 323 +
.../runners/capability-matrix/index.html | 10966 +++++++++++++++++++
.../documentation/runners/dataflow/index.html | 496 +
.../documentation/runners/direct/index.html | 336 +
.../documentation/runners/flink/index.html | 444 +
.../documentation/runners/gearpump/index.html | 387 +
.../documentation/runners/jstorm/index.html | 453 +
.../documentation/runners/mapreduce/index.html | 335 +
.../documentation/runners/samza/index.html | 413 +
.../documentation/runners/spark/index.html | 435 +
.../sdks/feature-comparison/index.html | 314 +
.../documentation/sdks/go/index.html | 331 +
.../documentation/sdks/java-extensions/index.html | 380 +
.../documentation/sdks/java-thirdparty/index.html | 434 +
.../documentation/sdks/java/euphoria/index.html | 386 +
.../documentation/sdks/java/index.html | 348 +
.../documentation/sdks/java/nexmark/index.html | 1128 ++
.../documentation/sdks/python-custom-io/index.html | 787 ++
.../sdks/python-pipeline-dependencies/index.html | 468 +
.../documentation/sdks/python-streaming/index.html | 503 +
.../sdks/python-type-safety/index.html | 522 +
.../documentation/sdks/python/index.html | 347 +
website/generated-content/downloads/beam-doap.rdf | 39 +
website/generated-content/downloads/logos.zip | Bin 0 -> 664457 bytes
website/generated-content/downloads/palette.pdf | Bin 0 -> 58247 bytes
website/generated-content/feed.xml | 2117 ++++
.../bootstrap/glyphicons-halflings-regular.eot | Bin 0 -> 20127 bytes
.../bootstrap/glyphicons-halflings-regular.svg | 306 +
.../bootstrap/glyphicons-halflings-regular.ttf | Bin 0 -> 45404 bytes
.../bootstrap/glyphicons-halflings-regular.woff | Bin 0 -> 23424 bytes
.../bootstrap/glyphicons-halflings-regular.woff2 | Bin 0 -> 18028 bytes
.../get-started/beam-overview/index.html | 330 +
.../get-started/downloads/index.html | 387 +
website/generated-content/get-started/index.html | 295 +
.../get-started/mobile-gaming-example/index.html | 1266 +++
.../get-started/quickstart-go/index.html | 338 +
.../get-started/quickstart-java/index.html | 603 +
.../get-started/quickstart-py/index.html | 493 +
.../get-started/quickstart/index.html | 10 +
.../get-started/releases/index.html | 10 +
.../get-started/support/index.html | 10 +
.../get-started/wordcount-example/index.html | 1717 +++
.../generated-content/getting-started/index.html | 10 +
.../images/apache_logo_circle.svg | 35 +
.../generated-content/images/beam_architecture.png | Bin 0 -> 26931 bytes
.../generated-content/images/beam_logo_circle.svg | 63 +
.../generated-content/images/beam_logo_navbar.png | Bin 0 -> 7298 bytes
website/generated-content/images/beam_logo_s.png | Bin 0 -> 11564 bytes
.../images/beam_sql_dsl_workflow.png | Bin 0 -> 73461 bytes
.../images/blog/2017-look-back/timeline.png | Bin 0 -> 12454 bytes
.../generated-content/images/blog/Facebook-AD.png | Bin 0 -> 1275709 bytes
.../images/blog/IMG_20160927_170455.jpg | Bin 0 -> 46625 bytes
.../images/blog/IMG_20160927_170956.jpg | Bin 0 -> 61284 bytes
.../images/blog/simple-wordcount-pipeline.png | Bin 0 -> 322878 bytes
.../images/blog/splittable-do-fn/blocks.png | Bin 0 -> 19493 bytes
.../blog/splittable-do-fn/jdbcio-expansion.png | Bin 0 -> 31429 bytes
.../blog/splittable-do-fn/kafka-splitting.png | Bin 0 -> 27762 bytes
.../images/blog/splittable-do-fn/restrictions.png | Bin 0 -> 34229 bytes
.../blog/splittable-do-fn/transform-expansion.png | Bin 0 -> 18690 bytes
.../blog/stateful-processing/assign-indices.png | Bin 0 -> 29308 bytes
.../images/blog/stateful-processing/combinefn.png | Bin 0 -> 18138 bytes
.../blog/stateful-processing/combiner-lifting.png | Bin 0 -> 37974 bytes
.../blog/stateful-processing/pardo-and-gbk.png | Bin 0 -> 27227 bytes
.../images/blog/stateful-processing/pipeline.png | Bin 0 -> 14308 bytes
.../images/blog/stateful-processing/plaid.png | Bin 0 -> 46216 bytes
.../blog/stateful-processing/stateful-dofn.png | Bin 0 -> 22229 bytes
.../blog/stateful-processing/stateful-pardo.png | Bin 0 -> 18035 bytes
.../blog/test-stream/elements-all-on-time.png | Bin 0 -> 14743 bytes
.../blog/test-stream/elements-droppably-late.png | Bin 0 -> 14800 bytes
.../blog/test-stream/elements-observably-late.png | Bin 0 -> 16219 bytes
.../elements-processing-speculative.png | Bin 0 -> 16987 bytes
.../test-stream/elements-unobservably-late.png | Bin 0 -> 14437 bytes
.../blog/timely-processing/BatchedRpcExpiry.png | Bin 0 -> 43015 bytes
.../blog/timely-processing/BatchedRpcStale.png | Bin 0 -> 51523 bytes
.../blog/timely-processing/BatchedRpcState.png | Bin 0 -> 32633 bytes
.../blog/timely-processing/CombinePerKey.png | Bin 0 -> 31517 bytes
.../images/blog/timely-processing/ParDo.png | Bin 0 -> 28247 bytes
.../blog/timely-processing/StateAndTimers.png | Bin 0 -> 21355 bytes
.../images/blog/timely-processing/UnifiedModel.png | Bin 0 -> 39982 bytes
.../blog/timely-processing/WindowingChoices.png | Bin 0 -> 20877 bytes
website/generated-content/images/card_border.svg | 37 +
website/generated-content/images/cards_bg.svg | 38 +
.../images/contribution-diversity.png | Bin 0 -> 38606 bytes
.../images/contribution-guide-1.png | Bin 0 -> 31400 bytes
.../design-your-pipeline-additional-outputs.png | Bin 0 -> 32797 bytes
.../images/design-your-pipeline-flatten.png | Bin 0 -> 47858 bytes
.../images/design-your-pipeline-join.png | Bin 0 -> 41878 bytes
.../images/design-your-pipeline-linear.png | Bin 0 -> 15218 bytes
.../design-your-pipeline-multiple-pcollections.png | Bin 0 -> 39095 bytes
.../images/execution_model_bundling.svg | 375 +
.../execution_model_bundling_coupled_failure.svg | 77 +
.../images/execution_model_bundling_gantt.svg | 68 +
.../images/execution_model_bundling_gantt_max.svg | 85 +
.../images/execution_model_bundling_multi.svg | 354 +
.../execution_model_bundling_multi_gantt.svg | 70 +
.../images/execution_model_failure_retry.svg | 74 +
.../images/external-link-icon.png | Bin 0 -> 1871 bytes
website/generated-content/images/favicon.ico | Bin 0 -> 370070 bytes
.../images/fixed-time-windows.png | Bin 0 -> 11717 bytes
.../images/gaming-example-basic.png | Bin 0 -> 63121 bytes
.../images/gaming-example-event-time-narrow.gif | Bin 0 -> 4857455 bytes
.../images/gaming-example-proc-time-narrow.gif | Bin 0 -> 4726095 bytes
.../images/gaming-example-session-windows.png | Bin 0 -> 17229 bytes
.../images/gaming-example-team-scores-narrow.gif | Bin 0 -> 4027933 bytes
.../generated-content/images/gaming-example.gif | Bin 0 -> 4135810 bytes
website/generated-content/images/hero_bg.svg | 51 +
website/generated-content/images/hero_bg_flat.svg | 64 +
website/generated-content/images/logo_apex.png | Bin 0 -> 8061 bytes
website/generated-content/images/logo_flink.png | Bin 0 -> 18261 bytes
website/generated-content/images/logo_gearpump.png | Bin 0 -> 4691 bytes
.../generated-content/images/logo_google_cloud.png | Bin 0 -> 13665 bytes
website/generated-content/images/logo_samza.png | Bin 0 -> 11728 bytes
website/generated-content/images/logo_spark.png | Bin 0 -> 13788 bytes
.../beam-logo-3-color-name-bottom-100.png | Bin 0 -> 2069 bytes
.../beam-logo-3-color-name-bottom-1000.png | Bin 0 -> 23684 bytes
.../beam-logo-3-color-name-bottom-200.png | Bin 0 -> 4085 bytes
.../beam-logo-3-color-name-bottom-500.png | Bin 0 -> 10905 bytes
.../name-bottom/beam-logo-3-color-name-bottom.svg | 19 +
.../beam-logo-3-color-name-right-100.png | Bin 0 -> 1506 bytes
.../beam-logo-3-color-name-right-1000.png | Bin 0 -> 17464 bytes
.../beam-logo-3-color-name-right-200.png | Bin 0 -> 2963 bytes
.../beam-logo-3-color-name-right-500.png | Bin 0 -> 7783 bytes
.../name-right/beam-logo-3-color-name-right.svg | 19 +
.../nameless/beam-logo-3-color-nameless-100.png | Bin 0 -> 1829 bytes
.../nameless/beam-logo-3-color-nameless-1000.png | Bin 0 -> 21040 bytes
.../nameless/beam-logo-3-color-nameless-200.png | Bin 0 -> 3522 bytes
.../nameless/beam-logo-3-color-nameless-500.png | Bin 0 -> 8950 bytes
.../nameless/beam-logo-3-color-nameless.svg | 19 +
.../name-bottom/beam-logo-bw-name-bottom-100.png | Bin 0 -> 2134 bytes
.../name-bottom/beam-logo-bw-name-bottom-1000.png | Bin 0 -> 24062 bytes
.../name-bottom/beam-logo-bw-name-bottom-200.png | Bin 0 -> 4172 bytes
.../name-bottom/beam-logo-bw-name-bottom-500.png | Bin 0 -> 11046 bytes
.../bw/name-bottom/beam-logo-bw-name-bottom.svg | 19 +
.../bw/name-right/beam-logo-bw-name-right-100.png | Bin 0 -> 1510 bytes
.../bw/name-right/beam-logo-bw-name-right-1000.png | Bin 0 -> 17836 bytes
.../bw/name-right/beam-logo-bw-name-right-200.png | Bin 0 -> 3040 bytes
.../bw/name-right/beam-logo-bw-name-right-500.png | Bin 0 -> 8059 bytes
.../bw/name-right/beam-logo-bw-name-right.svg | 19 +
.../bw/nameless/beam-logo-bw-nameless-100.png | Bin 0 -> 1778 bytes
.../bw/nameless/beam-logo-bw-nameless-1000.png | Bin 0 -> 20517 bytes
.../bw/nameless/beam-logo-bw-nameless-200.png | Bin 0 -> 3416 bytes
.../bw/nameless/beam-logo-bw-nameless-500.png | Bin 0 -> 9090 bytes
.../logos/bw/nameless/beam-logo-bw-nameless.svg | 19 +
.../beam-logo-full-color-name-bottom-100.png | Bin 0 -> 3395 bytes
.../beam-logo-full-color-name-bottom-1000.png | Bin 0 -> 50922 bytes
.../beam-logo-full-color-name-bottom-200.png | Bin 0 -> 7234 bytes
.../beam-logo-full-color-name-bottom-500.png | Bin 0 -> 20891 bytes
.../beam-logo-full-color-name-bottom.svg | 19 +
.../beam-logo-full-color-name-right-100.png | Bin 0 -> 2462 bytes
.../beam-logo-full-color-name-right-1000.png | Bin 0 -> 38278 bytes
.../beam-logo-full-color-name-right-200.png | Bin 0 -> 5222 bytes
.../beam-logo-full-color-name-right-500.png | Bin 0 -> 15369 bytes
.../name-right/beam-logo-full-color-name-right.svg | 19 +
.../nameless/beam-logo-full-color-nameless-100.png | Bin 0 -> 5346 bytes
.../beam-logo-full-color-nameless-1000.png | Bin 0 -> 133265 bytes
.../nameless/beam-logo-full-color-nameless-200.png | Bin 0 -> 12642 bytes
.../nameless/beam-logo-full-color-nameless-500.png | Bin 0 -> 46790 bytes
.../nameless/beam-logo-full-color-nameless.svg | 19 +
.../images/logos/runners/apex.png | Bin 0 -> 3717 bytes
.../images/logos/runners/dataflow.png | Bin 0 -> 8277 bytes
.../images/logos/runners/flink.png | Bin 0 -> 4584 bytes
.../images/logos/runners/gearpump.png | Bin 0 -> 2643 bytes
.../images/logos/runners/jstorm.png | Bin 0 -> 6375 bytes
.../images/logos/runners/samza.png | Bin 0 -> 11728 bytes
.../images/logos/runners/spark.png | Bin 0 -> 2701 bytes
website/generated-content/images/logos/sdks/go.png | Bin 0 -> 15475 bytes
.../generated-content/images/logos/sdks/java.png | Bin 0 -> 3726 bytes
.../generated-content/images/logos/sdks/python.png | Bin 0 -> 3735 bytes
.../generated-content/images/logos/sdks/scala.png | Bin 0 -> 2295 bytes
.../images/precommit_durations.png | Bin 0 -> 45673 bytes
.../images/precommit_graph_queuing_time.png | Bin 0 -> 25809 bytes
.../generated-content/images/release-guide-1.png | Bin 0 -> 363329 bytes
.../images/resources/se-radio-podcast.png | Bin 0 -> 165792 bytes
.../images/resources/streaming-101.png | Bin 0 -> 127023 bytes
.../images/resources/streaming-102.png | Bin 0 -> 126182 bytes
.../generated-content/images/session-windows.png | Bin 0 -> 16697 bytes
.../images/sliding-time-windows.png | Bin 0 -> 16537 bytes
.../images/trigger-accumulation.png | Bin 0 -> 11144 bytes
.../images/unwindowed-pipeline-bounded.png | Bin 0 -> 9589 bytes
.../images/windowing-pipeline-bounded.png | Bin 0 -> 13325 bytes
.../images/windowing-pipeline-unbounded.png | Bin 0 -> 21890 bytes
.../images/wordcount-pipeline.png | Bin 0 -> 19220 bytes
.../generated-content/js/bootstrap-sprockets.js | 12 +
website/generated-content/js/bootstrap.js | 2363 ++++
website/generated-content/js/bootstrap.min.js | 7 +
website/generated-content/js/bootstrap/affix.js | 162 +
website/generated-content/js/bootstrap/alert.js | 94 +
website/generated-content/js/bootstrap/button.js | 120 +
website/generated-content/js/bootstrap/carousel.js | 237 +
website/generated-content/js/bootstrap/collapse.js | 211 +
website/generated-content/js/bootstrap/dropdown.js | 165 +
website/generated-content/js/bootstrap/modal.js | 337 +
website/generated-content/js/bootstrap/popover.js | 108 +
.../generated-content/js/bootstrap/scrollspy.js | 172 +
website/generated-content/js/bootstrap/tab.js | 155 +
website/generated-content/js/bootstrap/tooltip.js | 514 +
.../generated-content/js/bootstrap/transition.js | 59 +
website/generated-content/js/fix-menu.js | 76 +
website/generated-content/js/language-switch.js | 131 +
website/generated-content/js/page-nav.js | 65 +
website/generated-content/js/section-nav.js | 169 +
website/generated-content/learn/index.html | 10 +
.../learn/presentation-materials/index.html | 10 +
.../learn/programming-guide/index.html | 10 +
.../generated-content/learn/resources/index.html | 10 +
.../learn/runners/Samza/index.html | 10 +
.../learn/runners/capability-matrix/index.html | 10 +
.../learn/runners/dataflow/index.html | 10 +
.../learn/runners/direct/index.html | 10 +
.../learn/runners/flink/index.html | 10 +
.../learn/runners/spark/index.html | 10 +
.../generated-content/learn/sdks/java/index.html | 10 +
website/generated-content/material/index.html | 10 +
.../generated-content/privacy_policy/index.html | 221 +
website/generated-content/project/logos/index.html | 10 +
website/generated-content/project/team/index.html | 10 +
website/generated-content/releases/index.html | 10 +
website/generated-content/team/index.html | 10 +
.../generated-content/use/beam-overview/index.html | 10 +
website/generated-content/use/index.html | 10 +
.../use/issue-tracking/index.html | 10 +
.../generated-content/use/mailing-lists/index.html | 10 +
.../use/mobile-gaming-example/index.html | 10 +
.../generated-content/use/quickstart/index.html | 10 +
website/generated-content/use/releases/index.html | 10 +
.../use/wordcount-example/index.html | 10 +
website/generated-content/v2/index.html | 265 +
326 files changed, 86598 insertions(+)
diff --git a/website/generated-content/.htaccess b/website/generated-content/.htaccess
new file mode 100644
index 0000000..06fc74b
--- /dev/null
+++ b/website/generated-content/.htaccess
@@ -0,0 +1,15 @@
+RewriteEngine On
+
+# This is a 301 (permanent) redirect from HTTP to HTTPS.
+
+# The next rule applies conditionally:
+# * the host is "beam.apache.org",
+# * the host comparison is case insensitive (NC),
+# * HTTPS is not used.
+RewriteCond %{HTTP_HOST} ^beam\.apache\.org [NC]
+RewriteCond %{HTTPS} !on
+
+# Rewrite the URL as follows:
+# * Redirect (R) permanently (301) to https://beam.apache.org/,
+# * Stop processing more rules (L).
+RewriteRule ^(.*)$ https://beam.apache.org/$1 [L,R=301]
diff --git a/website/generated-content/beam/capability/2016/03/17/capability-matrix.html b/website/generated-content/beam/capability/2016/03/17/capability-matrix.html
new file mode 100644
index 0000000..7cfe288
--- /dev/null
+++ b/website/generated-content/beam/capability/2016/03/17/capability-matrix.html
@@ -0,0 +1,2447 @@
+<!DOCTYPE html>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<html lang="en">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <title>Clarifying & Formalizing Runner Capabilities</title>
+ <meta name="description" content="With initial code drops complete (Dataflow SDK and Runner, Flink Runner, Spark Runner) and expressed interest in runner implementations for Storm, Hadoop, an...">
+ <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
+ <link rel="stylesheet" href="/css/site.css">
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <script src="/js/language-switch.js"></script>
+ <script src="/js/fix-menu.js"></script>
+ <script src="/js/section-nav.js"></script>
+ <script src="/js/page-nav.js"></script>
+ <link rel="canonical" href="https://beam.apache.org/beam/capability/2016/03/17/capability-matrix.html" data-proofer-ignore>
+ <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
+ <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+ ga('create', 'UA-73650088-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+</head>
+
+ <body class="body ">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<nav class="header navbar navbar-fixed-top">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+
+ <a href="/" class="navbar-brand" >
+ <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
+ </a>
+ </div>
+
+ <div class="navbar-mask closed"></div>
+
+ <div id="navbar" class="navbar-container closed">
+ <ul class="nav navbar-nav">
+ <li>
+ <a href="/get-started/beam-overview/">Get Started</a>
+ </li>
+ <li>
+ <a href="/documentation/">Documentation</a>
+ </li>
+ <li>
+ <a href="/documentation/sdks/java/">SDKS</a>
+ </li>
+ <li>
+ <a href="/documentation/runners/capability-matrix/">RUNNERS</a>
+ </li>
+ <li>
+ <a href="/contribute/">Contribute</a>
+ </li>
+ <li>
+ <a href="/community/contact-us/">Community</a>
+ </li>
+ <li><a href="/blog">Blog</a></li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:20px;"><span class="caret"></span></a>
+ <ul class="dropdown-menu dropdown-menu-right">
+ <li><a href="http://www.apache.org/">ASF Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">License</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
+ </ul>
+ </li>
+ </ul>
+ </div>
+</nav>
+
+ <div class="body__contained">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+
+
+<article class="post" itemscope itemtype="http://schema.org/BlogPosting">
+
+ <header class="post-header">
+ <h1 class="post-title" itemprop="name headline">Clarifying & Formalizing Runner Capabilities</h1>
+ <p class="post-meta"><time datetime="2016-03-17T11:00:00-07:00" itemprop="datePublished">Mar 17, 2016</time> •
+ Frances Perry [<a href="https://twitter.com/francesjperry">@francesjperry</a>] & Tyler Akidau [<a href="https://twitter.com/takidau">@takidau</a>]
+
+ </p>
+ </header>
+
+ <div class="post-content" itemprop="articleBody">
+ <!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+<p>With initial code drops complete (<a href="https://github.com/apache/beam/pull/1">Dataflow SDK and Runner</a>, <a href="https://github.com/apache/beam/pull/12">Flink Runner</a>, <a href="https://github.com/apache/beam/pull/42">Spark Runner</a>) and expressed interest in runner implementations for <a href="https://issues.apache.org/jira/browse/BEAM-9">Storm</a>, <a href="https://issues.apache.org/jira/browse/BEAM-19">Hadoop</a>, and <a href="https://issues.apache.org/jira/browse/BEAM-7 [...]
+
+<!--more-->
+
+<p>While we’d love to have a world where all runners support the full suite of semantics included in the Beam Model (formerly referred to as the <a href="http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf">Dataflow Model</a>), practically speaking, there will always be certain features that some runners can’t provide. For example, a Hadoop-based runner would be inherently batch-based and may be unable to (easily) implement support for unbounded collections. However, that doesn’t prevent it [...]
+
+<p>To help clarify things, we’ve been working on enumerating the key features of the Beam model in a <a href="/documentation/runners/capability-matrix/">capability matrix</a> for all existing runners, categorized around the four key questions addressed by the model: <span class="wwwh-what-dark">What</span> / <span class="wwwh-where-dark">Where</span> / <span class="wwwh-when-dark">When</span> / <span class="wwwh-how-dark">How</span> (if you’re not familiar with those questions, you might [...]
+
+<p>Included below is a summary snapshot of our current understanding of the capabilities of the existing runners (see the <a href="/documentation/runners/capability-matrix/">live version</a> for full details, descriptions, and Jira links); since integration is still under way, the system as whole isn’t yet in a completely stable, usable state. But that should be changing in the near future, and we’ll be updating loud and clear on this blog when the first supported Beam 1.0 release happens.</p>
+
+<p>In the meantime, these tables should help clarify where we expect to be in the very near term, and help guide expectations about what existing runners are capable of, and what features runner implementers will be tackling next.</p>
+
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<script type="text/javascript">
+ function ToggleTables(showDetails, anchor) {
+ document.getElementById("cap-summary").style.display = showDetails ? "none" : "block";
+ document.getElementById("cap-full").style.display = showDetails ? "block" : "none";
+ location.hash = anchor;
+ }
+</script>
+
+<!-- Summary table -->
+
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<div id="cap-blog" style="display:block">
+<table class="cap-summary">
+
+ <tr class="cap-summary" id="cap-blog-what">
+ <th class="cap-summary color-metadata format-category" colspan="5" style="color:#ca1">What is being computed?</th>
+ </tr>
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability"></th>
+
+ <th class="cap-summary color-platform format-platform" style="color:#ec3">Beam Model</th>
+
+ <th class="cap-summary color-platform format-platform" style="color:#ec3">Google Cloud Dataflow</th>
+
+ <th class="cap-summary color-platform format-platform" style="color:#ec3">Apache Flink</th>
+
+ <th class="cap-summary color-platform format-platform" style="color:#ec3">Apache Spark</th>
+
+ </tr>
+
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability format-capability" style="color:#ec3">ParDo</th>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+ </tr>
+
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability format-capability" style="color:#ec3">GroupByKey</th>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+
+ </tr>
+
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability format-capability" style="color:#ec3">Flatten</th>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+ </tr>
+
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability format-capability" style="color:#ec3">Combine</th>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+ </tr>
+
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability format-capability" style="color:#ec3">Composite Transforms</th>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+
+ </tr>
+
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability format-capability" style="color:#ec3">Side Inputs</th>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+
+ </tr>
+
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability format-capability" style="color:#ec3">Source API</th>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ec3;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+ </tr>
+
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability format-capability" style="color:#ec3">Aggregators</th>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+
+ </tr>
+
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability format-capability" style="color:#ec3">Keyed State</th>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
+ </tr>
+
+ <tr class="cap-summary">
+ <td class="cap-summary color-blank cap-blank" colspan="5"></td>
+ </tr>
+
+ <tr class="cap-summary" id="cap-blog-where">
+ <th class="cap-summary color-metadata format-category" colspan="5" style="color:#37d">Where in event time?</th>
+ </tr>
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability"></th>
+
+ <th class="cap-summary color-platform format-platform" style="color:#59f">Beam Model</th>
+
+ <th class="cap-summary color-platform format-platform" style="color:#59f">Google Cloud Dataflow</th>
+
+ <th class="cap-summary color-platform format-platform" style="color:#59f">Apache Flink</th>
+
+ <th class="cap-summary color-platform format-platform" style="color:#59f">Apache Spark</th>
+
+ </tr>
+
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability format-capability" style="color:#59f">Global windows</th>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+ </tr>
+
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability format-capability" style="color:#59f">Fixed windows</th>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#8cf;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+
+ </tr>
+
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability format-capability" style="color:#59f">Sliding windows</th>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
+ </tr>
+
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability format-capability" style="color:#59f">Session windows</th>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
+ </tr>
+
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability format-capability" style="color:#59f">Custom windows</th>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
+ </tr>
+
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability format-capability" style="color:#59f">Custom merging windows</th>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
+ </tr>
+
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability format-capability" style="color:#59f">Timestamp control</th>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
+ </tr>
+
+ <tr class="cap-summary">
+ <td class="cap-summary color-blank cap-blank" colspan="5"></td>
+ </tr>
+
+ <tr class="cap-summary" id="cap-blog-when">
+ <th class="cap-summary color-metadata format-category" colspan="5" style="color:#6a4">When in processing time?</th>
+ </tr>
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability"></th>
+
+ <th class="cap-summary color-platform format-platform" style="color:#8c6">Beam Model</th>
+
+ <th class="cap-summary color-platform format-platform" style="color:#8c6">Google Cloud Dataflow</th>
+
+ <th class="cap-summary color-platform format-platform" style="color:#8c6">Apache Flink</th>
+
+ <th class="cap-summary color-platform format-platform" style="color:#8c6">Apache Spark</th>
+
+ </tr>
+
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability format-capability" style="color:#8c6">Configurable triggering</th>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
+ </tr>
+
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability format-capability" style="color:#8c6">Event-time triggers</th>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
+ </tr>
+
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability format-capability" style="color:#8c6">Processing-time triggers</th>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+ </tr>
+
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability format-capability" style="color:#8c6">Count triggers</th>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
+ </tr>
+
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability format-capability" style="color:#8c6">[Meta]data driven triggers</th>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
+ </tr>
+
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability format-capability" style="color:#8c6">Composite triggers</th>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
+ </tr>
+
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability format-capability" style="color:#8c6">Allowed lateness</th>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#8c6;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
+ </tr>
+
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability format-capability" style="color:#8c6">Timers</th>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
+ </tr>
+
+ <tr class="cap-summary">
+ <td class="cap-summary color-blank cap-blank" colspan="5"></td>
+ </tr>
+
+ <tr class="cap-summary" id="cap-blog-how">
+ <th class="cap-summary color-metadata format-category" colspan="5" style="color:#b55">How do refinements relate?</th>
+ </tr>
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability"></th>
+
+ <th class="cap-summary color-platform format-platform" style="color:#d77">Beam Model</th>
+
+ <th class="cap-summary color-platform format-platform" style="color:#d77">Google Cloud Dataflow</th>
+
+ <th class="cap-summary color-platform format-platform" style="color:#d77">Apache Flink</th>
+
+ <th class="cap-summary color-platform format-platform" style="color:#d77">Apache Spark</th>
+
+ </tr>
+
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability format-capability" style="color:#d77">Discarding</th>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#d77;border-color:#b55"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#d77;border-color:#b55"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#d77;border-color:#b55"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#d77;border-color:#b55"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+ </tr>
+
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability format-capability" style="color:#d77">Accumulating</th>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#d77;border-color:#b55"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#d77;border-color:#b55"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#d77;border-color:#b55"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#b55"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
+ </tr>
+
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability format-capability" style="color:#d77">Accumulating & Retracting</th>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#b55"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#b55"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#b55"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary" style="background-color:#ddd;border-color:#b55"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
+ </tr>
+
+ <tr class="cap-summary">
+ <td class="cap-summary color-blank cap-blank" colspan="5"></td>
+ </tr>
+
+</table>
+</div>
+
+
+ </div>
+
+</article>
+
+ </div>
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<footer class="footer">
+ <div class="footer__contained">
+ <div class="footer__cols">
+ <div class="footer__cols__col">
+ <div class="footer__cols__col__logo">
+ <img src="/images/beam_logo_circle.svg" class="footer__logo" alt="Beam logo">
+ </div>
+ <div class="footer__cols__col__logo">
+ <img src="/images/apache_logo_circle.svg" class="footer__logo" alt="Apache logo">
+ </div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Start</div>
+ <div class="footer__cols__col__link"><a href="/get-started/beam-overview/">Overview</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-java/">Quickstart (Java)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-py/">Quickstart (Python)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-go/">Quickstart (Go)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/downloads/">Downloads</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Docs</div>
+ <div class="footer__cols__col__link"><a href="/documentation/programming-guide/">Concepts</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/pipelines/design-your-pipeline/">Pipelines</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/runners/capability-matrix/">Runners</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Community</div>
+ <div class="footer__cols__col__link"><a href="/contribute/">Contribute</a></div>
+ <div class="footer__cols__col__link"><a href="https://projects.apache.org/committee.html?beam" target="_blank">Team<img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a></div>
+ <div class="footer__cols__col__link"><a href="/contribute/presentation-materials/">Media</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Resources</div>
+ <div class="footer__cols__col__link"><a href="/blog/">Blog</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/support/">Support</a></div>
+ <div class="footer__cols__col__link"><a href="https://github.com/apache/beam">GitHub</a></div>
+ </div>
+ </div>
+ </div>
+ <div class="footer__bottom">
+ ©
+ <a href="http://www.apache.org">The Apache Software Foundation</a>
+ | <a href="/privacy_policy">Privacy Policy</a>
+ | <a href="/feed.xml">RSS Feed</a>
+ <br><br>
+ Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are
+ either registered trademarks or trademarks of The Apache Software
+ Foundation. All other products or name brands are trademarks of their
+ respective holders, including The Apache Software Foundation.
+ </div>
+</footer>
+
+ </body>
+</html>
diff --git a/website/generated-content/beam/capability/2016/04/03/presentation-materials.html b/website/generated-content/beam/capability/2016/04/03/presentation-materials.html
new file mode 100644
index 0000000..1dda77b
--- /dev/null
+++ b/website/generated-content/beam/capability/2016/04/03/presentation-materials.html
@@ -0,0 +1,255 @@
+<!DOCTYPE html>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<html lang="en">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <title>Apache Beam Presentation Materials</title>
+ <meta name="description" content="Are you interested in giving a presentation about Apache Beam? Perhaps you want to talk about Apache Beam at a local Meetup or a convention. Excellent! The ...">
+ <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
+ <link rel="stylesheet" href="/css/site.css">
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <script src="/js/language-switch.js"></script>
+ <script src="/js/fix-menu.js"></script>
+ <script src="/js/section-nav.js"></script>
+ <script src="/js/page-nav.js"></script>
+ <link rel="canonical" href="https://beam.apache.org/beam/capability/2016/04/03/presentation-materials.html" data-proofer-ignore>
+ <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
+ <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+ ga('create', 'UA-73650088-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+</head>
+
+ <body class="body ">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<nav class="header navbar navbar-fixed-top">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+
+ <a href="/" class="navbar-brand" >
+ <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
+ </a>
+ </div>
+
+ <div class="navbar-mask closed"></div>
+
+ <div id="navbar" class="navbar-container closed">
+ <ul class="nav navbar-nav">
+ <li>
+ <a href="/get-started/beam-overview/">Get Started</a>
+ </li>
+ <li>
+ <a href="/documentation/">Documentation</a>
+ </li>
+ <li>
+ <a href="/documentation/sdks/java/">SDKS</a>
+ </li>
+ <li>
+ <a href="/documentation/runners/capability-matrix/">RUNNERS</a>
+ </li>
+ <li>
+ <a href="/contribute/">Contribute</a>
+ </li>
+ <li>
+ <a href="/community/contact-us/">Community</a>
+ </li>
+ <li><a href="/blog">Blog</a></li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:20px;"><span class="caret"></span></a>
+ <ul class="dropdown-menu dropdown-menu-right">
+ <li><a href="http://www.apache.org/">ASF Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">License</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
+ </ul>
+ </li>
+ </ul>
+ </div>
+</nav>
+
+ <div class="body__contained">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+
+
+<article class="post" itemscope itemtype="http://schema.org/BlogPosting">
+
+ <header class="post-header">
+ <h1 class="post-title" itemprop="name headline">Apache Beam Presentation Materials</h1>
+ <p class="post-meta"><time datetime="2016-04-03T11:00:00-07:00" itemprop="datePublished">Apr 3, 2016</time> •
+ Frances Perry [<a href="https://twitter.com/francesjperry">@francesjperry</a>] & Tyler Akidau [<a href="https://twitter.com/takidau">@takidau</a>]
+
+ </p>
+ </header>
+
+ <div class="post-content" itemprop="articleBody">
+ <!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+<p>Are you interested in giving a presentation about Apache Beam? Perhaps you want to talk about Apache Beam at a local Meetup or a convention. Excellent! The Apache Beam community is excited to expand and grow the community. To help kickstart this process, we are excited to announce an initial set of <a href="/contribute/presentation-materials/">Apache Beam presentation materials</a> which anyone can use to give a presentation about Apache Beam.</p>
+
+<!--more-->
+
+<p>As a community, we want to build a shared collection of high quality presentation materials. This initial set includes the following slide decks:</p>
+
+<ul>
+ <li>The Apache Beam model
+ <ul>
+ <li><a href="https://goo.gl/r0nvWh">Long version</a></li>
+ <li><a href="https://goo.gl/h5D1yR">Short version</a></li>
+ </ul>
+ </li>
+ <li>Using Apache Beam with runners
+ <ul>
+ <li><a href="https://goo.gl/2ay8mi">Google Cloud Dataflow</a></li>
+ </ul>
+ </li>
+</ul>
+
+<p>As Apache Beam grows, so will this repository of presentation materials. We are excited to add new materials as the Apache Beam ecosystem grows with new runners, SDKs, and so on. If you are interested in contributing content or have a request, please see the <a href="/contribute/presentation-materials/">Apache Beam presentation materials</a> page or email the <a href="mailto:user@beam.apache.org"><code class="highlighter-rouge">user@beam.apache.org</code></a> mailing list with your id [...]
+
+ </div>
+
+</article>
+
+ </div>
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<footer class="footer">
+ <div class="footer__contained">
+ <div class="footer__cols">
+ <div class="footer__cols__col">
+ <div class="footer__cols__col__logo">
+ <img src="/images/beam_logo_circle.svg" class="footer__logo" alt="Beam logo">
+ </div>
+ <div class="footer__cols__col__logo">
+ <img src="/images/apache_logo_circle.svg" class="footer__logo" alt="Apache logo">
+ </div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Start</div>
+ <div class="footer__cols__col__link"><a href="/get-started/beam-overview/">Overview</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-java/">Quickstart (Java)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-py/">Quickstart (Python)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-go/">Quickstart (Go)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/downloads/">Downloads</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Docs</div>
+ <div class="footer__cols__col__link"><a href="/documentation/programming-guide/">Concepts</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/pipelines/design-your-pipeline/">Pipelines</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/runners/capability-matrix/">Runners</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Community</div>
+ <div class="footer__cols__col__link"><a href="/contribute/">Contribute</a></div>
+ <div class="footer__cols__col__link"><a href="https://projects.apache.org/committee.html?beam" target="_blank">Team<img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a></div>
+ <div class="footer__cols__col__link"><a href="/contribute/presentation-materials/">Media</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Resources</div>
+ <div class="footer__cols__col__link"><a href="/blog/">Blog</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/support/">Support</a></div>
+ <div class="footer__cols__col__link"><a href="https://github.com/apache/beam">GitHub</a></div>
+ </div>
+ </div>
+ </div>
+ <div class="footer__bottom">
+ ©
+ <a href="http://www.apache.org">The Apache Software Foundation</a>
+ | <a href="/privacy_policy">Privacy Policy</a>
+ | <a href="/feed.xml">RSS Feed</a>
+ <br><br>
+ Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are
+ either registered trademarks or trademarks of The Apache Software
+ Foundation. All other products or name brands are trademarks of their
+ respective holders, including The Apache Software Foundation.
+ </div>
+</footer>
+
+ </body>
+</html>
diff --git a/website/generated-content/beam/python/sdk/2016/02/25/python-sdk-now-public.html b/website/generated-content/beam/python/sdk/2016/02/25/python-sdk-now-public.html
new file mode 100644
index 0000000..511a719
--- /dev/null
+++ b/website/generated-content/beam/python/sdk/2016/02/25/python-sdk-now-public.html
@@ -0,0 +1,246 @@
+<!DOCTYPE html>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<html lang="en">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <title>Dataflow Python SDK is now public!</title>
+ <meta name="description" content="When the Apache Beam project proposed entry into the Apache Incubator the proposalincluded the Dataflow Java SDK. In the long term, however, Apache Beam aims...">
+ <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
+ <link rel="stylesheet" href="/css/site.css">
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <script src="/js/language-switch.js"></script>
+ <script src="/js/fix-menu.js"></script>
+ <script src="/js/section-nav.js"></script>
+ <script src="/js/page-nav.js"></script>
+ <link rel="canonical" href="https://beam.apache.org/beam/python/sdk/2016/02/25/python-sdk-now-public.html" data-proofer-ignore>
+ <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
+ <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+ ga('create', 'UA-73650088-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+</head>
+
+ <body class="body ">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<nav class="header navbar navbar-fixed-top">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+
+ <a href="/" class="navbar-brand" >
+ <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
+ </a>
+ </div>
+
+ <div class="navbar-mask closed"></div>
+
+ <div id="navbar" class="navbar-container closed">
+ <ul class="nav navbar-nav">
+ <li>
+ <a href="/get-started/beam-overview/">Get Started</a>
+ </li>
+ <li>
+ <a href="/documentation/">Documentation</a>
+ </li>
+ <li>
+ <a href="/documentation/sdks/java/">SDKS</a>
+ </li>
+ <li>
+ <a href="/documentation/runners/capability-matrix/">RUNNERS</a>
+ </li>
+ <li>
+ <a href="/contribute/">Contribute</a>
+ </li>
+ <li>
+ <a href="/community/contact-us/">Community</a>
+ </li>
+ <li><a href="/blog">Blog</a></li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:20px;"><span class="caret"></span></a>
+ <ul class="dropdown-menu dropdown-menu-right">
+ <li><a href="http://www.apache.org/">ASF Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">License</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
+ </ul>
+ </li>
+ </ul>
+ </div>
+</nav>
+
+ <div class="body__contained">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+
+
+<article class="post" itemscope itemtype="http://schema.org/BlogPosting">
+
+ <header class="post-header">
+ <h1 class="post-title" itemprop="name headline">Dataflow Python SDK is now public!</h1>
+ <p class="post-meta"><time datetime="2016-02-25T13:00:00-08:00" itemprop="datePublished">Feb 25, 2016</time> •
+ James Malone [<a href="https://twitter.com/chimerasaurus">@chimerasaurus</a>]
+
+ </p>
+ </header>
+
+ <div class="post-content" itemprop="articleBody">
+ <!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+<p>When the Apache Beam project proposed entry into the <a href="http://wiki.apache.org/incubator/BeamProposal">Apache Incubator</a> the proposal
+included the <a href="https://github.com/GoogleCloudPlatform/DataflowJavaSDK">Dataflow Java SDK</a>. In the long term, however, Apache Beam aims to support SDKs implemented in multiple languages, such as Python.</p>
+
+<!--more-->
+
+<p>Today, Google submitted the <a href="http://github.com/GoogleCloudPlatform/DataflowPythonSDK">Dataflow Python (2.x) SDK</a> on GitHub. Google is committed to including the in progress python SDK in Apache Beam and, in that spirit, we’ve moved development of the Python SDK to a public repository. While this SDK will not be included with the initial (incubating) releases of Apache Beam, our we plan on incorporating the Python SDK into beam during incubation. We want to take the time to [...]
+
+<p>You can look for the Apache Beam Python SDK in the coming months once we finish forking and refactoring the Java SDK.</p>
+
+<p>Best,</p>
+
+<p>Apache Beam Team</p>
+
+ </div>
+
+</article>
+
+ </div>
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<footer class="footer">
+ <div class="footer__contained">
+ <div class="footer__cols">
+ <div class="footer__cols__col">
+ <div class="footer__cols__col__logo">
+ <img src="/images/beam_logo_circle.svg" class="footer__logo" alt="Beam logo">
+ </div>
+ <div class="footer__cols__col__logo">
+ <img src="/images/apache_logo_circle.svg" class="footer__logo" alt="Apache logo">
+ </div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Start</div>
+ <div class="footer__cols__col__link"><a href="/get-started/beam-overview/">Overview</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-java/">Quickstart (Java)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-py/">Quickstart (Python)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-go/">Quickstart (Go)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/downloads/">Downloads</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Docs</div>
+ <div class="footer__cols__col__link"><a href="/documentation/programming-guide/">Concepts</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/pipelines/design-your-pipeline/">Pipelines</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/runners/capability-matrix/">Runners</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Community</div>
+ <div class="footer__cols__col__link"><a href="/contribute/">Contribute</a></div>
+ <div class="footer__cols__col__link"><a href="https://projects.apache.org/committee.html?beam" target="_blank">Team<img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a></div>
+ <div class="footer__cols__col__link"><a href="/contribute/presentation-materials/">Media</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Resources</div>
+ <div class="footer__cols__col__link"><a href="/blog/">Blog</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/support/">Support</a></div>
+ <div class="footer__cols__col__link"><a href="https://github.com/apache/beam">GitHub</a></div>
+ </div>
+ </div>
+ </div>
+ <div class="footer__bottom">
+ ©
+ <a href="http://www.apache.org">The Apache Software Foundation</a>
+ | <a href="/privacy_policy">Privacy Policy</a>
+ | <a href="/feed.xml">RSS Feed</a>
+ <br><br>
+ Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are
+ either registered trademarks or trademarks of The Apache Software
+ Foundation. All other products or name brands are trademarks of their
+ respective holders, including The Apache Software Foundation.
+ </div>
+</footer>
+
+ </body>
+</html>
diff --git a/website/generated-content/beam/release/2016/06/15/first-release.html b/website/generated-content/beam/release/2016/06/15/first-release.html
new file mode 100644
index 0000000..b47cc9c
--- /dev/null
+++ b/website/generated-content/beam/release/2016/06/15/first-release.html
@@ -0,0 +1,263 @@
+<!DOCTYPE html>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<html lang="en">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <title>The first release of Apache Beam!</title>
+ <meta name="description" content="I’m happy to announce that Apache Beam has officially released its firstversion – 0.1.0-incubating. This is an exciting milestone for the project,which joine...">
+ <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
+ <link rel="stylesheet" href="/css/site.css">
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <script src="/js/language-switch.js"></script>
+ <script src="/js/fix-menu.js"></script>
+ <script src="/js/section-nav.js"></script>
+ <script src="/js/page-nav.js"></script>
+ <link rel="canonical" href="https://beam.apache.org/beam/release/2016/06/15/first-release.html" data-proofer-ignore>
+ <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
+ <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+ ga('create', 'UA-73650088-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+</head>
+
+ <body class="body ">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<nav class="header navbar navbar-fixed-top">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+
+ <a href="/" class="navbar-brand" >
+ <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
+ </a>
+ </div>
+
+ <div class="navbar-mask closed"></div>
+
+ <div id="navbar" class="navbar-container closed">
+ <ul class="nav navbar-nav">
+ <li>
+ <a href="/get-started/beam-overview/">Get Started</a>
+ </li>
+ <li>
+ <a href="/documentation/">Documentation</a>
+ </li>
+ <li>
+ <a href="/documentation/sdks/java/">SDKS</a>
+ </li>
+ <li>
+ <a href="/documentation/runners/capability-matrix/">RUNNERS</a>
+ </li>
+ <li>
+ <a href="/contribute/">Contribute</a>
+ </li>
+ <li>
+ <a href="/community/contact-us/">Community</a>
+ </li>
+ <li><a href="/blog">Blog</a></li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:20px;"><span class="caret"></span></a>
+ <ul class="dropdown-menu dropdown-menu-right">
+ <li><a href="http://www.apache.org/">ASF Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">License</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
+ </ul>
+ </li>
+ </ul>
+ </div>
+</nav>
+
+ <div class="body__contained">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+
+
+<article class="post" itemscope itemtype="http://schema.org/BlogPosting">
+
+ <header class="post-header">
+ <h1 class="post-title" itemprop="name headline">The first release of Apache Beam!</h1>
+ <p class="post-meta"><time datetime="2016-06-15T00:00:01-07:00" itemprop="datePublished">Jun 15, 2016</time> •
+ Davor Bonaci [<a href="https://twitter.com/BonaciDavor">@BonaciDavor</a>]
+
+ </p>
+ </header>
+
+ <div class="post-content" itemprop="articleBody">
+ <!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+<p>I’m happy to announce that Apache Beam has officially released its first
+version – 0.1.0-incubating. This is an exciting milestone for the project,
+which joined the Apache Software Foundation and the Apache Incubator earlier
+this year.</p>
+
+<!--more-->
+
+<p>This release publishes the first set of Apache Beam binaries and source code,
+making them readily available for our users. The initial release includes the
+SDK for Java, along with three runners: Apache Flink, Apache Spark and Google
+Cloud Dataflow, a fully-managed cloud service. The release is available both
+in the <a href="http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.beam%22">Maven Central Repository</a>,
+as well as a download from the <a href="/get-started/downloads/">project’s website</a>.</p>
+
+<p>The goal of this release was process-oriented. In particular, the Beam
+community wanted to release existing functionality to our users, build and
+validate the release processes, and obtain validation from the Apache Software
+Foundation and the Apache Incubator.</p>
+
+<p>I’d like to encourage everyone to try out this release. Please keep in mind
+that this is the first incubating release – significant changes are to be
+expected. As we march toward stability, a rapid cadence of future releases is
+anticipated, perhaps one every 1-2 months.</p>
+
+<p>As always, the Beam community welcomes feedback. Stabilization, usability and
+the developer experience will be our focus for the next several months. If you
+have any comments or discover any issues, I’d like to invite you to reach out
+to us via <a href="/get-started/support/">user’s mailing list</a> or the
+<a href="https://issues.apache.org/jira/browse/BEAM/">Apache JIRA issue tracker</a>.</p>
+
+ </div>
+
+</article>
+
+ </div>
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<footer class="footer">
+ <div class="footer__contained">
+ <div class="footer__cols">
+ <div class="footer__cols__col">
+ <div class="footer__cols__col__logo">
+ <img src="/images/beam_logo_circle.svg" class="footer__logo" alt="Beam logo">
+ </div>
+ <div class="footer__cols__col__logo">
+ <img src="/images/apache_logo_circle.svg" class="footer__logo" alt="Apache logo">
+ </div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Start</div>
+ <div class="footer__cols__col__link"><a href="/get-started/beam-overview/">Overview</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-java/">Quickstart (Java)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-py/">Quickstart (Python)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-go/">Quickstart (Go)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/downloads/">Downloads</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Docs</div>
+ <div class="footer__cols__col__link"><a href="/documentation/programming-guide/">Concepts</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/pipelines/design-your-pipeline/">Pipelines</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/runners/capability-matrix/">Runners</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Community</div>
+ <div class="footer__cols__col__link"><a href="/contribute/">Contribute</a></div>
+ <div class="footer__cols__col__link"><a href="https://projects.apache.org/committee.html?beam" target="_blank">Team<img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a></div>
+ <div class="footer__cols__col__link"><a href="/contribute/presentation-materials/">Media</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Resources</div>
+ <div class="footer__cols__col__link"><a href="/blog/">Blog</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/support/">Support</a></div>
+ <div class="footer__cols__col__link"><a href="https://github.com/apache/beam">GitHub</a></div>
+ </div>
+ </div>
+ </div>
+ <div class="footer__bottom">
+ ©
+ <a href="http://www.apache.org">The Apache Software Foundation</a>
+ | <a href="/privacy_policy">Privacy Policy</a>
+ | <a href="/feed.xml">RSS Feed</a>
+ <br><br>
+ Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are
+ either registered trademarks or trademarks of The Apache Software
+ Foundation. All other products or name brands are trademarks of their
+ respective holders, including The Apache Software Foundation.
+ </div>
+</footer>
+
+ </body>
+</html>
diff --git a/website/generated-content/beam/update/2016/10/11/strata-hadoop-world-and-beam.html b/website/generated-content/beam/update/2016/10/11/strata-hadoop-world-and-beam.html
new file mode 100644
index 0000000..73f6fe1
--- /dev/null
+++ b/website/generated-content/beam/update/2016/10/11/strata-hadoop-world-and-beam.html
@@ -0,0 +1,256 @@
+<!DOCTYPE html>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<html lang="en">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <title>Strata+Hadoop World and Beam</title>
+ <meta name="description" content="Tyler Akidau and I gave a three-hour tutorial on Apache Beam at Strata+Hadoop World 2016. We had a plethora of help from our TAs: Kenn Knowles, Reuven Lax, F...">
+ <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
+ <link rel="stylesheet" href="/css/site.css">
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <script src="/js/language-switch.js"></script>
+ <script src="/js/fix-menu.js"></script>
+ <script src="/js/section-nav.js"></script>
+ <script src="/js/page-nav.js"></script>
+ <link rel="canonical" href="https://beam.apache.org/beam/update/2016/10/11/strata-hadoop-world-and-beam.html" data-proofer-ignore>
+ <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
+ <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+ ga('create', 'UA-73650088-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+</head>
+
+ <body class="body ">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<nav class="header navbar navbar-fixed-top">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+
+ <a href="/" class="navbar-brand" >
+ <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
+ </a>
+ </div>
+
+ <div class="navbar-mask closed"></div>
+
+ <div id="navbar" class="navbar-container closed">
+ <ul class="nav navbar-nav">
+ <li>
+ <a href="/get-started/beam-overview/">Get Started</a>
+ </li>
+ <li>
+ <a href="/documentation/">Documentation</a>
+ </li>
+ <li>
+ <a href="/documentation/sdks/java/">SDKS</a>
+ </li>
+ <li>
+ <a href="/documentation/runners/capability-matrix/">RUNNERS</a>
+ </li>
+ <li>
+ <a href="/contribute/">Contribute</a>
+ </li>
+ <li>
+ <a href="/community/contact-us/">Community</a>
+ </li>
+ <li><a href="/blog">Blog</a></li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:20px;"><span class="caret"></span></a>
+ <ul class="dropdown-menu dropdown-menu-right">
+ <li><a href="http://www.apache.org/">ASF Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">License</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
+ </ul>
+ </li>
+ </ul>
+ </div>
+</nav>
+
+ <div class="body__contained">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+
+
+<article class="post" itemscope itemtype="http://schema.org/BlogPosting">
+
+ <header class="post-header">
+ <h1 class="post-title" itemprop="name headline">Strata+Hadoop World and Beam</h1>
+ <p class="post-meta"><time datetime="2016-10-11T10:00:00-07:00" itemprop="datePublished">Oct 11, 2016</time> •
+ Jesse Anderson [<a href="https://twitter.com/jessetanderson">@jessetanderson</a>]
+
+ </p>
+ </header>
+
+ <div class="post-content" itemprop="articleBody">
+ <!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+<p>Tyler Akidau and I gave a <a href="http://conferences.oreilly.com/strata/hadoop-big-data-ny/public/schedule/detail/52129">three-hour tutorial</a> on Apache Beam at Strata+Hadoop World 2016. We had a plethora of help from our TAs: Kenn Knowles, Reuven Lax, Felipe Hoffa, Slava Chernyak, and Jamie Grier. There were a total of 66 people that attended the session.<!--more--></p>
+
+<p><img src="/images/blog/IMG_20160927_170956.jpg" alt="Exercise time" /></p>
+
+<p>If you want to take a look at the tutorial materials, we’ve put them up <a href="https://github.com/eljefe6a/beamexample">on GitHub</a>. This includes the <a href="https://github.com/eljefe6a/beamexample/blob/master/BeamTutorial/slides.pdf">actual slides</a> as well as the <a href="https://github.com/eljefe6a/beamexample/tree/master/BeamTutorial/src/main/java/org/apache/beam/examples/tutorial/game">exercises</a> that we covered. If you’re looking to learn a little about Beam, this is [...]
+
+<p>I want to share some of takeaways I had about Beam during the conference.</p>
+
+<p>The Data Engineers are looking to Beam as a way to <a href="https://www.oreilly.com/ideas/future-proof-and-scale-proof-your-code">future-proof</a>, meaning that code is portable between the various Big Data frameworks. In fact, many of the attendees were still on Hadoop MapReduce and looking to transition to a new framework. They’re realizing that continually rewriting code isn’t the most productive approach.</p>
+
+<p>Data Scientists are really interested in using Beam. They interested in having a single API for doing analysis instead of several different APIs. We talked about Beam’s progress on the Python API. If you want to take a peek, it’s being actively developed on a <a href="https://github.com/apache/beam/tree/master/sdks/python">feature branch</a>. As Beam matures, we’re looking to add other supported languages.</p>
+
+<p>We heard <a href="https://twitter.com/jessetanderson/status/781124173108305920">loud and clear</a> from Beam users that great runner support is crucial to adoption. We have great Apache Flink support. During the conference we had some more volunteers offer their help on the Spark runner.</p>
+
+<p>On management and thought leader side, Beam went from “what’s Beam?” at previous conferences to “I’m interested in Beam.” or “I’ve formed an informed opinion on Beam.” at this conference. This is one of the metrics I look for in early technology adoption.</p>
+
+<p><img src="/images/blog/IMG_20160927_170455.jpg" alt="So much brainpower answering questions" /></p>
+
+<p>We rounded out the tutorial with live demonstrations of Beam running on Apache Spark, Apache Flink, the local runner, and DataFlow runner. Then, we brought in the big brainpower and had a Q and A session.</p>
+
+<p>If you’re attending a conference, we encourage you to look for a Beam session. If you want to use these materials to give your own Beam talk or tutorial, we’re happy to help you. In addition to this tutorial, we have <a href="/contribute/presentation-materials/">other presentation materials</a>. You can reach out to us on the <a href="/get-started/support/">user mailing list</a>.</p>
+
+
+ </div>
+
+</article>
+
+ </div>
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<footer class="footer">
+ <div class="footer__contained">
+ <div class="footer__cols">
+ <div class="footer__cols__col">
+ <div class="footer__cols__col__logo">
+ <img src="/images/beam_logo_circle.svg" class="footer__logo" alt="Beam logo">
+ </div>
+ <div class="footer__cols__col__logo">
+ <img src="/images/apache_logo_circle.svg" class="footer__logo" alt="Apache logo">
+ </div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Start</div>
+ <div class="footer__cols__col__link"><a href="/get-started/beam-overview/">Overview</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-java/">Quickstart (Java)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-py/">Quickstart (Python)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-go/">Quickstart (Go)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/downloads/">Downloads</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Docs</div>
+ <div class="footer__cols__col__link"><a href="/documentation/programming-guide/">Concepts</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/pipelines/design-your-pipeline/">Pipelines</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/runners/capability-matrix/">Runners</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Community</div>
+ <div class="footer__cols__col__link"><a href="/contribute/">Contribute</a></div>
+ <div class="footer__cols__col__link"><a href="https://projects.apache.org/committee.html?beam" target="_blank">Team<img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a></div>
+ <div class="footer__cols__col__link"><a href="/contribute/presentation-materials/">Media</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Resources</div>
+ <div class="footer__cols__col__link"><a href="/blog/">Blog</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/support/">Support</a></div>
+ <div class="footer__cols__col__link"><a href="https://github.com/apache/beam">GitHub</a></div>
+ </div>
+ </div>
+ </div>
+ <div class="footer__bottom">
+ ©
+ <a href="http://www.apache.org">The Apache Software Foundation</a>
+ | <a href="/privacy_policy">Privacy Policy</a>
+ | <a href="/feed.xml">RSS Feed</a>
+ <br><br>
+ Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are
+ either registered trademarks or trademarks of The Apache Software
+ Foundation. All other products or name brands are trademarks of their
+ respective holders, including The Apache Software Foundation.
+ </div>
+</footer>
+
+ </body>
+</html>
diff --git a/website/generated-content/beam/update/website/2016/02/22/beam-has-a-logo.html b/website/generated-content/beam/update/website/2016/02/22/beam-has-a-logo.html
new file mode 100644
index 0000000..d96ac69
--- /dev/null
+++ b/website/generated-content/beam/update/website/2016/02/22/beam-has-a-logo.html
@@ -0,0 +1,252 @@
+<!DOCTYPE html>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<html lang="en">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <title>Apache Beam has a logo!</title>
+ <meta name="description" content="One of the major benefits of Apache Beam is the fact that it unifies bothboth batch and stream processing into one powerful model. In fact, this unificationi...">
+ <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
+ <link rel="stylesheet" href="/css/site.css">
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <script src="/js/language-switch.js"></script>
+ <script src="/js/fix-menu.js"></script>
+ <script src="/js/section-nav.js"></script>
+ <script src="/js/page-nav.js"></script>
+ <link rel="canonical" href="https://beam.apache.org/beam/update/website/2016/02/22/beam-has-a-logo.html" data-proofer-ignore>
+ <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
+ <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+ ga('create', 'UA-73650088-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+</head>
+
+ <body class="body ">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<nav class="header navbar navbar-fixed-top">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+
+ <a href="/" class="navbar-brand" >
+ <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
+ </a>
+ </div>
+
+ <div class="navbar-mask closed"></div>
+
+ <div id="navbar" class="navbar-container closed">
+ <ul class="nav navbar-nav">
+ <li>
+ <a href="/get-started/beam-overview/">Get Started</a>
+ </li>
+ <li>
+ <a href="/documentation/">Documentation</a>
+ </li>
+ <li>
+ <a href="/documentation/sdks/java/">SDKS</a>
+ </li>
+ <li>
+ <a href="/documentation/runners/capability-matrix/">RUNNERS</a>
+ </li>
+ <li>
+ <a href="/contribute/">Contribute</a>
+ </li>
+ <li>
+ <a href="/community/contact-us/">Community</a>
+ </li>
+ <li><a href="/blog">Blog</a></li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:20px;"><span class="caret"></span></a>
+ <ul class="dropdown-menu dropdown-menu-right">
+ <li><a href="http://www.apache.org/">ASF Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">License</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
+ </ul>
+ </li>
+ </ul>
+ </div>
+</nav>
+
+ <div class="body__contained">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+
+
+<article class="post" itemscope itemtype="http://schema.org/BlogPosting">
+
+ <header class="post-header">
+ <h1 class="post-title" itemprop="name headline">Apache Beam has a logo!</h1>
+ <p class="post-meta"><time datetime="2016-02-22T10:21:48-08:00" itemprop="datePublished">Feb 22, 2016</time> •
+ James Malone [<a href="https://twitter.com/chimerasaurus">@chimerasaurus</a>]
+
+ </p>
+ </header>
+
+ <div class="post-content" itemprop="articleBody">
+ <!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+<p>One of the major benefits of Apache Beam is the fact that it unifies both
+both batch and stream processing into one powerful model. In fact, this unification
+is so important, the name Beam itself comes from the union of <strong>B</strong>atch + str<strong>EAM</strong> = Beam</p>
+
+<p>When the project started, we wanted a logo which was both appealing and visually
+represented this unification. <!--more--> Thanks to the <strong>amazing</strong> work of Stephanie Smythies, the Apache Beam project
+now has a logo.</p>
+
+<p><em>drum roll</em> - <strong>Presenting, the Apache Beam Logo!</strong></p>
+
+<p><img src="/images/beam_logo_s.png" alt="Apache Beam Logo" /></p>
+
+<p>We are excited about this logo because it is <strong>simple</strong>, <strong>bright</strong>, and shows the
+unification of bath and streaming, as beams of light, within the ‘B’. We will base
+our future website and documentation design around this logo and its coloring. We
+will also make various permutations and resolutions of this logo available in the
+coming weeks. For any questions or comments, send an email to the <code class="highlighter-rouge">dev@</code> email list
+for Apache Beam.</p>
+
+ </div>
+
+</article>
+
+ </div>
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<footer class="footer">
+ <div class="footer__contained">
+ <div class="footer__cols">
+ <div class="footer__cols__col">
+ <div class="footer__cols__col__logo">
+ <img src="/images/beam_logo_circle.svg" class="footer__logo" alt="Beam logo">
+ </div>
+ <div class="footer__cols__col__logo">
+ <img src="/images/apache_logo_circle.svg" class="footer__logo" alt="Apache logo">
+ </div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Start</div>
+ <div class="footer__cols__col__link"><a href="/get-started/beam-overview/">Overview</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-java/">Quickstart (Java)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-py/">Quickstart (Python)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-go/">Quickstart (Go)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/downloads/">Downloads</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Docs</div>
+ <div class="footer__cols__col__link"><a href="/documentation/programming-guide/">Concepts</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/pipelines/design-your-pipeline/">Pipelines</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/runners/capability-matrix/">Runners</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Community</div>
+ <div class="footer__cols__col__link"><a href="/contribute/">Contribute</a></div>
+ <div class="footer__cols__col__link"><a href="https://projects.apache.org/committee.html?beam" target="_blank">Team<img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a></div>
+ <div class="footer__cols__col__link"><a href="/contribute/presentation-materials/">Media</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Resources</div>
+ <div class="footer__cols__col__link"><a href="/blog/">Blog</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/support/">Support</a></div>
+ <div class="footer__cols__col__link"><a href="https://github.com/apache/beam">GitHub</a></div>
+ </div>
+ </div>
+ </div>
+ <div class="footer__bottom">
+ ©
+ <a href="http://www.apache.org">The Apache Software Foundation</a>
+ | <a href="/privacy_policy">Privacy Policy</a>
+ | <a href="/feed.xml">RSS Feed</a>
+ <br><br>
+ Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are
+ either registered trademarks or trademarks of The Apache Software
+ Foundation. All other products or name brands are trademarks of their
+ respective holders, including The Apache Software Foundation.
+ </div>
+</footer>
+
+ </body>
+</html>
diff --git a/website/generated-content/blog/2016/05/18/splitAtFraction-method.html b/website/generated-content/blog/2016/05/18/splitAtFraction-method.html
new file mode 100644
index 0000000..14e57e1
--- /dev/null
+++ b/website/generated-content/blog/2016/05/18/splitAtFraction-method.html
@@ -0,0 +1,241 @@
+<!DOCTYPE html>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<html lang="en">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <title>Dynamic work rebalancing for Beam</title>
+ <meta name="description" content="This morning, Eugene and Malo from the Google Cloud Dataflow team posted No shard left behind: dynamic work rebalancing in Google Cloud Dataflow. This articl...">
+ <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
+ <link rel="stylesheet" href="/css/site.css">
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <script src="/js/language-switch.js"></script>
+ <script src="/js/fix-menu.js"></script>
+ <script src="/js/section-nav.js"></script>
+ <script src="/js/page-nav.js"></script>
+ <link rel="canonical" href="https://beam.apache.org/blog/2016/05/18/splitAtFraction-method.html" data-proofer-ignore>
+ <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
+ <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+ ga('create', 'UA-73650088-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+</head>
+
+ <body class="body ">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<nav class="header navbar navbar-fixed-top">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+
+ <a href="/" class="navbar-brand" >
+ <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
+ </a>
+ </div>
+
+ <div class="navbar-mask closed"></div>
+
+ <div id="navbar" class="navbar-container closed">
+ <ul class="nav navbar-nav">
+ <li>
+ <a href="/get-started/beam-overview/">Get Started</a>
+ </li>
+ <li>
+ <a href="/documentation/">Documentation</a>
+ </li>
+ <li>
+ <a href="/documentation/sdks/java/">SDKS</a>
+ </li>
+ <li>
+ <a href="/documentation/runners/capability-matrix/">RUNNERS</a>
+ </li>
+ <li>
+ <a href="/contribute/">Contribute</a>
+ </li>
+ <li>
+ <a href="/community/contact-us/">Community</a>
+ </li>
+ <li><a href="/blog">Blog</a></li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:20px;"><span class="caret"></span></a>
+ <ul class="dropdown-menu dropdown-menu-right">
+ <li><a href="http://www.apache.org/">ASF Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">License</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
+ </ul>
+ </li>
+ </ul>
+ </div>
+</nav>
+
+ <div class="body__contained">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+
+
+<article class="post" itemscope itemtype="http://schema.org/BlogPosting">
+
+ <header class="post-header">
+ <h1 class="post-title" itemprop="name headline">Dynamic work rebalancing for Beam</h1>
+ <p class="post-meta"><time datetime="2016-05-18T11:00:00-07:00" itemprop="datePublished">May 18, 2016</time> •
+ Dan Halperin
+
+ </p>
+ </header>
+
+ <div class="post-content" itemprop="articleBody">
+ <!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+<p>This morning, Eugene and Malo from the Google Cloud Dataflow team posted <a href="https://cloud.google.com/blog/big-data/2016/05/no-shard-left-behind-dynamic-work-rebalancing-in-google-cloud-dataflow"><em>No shard left behind: dynamic work rebalancing in Google Cloud Dataflow</em></a>. This article discusses Cloud Dataflow’s solution to the well-known straggler problem.</p>
+
+<!--more-->
+
+<p>In a large batch processing job with many tasks executing in parallel, some of the tasks – the stragglers – can take a much longer time to complete than others, perhaps due to imperfect splitting of the work into parallel chunks when issuing the job. Typically, waiting for stragglers means that the overall job completes later than it should, and may also reserve too many machines that may be underutilized at the end. Cloud Dataflow’s dynamic work rebalancing can mitigate stragglers in [...]
+
+<p>What I’d like to highlight for the Apache Beam (incubating) community is that Cloud Dataflow’s dynamic work rebalancing is implemented using <em>runner-specific</em> control logic on top of Beam’s <em>runner-independent</em> <a href="https://github.com/apache/beam/blob/9fa97fb2491bc784df53fb0f044409dbbc2af3d7/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedSource.java"><code class="highlighter-rouge">BoundedSource API</code></a>. Specifically, to steal work from a straggler [...]
+
+ </div>
+
+</article>
+
+ </div>
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<footer class="footer">
+ <div class="footer__contained">
+ <div class="footer__cols">
+ <div class="footer__cols__col">
+ <div class="footer__cols__col__logo">
+ <img src="/images/beam_logo_circle.svg" class="footer__logo" alt="Beam logo">
+ </div>
+ <div class="footer__cols__col__logo">
+ <img src="/images/apache_logo_circle.svg" class="footer__logo" alt="Apache logo">
+ </div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Start</div>
+ <div class="footer__cols__col__link"><a href="/get-started/beam-overview/">Overview</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-java/">Quickstart (Java)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-py/">Quickstart (Python)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-go/">Quickstart (Go)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/downloads/">Downloads</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Docs</div>
+ <div class="footer__cols__col__link"><a href="/documentation/programming-guide/">Concepts</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/pipelines/design-your-pipeline/">Pipelines</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/runners/capability-matrix/">Runners</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Community</div>
+ <div class="footer__cols__col__link"><a href="/contribute/">Contribute</a></div>
+ <div class="footer__cols__col__link"><a href="https://projects.apache.org/committee.html?beam" target="_blank">Team<img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a></div>
+ <div class="footer__cols__col__link"><a href="/contribute/presentation-materials/">Media</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Resources</div>
+ <div class="footer__cols__col__link"><a href="/blog/">Blog</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/support/">Support</a></div>
+ <div class="footer__cols__col__link"><a href="https://github.com/apache/beam">GitHub</a></div>
+ </div>
+ </div>
+ </div>
+ <div class="footer__bottom">
+ ©
+ <a href="http://www.apache.org">The Apache Software Foundation</a>
+ | <a href="/privacy_policy">Privacy Policy</a>
+ | <a href="/feed.xml">RSS Feed</a>
+ <br><br>
+ Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are
+ either registered trademarks or trademarks of The Apache Software
+ Foundation. All other products or name brands are trademarks of their
+ respective holders, including The Apache Software Foundation.
+ </div>
+</footer>
+
+ </body>
+</html>
diff --git a/website/generated-content/blog/2016/05/27/where-is-my-pcollection-dot-map.html b/website/generated-content/blog/2016/05/27/where-is-my-pcollection-dot-map.html
new file mode 100644
index 0000000..19890b6
--- /dev/null
+++ b/website/generated-content/blog/2016/05/27/where-is-my-pcollection-dot-map.html
@@ -0,0 +1,316 @@
+<!DOCTYPE html>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<html lang="en">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <title>Where's my PCollection.map()?</title>
+ <meta name="description" content="Have you ever wondered why Beam has PTransforms for everything instead of having methods on PCollection? Take a look at the history that led to this (and oth...">
+ <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
+ <link rel="stylesheet" href="/css/site.css">
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <script src="/js/language-switch.js"></script>
+ <script src="/js/fix-menu.js"></script>
+ <script src="/js/section-nav.js"></script>
+ <script src="/js/page-nav.js"></script>
+ <link rel="canonical" href="https://beam.apache.org/blog/2016/05/27/where-is-my-pcollection-dot-map.html" data-proofer-ignore>
+ <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
+ <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+ ga('create', 'UA-73650088-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+</head>
+
+ <body class="body ">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<nav class="header navbar navbar-fixed-top">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+
+ <a href="/" class="navbar-brand" >
+ <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
+ </a>
+ </div>
+
+ <div class="navbar-mask closed"></div>
+
+ <div id="navbar" class="navbar-container closed">
+ <ul class="nav navbar-nav">
+ <li>
+ <a href="/get-started/beam-overview/">Get Started</a>
+ </li>
+ <li>
+ <a href="/documentation/">Documentation</a>
+ </li>
+ <li>
+ <a href="/documentation/sdks/java/">SDKS</a>
+ </li>
+ <li>
+ <a href="/documentation/runners/capability-matrix/">RUNNERS</a>
+ </li>
+ <li>
+ <a href="/contribute/">Contribute</a>
+ </li>
+ <li>
+ <a href="/community/contact-us/">Community</a>
+ </li>
+ <li><a href="/blog">Blog</a></li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:20px;"><span class="caret"></span></a>
+ <ul class="dropdown-menu dropdown-menu-right">
+ <li><a href="http://www.apache.org/">ASF Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">License</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
+ </ul>
+ </li>
+ </ul>
+ </div>
+</nav>
+
+ <div class="body__contained">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+
+
+<article class="post" itemscope itemtype="http://schema.org/BlogPosting">
+
+ <header class="post-header">
+ <h1 class="post-title" itemprop="name headline">Where's my PCollection.map()?</h1>
+ <p class="post-meta"><time datetime="2016-05-27T09:00:00-07:00" itemprop="datePublished">May 27, 2016</time> •
+ Robert Bradshaw
+
+ </p>
+ </header>
+
+ <div class="post-content" itemprop="articleBody">
+ <!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<p>Have you ever wondered why Beam has PTransforms for everything instead of having methods on PCollection? Take a look at the history that led to this (and other) design decisions.</p>
+
+<!--more-->
+
+<p>Though Beam is relatively new, its design draws heavily on many years of
+experience with real-world pipelines. One of the primary inspirations is
+<a href="https://ai.google/research/pubs/pub35650">FlumeJava</a>, which is Google’s
+internal successor to MapReduce first introduced in 2009.</p>
+
+<p>The original FlumeJava API has methods like <code class="highlighter-rouge">count</code> and <code class="highlighter-rouge">parallelDo</code> on the PCollections. Though slightly more succinct, this approach has many disadvantages to extensibility. Every new user to FlumeJava wanted to add transforms, and adding them as methods to PCollection simply doesn’t scale well. In contrast, a PCollection in Beam has a single <code class="highlighter-rouge">apply</code> method which takes any [...]
+
+<table class="table">
+ <tr>
+ <th>FlumeJava</th>
+ <th>Beam</th>
+ </tr>
+ <tr>
+ <td><pre>
+PCollection<T> input = …
+PCollection<O> output = input.count()
+ .parallelDo(...);
+ </pre></td>
+ <td><pre>
+PCollection<T> input = …
+PCollection<O> output = input.apply(Count.perElement())
+ .apply(ParDo.of(...));
+ </pre></td>
+ </tr>
+</table>
+
+<p>This is a more scalable approach for several reasons.</p>
+
+<h2 id="where-to-draw-the-line">Where to draw the line?</h2>
+<p>Adding methods to PCollection forces a line to be drawn between operations that are “useful” enough to merit this special treatment and those that are not. It is easy to make the case for flat map, group by key, and combine per key. But what about filter? Count? Approximate count? Approximate quantiles? Most frequent? WriteToMyFavoriteSource? Going too far down this path leads to a single enormous class that contains nearly everything one could want to do. (FlumeJava’s PCollection cla [...]
+
+<p>Instead in Beam we’ve chosen a style that places all transforms–whether they be primitive operations, composite operations bundled in the SDK, or part of an external library–on equal footing. This also facilitates alternative implementations (which may even take different options) that are easily interchangeable.</p>
+
+<table class="table">
+ <tr>
+ <th>FlumeJava</th>
+ <th>Beam</th>
+ </tr>
+ <tr>
+ <td><pre>
+PCollection<O> output =
+ ExternalLibrary.doStuff(
+ MyLibrary.transform(input, myArgs)
+ .parallelDo(...),
+ externalLibArgs);
+ </pre></td>
+ <td><pre>
+PCollection<O> output = input
+ .apply(MyLibrary.transform(myArgs))
+ .apply(ParDo.of(...))
+ .apply(ExternalLibrary.doStuff(externalLibArgs));
+
+ </pre></td>
+ </tr>
+</table>
+
+<h2 id="configurability">Configurability</h2>
+<p>It makes for a fluent style to let values (PCollections) be the objects passed around and manipulated (i.e. the handles to the deferred execution graph), but it is the operations themselves that need to be composable, configurable, and extendable. Using PCollection methods for the operations doesn’t scale well here, especially in a language without default or keyword arguments. For example, a ParDo operation can have any number of side inputs and side outputs, or a write operation may [...]
+
+<h2 id="type-safety">Type Safety</h2>
+<p>Many operations can only be applied to collections whose elements are of a specific type. For example, the GroupByKey operation should only be applied to <code class="highlighter-rouge">PCollection<KV<K, V>></code>s. In Java at least, it’s not possible to restrict methods based on the element type parameter alone. In FlumeJava, this led us to add a <code class="highlighter-rouge">PTable<K, V></code> subclassing <code class="highlighter-rouge">PCollection<KV<K, [...]
+
+<p>This is particularly inconvenient for transforms that produce outputs whose element types are the same as (or related to) their input’s element types, requiring extra support to generate the right subclasses (e.g. a filter on a PTable should produce another PTable rather than just a raw PCollection of key-value pairs).</p>
+
+<p>Using PTransforms allows us to sidestep this entire issue. We can place arbitrary constraints on the context in which a transform may be used based on the type of its inputs; for instance GroupByKey is statically typed to only apply to a <code class="highlighter-rouge">PCollection<KV<K, V>></code>. The way this happens is generalizable to arbitrary shapes, without needing to introduce specialized types like PTable.</p>
+
+<h2 id="reusability-and-structure">Reusability and Structure</h2>
+<p>Though PTransforms are generally constructed at the site at which they’re used, by pulling them out as separate objects one is able to store them and pass them around.</p>
+
+<p>As pipelines grow and evolve, it is useful to structure your pipeline into modular, often reusable components, and PTransforms allow one to do this nicely in a data-processing pipeline. In addition, modular PTransforms also expose the logical structure of your code to the system (e.g. for monitoring). Of the three different representations of the WordCount pipeline below, only the structured view captures the high-level intent of the pipeline. Letting even the simple operations be PTr [...]
+
+<p><img class="center-block" src="/images/blog/simple-wordcount-pipeline.png" alt="Three different visualizations of a simple WordCount pipeline" width="500" /></p>
+
+<div class="text-center">
+<i>Three different visualizations of a simple WordCount pipeline which computes the number of occurrences of every word in a set of text files. The flat view gives the full DAG of all operations performed. The execution view groups operations according to how they're executed, e.g. after performing runner-specific optimizations like function composition. The structured view nests operations according to their grouping in PTransforms.</i>
+</div>
+
+<h2 id="summary">Summary</h2>
+<p>Although it’s tempting to add methods to PCollections, such an approach is not scalable, extensible, or sufficiently expressive. Putting a single apply method on PCollection and all the logic into the operation itself lets us have the best of both worlds, and avoids hard cliffs of complexity by having a single consistent style across simple and complex pipelines, and between predefined and user-defined operations.</p>
+
+ </div>
+
+</article>
+
+ </div>
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<footer class="footer">
+ <div class="footer__contained">
+ <div class="footer__cols">
+ <div class="footer__cols__col">
+ <div class="footer__cols__col__logo">
+ <img src="/images/beam_logo_circle.svg" class="footer__logo" alt="Beam logo">
+ </div>
+ <div class="footer__cols__col__logo">
+ <img src="/images/apache_logo_circle.svg" class="footer__logo" alt="Apache logo">
+ </div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Start</div>
+ <div class="footer__cols__col__link"><a href="/get-started/beam-overview/">Overview</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-java/">Quickstart (Java)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-py/">Quickstart (Python)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-go/">Quickstart (Go)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/downloads/">Downloads</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Docs</div>
+ <div class="footer__cols__col__link"><a href="/documentation/programming-guide/">Concepts</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/pipelines/design-your-pipeline/">Pipelines</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/runners/capability-matrix/">Runners</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Community</div>
+ <div class="footer__cols__col__link"><a href="/contribute/">Contribute</a></div>
+ <div class="footer__cols__col__link"><a href="https://projects.apache.org/committee.html?beam" target="_blank">Team<img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a></div>
+ <div class="footer__cols__col__link"><a href="/contribute/presentation-materials/">Media</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Resources</div>
+ <div class="footer__cols__col__link"><a href="/blog/">Blog</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/support/">Support</a></div>
+ <div class="footer__cols__col__link"><a href="https://github.com/apache/beam">GitHub</a></div>
+ </div>
+ </div>
+ </div>
+ <div class="footer__bottom">
+ ©
+ <a href="http://www.apache.org">The Apache Software Foundation</a>
+ | <a href="/privacy_policy">Privacy Policy</a>
+ | <a href="/feed.xml">RSS Feed</a>
+ <br><br>
+ Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are
+ either registered trademarks or trademarks of The Apache Software
+ Foundation. All other products or name brands are trademarks of their
+ respective holders, including The Apache Software Foundation.
+ </div>
+</footer>
+
+ </body>
+</html>
diff --git a/website/generated-content/blog/2016/06/13/flink-batch-runner-milestone.html b/website/generated-content/blog/2016/06/13/flink-batch-runner-milestone.html
new file mode 100644
index 0000000..3dd6c40
--- /dev/null
+++ b/website/generated-content/blog/2016/06/13/flink-batch-runner-milestone.html
@@ -0,0 +1,256 @@
+<!DOCTYPE html>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<html lang="en">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <title>How We Added Windowing to the Apache Flink Batch Runner</title>
+ <meta name="description" content="We recently achieved a major milestone by adding support for windowing to the Apache Flink Batch runner. In this post we would like to explain what this mean...">
+ <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
+ <link rel="stylesheet" href="/css/site.css">
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <script src="/js/language-switch.js"></script>
+ <script src="/js/fix-menu.js"></script>
+ <script src="/js/section-nav.js"></script>
+ <script src="/js/page-nav.js"></script>
+ <link rel="canonical" href="https://beam.apache.org/blog/2016/06/13/flink-batch-runner-milestone.html" data-proofer-ignore>
+ <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
+ <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+ ga('create', 'UA-73650088-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+</head>
+
+ <body class="body ">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<nav class="header navbar navbar-fixed-top">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+
+ <a href="/" class="navbar-brand" >
+ <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
+ </a>
+ </div>
+
+ <div class="navbar-mask closed"></div>
+
+ <div id="navbar" class="navbar-container closed">
+ <ul class="nav navbar-nav">
+ <li>
+ <a href="/get-started/beam-overview/">Get Started</a>
+ </li>
+ <li>
+ <a href="/documentation/">Documentation</a>
+ </li>
+ <li>
+ <a href="/documentation/sdks/java/">SDKS</a>
+ </li>
+ <li>
+ <a href="/documentation/runners/capability-matrix/">RUNNERS</a>
+ </li>
+ <li>
+ <a href="/contribute/">Contribute</a>
+ </li>
+ <li>
+ <a href="/community/contact-us/">Community</a>
+ </li>
+ <li><a href="/blog">Blog</a></li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:20px;"><span class="caret"></span></a>
+ <ul class="dropdown-menu dropdown-menu-right">
+ <li><a href="http://www.apache.org/">ASF Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">License</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
+ </ul>
+ </li>
+ </ul>
+ </div>
+</nav>
+
+ <div class="body__contained">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+
+
+<article class="post" itemscope itemtype="http://schema.org/BlogPosting">
+
+ <header class="post-header">
+ <h1 class="post-title" itemprop="name headline">How We Added Windowing to the Apache Flink Batch Runner</h1>
+ <p class="post-meta"><time datetime="2016-06-13T09:00:00-07:00" itemprop="datePublished">Jun 13, 2016</time> •
+ Aljoscha Krettek [<a href="https://twitter.com/aljoscha">@aljoscha</a>]
+
+ </p>
+ </header>
+
+ <div class="post-content" itemprop="articleBody">
+ <!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<p>We recently achieved a major milestone by adding support for windowing to the <a href="http://flink.apache.org">Apache Flink</a> Batch runner. In this post we would like to explain what this means for users of Apache Beam and highlight some of the implementation details.</p>
+
+<!--more-->
+
+<p>Before we start, though, let’s quickly talk about the execution of Beam programs and how this is relevant to today’s post. A Beam pipeline can contain bounded and unbounded sources. If the pipeline only contains bounded sources it can be executed in a batch fashion, if it contains some unbounded sources it must be executed in a streaming fashion. When executing a Beam pipeline on Flink, you don’t have to choose the execution mode. Internally, the Flink runner either translates the pip [...]
+
+<h2 id="what-does-this-mean-for-users">What does this mean for users?</h2>
+
+<p>Support for windowing was the last missing puzzle piece for making the Flink Batch runner compatible with the Beam model. With the latest change to the Batch runner users can now run any pipeline that only contains bounded sources and be certain that the results match those of the original reference-implementation runners that were provided by Google as part of the initial code drop coming from the Google Dataflow SDK.</p>
+
+<p>The most obvious part of the change is that windows can now be assigned to elements and that the runner respects these windows for the <code class="highlighter-rouge">GroupByKey</code> and <code class="highlighter-rouge">Combine</code> operations. A not-so-obvious change concerns side-inputs. In the Beam model, side inputs respect windows; when a value of the main input is being processed only the side input that corresponds to the correct window is available to the processing functio [...]
+
+<p>Getting side-input semantics right is an important milestone in it’s own because it allows to use a big suite of unit tests for verifying the correctness of a runner implementation. These tests exercise every obscure detail of the Beam programming model and verify that the results produced by a runner match what you would expect from a correct implementation. In the suite, side inputs are used to compare the expected result to the actual result. With these tests being executed regular [...]
+
+<h2 id="under-the-hood">Under the Hood</h2>
+<p>The basis for the changes is the introduction of <code class="highlighter-rouge">WindowedValue</code> in the generated Flink transformations. Before, a Beam <code class="highlighter-rouge">PCollection<T></code> would be transformed to a <code class="highlighter-rouge">DataSet<T></code>. Now, we instead create a <code class="highlighter-rouge">DataSet<WindowedValue<T>></code>. The <code class="highlighter-rouge">WindowedValue<T></code> stores meta data abo [...]
+
+<p>With this basic change out of the way we just had to make sure that windows were respected for side inputs and that <code class="highlighter-rouge">Combine</code> and <code class="highlighter-rouge">GroupByKey</code> correctly handled windows. The tricky part there is the handling of merging windows such as session windows. For these we essentially emulate the behavior of a merging <code class="highlighter-rouge">WindowFn</code> in our own code.</p>
+
+<p>After we got side inputs working we could enable the aforementioned suite of tests to check how well the runner behaves with respect to the Beam model. As can be expected there were quite some discrepancies but we managed to resolve them all. In the process, we also slimmed down the runner implementation. For example, we removed all custom translations for sources and sinks and are now relying only on Beam code for these, thereby greatly reducing the maintenance overhead.</p>
+
+<h2 id="summary">Summary</h2>
+<p>We reached a major milestone in adding windowing support to the Flink Batch runner, thereby making it compatible with the Beam model. Because of the large suite of tests that can now be executed on the runner we are also confident about the correctness of the implementation and about it staying that way in the future.</p>
+
+ </div>
+
+</article>
+
+ </div>
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<footer class="footer">
+ <div class="footer__contained">
+ <div class="footer__cols">
+ <div class="footer__cols__col">
+ <div class="footer__cols__col__logo">
+ <img src="/images/beam_logo_circle.svg" class="footer__logo" alt="Beam logo">
+ </div>
+ <div class="footer__cols__col__logo">
+ <img src="/images/apache_logo_circle.svg" class="footer__logo" alt="Apache logo">
+ </div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Start</div>
+ <div class="footer__cols__col__link"><a href="/get-started/beam-overview/">Overview</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-java/">Quickstart (Java)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-py/">Quickstart (Python)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-go/">Quickstart (Go)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/downloads/">Downloads</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Docs</div>
+ <div class="footer__cols__col__link"><a href="/documentation/programming-guide/">Concepts</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/pipelines/design-your-pipeline/">Pipelines</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/runners/capability-matrix/">Runners</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Community</div>
+ <div class="footer__cols__col__link"><a href="/contribute/">Contribute</a></div>
+ <div class="footer__cols__col__link"><a href="https://projects.apache.org/committee.html?beam" target="_blank">Team<img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a></div>
+ <div class="footer__cols__col__link"><a href="/contribute/presentation-materials/">Media</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Resources</div>
+ <div class="footer__cols__col__link"><a href="/blog/">Blog</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/support/">Support</a></div>
+ <div class="footer__cols__col__link"><a href="https://github.com/apache/beam">GitHub</a></div>
+ </div>
+ </div>
+ </div>
+ <div class="footer__bottom">
+ ©
+ <a href="http://www.apache.org">The Apache Software Foundation</a>
+ | <a href="/privacy_policy">Privacy Policy</a>
+ | <a href="/feed.xml">RSS Feed</a>
+ <br><br>
+ Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are
+ either registered trademarks or trademarks of The Apache Software
+ Foundation. All other products or name brands are trademarks of their
+ respective holders, including The Apache Software Foundation.
+ </div>
+</footer>
+
+ </body>
+</html>
diff --git a/website/generated-content/blog/2016/08/03/six-months.html b/website/generated-content/blog/2016/08/03/six-months.html
new file mode 100644
index 0000000..7a12c93
--- /dev/null
+++ b/website/generated-content/blog/2016/08/03/six-months.html
@@ -0,0 +1,273 @@
+<!DOCTYPE html>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<html lang="en">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <title>Apache Beam: Six Months in Incubation</title>
+ <meta name="description" content="It’s been just over six months since Apache Beam was formally accepted into incubation with the Apache Software Foundation. As a community, we’ve been hard a...">
+ <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
+ <link rel="stylesheet" href="/css/site.css">
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <script src="/js/language-switch.js"></script>
+ <script src="/js/fix-menu.js"></script>
+ <script src="/js/section-nav.js"></script>
+ <script src="/js/page-nav.js"></script>
+ <link rel="canonical" href="https://beam.apache.org/blog/2016/08/03/six-months.html" data-proofer-ignore>
+ <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
+ <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+ ga('create', 'UA-73650088-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+</head>
+
+ <body class="body ">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<nav class="header navbar navbar-fixed-top">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+
+ <a href="/" class="navbar-brand" >
+ <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
+ </a>
+ </div>
+
+ <div class="navbar-mask closed"></div>
+
+ <div id="navbar" class="navbar-container closed">
+ <ul class="nav navbar-nav">
+ <li>
+ <a href="/get-started/beam-overview/">Get Started</a>
+ </li>
+ <li>
+ <a href="/documentation/">Documentation</a>
+ </li>
+ <li>
+ <a href="/documentation/sdks/java/">SDKS</a>
+ </li>
+ <li>
+ <a href="/documentation/runners/capability-matrix/">RUNNERS</a>
+ </li>
+ <li>
+ <a href="/contribute/">Contribute</a>
+ </li>
+ <li>
+ <a href="/community/contact-us/">Community</a>
+ </li>
+ <li><a href="/blog">Blog</a></li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:20px;"><span class="caret"></span></a>
+ <ul class="dropdown-menu dropdown-menu-right">
+ <li><a href="http://www.apache.org/">ASF Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">License</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
+ </ul>
+ </li>
+ </ul>
+ </div>
+</nav>
+
+ <div class="body__contained">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+
+
+<article class="post" itemscope itemtype="http://schema.org/BlogPosting">
+
+ <header class="post-header">
+ <h1 class="post-title" itemprop="name headline">Apache Beam: Six Months in Incubation</h1>
+ <p class="post-meta"><time datetime="2016-08-03T00:00:01-07:00" itemprop="datePublished">Aug 3, 2016</time> •
+ Frances Perry [<a href="https://twitter.com/francesjperry">@francesjperry</a>]
+
+ </p>
+ </header>
+
+ <div class="post-content" itemprop="articleBody">
+ <!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+<p>It’s been just over six months since Apache Beam was formally accepted into incubation with the <a href="http://www.apache.org">Apache Software Foundation</a>. As a community, we’ve been hard at work getting Beam off the ground.</p>
+
+<!--more-->
+
+<p>Looking just at raw numbers for those first six months, that’s:</p>
+
+<ul>
+ <li>48,238 lines of preexisting code donated by Cloudera, dataArtisans, and Google.</li>
+ <li>761 pull requests from 45 contributors.</li>
+ <li>498 Jira issues opened and 245 resolved.</li>
+ <li>1 incubating release (and another 1 in progress).</li>
+ <li>4,200 hours of automated tests.</li>
+ <li>161 subscribers / 606 messages on user@.</li>
+ <li>217 subscribers / 1205 messages on dev@.</li>
+ <li>277 stars and 174 forks on GitHub.</li>
+</ul>
+
+<p>And behind those numbers, there’s been a ton of technical progress, including:</p>
+
+<ul>
+ <li>Refactoring of the entire codebase, examples, and tests to be truly runner-independent.</li>
+ <li>New functionality in the Apache Flink runner for timestamps/windows in batch and bounded sources and side inputs in streaming mode.</li>
+ <li>Work in progress to upgrade the Apache Spark runner to use Spark 2.0.</li>
+ <li>Several new runners from the wider Apache community – Apache Gearpump has its own feature branch, Apache Apex has a PR, and conversations are starting on Apache Storm and others.</li>
+ <li>New SDKs/DSLs for exposing the Beam model – the Python SDK from Google is in on a feature branch, and there are plans to add the Scio DSL from Spotify.</li>
+ <li>Support for additional data sources and sinks – Apache Kafka and JMS are in, there are PRs for Amazon Kinesis, Apache Cassandra, and MongoDB, and more connectors are being planned.</li>
+</ul>
+
+<p>But perhaps most importantly, we’re committed to building an involved, welcoming community. So far, we’ve:</p>
+
+<ul>
+ <li>Started building a vibrant developer community, with detailed design discussions on features like DoFn reuse semantics, serialization technology, and an API for accessing state.</li>
+ <li>Started building a user community with an active mailing list and improvements to the website and documentation.</li>
+ <li>Had multiple talks on Beam at venues including ApacheCon, Hadoop Summit, Kafka Summit, JBCN Barcelona, and Strata.</li>
+ <li>Presented at multiple existing meetups and are starting to organize some of our own.</li>
+</ul>
+
+<p>While it’s nice to reflect back on all we’ve done, we’re working full <em>stream</em> ahead towards a stable release and graduation from incubator. And we’d love your help – join the <a href="/get-started/support/">mailing lists</a>, check out the <a href="/contribute/contribution-guide/">contribution guide</a>, and grab a <a href="https://issues.apache.org/jira/browse/BEAM-520?jql=project%20%3D%20BEAM%20AND%20resolution%20%3D%20Unresolved%20AND%20labels%20in%20(newbie%2C%20starter)"> [...]
+
+
+ </div>
+
+</article>
+
+ </div>
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<footer class="footer">
+ <div class="footer__contained">
+ <div class="footer__cols">
+ <div class="footer__cols__col">
+ <div class="footer__cols__col__logo">
+ <img src="/images/beam_logo_circle.svg" class="footer__logo" alt="Beam logo">
+ </div>
+ <div class="footer__cols__col__logo">
+ <img src="/images/apache_logo_circle.svg" class="footer__logo" alt="Apache logo">
+ </div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Start</div>
+ <div class="footer__cols__col__link"><a href="/get-started/beam-overview/">Overview</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-java/">Quickstart (Java)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-py/">Quickstart (Python)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-go/">Quickstart (Go)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/downloads/">Downloads</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Docs</div>
+ <div class="footer__cols__col__link"><a href="/documentation/programming-guide/">Concepts</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/pipelines/design-your-pipeline/">Pipelines</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/runners/capability-matrix/">Runners</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Community</div>
+ <div class="footer__cols__col__link"><a href="/contribute/">Contribute</a></div>
+ <div class="footer__cols__col__link"><a href="https://projects.apache.org/committee.html?beam" target="_blank">Team<img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a></div>
+ <div class="footer__cols__col__link"><a href="/contribute/presentation-materials/">Media</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Resources</div>
+ <div class="footer__cols__col__link"><a href="/blog/">Blog</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/support/">Support</a></div>
+ <div class="footer__cols__col__link"><a href="https://github.com/apache/beam">GitHub</a></div>
+ </div>
+ </div>
+ </div>
+ <div class="footer__bottom">
+ ©
+ <a href="http://www.apache.org">The Apache Software Foundation</a>
+ | <a href="/privacy_policy">Privacy Policy</a>
+ | <a href="/feed.xml">RSS Feed</a>
+ <br><br>
+ Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are
+ either registered trademarks or trademarks of The Apache Software
+ Foundation. All other products or name brands are trademarks of their
+ respective holders, including The Apache Software Foundation.
+ </div>
+</footer>
+
+ </body>
+</html>
diff --git a/website/generated-content/blog/2016/10/20/test-stream.html b/website/generated-content/blog/2016/10/20/test-stream.html
new file mode 100644
index 0000000..8b0566d
--- /dev/null
+++ b/website/generated-content/blog/2016/10/20/test-stream.html
@@ -0,0 +1,535 @@
+<!DOCTYPE html>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<html lang="en">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <title>Testing Unbounded Pipelines in Apache Beam</title>
+ <meta name="description" content="The Beam Programming Model unifies writing pipelines for Batch and Streamingpipelines. We’ve recently introduced a new PTransform to write tests forpipelines...">
+ <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
+ <link rel="stylesheet" href="/css/site.css">
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <script src="/js/language-switch.js"></script>
+ <script src="/js/fix-menu.js"></script>
+ <script src="/js/section-nav.js"></script>
+ <script src="/js/page-nav.js"></script>
+ <link rel="canonical" href="https://beam.apache.org/blog/2016/10/20/test-stream.html" data-proofer-ignore>
+ <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
+ <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+ ga('create', 'UA-73650088-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+</head>
+
+ <body class="body ">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<nav class="header navbar navbar-fixed-top">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+
+ <a href="/" class="navbar-brand" >
+ <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
+ </a>
+ </div>
+
+ <div class="navbar-mask closed"></div>
+
+ <div id="navbar" class="navbar-container closed">
+ <ul class="nav navbar-nav">
+ <li>
+ <a href="/get-started/beam-overview/">Get Started</a>
+ </li>
+ <li>
+ <a href="/documentation/">Documentation</a>
+ </li>
+ <li>
+ <a href="/documentation/sdks/java/">SDKS</a>
+ </li>
+ <li>
+ <a href="/documentation/runners/capability-matrix/">RUNNERS</a>
+ </li>
+ <li>
+ <a href="/contribute/">Contribute</a>
+ </li>
+ <li>
+ <a href="/community/contact-us/">Community</a>
+ </li>
+ <li><a href="/blog">Blog</a></li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:20px;"><span class="caret"></span></a>
+ <ul class="dropdown-menu dropdown-menu-right">
+ <li><a href="http://www.apache.org/">ASF Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">License</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
+ </ul>
+ </li>
+ </ul>
+ </div>
+</nav>
+
+ <div class="body__contained">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+
+
+<article class="post" itemscope itemtype="http://schema.org/BlogPosting">
+
+ <header class="post-header">
+ <h1 class="post-title" itemprop="name headline">Testing Unbounded Pipelines in Apache Beam</h1>
+ <p class="post-meta"><time datetime="2016-10-20T11:00:00-07:00" itemprop="datePublished">Oct 20, 2016</time> •
+ Thomas Groh
+
+ </p>
+ </header>
+
+ <div class="post-content" itemprop="articleBody">
+ <!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+<p>The Beam Programming Model unifies writing pipelines for Batch and Streaming
+pipelines. We’ve recently introduced a new PTransform to write tests for
+pipelines that will be run over unbounded datasets and must handle out-of-order
+and delayed data.
+<!--more--></p>
+
+<p>Watermarks, Windows and Triggers form a core part of the Beam programming model
+– they respectively determine how your data are grouped, when your input is
+complete, and when to produce results. This is true for all pipelines,
+regardless of if they are processing bounded or unbounded inputs. If you’re not
+familiar with watermarks, windowing, and triggering in the Beam model,
+<a href="https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101">Streaming 101</a>
+and <a href="https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102">Streaming 102</a>
+are an excellent place to get started. A key takeaway from
+these articles: in realistic streaming scenarios with intermittent failures and
+disconnected users, data can arrive out of order or be delayed. Beam’s
+primitives provide a way for users to perform useful, powerful, and correct
+computations in spite of these challenges.</p>
+
+<p>As Beam pipeline authors, we need comprehensive tests that cover crucial
+failure scenarios and corner cases to gain real confidence that a pipeline is
+ready for production. The existing testing infrastructure within the Beam SDKs
+permits tests to be written which examine the contents of a Pipeline at
+execution time. However, writing unit tests for pipelines that may receive
+late data or trigger multiple times has historically ranged from complex to
+not possible, as pipelines that read from unbounded sources do not shut down
+without external intervention, while pipelines that read from bounded sources
+exclusively cannot test behavior with late data nor most speculative triggers.
+Without additional tools, pipelines that use custom triggers and handle
+out-of-order data could not be easily tested.</p>
+
+<p>This blog post introduces our new framework for writing tests for pipelines that
+handle delayed and out-of-order data in the context of the LeaderBoard pipeline
+from the Mobile Gaming example series.</p>
+
+<h2 id="leaderboard-and-the-mobile-gaming-example">LeaderBoard and the Mobile Gaming Example</h2>
+
+<p><a href="https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java#L177">LeaderBoard</a>
+is part of the <a href="https://github.com/apache/beam/tree/master/examples/java/src/main/java/org/apache/beam/examples/complete/game">Beam mobile gaming examples</a>
+(and <a href="/get-started/mobile-gaming-example/">walkthroughs</a>)
+which produces a continuous accounting of user and team scores. User scores are
+calculated over the lifetime of the program, while team scores are calculated
+within fixed windows with a default duration of one hour. The LeaderBoard
+pipeline produces speculative and late panes as appropriate, based on the
+configured triggering and allowed lateness of the pipeline. The expected outputs
+of the LeaderBoard pipeline vary depending on when elements arrive in relation
+to the watermark and the progress of processing time, which could not previously
+be controlled within a test.</p>
+
+<h2 id="writing-deterministic-tests-to-emulate-nondeterminism">Writing Deterministic Tests to Emulate Nondeterminism</h2>
+
+<p>The Beam testing infrastructure provides the
+<a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/testing/PAssert.html">PAssert</a>
+methods, which assert properties about the contents of a PCollection from within
+a pipeline. We have expanded this infrastructure to include
+<a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestStream.java">TestStream</a>,
+which is a PTransform that performs a series of events, consisting of adding
+additional elements to a pipeline, advancing the watermark of the TestStream,
+and advancing the pipeline processing time clock. TestStream permits tests which
+observe the effects of triggers on the output a pipeline produces.</p>
+
+<p>While executing a pipeline that reads from a TestStream, the read waits for all
+of the consequences of each event to complete before continuing on to the next
+event, ensuring that when processing time advances, triggers that are based on
+processing time fire as appropriate. With this transform, the effect of
+triggering and allowed lateness can be observed on a pipeline, including
+reactions to speculative and late panes and dropped data.</p>
+
+<h2 id="element-timings">Element Timings</h2>
+
+<p>Elements arrive either before, with, or after the watermark, which categorizes
+them into the “early”, “on-time”, and “late” divisions. “Late” elements can be
+further subdivided into “unobservably”, “observably”, and “droppably” late,
+depending on the window to which they are assigned and the maximum allowed
+lateness, as specified by the windowing strategy. Elements that arrive with
+these timings are emitted into panes, which can be “EARLY”, “ON-TIME”, or
+“LATE”, depending on the position of the watermark when the pane was emitted.</p>
+
+<p>Using TestStream, we can write tests that demonstrate that speculative panes are
+output after their trigger condition is met, that the advancing of the watermark
+causes the on-time pane to be produced, and that late-arriving data produces
+refinements when it arrives before the maximum allowed lateness, and is dropped
+after.</p>
+
+<p>The following examples demonstrate how you can use TestStream to provide a
+sequence of events to the Pipeline, where the arrival of elements is interspersed
+with updates to the watermark and the advance of processing time. Each of these
+events runs to completion before additional events occur.</p>
+
+<p>In the diagrams, the time at which events occurred in “real” (event) time
+progresses as the graph moves to the right. The time at which the pipeline
+receives them progresses as the graph goes upwards. The watermark is represented
+by the squiggly red line, and each starburst is the firing of a trigger and the
+associated pane.</p>
+
+<p><img class="center-block" src="/images/blog/test-stream/elements-all-on-time.png" alt="Elements on the Event and Processing time axes, with the Watermark and produced panes" width="442" /></p>
+
+<h3 id="everything-arrives-on-time">Everything arrives on-time</h3>
+
+<p>For example, if we create a TestStream where all the data arrives before the
+watermark and provide the result PCollection as input to the CalculateTeamScores
+PTransform:</p>
+
+<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">TestStream</span><span class="o"><</span><span class="n">GameActionInfo</span><span class="o">></span> <span class="n">infos</span> <span class="o">=</span> <span class="n">TestStream</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">AvroCoder</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">GameAc [...]
+ <span class="o">.</span><span class="na">addElements</span><span class="o">(</span><span class="k">new</span> <span class="n">GameActionInfo</span><span class="o">(</span><span class="s">"sky"</span><span class="o">,</span> <span class="s">"blue"</span><span class="o">,</span> <span class="mi">12</span><span class="o">,</span> <span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="mi">0L</span><span class="o">)),</span>
+ <span class="err"> </span><span class="k">new</span> <span class="n">GameActionInfo</span><span class="o">(</span><span class="s">"navy"</span><span class="o">,</span> <span class="s">"blue"</span><span class="o">,</span> <span class="mi">3</span><span class="o">,</span> <span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="mi">0L</span><span class="o">)),</span>
+ <span class="err"> </span><span class="k">new</span> <span class="n">GameActionInfo</span><span class="o">(</span><span class="s">"navy"</span><span class="o">,</span> <span class="s">"blue"</span><span class="o">,</span> <span class="mi">3</span><span class="o">,</span> <span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="mi">0L</span><span class="o">).</span><span class="na">plus</span><span class="o">(</span><span class="n">Dura [...]
+ <span class="err"> </span><span class="c1">// Move the watermark past the end the end of the window</span>
+ <span class="o">.</span><span class="na">advanceWatermarkTo</span><span class="o">(</span><span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="mi">0L</span><span class="o">).</span><span class="na">plus</span><span class="o">(</span><span class="n">TEAM_WINDOW_DURATION</span><span class="o">)</span>
+ <span class="err"> </span> <span class="o">.</span><span class="na">plus</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="mi">1</span><span class="o">)))</span>
+ <span class="o">.</span><span class="na">advanceWatermarkToInfinity</span><span class="o">();</span>
+
+<span class="n">PCollection</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">>></span> <span class="n">teamScores</span> <span class="o">=</span> <span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">createEvents</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="k">new</span> <span class="n">CalculateTeamScores</span><span class="o">(</span><span class="n">TEAM_WINDOW_DURATION</span><span class="o">,</span> <span class="n">ALLOWED_LATENESS</span><span class="o">));</span>
+</code></pre>
+</div>
+
+<p>we can then assert that the result PCollection contains elements that arrived:</p>
+
+<p><img class="center-block" src="/images/blog/test-stream/elements-all-on-time.png" alt="Elements all arrive before the watermark, and are produced in the on-time pane" width="442" /></p>
+
+<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="c1">// Only one value is emitted for the blue team</span>
+<span class="n">PAssert</span><span class="o">.</span><span class="na">that</span><span class="o">(</span><span class="n">teamScores</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">inWindow</span><span class="o">(</span><span class="n">window</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">containsInAnyOrder</span><span class="o">(</span><span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">"blue"</span><span class="o">,</span> <span class="mi">18</span><span class="o">));</span>
+<span class="n">p</span><span class="o">.</span><span class="na">run</span><span class="o">();</span>
+</code></pre>
+</div>
+
+<h3 id="some-elements-are-late-but-arrive-before-the-end-of-the-window">Some elements are late, but arrive before the end of the window</h3>
+
+<p>We can also add data to the TestStream after the watermark, but before the end
+of the window (shown below to the left of the red watermark), which demonstrates
+“unobservably late” data - that is, data that arrives late, but is promoted by
+the system to be on time, as it arrives before the watermark passes the end of
+the window</p>
+
+<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">TestStream</span><span class="o"><</span><span class="n">GameActionInfo</span><span class="o">></span> <span class="n">infos</span> <span class="o">=</span> <span class="n">TestStream</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">AvroCoder</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">GameAc [...]
+ <span class="o">.</span><span class="na">addElements</span><span class="o">(</span><span class="k">new</span> <span class="n">GameActionInfo</span><span class="o">(</span><span class="s">"sky"</span><span class="o">,</span> <span class="s">"blue"</span><span class="o">,</span> <span class="mi">3</span><span class="o">,</span> <span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="mi">0L</span><span class="o">)),</span>
+ <span class="err"> </span> <span class="k">new</span> <span class="n">GameActionInfo</span><span class="o">(</span><span class="s">"navy"</span><span class="o">,</span> <span class="s">"blue"</span><span class="o">,</span> <span class="mi">3</span><span class="o">,</span> <span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="mi">0L</span><span class="o">).</span><span class="na">plus</span><span class="o">(</span><span class="n">Dura [...]
+ <span class="err"> </span><span class="c1">// Move the watermark up to "near" the end of the window</span>
+ <span class="o">.</span><span class="na">advanceWatermarkTo</span><span class="o">(</span><span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="mi">0L</span><span class="o">).</span><span class="na">plus</span><span class="o">(</span><span class="n">TEAM_WINDOW_DURATION</span><span class="o">)</span>
+ <span class="err"> </span> <span class="err"> </span><span class="o">.</span><span class="na">minus</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="mi">1</span><span class="o">)))</span>
+ <span class="o">.</span><span class="na">addElements</span><span class="o">(</span><span class="k">new</span> <span class="n">GameActionInfo</span><span class="o">(</span><span class="s">"sky"</span><span class="o">,</span> <span class="s">"blue"</span><span class="o">,</span> <span class="mi">12</span><span class="o">,</span> <span class="n">Duration</span><span class="o">.</span><span class="na">ZERO</span><span class="o">))</span>
+ <span class="o">.</span><span class="na">advanceWatermarkToInfinity</span><span class="o">();</span>
+
+<span class="n">PCollection</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">>></span> <span class="n">teamScores</span> <span class="o">=</span> <span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">createEvents</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="k">new</span> <span class="n">CalculateTeamScores</span><span class="o">(</span><span class="n">TEAM_WINDOW_DURATION</span><span class="o">,</span> <span class="n">ALLOWED_LATENESS</span><span class="o">));</span>
+</code></pre>
+</div>
+
+<p><img class="center-block" src="/images/blog/test-stream/elements-unobservably-late.png" alt="An element arrives late, but before the watermark passes the end of the window, and is produced in the on-time pane" width="442" /></p>
+
+<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="c1">// Only one value is emitted for the blue team</span>
+<span class="n">PAssert</span><span class="o">.</span><span class="na">that</span><span class="o">(</span><span class="n">teamScores</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">inWindow</span><span class="o">(</span><span class="n">window</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">containsInAnyOrder</span><span class="o">(</span><span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">"blue"</span><span class="o">,</span> <span class="mi">18</span><span class="o">));</span>
+<span class="n">p</span><span class="o">.</span><span class="na">run</span><span class="o">();</span>
+</code></pre>
+</div>
+
+<h3 id="elements-are-late-and-arrive-after-the-end-of-the-window">Elements are late, and arrive after the end of the window</h3>
+
+<p>By advancing the watermark farther in time before adding the late data, we can
+demonstrate the triggering behavior that causes the system to emit an on-time
+pane, and then after the late data arrives, a pane that refines the result.</p>
+
+<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">TestStream</span><span class="o"><</span><span class="n">GameActionInfo</span><span class="o">></span> <span class="n">infos</span> <span class="o">=</span> <span class="n">TestStream</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">AvroCoder</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">GameAc [...]
+ <span class="o">.</span><span class="na">addElements</span><span class="o">(</span><span class="k">new</span> <span class="n">GameActionInfo</span><span class="o">(</span><span class="s">"sky"</span><span class="o">,</span> <span class="s">"blue"</span><span class="o">,</span> <span class="mi">3</span><span class="o">,</span> <span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="mi">0L</span><span class="o">)),</span>
+ <span class="err"> </span><span class="k">new</span> <span class="n">GameActionInfo</span><span class="o">(</span><span class="s">"navy"</span><span class="o">,</span> <span class="s">"blue"</span><span class="o">,</span> <span class="mi">3</span><span class="o">,</span> <span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="mi">0L</span><span class="o">).</span><span class="na">plus</span><span class="o">(</span><span class="n">Dura [...]
+<span class="err"> </span> <span class="err"> </span><span class="c1">// Move the watermark up to "near" the end of the window</span>
+ <span class="o">.</span><span class="na">advanceWatermarkTo</span><span class="o">(</span><span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="mi">0L</span><span class="o">).</span><span class="na">plus</span><span class="o">(</span><span class="n">TEAM_WINDOW_DURATION</span><span class="o">)</span>
+<span class="err"> </span> <span class="o">.</span><span class="na">minus</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="mi">1</span><span class="o">)))</span>
+ <span class="o">.</span><span class="na">addElements</span><span class="o">(</span><span class="k">new</span> <span class="n">GameActionInfo</span><span class="o">(</span><span class="s">"sky"</span><span class="o">,</span> <span class="s">"blue"</span><span class="o">,</span> <span class="mi">12</span><span class="o">,</span> <span class="n">Duration</span><span class="o">.</span><span class="na">ZERO</span><span class="o">))</span>
+ <span class="o">.</span><span class="na">advanceWatermarkToInfinity</span><span class="o">();</span>
+
+<span class="n">PCollection</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">>></span> <span class="n">teamScores</span> <span class="o">=</span> <span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">createEvents</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="k">new</span> <span class="n">CalculateTeamScores</span><span class="o">(</span><span class="n">TEAM_WINDOW_DURATION</span><span class="o">,</span> <span class="n">ALLOWED_LATENESS</span><span class="o">));</span>
+</code></pre>
+</div>
+
+<p><img class="center-block" src="/images/blog/test-stream/elements-observably-late.png" alt="Elements all arrive before the watermark, and are produced in the on-time pane" width="442" /></p>
+
+<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="c1">// An on-time pane is emitted with the events that arrived before the window closed</span>
+<span class="n">PAssert</span><span class="o">.</span><span class="na">that</span><span class="o">(</span><span class="n">teamScores</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">inOnTimePane</span><span class="o">(</span><span class="n">window</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">containsInAnyOrder</span><span class="o">(</span><span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">"blue"</span><span class="o">,</span> <span class="mi">6</span><span class="o">));</span>
+<span class="c1">// The final pane contains the late refinement</span>
+<span class="n">PAssert</span><span class="o">.</span><span class="na">that</span><span class="o">(</span><span class="n">teamScores</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">inFinalPane</span><span class="o">(</span><span class="n">window</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">containsInAnyOrder</span><span class="o">(</span><span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">"blue"</span><span class="o">,</span> <span class="mi">18</span><span class="o">));</span>
+<span class="n">p</span><span class="o">.</span><span class="na">run</span><span class="o">();</span>
+</code></pre>
+</div>
+
+<h3 id="elements-are-late-and-after-the-end-of-the-window-plus-the-allowed-lateness">Elements are late, and after the end of the window plus the allowed lateness</h3>
+
+<p>If we push the watermark even further into the future, beyond the maximum
+configured allowed lateness, we can demonstrate that the late element is dropped
+by the system.</p>
+
+<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">TestStream</span><span class="o"><</span><span class="n">GameActionInfo</span><span class="o">></span> <span class="n">infos</span> <span class="o">=</span> <span class="n">TestStream</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">AvroCoder</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">GameAc [...]
+ <span class="o">.</span><span class="na">addElements</span><span class="o">(</span><span class="k">new</span> <span class="n">GameActionInfo</span><span class="o">(</span><span class="s">"sky"</span><span class="o">,</span> <span class="s">"blue"</span><span class="o">,</span> <span class="mi">3</span><span class="o">,</span> <span class="n">Duration</span><span class="o">.</span><span class="na">ZERO</span><span class="o">),</span>
+ <span class="err"> </span><span class="k">new</span> <span class="n">GameActionInfo</span><span class="o">(</span><span class="s">"navy"</span><span class="o">,</span> <span class="s">"blue"</span><span class="o">,</span> <span class="mi">3</span><span class="o">,</span> <span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="mi">3</span><span class="o">)))</span>
+<span class="err"> </span><span class="c1">// Move the watermark up to "near" the end of the window</span>
+ <span class="o">.</span><span class="na">advanceWatermarkTo</span><span class="o">(</span><span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="mi">0</span><span class="o">).</span><span class="na">plus</span><span class="o">(</span><span class="n">TEAM_WINDOW_DURATION</span><span class="o">)</span>
+ <span class="err"> </span><span class="o">.</span><span class="na">plus</span><span class="o">(</span><span class="n">ALLOWED_LATENESS</span><span class="o">)</span>
+ <span class="err"> </span><span class="o">.</span><span class="na">plus</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="mi">1</span><span class="o">)))</span>
+ <span class="o">.</span><span class="na">addElements</span><span class="o">(</span><span class="k">new</span> <span class="n">GameActionInfo</span><span class="o">(</span>
+<span class="err"> </span><span class="s">"sky"</span><span class="o">,</span>
+<span class="err"> </span><span class="s">"blue"</span><span class="o">,</span>
+<span class="err"> </span><span class="mi">12</span><span class="o">,</span>
+<span class="err"> </span><span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="mi">0</span><span class="o">).</span><span class="na">plus</span><span class="o">(</span><span class="n">TEAM_WINDOW_DURATION</span><span class="o">).</span><span class="na">minus</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="mi">1</span [...]
+ <span class="o">.</span><span class="na">advanceWatermarkToInfinity</span><span class="o">();</span>
+
+<span class="n">PCollection</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">>></span> <span class="n">teamScores</span> <span class="o">=</span> <span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">createEvents</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="k">new</span> <span class="n">CalculateTeamScores</span><span class="o">(</span><span class="n">TEAM_WINDOW_DURATION</span><span class="o">,</span> <span class="n">ALLOWED_LATENESS</span><span class="o">));</span>
+</code></pre>
+</div>
+
+<p><img class="center-block" src="/images/blog/test-stream/elements-droppably-late.png" alt="Elements all arrive before the watermark, and are produced in the on-time pane" width="442" /></p>
+
+<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="c1">// An on-time pane is emitted with the events that arrived before the window closed</span>
+<span class="n">PAssert</span><span class="o">.</span><span class="na">that</span><span class="o">(</span><span class="n">teamScores</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">inWindow</span><span class="o">(</span><span class="n">window</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">containsInAnyOrder</span><span class="o">(</span><span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">"blue"</span><span class="o">,</span> <span class="mi">6</span><span class="o">));</span>
+
+<span class="n">p</span><span class="o">.</span><span class="na">run</span><span class="o">();</span>
+</code></pre>
+</div>
+
+<h3 id="elements-arrive-before-the-end-of-the-window-and-some-processing-time-passes">Elements arrive before the end of the window, and some processing time passes</h3>
+<p>Using additional methods, we can demonstrate the behavior of speculative
+triggers by advancing the processing time of the TestStream. If we add elements
+to an input PCollection, occasionally advancing the processing time clock, and
+apply <code class="highlighter-rouge">CalculateUserScores</code></p>
+
+<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">TestStream</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">AvroCoder</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">GameActionInfo</span><span class="o">.</span><span class="na">class</span><span class="o">))</span>
+ <span class="err"> </span><span class="o">.</span><span class="na">addElements</span><span class="o">(</span><span class="k">new</span> <span class="n">GameActionInfo</span><span class="o">(</span><span class="s">"scarlet"</span><span class="o">,</span> <span class="s">"red"</span><span class="o">,</span> <span class="mi">3</span><span class="o">,</span> <span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="mi">0L</span><span class="o">)),</span>
+ <span class="err"> </span> <span class="k">new</span> <span class="n">GameActionInfo</span><span class="o">(</span><span class="s">"scarlet"</span><span class="o">,</span> <span class="s">"red"</span><span class="o">,</span> <span class="mi">2</span><span class="o">,</span> <span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="mi">0L</span><span class="o">).</span><span class="na">plus</span><span class="o">(</span><span class="n">Du [...]
+ <span class="o">.</span><span class="na">advanceProcessingTime</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="mi">12</span><span class="o">))</span>
+ <span class="err"> </span><span class="o">.</span><span class="na">addElements</span><span class="o">(</span><span class="k">new</span> <span class="n">GameActionInfo</span><span class="o">(</span><span class="s">"oxblood"</span><span class="o">,</span> <span class="s">"red"</span><span class="o">,</span> <span class="mi">2</span><span class="o">,</span> <span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="mi">0L</span><span class="o">)).</span [...]
+ <span class="err"> </span> <span class="k">new</span> <span class="n">GameActionInfo</span><span class="o">(</span><span class="s">"scarlet"</span><span class="o">,</span> <span class="s">"red"</span><span class="o">,</span> <span class="mi">4</span><span class="o">,</span> <span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="mi">0L</span><span class="o">).</span><span class="na">plus</span><span class="o">(</span><span class="n">Du [...]
+ <span class="o">.</span><span class="na">advanceProcessingTime</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="mi">15</span><span class="o">))</span>
+ <span class="o">.</span><span class="na">advanceWatermarkToInfinity</span><span class="o">();</span>
+
+<span class="n">PCollection</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">>></span> <span class="n">userScores</span> <span class="o">=</span>
+ <span class="err"> </span><span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">infos</span><span class="o">).</span><span class="na">apply</span><span class="o">(</span><span class="k">new</span> <span class="n">CalculateUserScores</span><span class="o">(</span><span class="n">ALLOWED_LATENESS</span><span class="o">));</span>
+</code></pre>
+</div>
+
+<p><img class="center-block" src="/images/blog/test-stream/elements-processing-speculative.png" alt="Elements all arrive before the watermark, and are produced in the on-time pane" width="442" /></p>
+
+<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">PAssert</span><span class="o">.</span><span class="na">that</span><span class="o">(</span><span class="n">userScores</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">inEarlyGlobalWindowPanes</span><span class="o">()</span>
+ <span class="o">.</span><span class="na">containsInAnyOrder</span><span class="o">(</span><span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">"scarlet"</span><span class="o">,</span> <span class="mi">5</span><span class="o">),</span>
+ <span class="err"> </span><span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">"scarlet"</span><span class="o">,</span> <span class="mi">9</span><span class="o">),</span>
+ <span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">"oxblood"</span><span class="o">,</span> <span class="mi">2</span><span class="o">));</span>
+
+<span class="n">p</span><span class="o">.</span><span class="na">run</span><span class="o">();</span>
+</code></pre>
+</div>
+
+<h2 id="teststream---under-the-hood">TestStream - Under the Hood</h2>
+
+<p>TestStream relies on a pipeline concept we’ve introduced, called quiescence, to
+utilize the existing runner infrastructure while providing guarantees about when
+a root transform will called by the runner. This consists of properties about
+pending elements and triggers, namely:</p>
+
+<ul>
+ <li>No trigger is permitted to fire but has not fired</li>
+ <li>All elements are either buffered in state or cannot progress until a side input becomes available</li>
+</ul>
+
+<p>Simplified, this means that, in the absence of an advancement in input
+watermarks or processing time, or additional elements being added to the
+pipeline, the pipeline will not make progress. Whenever the TestStream PTransform
+performs an action, the runner must not reinvoke the same instance until the
+pipeline has quiesced. This ensures that the events specified by TestStream
+happen “in-order”, which ensures that input watermarks and the system clock do
+not advance ahead of the elements they hoped to hold up.</p>
+
+<p>The DirectRunner has been modified to use quiescence as the signal that it
+should add more work to the Pipeline, and the implementation of TestStream in
+that runner uses this fact to perform a single output per event. The DirectRunner
+implementation also directly controls the runner’s system clock, ensuring that
+tests will complete promptly even if there is a multi-minute processing time
+trigger located within the pipeline.</p>
+
+<p>The TestStream transform is supported in the DirectRunner. For most users, tests
+written using TestPipeline and PAsserts will automatically function while using
+TestStream.</p>
+
+<h2 id="summary">Summary</h2>
+
+<p>The addition of TestStream alongside window and pane-specific matchers in PAssert
+has enabled the testing of Pipelines which produce speculative and late panes.
+This permits tests for all styles of pipeline to be expressed directly within the
+Java SDK. If you have questions or comments, we’d love to hear them on the
+<a href="/get-started/support/">mailing lists</a>.</p>
+
+ </div>
+
+</article>
+
+ </div>
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<footer class="footer">
+ <div class="footer__contained">
+ <div class="footer__cols">
+ <div class="footer__cols__col">
+ <div class="footer__cols__col__logo">
+ <img src="/images/beam_logo_circle.svg" class="footer__logo" alt="Beam logo">
+ </div>
+ <div class="footer__cols__col__logo">
+ <img src="/images/apache_logo_circle.svg" class="footer__logo" alt="Apache logo">
+ </div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Start</div>
+ <div class="footer__cols__col__link"><a href="/get-started/beam-overview/">Overview</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-java/">Quickstart (Java)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-py/">Quickstart (Python)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-go/">Quickstart (Go)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/downloads/">Downloads</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Docs</div>
+ <div class="footer__cols__col__link"><a href="/documentation/programming-guide/">Concepts</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/pipelines/design-your-pipeline/">Pipelines</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/runners/capability-matrix/">Runners</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Community</div>
+ <div class="footer__cols__col__link"><a href="/contribute/">Contribute</a></div>
+ <div class="footer__cols__col__link"><a href="https://projects.apache.org/committee.html?beam" target="_blank">Team<img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a></div>
+ <div class="footer__cols__col__link"><a href="/contribute/presentation-materials/">Media</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Resources</div>
+ <div class="footer__cols__col__link"><a href="/blog/">Blog</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/support/">Support</a></div>
+ <div class="footer__cols__col__link"><a href="https://github.com/apache/beam">GitHub</a></div>
+ </div>
+ </div>
+ </div>
+ <div class="footer__bottom">
+ ©
+ <a href="http://www.apache.org">The Apache Software Foundation</a>
+ | <a href="/privacy_policy">Privacy Policy</a>
+ | <a href="/feed.xml">RSS Feed</a>
+ <br><br>
+ Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are
+ either registered trademarks or trademarks of The Apache Software
+ Foundation. All other products or name brands are trademarks of their
+ respective holders, including The Apache Software Foundation.
+ </div>
+</footer>
+
+ </body>
+</html>
diff --git a/website/generated-content/blog/2017/01/09/added-apex-runner.html b/website/generated-content/blog/2017/01/09/added-apex-runner.html
new file mode 100644
index 0000000..97a5d75
--- /dev/null
+++ b/website/generated-content/blog/2017/01/09/added-apex-runner.html
@@ -0,0 +1,263 @@
+<!DOCTYPE html>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<html lang="en">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <title>Release 0.4.0 adds a runner for Apache Apex</title>
+ <meta name="description" content="The latest release 0.4.0 of Apache Beam adds a new runner for Apache Apex. We are excited to reach this initial milestone and are looking forward to continue...">
+ <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
+ <link rel="stylesheet" href="/css/site.css">
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <script src="/js/language-switch.js"></script>
+ <script src="/js/fix-menu.js"></script>
+ <script src="/js/section-nav.js"></script>
+ <script src="/js/page-nav.js"></script>
+ <link rel="canonical" href="https://beam.apache.org/blog/2017/01/09/added-apex-runner.html" data-proofer-ignore>
+ <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
+ <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+ ga('create', 'UA-73650088-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+</head>
+
+ <body class="body ">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<nav class="header navbar navbar-fixed-top">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+
+ <a href="/" class="navbar-brand" >
+ <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
+ </a>
+ </div>
+
+ <div class="navbar-mask closed"></div>
+
+ <div id="navbar" class="navbar-container closed">
+ <ul class="nav navbar-nav">
+ <li>
+ <a href="/get-started/beam-overview/">Get Started</a>
+ </li>
+ <li>
+ <a href="/documentation/">Documentation</a>
+ </li>
+ <li>
+ <a href="/documentation/sdks/java/">SDKS</a>
+ </li>
+ <li>
+ <a href="/documentation/runners/capability-matrix/">RUNNERS</a>
+ </li>
+ <li>
+ <a href="/contribute/">Contribute</a>
+ </li>
+ <li>
+ <a href="/community/contact-us/">Community</a>
+ </li>
+ <li><a href="/blog">Blog</a></li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:20px;"><span class="caret"></span></a>
+ <ul class="dropdown-menu dropdown-menu-right">
+ <li><a href="http://www.apache.org/">ASF Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">License</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
+ </ul>
+ </li>
+ </ul>
+ </div>
+</nav>
+
+ <div class="body__contained">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+
+
+<article class="post" itemscope itemtype="http://schema.org/BlogPosting">
+
+ <header class="post-header">
+ <h1 class="post-title" itemprop="name headline">Release 0.4.0 adds a runner for Apache Apex</h1>
+ <p class="post-meta"><time datetime="2017-01-09T09:00:01-08:00" itemprop="datePublished">Jan 9, 2017</time> •
+ Thomas Weise [<a href="https://twitter.com/thweise">@thweise</a>]
+
+ </p>
+ </header>
+
+ <div class="post-content" itemprop="articleBody">
+ <!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+<p>The latest release 0.4.0 of <a href="/">Apache Beam</a> adds a new runner for <a href="http://apex.apache.org/">Apache Apex</a>. We are excited to reach this initial milestone and are looking forward to continued collaboration between the Beam and Apex communities to advance the runner.</p>
+
+<!--more-->
+
+<p>Beam evolved from the Google Dataflow SDK and as incubator project has quickly adapted the Apache way, grown the community and attracts increasing interest from users that hope to benefit from a conceptual strong unified programming model that is portable between different big data processing frameworks (see <a href="https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101">Streaming-101</a> and <a href="https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102">Str [...]
+
+<p>Apex is a stream processing framework for low-latency, high-throughput, stateful and reliable processing of complex analytics pipelines on clusters. Apex was developed since 2012 and is used in production by large companies for real-time and batch processing at scale.</p>
+
+<p>The initial revision of the runner was focussed on broad coverage of the Beam model on a functional level. That means, there will be follow up work in several areas to take the runner from functional to scalable and high performance to match the capabilities of Apex and its native API. The runner capabilities matrix shows that the Apex capabilities are well aligned with the Beam model. Specifically, the ability to track computational state in a fault tolerant and efficient manner is n [...]
+
+<h2 id="stateful-stream-processor">Stateful Stream Processor</h2>
+
+<p>Apex was built as stateful stream processor from the ground up. Operators <a href="https://www.datatorrent.com/blog/blog-introduction-to-checkpoint/">checkpoint</a> state in a distributed and asynchronous manner that produces a consistent snapshot for the entire processing graph, which can be used for recovery. Apex also supports such recovery in an incremental, or fine grained, manner. This means only the portion of the DAG that is actually affected by a failure will be recovered whi [...]
+
+<h2 id="translation-to-apex-dag">Translation to Apex DAG</h2>
+
+<p>A Beam runner needs to implement the translation from the Beam model to the underlying frameworks execution model. In the case of Apex, the runner will translate the pipeline into the <a href="https://www.datatorrent.com/blog/tracing-dags-from-specification-to-execution/">native (compositional, low level) DAG API</a> (which is also the base for a number of other API that are available to specify applications that run on Apex). The DAG consists of operators (functional building blocks [...]
+
+<h2 id="execution-and-testing">Execution and Testing</h2>
+
+<p>In this release, the Apex runner executes the pipelines in embedded mode, where, similar to the direct runner, everything is executed in a single JVM. See <a href="/get-started/quickstart/">quickstart</a> on how to run the Beam examples with the Apex runner.</p>
+
+<p>Embedded mode is useful for development and debugging. Apex in production runs distributed on Apache Hadoop YARN clusters. An example how a Beam pipeline can be embedded into an Apex application package to run on YARN can be found <a href="https://github.com/tweise/apex-samples/tree/master/beam-apex-wordcount">here</a> and support for direct launch in the runner is currently being worked on.</p>
+
+<p>The Beam project has a strong focus on development process and tooling, including testing. For the runners, there is a comprehensive test suite with more than 200 integration tests that are executed against each runner to ensure they don’t break as changes are made. The tests cover the capabilities of the matrix and thus are a measure of completeness and correctness of the runner implementations. The suite was very helpful when developing the Apex runner.</p>
+
+<h2 id="outlook">Outlook</h2>
+
+<p>The next step is to take the Apex runner from functional to ready for real applications that run distributed, leveraging the scalability and performance features of Apex, similar to its native API. This includes chaining of ParDos, partitioning, optimizing combine operations etc. To get involved, please see <a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20and%20component%20%3D%20runner-apex%20and%20resolution%20%3D%20unresolved">JIRA</a> and join the Beam com [...]
+
+ </div>
+
+</article>
+
+ </div>
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<footer class="footer">
+ <div class="footer__contained">
+ <div class="footer__cols">
+ <div class="footer__cols__col">
+ <div class="footer__cols__col__logo">
+ <img src="/images/beam_logo_circle.svg" class="footer__logo" alt="Beam logo">
+ </div>
+ <div class="footer__cols__col__logo">
+ <img src="/images/apache_logo_circle.svg" class="footer__logo" alt="Apache logo">
+ </div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Start</div>
+ <div class="footer__cols__col__link"><a href="/get-started/beam-overview/">Overview</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-java/">Quickstart (Java)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-py/">Quickstart (Python)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-go/">Quickstart (Go)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/downloads/">Downloads</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Docs</div>
+ <div class="footer__cols__col__link"><a href="/documentation/programming-guide/">Concepts</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/pipelines/design-your-pipeline/">Pipelines</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/runners/capability-matrix/">Runners</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Community</div>
+ <div class="footer__cols__col__link"><a href="/contribute/">Contribute</a></div>
+ <div class="footer__cols__col__link"><a href="https://projects.apache.org/committee.html?beam" target="_blank">Team<img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a></div>
+ <div class="footer__cols__col__link"><a href="/contribute/presentation-materials/">Media</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Resources</div>
+ <div class="footer__cols__col__link"><a href="/blog/">Blog</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/support/">Support</a></div>
+ <div class="footer__cols__col__link"><a href="https://github.com/apache/beam">GitHub</a></div>
+ </div>
+ </div>
+ </div>
+ <div class="footer__bottom">
+ ©
+ <a href="http://www.apache.org">The Apache Software Foundation</a>
+ | <a href="/privacy_policy">Privacy Policy</a>
+ | <a href="/feed.xml">RSS Feed</a>
+ <br><br>
+ Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are
+ either registered trademarks or trademarks of The Apache Software
+ Foundation. All other products or name brands are trademarks of their
+ respective holders, including The Apache Software Foundation.
+ </div>
+</footer>
+
+ </body>
+</html>
diff --git a/website/generated-content/blog/2017/01/10/beam-graduates.html b/website/generated-content/blog/2017/01/10/beam-graduates.html
new file mode 100644
index 0000000..fff8666
--- /dev/null
+++ b/website/generated-content/blog/2017/01/10/beam-graduates.html
@@ -0,0 +1,289 @@
+<!DOCTYPE html>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<html lang="en">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <title>Apache Beam established as a new top-level project</title>
+ <meta name="description" content="Today, the Apache Software Foundation announcedthat Apache Beam has successfully graduated from incubation, becoming a newTop-Level Project at the foundation...">
+ <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
+ <link rel="stylesheet" href="/css/site.css">
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <script src="/js/language-switch.js"></script>
+ <script src="/js/fix-menu.js"></script>
+ <script src="/js/section-nav.js"></script>
+ <script src="/js/page-nav.js"></script>
+ <link rel="canonical" href="https://beam.apache.org/blog/2017/01/10/beam-graduates.html" data-proofer-ignore>
+ <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
+ <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+ ga('create', 'UA-73650088-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+</head>
+
+ <body class="body ">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<nav class="header navbar navbar-fixed-top">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+
+ <a href="/" class="navbar-brand" >
+ <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
+ </a>
+ </div>
+
+ <div class="navbar-mask closed"></div>
+
+ <div id="navbar" class="navbar-container closed">
+ <ul class="nav navbar-nav">
+ <li>
+ <a href="/get-started/beam-overview/">Get Started</a>
+ </li>
+ <li>
+ <a href="/documentation/">Documentation</a>
+ </li>
+ <li>
+ <a href="/documentation/sdks/java/">SDKS</a>
+ </li>
+ <li>
+ <a href="/documentation/runners/capability-matrix/">RUNNERS</a>
+ </li>
+ <li>
+ <a href="/contribute/">Contribute</a>
+ </li>
+ <li>
+ <a href="/community/contact-us/">Community</a>
+ </li>
+ <li><a href="/blog">Blog</a></li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:20px;"><span class="caret"></span></a>
+ <ul class="dropdown-menu dropdown-menu-right">
+ <li><a href="http://www.apache.org/">ASF Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">License</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
+ </ul>
+ </li>
+ </ul>
+ </div>
+</nav>
+
+ <div class="body__contained">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+
+
+<article class="post" itemscope itemtype="http://schema.org/BlogPosting">
+
+ <header class="post-header">
+ <h1 class="post-title" itemprop="name headline">Apache Beam established as a new top-level project</h1>
+ <p class="post-meta"><time datetime="2017-01-10T00:00:01-08:00" itemprop="datePublished">Jan 10, 2017</time> •
+ Davor Bonaci [<a href="https://twitter.com/BonaciDavor">@BonaciDavor</a>]
+
+ </p>
+ </header>
+
+ <div class="post-content" itemprop="articleBody">
+ <!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+<p>Today, the Apache Software Foundation <a href="https://blogs.apache.org/foundation/entry/the-apache-software-foundation-announces">announced</a>
+that Apache Beam has successfully graduated from incubation, becoming a new
+Top-Level Project at the foundation and signifying that its “community and
+products have been well-governed under the foundation’s meritocratic process
+and principles”.</p>
+
+<!--more-->
+
+<p>Graduation is an exciting milestone for Apache Beam. Becoming a top-level
+project is a recognition of the amazing growth of the Apache Beam community,
+both in terms of size and diversity. It is fantastic to see ever-increasing
+participation on our development and user mailing lists, in pull requests,
+and at conferences and meetups. Contributions of additional runners and IO
+connectors further show the significant interest from other projects and
+organizations in being part of our growing community.</p>
+
+<p>Although graduation is a statement about community, our technical progress has
+also been impressive. We started with code donations from several companies
+that were tailored with a specific vendor and scenario in mind and had fragile
+component boundaries. The whole community worked extremely hard over the past
+year to refactor the codebase and documentation into a truly vendor-neutral and
+extensible framework. We have established an impressive engineering system and
+processes that promote test-driven development. This concerted focus on quality
+and execution excellence enabled us to publish four releases, each significantly
+better than the previous.</p>
+
+<p>Going forward, we will continue to extend the core abstractions to distill
+additional complex data processing patterns into intuitive APIs, and, at the
+same time, enhance the ability to interconnect additional storage/messaging
+systems and execution engines. Together, we are excited to push forward the
+state of the art in distributed data processing.</p>
+
+<p>This is also an opportunity to thank those who helped and supported us reaching
+this milestone. None of this would be possible without the Apache Software
+Foundation and its amazing volunteers who have wholeheartedly welcomed us into
+the family. In particular, we’d like to thank the Apache Incubator community
+for their guidance producing releases and following processes. We are especially
+thankful to the Apache Infra team for their assistance establishing Beam’s
+engineering system, particularly in the areas of testing infrastructure and
+development productivity.</p>
+
+<p>On behalf of the whole Apache Beam community, thank you to our incubation
+mentors Ted Dunning, Venkatesh Seetharam, and, previously, Bertrand Delacretaz
+and Jim Jagielski, for teaching us the Apache Way and guiding us through
+incubation. In particular, special thanks goes to our incubation champion and
+mentor Jean-Baptiste Onofré, whose extraordinary help, day after day, was
+instrumental in getting us here, and who continues as an active code contributor
+and a member of the project management committee.</p>
+
+<p>Please consider joining us, whether as a user or a contributor, as we work
+towards our first release with API stability. If you’d like to try out Apache
+Beam today, check out the latest
+<a href="/get-started/downloads/">0.4.0 release</a>. We welcome
+contribution and participation from anyone through our mailing lists, issue
+tracker, pull requests, and events.</p>
+
+ </div>
+
+</article>
+
+ </div>
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<footer class="footer">
+ <div class="footer__contained">
+ <div class="footer__cols">
+ <div class="footer__cols__col">
+ <div class="footer__cols__col__logo">
+ <img src="/images/beam_logo_circle.svg" class="footer__logo" alt="Beam logo">
+ </div>
+ <div class="footer__cols__col__logo">
+ <img src="/images/apache_logo_circle.svg" class="footer__logo" alt="Apache logo">
+ </div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Start</div>
+ <div class="footer__cols__col__link"><a href="/get-started/beam-overview/">Overview</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-java/">Quickstart (Java)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-py/">Quickstart (Python)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-go/">Quickstart (Go)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/downloads/">Downloads</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Docs</div>
+ <div class="footer__cols__col__link"><a href="/documentation/programming-guide/">Concepts</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/pipelines/design-your-pipeline/">Pipelines</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/runners/capability-matrix/">Runners</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Community</div>
+ <div class="footer__cols__col__link"><a href="/contribute/">Contribute</a></div>
+ <div class="footer__cols__col__link"><a href="https://projects.apache.org/committee.html?beam" target="_blank">Team<img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a></div>
+ <div class="footer__cols__col__link"><a href="/contribute/presentation-materials/">Media</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Resources</div>
+ <div class="footer__cols__col__link"><a href="/blog/">Blog</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/support/">Support</a></div>
+ <div class="footer__cols__col__link"><a href="https://github.com/apache/beam">GitHub</a></div>
+ </div>
+ </div>
+ </div>
+ <div class="footer__bottom">
+ ©
+ <a href="http://www.apache.org">The Apache Software Foundation</a>
+ | <a href="/privacy_policy">Privacy Policy</a>
+ | <a href="/feed.xml">RSS Feed</a>
+ <br><br>
+ Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are
+ either registered trademarks or trademarks of The Apache Software
+ Foundation. All other products or name brands are trademarks of their
+ respective holders, including The Apache Software Foundation.
+ </div>
+</footer>
+
+ </body>
+</html>
diff --git a/website/generated-content/blog/2017/02/01/graduation-media-recap.html b/website/generated-content/blog/2017/02/01/graduation-media-recap.html
new file mode 100644
index 0000000..d4bd442
--- /dev/null
+++ b/website/generated-content/blog/2017/02/01/graduation-media-recap.html
@@ -0,0 +1,280 @@
+<!DOCTYPE html>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<html lang="en">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <title>Media recap of the Apache Beam graduation</title>
+ <meta name="description" content="One year ago today Apache Beam was accepted into incubation at the ApacheSoftware Foundation. The community’s work over the past year culminated, justover th...">
+ <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
+ <link rel="stylesheet" href="/css/site.css">
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <script src="/js/language-switch.js"></script>
+ <script src="/js/fix-menu.js"></script>
+ <script src="/js/section-nav.js"></script>
+ <script src="/js/page-nav.js"></script>
+ <link rel="canonical" href="https://beam.apache.org/blog/2017/02/01/graduation-media-recap.html" data-proofer-ignore>
+ <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
+ <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+ ga('create', 'UA-73650088-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+</head>
+
+ <body class="body ">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<nav class="header navbar navbar-fixed-top">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+
+ <a href="/" class="navbar-brand" >
+ <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
+ </a>
+ </div>
+
+ <div class="navbar-mask closed"></div>
+
+ <div id="navbar" class="navbar-container closed">
+ <ul class="nav navbar-nav">
+ <li>
+ <a href="/get-started/beam-overview/">Get Started</a>
+ </li>
+ <li>
+ <a href="/documentation/">Documentation</a>
+ </li>
+ <li>
+ <a href="/documentation/sdks/java/">SDKS</a>
+ </li>
+ <li>
+ <a href="/documentation/runners/capability-matrix/">RUNNERS</a>
+ </li>
+ <li>
+ <a href="/contribute/">Contribute</a>
+ </li>
+ <li>
+ <a href="/community/contact-us/">Community</a>
+ </li>
+ <li><a href="/blog">Blog</a></li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:20px;"><span class="caret"></span></a>
+ <ul class="dropdown-menu dropdown-menu-right">
+ <li><a href="http://www.apache.org/">ASF Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">License</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
+ </ul>
+ </li>
+ </ul>
+ </div>
+</nav>
+
+ <div class="body__contained">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+
+
+<article class="post" itemscope itemtype="http://schema.org/BlogPosting">
+
+ <header class="post-header">
+ <h1 class="post-title" itemprop="name headline">Media recap of the Apache Beam graduation</h1>
+ <p class="post-meta"><time datetime="2017-02-01T00:00:01-08:00" itemprop="datePublished">Feb 1, 2017</time> •
+ Davor Bonaci [<a href="https://twitter.com/BonaciDavor">@BonaciDavor</a>]
+
+ </p>
+ </header>
+
+ <div class="post-content" itemprop="articleBody">
+ <!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+<p>One year ago today Apache Beam was accepted into incubation at the Apache
+Software Foundation. The community’s work over the past year culminated, just
+over three weeks ago, with an <a href="/blog/2017/01/10/beam-graduates.html">announcement</a>
+that Apache Beam has successfully graduated as a new Top-Level Project at the
+foundation. Graduation sparked an additional interest in the project, from
+corporate endorsements, news articles, interviews, to the volume of traffic to
+our website and mailing lists.</p>
+
+<!--more-->
+
+<p>Corporate endorsements include Google, PayPal, Talend, data Artisans, and
+others. You can read more in the following blog posts:</p>
+<ul>
+ <li>Google: “<a href="https://opensource.googleblog.com/2017/01/apache-beam-graduates.html">Apache Beam graduates to a top-level project</a>” by Tyler Akidau.</li>
+ <li>Talend: “<a href="https://www.talend.com/blog/2017/01/13/future-apache-beam-now-top-level-apache-software-foundation-project/">The Future of Apache Beam, Now a Top-Level Apache Software Foundation Project</a>” by Jean-Baptiste Onofré.</li>
+ <li>Talend: “<a href="https://www.talend.com/blog/2017/01/23/apache-beam-way-greater-data-agility/?utm_medium=socialpost&utm_source=twitter&utm_campaign=blog">Apache Beam Your Way to Greater Data Agility</a>” by Shane Kent.</li>
+ <li>Google: “<a href="https://cloud.google.com/blog/big-data/2017/01/apache-beam-graduates-from-incubation-try-it-today-on-google-cloud-dataflow">Apache Beam graduates from incubation: Try it today on Google Cloud Dataflow</a>” by Frances Perry.</li>
+</ul>
+
+<p>News coverage started with the Apache Software Foundation’s press release in
+<a href="https://globenewswire.com/news-release/2017/01/10/904692/0/en/The-Apache-Software-Foundation-Announces-Apache-Beam-as-a-Top-Level-Project.html">Nasdaq GlobeNewswire</a>,
+and followed by coverage in many independent outlets. Some of those in English
+include:</p>
+<ul>
+ <li>ZDNet: “<a href="http://www.zdnet.com/article/apache-beam-and-spark-new-coopetition-for-squashing-the-lambda-architecture/">Apache Beam and Spark: New coopetition for squashing the Lambda Architecture?</a>” by Tony Baer.</li>
+ <li>Datanami: “<a href="https://www.datanami.com/2017/01/10/google-lauds-outside-influence-apache-beam/">Google Lauds Outside Influence on Apache Beam</a>” by Alex Woodie.</li>
+ <li>InfoWorld / JavaWorld: “<a href="http://www.infoworld.com/article/3156598/big-data/apache-beam-unifies-batch-and-streaming-for-big-data.html">Apache Beam unifies batch and streaming for big data</a>” by Serdar Yegulalp, and republished in <a href="http://www.javaworld.com/article/3156598/big-data/apache-beam-unifies-batch-and-streaming-for-big-data.html">JavaWorld</a>.</li>
+ <li>JAXenter: “<a href="https://jaxenter.com/apache-beam-interview-131314.html">In a way, Apache Beam is the glue that connects many big data systems together</a>” by Kypriani Sinaris.</li>
+ <li>OStatic: “Apache Beam Unifies Batch and Streaming Data Processing” by Sam Dean. <!-- http://ostatic.com/blog/apache-beam-unifies-batch-and-streaming-data-processing --></li>
+ <li>Enterprise Apps Today: “<a href="http://www.enterpriseappstoday.com/business-intelligence/data-analytics/apache-beam-graduates-to-help-define-streaming-data-processing.html">Apache Beam Graduates to Help Define Streaming Data Processing</a>” by Sean Michael Kerner.</li>
+ <li>The Register: “<a href="http://www.theregister.co.uk/2017/01/10/google_must_be_ibeamiing_as_apache_announces_its_new_top_level_projects/">Google must be Beaming as Apache announces its new top-level projects</a>” by Alexander J. Martin.</li>
+ <li>SiliconANGLE: “<a href="http://siliconangle.com/blog/2017/01/11/apache-software-foundation-announces-2-top-level-projects/">Apache Software Foundation announces two more top-level open source projects</a>” by Mike Wheatley.</li>
+ <li>SD Times: “<a href="http://sdtimes.com/apache-beam-goes-top-level/">Apache Beam goes top level</a>” by Alex Handy.</li>
+</ul>
+
+<p>Graduation and media coverage helped push Beam website traffic to record levels.
+The website traffic, measured in unique sessions per hour, peaked at more than
+15 times above the previous week’s numbers. In a steady state, the traffic is
+several times larger than before graduation.</p>
+
+<p>Hopefully these perspectives entice you to join us on this exciting ride, either
+as a user or a contributor, as we work towards our first release with API
+stability. If you’d like to try out Apache Beam today, check out the latest
+<a href="/get-started/downloads/">0.4.0 release</a>. We welcome
+contribution and participation from anyone through our mailing lists, issue
+tracker, pull requests, and events.</p>
+
+ </div>
+
+</article>
+
+ </div>
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<footer class="footer">
+ <div class="footer__contained">
+ <div class="footer__cols">
+ <div class="footer__cols__col">
+ <div class="footer__cols__col__logo">
+ <img src="/images/beam_logo_circle.svg" class="footer__logo" alt="Beam logo">
+ </div>
+ <div class="footer__cols__col__logo">
+ <img src="/images/apache_logo_circle.svg" class="footer__logo" alt="Apache logo">
+ </div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Start</div>
+ <div class="footer__cols__col__link"><a href="/get-started/beam-overview/">Overview</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-java/">Quickstart (Java)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-py/">Quickstart (Python)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-go/">Quickstart (Go)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/downloads/">Downloads</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Docs</div>
+ <div class="footer__cols__col__link"><a href="/documentation/programming-guide/">Concepts</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/pipelines/design-your-pipeline/">Pipelines</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/runners/capability-matrix/">Runners</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Community</div>
+ <div class="footer__cols__col__link"><a href="/contribute/">Contribute</a></div>
+ <div class="footer__cols__col__link"><a href="https://projects.apache.org/committee.html?beam" target="_blank">Team<img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a></div>
+ <div class="footer__cols__col__link"><a href="/contribute/presentation-materials/">Media</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Resources</div>
+ <div class="footer__cols__col__link"><a href="/blog/">Blog</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/support/">Support</a></div>
+ <div class="footer__cols__col__link"><a href="https://github.com/apache/beam">GitHub</a></div>
+ </div>
+ </div>
+ </div>
+ <div class="footer__bottom">
+ ©
+ <a href="http://www.apache.org">The Apache Software Foundation</a>
+ | <a href="/privacy_policy">Privacy Policy</a>
+ | <a href="/feed.xml">RSS Feed</a>
+ <br><br>
+ Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are
+ either registered trademarks or trademarks of The Apache Software
+ Foundation. All other products or name brands are trademarks of their
+ respective holders, including The Apache Software Foundation.
+ </div>
+</footer>
+
+ </body>
+</html>
diff --git a/website/generated-content/blog/2017/02/13/stateful-processing.html b/website/generated-content/blog/2017/02/13/stateful-processing.html
new file mode 100644
index 0000000..cf15ace
--- /dev/null
+++ b/website/generated-content/blog/2017/02/13/stateful-processing.html
@@ -0,0 +1,803 @@
+<!DOCTYPE html>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<html lang="en">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <title>Stateful processing with Apache Beam</title>
+ <meta name="description" content="Beam lets you process unbounded, out-of-order, global-scale data with portablehigh-level pipelines. Stateful processing is a new feature of the Beam modeltha...">
+ <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
+ <link rel="stylesheet" href="/css/site.css">
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <script src="/js/language-switch.js"></script>
+ <script src="/js/fix-menu.js"></script>
+ <script src="/js/section-nav.js"></script>
+ <script src="/js/page-nav.js"></script>
+ <link rel="canonical" href="https://beam.apache.org/blog/2017/02/13/stateful-processing.html" data-proofer-ignore>
+ <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
+ <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+ ga('create', 'UA-73650088-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+</head>
+
+ <body class="body ">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<nav class="header navbar navbar-fixed-top">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+
+ <a href="/" class="navbar-brand" >
+ <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
+ </a>
+ </div>
+
+ <div class="navbar-mask closed"></div>
+
+ <div id="navbar" class="navbar-container closed">
+ <ul class="nav navbar-nav">
+ <li>
+ <a href="/get-started/beam-overview/">Get Started</a>
+ </li>
+ <li>
+ <a href="/documentation/">Documentation</a>
+ </li>
+ <li>
+ <a href="/documentation/sdks/java/">SDKS</a>
+ </li>
+ <li>
+ <a href="/documentation/runners/capability-matrix/">RUNNERS</a>
+ </li>
+ <li>
+ <a href="/contribute/">Contribute</a>
+ </li>
+ <li>
+ <a href="/community/contact-us/">Community</a>
+ </li>
+ <li><a href="/blog">Blog</a></li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:20px;"><span class="caret"></span></a>
+ <ul class="dropdown-menu dropdown-menu-right">
+ <li><a href="http://www.apache.org/">ASF Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">License</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
+ </ul>
+ </li>
+ </ul>
+ </div>
+</nav>
+
+ <div class="body__contained">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+
+
+<article class="post" itemscope itemtype="http://schema.org/BlogPosting">
+
+ <header class="post-header">
+ <h1 class="post-title" itemprop="name headline">Stateful processing with Apache Beam</h1>
+ <p class="post-meta"><time datetime="2017-02-13T00:00:01-08:00" itemprop="datePublished">Feb 13, 2017</time> •
+ Kenneth Knowles [<a href="https://twitter.com/KennKnowles">@KennKnowles</a>]
+
+ </p>
+ </header>
+
+ <div class="post-content" itemprop="articleBody">
+ <!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+<p>Beam lets you process unbounded, out-of-order, global-scale data with portable
+high-level pipelines. Stateful processing is a new feature of the Beam model
+that expands the capabilities of Beam, unlocking new use cases and new
+efficiencies. In this post, I will guide you through stateful processing in
+Beam: how it works, how it fits in with the other features of the Beam model,
+what you might use it for, and what it looks like in code.</p>
+
+<!--more-->
+
+<blockquote>
+ <p><strong>Warning: new features ahead!</strong>: This is a very new aspect of the Beam
+model. Runners are still adding support. You can try it out today on multiple
+runners, but do check the <a href="/documentation/runners/capability-matrix/">runner capability
+matrix</a> for
+the current status in each runner.</p>
+</blockquote>
+
+<p>First, a quick recap: In Beam, a big data processing <em>pipeline</em> is a directed,
+acyclic graph of parallel operations called <em><code class="highlighter-rouge">PTransforms</code></em> processing data
+from <em><code class="highlighter-rouge">PCollections</code></em>. I’ll expand on that by walking through this illustration:</p>
+
+<p><img class="center-block" src="/images/blog/stateful-processing/pipeline.png" alt="A Beam Pipeline - PTransforms are boxes - PCollections are arrows" width="300" /></p>
+
+<p>The boxes are <code class="highlighter-rouge">PTransforms</code> and the edges represent the data in <code class="highlighter-rouge">PCollections</code>
+flowing from one <code class="highlighter-rouge">PTransform</code> to the next. A <code class="highlighter-rouge">PCollection</code> may be <em>bounded</em> (which
+means it is finite and you know it) or <em>unbounded</em> (which means you don’t know if
+it is finite or not - basically, it is like an incoming stream of data that may
+or may not ever terminate). The cylinders are the data sources and sinks at the
+edges of your pipeline, such as bounded collections of log files or unbounded
+data streaming over a Kafka topic. This blog post isn’t about sources or sinks,
+but about what happens in between - your data processing.</p>
+
+<p>There are two main building blocks for processing your data in Beam: <em><code class="highlighter-rouge">ParDo</code></em>,
+for performing an operation in parallel across all elements, and <em><code class="highlighter-rouge">GroupByKey</code></em>
+(and the closely related <code class="highlighter-rouge">CombinePerKey</code> that I will talk about quite soon)
+for aggregating elements to which you have assigned the same key. In the
+picture below (featured in many of our presentations) the color indicates the
+key of the element. Thus the <code class="highlighter-rouge">GroupByKey</code>/<code class="highlighter-rouge">CombinePerKey</code> transform gathers all the
+green squares to produce a single output element.</p>
+
+<p><img class="center-block" src="/images/blog/stateful-processing/pardo-and-gbk.png" alt="ParDo and GroupByKey/CombinePerKey: Elementwise versus aggregating computations" width="400" /></p>
+
+<p>But not all use cases are easily expressed as pipelines of simple <code class="highlighter-rouge">ParDo</code>/<code class="highlighter-rouge">Map</code> and
+<code class="highlighter-rouge">GroupByKey</code>/<code class="highlighter-rouge">CombinePerKey</code> transforms. The topic of this blog post is a new
+extension to the Beam programming model: <strong>per-element operation augmented with
+mutable state</strong>.</p>
+
+<p><img class="center-block" src="/images/blog/stateful-processing/stateful-pardo.png" alt="Stateful ParDo - sequential per-key processing with persistent state" width="300" /></p>
+
+<p>In the illustration above, ParDo now has a bit of durable, consistent state on
+the side, which can be read and written during the processing of each element.
+The state is partitioned by key, so it is drawn as having disjoint sections for
+each color. It is also partitioned per window, but I thought plaid
+<img src="/images/blog/stateful-processing/plaid.png" alt="A plaid storage cylinder" width="20" />
+would be a bit much :-). I’ll talk about
+why state is partitioned this way a bit later, via my first example.</p>
+
+<p>For the rest of this post, I will describe this new feature of Beam in detail -
+how it works at a high level, how it differs from existing features, how to
+make sure it is still massively scalable. After that introduction at the model
+level, I’ll walk through a simple example of how you use it in the Beam Java
+SDK.</p>
+
+<h2 id="how-does-stateful-processing-in-beam-work">How does stateful processing in Beam work?</h2>
+
+<p>The processing logic of your <code class="highlighter-rouge">ParDo</code> transform is expressed through the <code class="highlighter-rouge">DoFn</code>
+that it applies to each element. Without stateful augmentations, a <code class="highlighter-rouge">DoFn</code> is a
+mostly-pure function from inputs to one or more outputs, corresponding to the
+Mapper in a MapReduce. With state, a <code class="highlighter-rouge">DoFn</code> has the ability to access
+persistent mutable state while processing each input element. Consider this
+illustration:</p>
+
+<p><img class="center-block" src="/images/blog/stateful-processing/stateful-dofn.png" alt="Stateful DoFn - the runner controls input but the DoFn controls storage and output" width="300" /></p>
+
+<p>The first thing to note is that all the data - the little squares, circles, and
+triangles - are red. This is to illustrate that stateful processing occurs in
+the context of a single key - all of the elements are key-value pairs with the
+same key. Calls from your chosen Beam runner to the <code class="highlighter-rouge">DoFn</code> are colored in
+yellow, while calls from the <code class="highlighter-rouge">DoFn</code> to the runner are in purple:</p>
+
+<ul>
+ <li>The runner invokes the <code class="highlighter-rouge">DoFn</code>’s <code class="highlighter-rouge">@ProcessElement</code> method on each element for a
+key+window.</li>
+ <li>The <code class="highlighter-rouge">DoFn</code> reads and writes state - the curved arrows to/from the storage on
+the side.</li>
+ <li>The <code class="highlighter-rouge">DoFn</code> emits output (or side output) to the runner as usual via
+<code class="highlighter-rouge">ProcessContext.output</code> (resp. <code class="highlighter-rouge">ProcessContext.sideOutput</code>).</li>
+</ul>
+
+<p>At this very high level, it is pretty intuitive: In your programming
+experience, you have probably at some point written a loop over elements that
+updates some mutable variables while performing other actions. The interesting
+question is how does this fit into the Beam model: how does it relate with
+other features? How does it scale, since state implies some synchronization?
+When should it be used versus other features?</p>
+
+<h2 id="how-does-stateful-processing-fit-into-the-beam-model">How does stateful processing fit into the Beam model?</h2>
+
+<p>To see where stateful processing fits in the Beam model, consider another
+way that you can keep some “state” while processing many elements: CombineFn. In
+Beam, you can write <code class="highlighter-rouge">Combine.perKey(CombineFn)</code> in Java or Python to apply an
+associative, commutative accumulating operation across all the elements with a
+common key (and window).</p>
+
+<p>Here is a diagram illustrating the basics of a <code class="highlighter-rouge">CombineFn</code>, the simplest way
+that a runner might invoke it on a per-key basis to build an accumulator and
+extract an output from the final accumulator:</p>
+
+<p><img class="center-block" src="/images/blog/stateful-processing/combinefn.png" alt="CombineFn - the runner controls input, storage, and output" width="300" /></p>
+
+<p>As with the illustration of stateful <code class="highlighter-rouge">DoFn</code>, all the data is colored red, since
+this is the processing of Combine for a single key. The illustrated method
+calls are colored yellow, since they are all controlled by the runner: The
+runner invokes <code class="highlighter-rouge">addInput</code> on each method to add it to the current accumulator.</p>
+
+<ul>
+ <li>The runner persists the accumulator when it chooses.</li>
+ <li>The runner calls <code class="highlighter-rouge">extractOutput</code> when ready to emit an output element.</li>
+</ul>
+
+<p>At this point, the diagram for <code class="highlighter-rouge">CombineFn</code> looks a whole lot like the diagram
+for stateful <code class="highlighter-rouge">DoFn</code>. In practice, the flow of data is, indeed, quite similar.
+But there are important differences, even so:</p>
+
+<ul>
+ <li>The runner controls all invocations and storage here. You do not decide when
+or how state is persisted, when an accumulator is discarded (based on
+triggering) or when output is extracted from an accumulator.</li>
+ <li>You can only have one piece of state - the accumulator. In a stateful DoFn
+you can read only what you need to know and write only what has changed.</li>
+ <li>You don’t have the extended features of <code class="highlighter-rouge">DoFn</code>, such as multiple outputs per
+input or side outputs. (These could be simulated by a sufficient complex
+accumulator, but it would not be natural or efficient. Some other features of
+<code class="highlighter-rouge">DoFn</code> such as side inputs and access to the window make perfect sense for
+<code class="highlighter-rouge">CombineFn</code>)</li>
+</ul>
+
+<p>But the main thing that <code class="highlighter-rouge">CombineFn</code> allows a runner to do is to
+<code class="highlighter-rouge">mergeAccumulators</code>, the concrete expression of the <code class="highlighter-rouge">CombineFn</code>’s associativity.
+This unlocks some huge optimizations: the runner can invoke multiple instances
+of a <code class="highlighter-rouge">CombineFn</code> on a number of inputs and later combine them in a classic
+divide-and-conquer architecture, as in this picture:</p>
+
+<p><img class="center-block" src="/images/blog/stateful-processing/combiner-lifting.png" alt="Divide-and-conquer aggregation with a CombineFn" width="600" /></p>
+
+<p>The contract of a <code class="highlighter-rouge">CombineFn</code> is that the result should be exactly the same,
+whether or not the runner decides to actually do such a thing, or even more
+complex trees with hot-key fanout, etc.</p>
+
+<p>This merge operation is not (necessarily) provided by a stateful <code class="highlighter-rouge">DoFn</code>: the
+runner cannot freely branch its execution and recombine the states. Note that
+the input elements are still received in an arbitrary order, so the <code class="highlighter-rouge">DoFn</code> should
+be insensitive to ordering and bundling but it doesn’t mean the output must be
+exactly equal. (fun and easy fact: if the outputs are actually always equal,
+then the <code class="highlighter-rouge">DoFn</code> is an associative and commutative operator)</p>
+
+<p>So now you can see how a stateful <code class="highlighter-rouge">DoFn</code> differs from <code class="highlighter-rouge">CombineFn</code>, but I want to
+step back and extrapolate this to a high level picture of how state in Beam
+relates to using other features to achieve the same or similar goals: In a lot
+of cases, what stateful processing represents is a chance to “get under the
+hood” of the highly abstract mostly-deterministic functional paradigm of Beam
+and do potentially-nondeterministic imperative-style programming that is hard
+to express any other way.</p>
+
+<h2 id="example-arbitrary-but-consistent-index-assignment">Example: arbitrary-but-consistent index assignment</h2>
+
+<p>Suppose that you want to give an index to every incoming element for a
+key-and-window. You don’t care what the indices are, just as long as they are
+unique and consistent. Before diving into the code for how to do this in a Beam
+SDK, I’ll go over this example from the level of the model. In pictures, you
+want to write a transform that maps input to output like this:</p>
+
+<p><img class="center-block" src="/images/blog/stateful-processing/assign-indices.png" alt="Assigning arbitrary but unique indices to each element" width="180" /></p>
+
+<p>The order of the elements A, B, C, D, E is arbitrary, hence their assigned
+indices are arbitrary, but downstream transforms just need to be OK with this.
+There is no associativity or commutativity as far as the actual values are
+concerned. The order-insensitivity of this transform only extends to the point
+of ensuring the necessary properties of the output: no duplicated indices, no
+gaps, and every element gets an index.</p>
+
+<p>Conceptually expressing this as a stateful loop is as trivial as you can
+imagine: The state you should store is the next index.</p>
+
+<ul>
+ <li>As an element comes in, output it along with the next index.</li>
+ <li>Increment the index.</li>
+</ul>
+
+<p>This presents a good opportunity to talk about big data and parallelism,
+because the algorithm in those bullet points is not parallelizable at all! If
+you wanted to apply this logic over an entire <code class="highlighter-rouge">PCollection</code>, you would have to
+process each element of the <code class="highlighter-rouge">PCollection</code> one-at-a-time… this is obvious a
+bad idea. State in Beam is tightly scoped so that most of the time a stateful
+<code class="highlighter-rouge">ParDo</code> transform should still be possible for a runner to execute in parallel,
+though you still have to be thoughtful about it.</p>
+
+<p>A state cell in Beam is scoped to a key+window pair. When your DoFn reads or
+writes state by the name of <code class="highlighter-rouge">"index"</code>, it is actually accessing a mutable cell
+specified by <code class="highlighter-rouge">"index"</code> <em>along with</em> the key and window currently being
+processed. So, when thinking about a state cell, it may be helpful to consider
+the full state of your transform as a table, where the rows are named according
+to names you use in your program, like <code class="highlighter-rouge">"index"</code>, and the columns are
+key+window pairs, like this:</p>
+
+<table class="table">
+ <thead>
+ <tr>
+ <th> </th>
+ <th>(key, window)<sub>1</sub></th>
+ <th>(key, window)<sub>2</sub></th>
+ <th>(key, window)<sub>3</sub></th>
+ <th>…</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><code class="highlighter-rouge">"index"</code></td>
+ <td><code class="highlighter-rouge">3</code></td>
+ <td><code class="highlighter-rouge">7</code></td>
+ <td><code class="highlighter-rouge">15</code></td>
+ <td>…</td>
+ </tr>
+ <tr>
+ <td><code class="highlighter-rouge">"fizzOrBuzz?"</code></td>
+ <td><code class="highlighter-rouge">"fizz"</code></td>
+ <td><code class="highlighter-rouge">"7"</code></td>
+ <td><code class="highlighter-rouge">"fizzbuzz"</code></td>
+ <td>…</td>
+ </tr>
+ <tr>
+ <td>…</td>
+ <td>…</td>
+ <td>…</td>
+ <td>…</td>
+ <td>…</td>
+ </tr>
+ </tbody>
+</table>
+
+<p>(if you have a superb spatial sense, feel free to imagine this as a cube where
+keys and windows are independent dimensions)</p>
+
+<p>You can provide the opportunity for parallelism by making sure that table has
+enough columns. You might have many keys and many windows, or you might have
+many of just one or the other:</p>
+
+<ul>
+ <li>Many keys in few windows, for example a globally windowed stateful computation
+keyed by user ID.</li>
+ <li>Many windows over few keys, for example a fixed windowed stateful computation
+over a global key.</li>
+</ul>
+
+<p>Caveat: all Beam runners today parallelize only over the key.</p>
+
+<p>Most often your mental model of state can be focused on only a single column of
+the table, a single key+window pair. Cross-column interactions do not occur
+directly, by design.</p>
+
+<h2 id="state-in-beams-java-sdk">State in Beam’s Java SDK</h2>
+
+<p>Now that I have talked a bit about stateful processing in the Beam model and
+worked through an abstract example, I’d like to show you what it looks like to
+write stateful processing code using Beam’s Java SDK. Here is the code for a
+stateful <code class="highlighter-rouge">DoFn</code> that assigns an arbitrary-but-consistent index to each element
+on a per key-and-window basis:</p>
+
+<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="k">new</span> <span class="n">DoFn</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">MyKey</span><span class="o">,</span> <span class="n">MyValue</span><span class="o">>,</span> <span class="n">KV</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">KV</span><span class="o"><</span><span class="n [...]
+
+ <span class="c1">// A state cell holding a single Integer per key+window</span>
+ <span class="nd">@StateId</span><span class="o">(</span><span class="s">"index"</span><span class="o">)</span>
+ <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o"><</span><span class="n">ValueState</span><span class="o"><</span><span class="n">Integer</span><span class="o">>></span> <span class="n">indexSpec</span> <span class="o">=</span>
+ <span class="n">StateSpecs</span><span class="o">.</span><span class="na">value</span><span class="o">(</span><span class="n">VarIntCoder</span><span class="o">.</span><span class="na">of</span><span class="o">());</span>
+
+ <span class="nd">@ProcessElement</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span>
+ <span class="n">ProcessContext</span> <span class="n">context</span><span class="o">,</span>
+ <span class="nd">@StateId</span><span class="o">(</span><span class="s">"index"</span><span class="o">)</span> <span class="n">ValueState</span><span class="o"><</span><span class="n">Integer</span><span class="o">></span> <span class="n">index</span><span class="o">)</span> <span class="o">{</span>
+ <span class="kt">int</span> <span class="n">current</span> <span class="o">=</span> <span class="n">firstNonNull</span><span class="o">(</span><span class="n">index</span><span class="o">.</span><span class="na">read</span><span class="o">(),</span> <span class="mi">0</span><span class="o">);</span>
+ <span class="n">context</span><span class="o">.</span><span class="na">output</span><span class="o">(</span><span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">current</span><span class="o">,</span> <span class="n">context</span><span class="o">.</span><span class="na">element</span><span class="o">()));</span>
+ <span class="n">index</span><span class="o">.</span><span class="na">write</span><span class="o">(</span><span class="n">current</span><span class="o">+</span><span class="mi">1</span><span class="o">);</span>
+ <span class="o">}</span>
+<span class="o">}</span>
+</code></pre>
+</div>
+
+<div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="c"># State and timers are not yet supported in Beam's Python SDK.</span>
+<span class="c"># Watch this space!</span>
+</code></pre>
+</div>
+
+<p>Let’s dissect this:</p>
+
+<ul>
+ <li>The first thing to look at is the presence of a couple of <code class="highlighter-rouge">@StateId("index")</code>
+annotations. This calls out that you are using a mutable state cell named
+“index” in this <code class="highlighter-rouge">DoFn</code>. The Beam Java SDK, and from there your chosen runner,
+will also note these annotations and use them to wire up your DoFn correctly.</li>
+ <li>The first <code class="highlighter-rouge">@StateId("index")</code> is annotated on a field of type <code class="highlighter-rouge">StateSpec</code> (for
+“state specification”). This declares and configures the state cell. The
+type parameter <code class="highlighter-rouge">ValueState</code> describes the kind of state you can get out of this
+cell - <code class="highlighter-rouge">ValueState</code> stores just a single value. Note that the spec itself is not
+a usable state cell - you need the runner to provide that during pipeline
+execution.</li>
+ <li>To fully specify a <code class="highlighter-rouge">ValueState</code> cell, you need to provide the coder
+that the runner will use (as necessary) to serialize the value
+you will be storing. This is the invocation <code class="highlighter-rouge">StateSpecs.value(VarIntCoder.of())</code>.</li>
+ <li>The second <code class="highlighter-rouge">@StateId("index")</code> annotation is on a parameter to your
+<code class="highlighter-rouge">@ProcessElement</code> method. This indicates access to the ValueState cell that
+was specified earlier.</li>
+ <li>The state is accessed in the simplest way: <code class="highlighter-rouge">read()</code> to read it, and
+<code class="highlighter-rouge">write(newvalue)</code> to write it.</li>
+ <li>The other features of <code class="highlighter-rouge">DoFn</code> are available in the usual way - such as
+<code class="highlighter-rouge">context.output(...)</code>. You can also use side inputs, side outputs, gain access
+to the window, etc.</li>
+</ul>
+
+<p>A few notes on how the SDK and runners see this DoFn:</p>
+
+<ul>
+ <li>Your state cells are all explicitly declared so a Beam SDK or runner can
+reason about them, for example to clear them out when a window expires.</li>
+ <li>If you declare a state cell and then use it with the wrong type, the Beam
+Java SDK will catch that error for you.</li>
+ <li>If you declare two state cells with the same ID, the SDK will catch that,
+too.</li>
+ <li>The runner knows that this is a stateful <code class="highlighter-rouge">DoFn</code> and may run it quite
+differently, for example by additional data shuffling and synchronization in
+order to avoid concurrent access to state cells.</li>
+</ul>
+
+<p>Let’s look at one more example of how to use this API, this time a bit more real-world.</p>
+
+<h2 id="example-anomaly-detection">Example: anomaly detection</h2>
+
+<p>Suppose you are feeding a stream of actions by your user into some complex
+model to predict some quantitative expression of the sorts of actions they
+take, for example to detect fraudulent activity. You will build up the model
+from events, and also compare incoming events against the latest model to
+determine if something has changed.</p>
+
+<p>If you try to express the building of your model as a <code class="highlighter-rouge">CombineFn</code>, you may have
+trouble with <code class="highlighter-rouge">mergeAccumulators</code>. Assuming you could express that, it might
+look something like this:</p>
+
+<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">ModelFromEventsFn</span> <span class="kd">extends</span> <span class="n">CombineFn</span><span class="o"><</span><span class="n">Event</span><span class="o">,</span> <span class="n">Model</span><span class="o">,</span> <span class="n">Model</span><span class="o">></span> <span class="o">{</span>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kd">abstract</span> <span class="n">Model</span> <span class="nf">createAccumulator</span><span class="o">()</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="n">Model</span><span class="o">.</span><span class="na">empty</span><span class="o">();</span>
+ <span class="o">}</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kd">abstract</span> <span class="n">Model</span> <span class="nf">addInput</span><span class="o">(</span><span class="n">Model</span> <span class="n">accumulator</span><span class="o">,</span> <span class="n">Event</span> <span class="n">input</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="n">accumulator</span><span class="o">.</span><span class="na">update</span><span class="o">(</span><span class="n">input</span><span class="o">);</span> <span class="c1">// this is encouraged to mutate, for efficiency</span>
+ <span class="o">}</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kd">abstract</span> <span class="n">Model</span> <span class="nf">mergeAccumulators</span><span class="o">(</span><span class="n">Iterable</span><span class="o"><</span><span class="n">Model</span><span class="o">></span> <span class="n">accumulators</span><span class="o">)</span> <span class="o">{</span>
+ <span class="c1">// ?? can you write this ??</span>
+ <span class="o">}</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kd">abstract</span> <span class="n">Model</span> <span class="nf">extractOutput</span><span class="o">(</span><span class="n">Model</span> <span class="n">accumulator</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="n">accumulator</span><span class="o">;</span> <span class="o">}</span>
+<span class="o">}</span>
+</code></pre>
+</div>
+
+<div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="c"># State and timers are not yet supported in Beam's Python SDK.</span>
+<span class="c"># Watch this space!</span>
+</code></pre>
+</div>
+
+<p>Now you have a way to compute the model of a particular user for a window as
+<code class="highlighter-rouge">Combine.perKey(new ModelFromEventsFn())</code>. How would you apply this model to
+the same stream of events from which it is calculated? A standard way to do
+take the result of a <code class="highlighter-rouge">Combine</code> transform and use it while processing the
+elements of a <code class="highlighter-rouge">PCollection</code> is to read it as a side input to a <code class="highlighter-rouge">ParDo</code>
+transform. So you could side input the model and check the stream of events
+against it, outputting the prediction, like so:</p>
+
+<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">PCollection</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">UserId</span><span class="o">,</span> <span class="n">Event</span><span class="o">>></span> <span class="n">events</span> <span class="o">=</span> <span class="o">...</span>
+
+<span class="kd">final</span> <span class="n">PCollectionView</span><span class="o"><</span><span class="n">Map</span><span class="o"><</span><span class="n">UserId</span><span class="o">,</span> <span class="n">Model</span><span class="o">>></span> <span class="n">userModels</span> <span class="o">=</span> <span class="n">events</span>
+ <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Combine</span><span class="o">.</span><span class="na">perKey</span><span class="o">(</span><span class="k">new</span> <span class="n">ModelFromEventsFn</span><span class="o">()))</span>
+ <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">View</span><span class="o">.</span><span class="na">asMap</span><span class="o">());</span>
+
+<span class="n">PCollection</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">UserId</span><span class="o">,</span> <span class="n">Prediction</span><span class="o">>></span> <span class="n">predictions</span> <span class="o">=</span> <span class="n">events</span>
+ <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">UserId</span><span class="o">,</span> <span class="n">Event</span><span class="o">>>()</span> <span class="o">{</span>
+
+ <span class="nd">@ProcessElement</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span><span class="n">ProcessContext</span> <span class="n">ctx</span><span class="o">)</span> <span class="o">{</span>
+ <span class="n">UserId</span> <span class="n">userId</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="na">element</span><span class="o">().</span><span class="na">getKey</span><span class="o">();</span>
+ <span class="n">Event</span> <span class="n">event</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="na">element</span><span class="o">().</span><span class="na">getValue</span><span class="o">();</span>
+
+ <span class="n">Model</span> <span class="n">model</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="na">sideinput</span><span class="o">(</span><span class="n">userModels</span><span class="o">).</span><span class="na">get</span><span class="o">(</span><span class="n">userId</span><span class="o">);</span>
+
+ <span class="c1">// Perhaps some logic around when to output a new prediction</span>
+ <span class="err">…</span> <span class="n">c</span><span class="o">.</span><span class="na">output</span><span class="o">(</span><span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">userId</span><span class="o">,</span> <span class="n">model</span><span class="o">.</span><span class="na">prediction</span><span class="o">(</span><span class="n">event</span><span class="o">)))</span> <span class="err">…</span>
+ <span class="o">}</span>
+ <span class="o">}));</span>
+</code></pre>
+</div>
+
+<div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="c"># State and timers are not yet supported in Beam's Python SDK.</span>
+<span class="c"># Watch this space!</span>
+</code></pre>
+</div>
+
+<p>In this pipeline, there is just one model emitted by the <code class="highlighter-rouge">Combine.perKey(...)</code>
+per user, per window, which is then prepared for side input by the <code class="highlighter-rouge">View.asMap()</code>
+transform. The processing of the <code class="highlighter-rouge">ParDo</code> over events will block until that side
+input is ready, buffering events, and will then check each event against the
+model. This is a high latency, high completeness solution: The model takes into
+account all user behavior in the window, but there can be no output until the
+window is complete.</p>
+
+<p>Suppose you want to get some results earlier, or don’t even have any
+natural windowing, but just want continuous analysis with the “model so far”,
+even though your model may not be as complete. How can you control the updates
+to the model against which you are checking your events? Triggers are the
+generic Beam feature for managing completeness versus latency tradeoffs. So here
+is the same pipeline with an added trigger that outputs a new model one second
+after input arrives:</p>
+
+<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">PCollection</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">UserId</span><span class="o">,</span> <span class="n">Event</span><span class="o">>></span> <span class="n">events</span> <span class="o">=</span> <span class="o">...</span>
+
+<span class="n">PCollectionView</span><span class="o"><</span><span class="n">Map</span><span class="o"><</span><span class="n">UserId</span><span class="o">,</span> <span class="n">Model</span><span class="o">>></span> <span class="n">userModels</span> <span class="o">=</span> <span class="n">events</span>
+
+ <span class="c1">// A tradeoff between latency and cost</span>
+ <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Window</span><span class="o">.</span><span class="na">triggering</span><span class="o">(</span>
+ <span class="n">AfterProcessingTime</span><span class="o">.</span><span class="na">pastFirstElementInPane</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardSeconds</span><span class="o">(</span><span class="mi">1</span><span class="o">)))</span>
+
+ <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Combine</span><span class="o">.</span><span class="na">perKey</span><span class="o">(</span><span class="k">new</span> <span class="n">ModelFromEventsFn</span><span class="o">()))</span>
+ <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">View</span><span class="o">.</span><span class="na">asMap</span><span class="o">());</span>
+</code></pre>
+</div>
+
+<div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="c"># State and timers are not yet supported in Beam's Python SDK.</span>
+<span class="c"># Watch this space!</span>
+</code></pre>
+</div>
+
+<p>This is often a pretty nice tradeoff between latency and cost: If a huge flood
+of events comes in a second, then you will only emit one new model, so you
+won’t be flooded with model outputs that you cannot even use before they are
+obsolete. In practice, the new model may not be present on the side input
+channel until many more seconds have passed, due to caches and processing
+delays preparing the side input. Many events (maybe an entire batch of
+activity) will have passed through the <code class="highlighter-rouge">ParDo</code> and had their predictions
+calculated according to the prior model. If the runner gave a tight enough
+bound on cache expirations and you used a more aggressive trigger, you might be
+able to improve latency at additional cost.</p>
+
+<p>But there is another cost to consider: you are outputting many uninteresting
+outputs from the <code class="highlighter-rouge">ParDo</code> that will be processed downstream. If the
+“interestingness” of the output is only well-defined relative to the prior
+output, then you cannot use a <code class="highlighter-rouge">Filter</code> transform to reduce data volume downstream.</p>
+
+<p>Stateful processing lets you address both the latency problem of side inputs
+and the cost problem of excessive uninteresting output. Here is the code, using
+only features I have already introduced:</p>
+
+<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="k">new</span> <span class="n">DoFn</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">UserId</span><span class="o">,</span> <span class="n">Event</span><span class="o">>,</span> <span class="n">KV</span><span class="o"><</span><span class="n">UserId</span><span class="o">,</span> <span class="n">Prediction</span><span class="o">>>()</span> < [...]
+
+ <span class="nd">@StateId</span><span class="o">(</span><span class="s">"model"</span><span class="o">)</span>
+ <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o"><</span><span class="n">ValueState</span><span class="o"><</span><span class="n">Model</span><span class="o">>></span> <span class="n">modelSpec</span> <span class="o">=</span>
+ <span class="n">StateSpecs</span><span class="o">.</span><span class="na">value</span><span class="o">(</span><span class="n">Model</span><span class="o">.</span><span class="na">coder</span><span class="o">());</span>
+
+ <span class="nd">@StateId</span><span class="o">(</span><span class="s">"previousPrediction"</span><span class="o">)</span>
+ <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o"><</span><span class="n">ValueState</span><span class="o"><</span><span class="n">Prediction</span><span class="o">>></span> <span class="n">previousPredictionSpec</span> <span class="o">=</span>
+ <span class="n">StateSpecs</span><span class="o">.</span><span class="na">value</span><span class="o">(</span><span class="n">Prediction</span><span class="o">.</span><span class="na">coder</span><span class="o">());</span>
+
+ <span class="nd">@ProcessElement</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span>
+ <span class="n">ProcessContext</span> <span class="n">c</span><span class="o">,</span>
+ <span class="nd">@StateId</span><span class="o">(</span><span class="s">"previousPrediction"</span><span class="o">)</span> <span class="n">ValueState</span><span class="o"><</span><span class="n">Prediction</span><span class="o">></span> <span class="n">previousPredictionState</span><span class="o">,</span>
+ <span class="nd">@StateId</span><span class="o">(</span><span class="s">"model"</span><span class="o">)</span> <span class="n">ValueState</span><span class="o"><</span><span class="n">Model</span><span class="o">></span> <span class="n">modelState</span><span class="o">)</span> <span class="o">{</span>
+ <span class="n">UserId</span> <span class="n">userId</span> <span class="o">=</span> <span class="n">c</span><span class="o">.</span><span class="na">element</span><span class="o">().</span><span class="na">getKey</span><span class="o">();</span>
+ <span class="n">Event</span> <span class="n">event</span> <span class="o">=</span> <span class="n">c</span><span class="o">.</span><span class="na">element</span><span class="o">().</span><span class="na">getValue</span><span class="o">()</span>
+
+ <span class="n">Model</span> <span class="n">model</span> <span class="o">=</span> <span class="n">modelState</span><span class="o">.</span><span class="na">read</span><span class="o">();</span>
+ <span class="n">Prediction</span> <span class="n">previousPrediction</span> <span class="o">=</span> <span class="n">previousPredictionState</span><span class="o">.</span><span class="na">read</span><span class="o">();</span>
+ <span class="n">Prediction</span> <span class="n">newPrediction</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="na">prediction</span><span class="o">(</span><span class="n">event</span><span class="o">);</span>
+ <span class="n">model</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">event</span><span class="o">);</span>
+ <span class="n">modelState</span><span class="o">.</span><span class="na">write</span><span class="o">(</span><span class="n">model</span><span class="o">);</span>
+ <span class="k">if</span> <span class="o">(</span><span class="n">previousPrediction</span> <span class="o">==</span> <span class="kc">null</span>
+ <span class="o">||</span> <span class="n">shouldOutputNewPrediction</span><span class="o">(</span><span class="n">previousPrediction</span><span class="o">,</span> <span class="n">newPrediction</span><span class="o">))</span> <span class="o">{</span>
+ <span class="n">c</span><span class="o">.</span><span class="na">output</span><span class="o">(</span><span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">userId</span><span class="o">,</span> <span class="n">newPrediction</span><span class="o">));</span>
+ <span class="n">previousPredictionState</span><span class="o">.</span><span class="na">write</span><span class="o">(</span><span class="n">newPrediction</span><span class="o">);</span>
+ <span class="o">}</span>
+ <span class="o">}</span>
+<span class="o">};</span>
+</code></pre>
+</div>
+
+<div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="c"># State and timers are not yet supported in Beam's Python SDK.</span>
+<span class="c"># Watch this space!</span>
+</code></pre>
+</div>
+
+<p>Let’s walk through it,</p>
+
+<ul>
+ <li>You have two state cells declared, <code class="highlighter-rouge">@StateId("model")</code> to hold the current
+state of the model for a user and <code class="highlighter-rouge">@StateId("previousPrediction")</code> to hold
+the prediction output previously.</li>
+ <li>Access to the two state cells by annotation in the <code class="highlighter-rouge">@ProcessElement</code> method
+is as before.</li>
+ <li>You read the current model via <code class="highlighter-rouge">modelState.read()</code>. Because state is also
+per-key-and-window, this is a model just for the UserId of the Event
+currently being processed.</li>
+ <li>You derive a new prediction <code class="highlighter-rouge">model.prediction(event)</code> and compare it against
+the last one you output, accessed via <code class="highlighter-rouge">previousPredicationState.read()</code>.</li>
+ <li>You then update the model <code class="highlighter-rouge">model.update()</code> and write it via
+<code class="highlighter-rouge">modelState.write(...)</code>. It is perfectly fine to mutate the value you pulled
+out of state as long as you also remember to write the mutated value, in the
+same way you are encouraged to mutate <code class="highlighter-rouge">CombineFn</code> accumulators.</li>
+ <li>If the prediction has changed a significant amount since the last time you
+output, you emit it via <code class="highlighter-rouge">context.output(...)</code> and save the prediction using
+<code class="highlighter-rouge">previousPredictionState.write(...)</code>. Here the decision is relative to the
+prior prediction output, not the last one computed - realistically you might
+have some complex conditions here.</li>
+</ul>
+
+<p>Most of the above is just talking through Java! But before you go out and
+convert all of your pipelines to use stateful processing, I want to go over
+some considerations as to whether it is a good fit for your use case.</p>
+
+<h2 id="performance-considerations">Performance considerations</h2>
+
+<p>To decide whether to use per-key-and-window state, you need to consider how it
+executes. You can dig into how a particular runner manages state, but there are
+some general things to keep in mind:</p>
+
+<ul>
+ <li>Partitioning per-key-and-window: perhaps the most important thing to
+consider is that the runner may have to shuffle your data to colocate all
+the data for a particular key+window. If the data is already shuffled
+correctly, the runner may take advantage of this.</li>
+ <li>Synchronization overhead: the API is designed so the runner takes care of
+concurrency control, but this means that the runner cannot parallelize
+processing of elements for a particular key+window even when it would otherwise
+be advantageous.</li>
+ <li>Storage and fault tolerance of state: since state is per-key-and-window, the
+more keys and windows you expect to process simultaneously, the more storage
+you will incur. Because state benefits from all the fault tolerance /
+consistency properties of your other data in Beam, it also adds to the cost of
+committing the results of processing.</li>
+ <li>Expiration of state: also since state is per-window, the runner can reclaim
+the resources when a window expires (when the watermark exceeds its allowed
+lateness) but this could mean that the runner is tracking an additional timer
+per key and window to cause reclamation code to execute.</li>
+</ul>
+
+<h2 id="go-use-it">Go use it!</h2>
+
+<p>If you are new to Beam, I hope you are now interested in seeing if Beam with
+stateful processing addresses your use case. If you are already using Beam, I
+hope this new addition to the model unlocks new use cases for you. Do check
+the <a href="/documentation/runners/capability-matrix/">capability
+matrix</a> to
+see the level of support for this new model feature on your favorite
+backend(s).</p>
+
+<p>And please do join the community at
+<a href="/get-started/support">user@beam.apache.org</a>. We’d love to
+hear from you.</p>
+
+ </div>
+
+</article>
+
+ </div>
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<footer class="footer">
+ <div class="footer__contained">
+ <div class="footer__cols">
+ <div class="footer__cols__col">
+ <div class="footer__cols__col__logo">
+ <img src="/images/beam_logo_circle.svg" class="footer__logo" alt="Beam logo">
+ </div>
+ <div class="footer__cols__col__logo">
+ <img src="/images/apache_logo_circle.svg" class="footer__logo" alt="Apache logo">
+ </div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Start</div>
+ <div class="footer__cols__col__link"><a href="/get-started/beam-overview/">Overview</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-java/">Quickstart (Java)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-py/">Quickstart (Python)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-go/">Quickstart (Go)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/downloads/">Downloads</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Docs</div>
+ <div class="footer__cols__col__link"><a href="/documentation/programming-guide/">Concepts</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/pipelines/design-your-pipeline/">Pipelines</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/runners/capability-matrix/">Runners</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Community</div>
+ <div class="footer__cols__col__link"><a href="/contribute/">Contribute</a></div>
+ <div class="footer__cols__col__link"><a href="https://projects.apache.org/committee.html?beam" target="_blank">Team<img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a></div>
+ <div class="footer__cols__col__link"><a href="/contribute/presentation-materials/">Media</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Resources</div>
+ <div class="footer__cols__col__link"><a href="/blog/">Blog</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/support/">Support</a></div>
+ <div class="footer__cols__col__link"><a href="https://github.com/apache/beam">GitHub</a></div>
+ </div>
+ </div>
+ </div>
+ <div class="footer__bottom">
+ ©
+ <a href="http://www.apache.org">The Apache Software Foundation</a>
+ | <a href="/privacy_policy">Privacy Policy</a>
+ | <a href="/feed.xml">RSS Feed</a>
+ <br><br>
+ Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are
+ either registered trademarks or trademarks of The Apache Software
+ Foundation. All other products or name brands are trademarks of their
+ respective holders, including The Apache Software Foundation.
+ </div>
+</footer>
+
+ </body>
+</html>
diff --git a/website/generated-content/blog/2017/03/16/python-sdk-release.html b/website/generated-content/blog/2017/03/16/python-sdk-release.html
new file mode 100644
index 0000000..a1a3cdc
--- /dev/null
+++ b/website/generated-content/blog/2017/03/16/python-sdk-release.html
@@ -0,0 +1,296 @@
+<!DOCTYPE html>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<html lang="en">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <title>Python SDK released in Apache Beam 0.6.0</title>
+ <meta name="description" content="Apache Beam’s latest release, version 0.6.0, introduces a new SDK – this time, for the Python programming language. The Python SDK joins the Java SDK as the ...">
+ <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
+ <link rel="stylesheet" href="/css/site.css">
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <script src="/js/language-switch.js"></script>
+ <script src="/js/fix-menu.js"></script>
+ <script src="/js/section-nav.js"></script>
+ <script src="/js/page-nav.js"></script>
+ <link rel="canonical" href="https://beam.apache.org/blog/2017/03/16/python-sdk-release.html" data-proofer-ignore>
+ <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
+ <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+ ga('create', 'UA-73650088-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+</head>
+
+ <body class="body ">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<nav class="header navbar navbar-fixed-top">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+
+ <a href="/" class="navbar-brand" >
+ <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
+ </a>
+ </div>
+
+ <div class="navbar-mask closed"></div>
+
+ <div id="navbar" class="navbar-container closed">
+ <ul class="nav navbar-nav">
+ <li>
+ <a href="/get-started/beam-overview/">Get Started</a>
+ </li>
+ <li>
+ <a href="/documentation/">Documentation</a>
+ </li>
+ <li>
+ <a href="/documentation/sdks/java/">SDKS</a>
+ </li>
+ <li>
+ <a href="/documentation/runners/capability-matrix/">RUNNERS</a>
+ </li>
+ <li>
+ <a href="/contribute/">Contribute</a>
+ </li>
+ <li>
+ <a href="/community/contact-us/">Community</a>
+ </li>
+ <li><a href="/blog">Blog</a></li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:20px;"><span class="caret"></span></a>
+ <ul class="dropdown-menu dropdown-menu-right">
+ <li><a href="http://www.apache.org/">ASF Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">License</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
+ </ul>
+ </li>
+ </ul>
+ </div>
+</nav>
+
+ <div class="body__contained">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+
+
+<article class="post" itemscope itemtype="http://schema.org/BlogPosting">
+
+ <header class="post-header">
+ <h1 class="post-title" itemprop="name headline">Python SDK released in Apache Beam 0.6.0</h1>
+ <p class="post-meta"><time datetime="2017-03-16T01:00:01-07:00" itemprop="datePublished">Mar 16, 2017</time> •
+ Ahmet Altay
+
+ </p>
+ </header>
+
+ <div class="post-content" itemprop="articleBody">
+ <!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+<p>Apache Beam’s latest release, version <a href="/get-started/downloads/">0.6.0</a>, introduces a new SDK – this time, for the Python programming language. The Python SDK joins the Java SDK as the second implementation of the Beam programming model.</p>
+
+<!--more-->
+
+<p>The Python SDK incorporates all of the main concepts of the Beam model, including ParDo, GroupByKey, Windowing, and others. It features extensible IO APIs for writing bounded sources and sinks, and provides built-in implementation for reading and writing Text, Avro, and TensorFlow record files, as well as connectors to Google BigQuery and Google Cloud Datastore.</p>
+
+<p>There are two runners capable of executing pipelines written with the Python SDK today: <a href="/documentation/runners/direct/">Direct Runner</a> and <a href="/documentation/runners/dataflow/">Dataflow Runner</a>, both of which are currently limited to batch execution only. Upcoming features will shortly bring the benefits of the Python SDK to additional runners.</p>
+
+<h4 id="try-the-apache-beam-python-sdk">Try the Apache Beam Python SDK</h4>
+
+<p>If you would like to try out the Python SDK, a good place to start is the <a href="/get-started/quickstart-py/">Quickstart</a>. After that, you can take a look at additional <a href="https://github.com/apache/beam/tree/v0.6.0/sdks/python/apache_beam/examples">examples</a>, and deep dive into the <a href="/documentation/sdks/pydoc/">API reference</a>.</p>
+
+<p>Let’s take a look at a quick example together. First, install the <code class="highlighter-rouge">apache-beam</code> package from PyPI and start your Python interpreter.</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>$ pip install apache-beam
+$ python
+</code></pre>
+</div>
+
+<p>We will harness the power of Apache Beam to estimate Pi in honor of the recently passed Pi Day.</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>import random
+import apache_beam as beam
+
+def run_trials(count):
+ """Throw darts into unit square and count how many fall into unit circle."""
+ inside = 0
+ for _ in xrange(count):
+ x, y = random.uniform(0, 1), random.uniform(0, 1)
+ inside += 1 if x*x + y*y <= 1.0 else 0
+ return count, inside
+
+def combine_results(results):
+ """Given all the trial results, estimate pi."""
+ total, inside = sum(r[0] for r in results), sum(r[1] for r in results)
+ return total, inside, 4 * float(inside) / total if total > 0 else 0
+
+p = beam.Pipeline()
+(p | beam.Create([500] * 10) # Create 10 experiments with 500 samples each.
+ | beam.Map(run_trials) # Run experiments in parallel.
+ | beam.CombineGlobally(combine_results) # Combine the results.
+ | beam.io.WriteToText('./pi_estimate.txt')) # Write PI estimate to a file.
+
+p.run()
+</code></pre>
+</div>
+
+<p>This example estimates Pi by throwing random darts into the unit square and keeping track of the fraction of those darts that fell into the unit circle (see the full <a href="https://github.com/apache/beam/blob/v0.6.0/sdks/python/apache_beam/examples/complete/estimate_pi.py">example</a> for details). If you are curious, you can check the result of our estimation by looking at the output file.</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>$ cat pi_estimate.txt*
+</code></pre>
+</div>
+
+<h4 id="roadmap">Roadmap</h4>
+
+<p>The first thing on the Python SDK’s roadmap is to address two of its limitations. First, the existing runners are currently limited to bounded PCollections, and we are looking forward to extending the SDK to support unbounded PCollections (“streaming”). Additionally, we are working on extending support to more Apache Beam runners, and the upcoming Fn API will do the heavy lifting.</p>
+
+<p>Both of these improvements will enable the Python SDK to fulfill the mission of Apache Beam: a unified programming model for batch and streaming data processing that can run on any execution engine.</p>
+
+<h4 id="join-us">Join us!</h4>
+
+<p>Please consider joining us, whether as a user or a contributor, as we work towards our first release with API stability. If you’d like to try out Apache Beam today, check out the latest <a href="/get-started/downloads/">0.6.0</a> release. We welcome contributions and participation from anyone through our mailing lists, issue tracker, pull requests, and events.</p>
+
+ </div>
+
+</article>
+
+ </div>
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<footer class="footer">
+ <div class="footer__contained">
+ <div class="footer__cols">
+ <div class="footer__cols__col">
+ <div class="footer__cols__col__logo">
+ <img src="/images/beam_logo_circle.svg" class="footer__logo" alt="Beam logo">
+ </div>
+ <div class="footer__cols__col__logo">
+ <img src="/images/apache_logo_circle.svg" class="footer__logo" alt="Apache logo">
+ </div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Start</div>
+ <div class="footer__cols__col__link"><a href="/get-started/beam-overview/">Overview</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-java/">Quickstart (Java)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-py/">Quickstart (Python)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-go/">Quickstart (Go)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/downloads/">Downloads</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Docs</div>
+ <div class="footer__cols__col__link"><a href="/documentation/programming-guide/">Concepts</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/pipelines/design-your-pipeline/">Pipelines</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/runners/capability-matrix/">Runners</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Community</div>
+ <div class="footer__cols__col__link"><a href="/contribute/">Contribute</a></div>
+ <div class="footer__cols__col__link"><a href="https://projects.apache.org/committee.html?beam" target="_blank">Team<img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a></div>
+ <div class="footer__cols__col__link"><a href="/contribute/presentation-materials/">Media</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Resources</div>
+ <div class="footer__cols__col__link"><a href="/blog/">Blog</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/support/">Support</a></div>
+ <div class="footer__cols__col__link"><a href="https://github.com/apache/beam">GitHub</a></div>
+ </div>
+ </div>
+ </div>
+ <div class="footer__bottom">
+ ©
+ <a href="http://www.apache.org">The Apache Software Foundation</a>
+ | <a href="/privacy_policy">Privacy Policy</a>
+ | <a href="/feed.xml">RSS Feed</a>
+ <br><br>
+ Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are
+ either registered trademarks or trademarks of The Apache Software
+ Foundation. All other products or name brands are trademarks of their
+ respective holders, including The Apache Software Foundation.
+ </div>
+</footer>
+
+ </body>
+</html>
diff --git a/website/generated-content/blog/2017/05/17/beam-first-stable-release.html b/website/generated-content/blog/2017/05/17/beam-first-stable-release.html
new file mode 100644
index 0000000..e6b3151
--- /dev/null
+++ b/website/generated-content/blog/2017/05/17/beam-first-stable-release.html
@@ -0,0 +1,335 @@
+<!DOCTYPE html>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<html lang="en">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <title>Apache Beam publishes the first stable release</title>
+ <meta name="description" content="The Apache Beam community is pleased to announce the availability of version 2.0.0. This is the first stable release of Apache Beam, signifying a statement f...">
+ <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
+ <link rel="stylesheet" href="/css/site.css">
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <script src="/js/language-switch.js"></script>
+ <script src="/js/fix-menu.js"></script>
+ <script src="/js/section-nav.js"></script>
+ <script src="/js/page-nav.js"></script>
+ <link rel="canonical" href="https://beam.apache.org/blog/2017/05/17/beam-first-stable-release.html" data-proofer-ignore>
+ <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
+ <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+ ga('create', 'UA-73650088-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+</head>
+
+ <body class="body ">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<nav class="header navbar navbar-fixed-top">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+
+ <a href="/" class="navbar-brand" >
+ <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
+ </a>
+ </div>
+
+ <div class="navbar-mask closed"></div>
+
+ <div id="navbar" class="navbar-container closed">
+ <ul class="nav navbar-nav">
+ <li>
+ <a href="/get-started/beam-overview/">Get Started</a>
+ </li>
+ <li>
+ <a href="/documentation/">Documentation</a>
+ </li>
+ <li>
+ <a href="/documentation/sdks/java/">SDKS</a>
+ </li>
+ <li>
+ <a href="/documentation/runners/capability-matrix/">RUNNERS</a>
+ </li>
+ <li>
+ <a href="/contribute/">Contribute</a>
+ </li>
+ <li>
+ <a href="/community/contact-us/">Community</a>
+ </li>
+ <li><a href="/blog">Blog</a></li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:20px;"><span class="caret"></span></a>
+ <ul class="dropdown-menu dropdown-menu-right">
+ <li><a href="http://www.apache.org/">ASF Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">License</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
+ </ul>
+ </li>
+ </ul>
+ </div>
+</nav>
+
+ <div class="body__contained">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+
+
+<article class="post" itemscope itemtype="http://schema.org/BlogPosting">
+
+ <header class="post-header">
+ <h1 class="post-title" itemprop="name headline">Apache Beam publishes the first stable release</h1>
+ <p class="post-meta"><time datetime="2017-05-17T01:00:01-07:00" itemprop="datePublished">May 17, 2017</time> •
+ Davor Bonaci [<a href="https://twitter.com/BonaciDavor">@BonaciDavor</a>] & Dan Halperin
+
+ </p>
+ </header>
+
+ <div class="post-content" itemprop="articleBody">
+ <!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+<p>The Apache Beam community is pleased to <a href="https://blogs.apache.org/foundation/entry/the-apache-software-foundation-announces12">announce the availability of version 2.0.0</a>. This is the first stable release of Apache Beam, signifying a statement from the community that it intends to maintain API stability with all releases for the foreseeable future, and making Beam suitable for enterprise deployment.</p>
+
+<!--more-->
+
+<p>This first stable release is the third important milestone for the Apache Beam community. Beam joined the Apache Incubator in February 2016 and graduated as a top-level project of The Apache Software Foundation in December. Through these fifteen months of concentrated effort, a slightly chaotic codebase, merged from multiple organizations, has been developed into a generalized framework for data processing that is truly engine- and environment-independent. Apache Beam has evolved and [...]
+
+<p>In the five months since graduation, Apache Beam has seen a significant growth, both in terms of adoption and community contribution. Apache Beam is <a href="https://blogs.apache.org/foundation/entry/the-apache-software-foundation-announces12">in use</a> at Google Cloud, PayPal, and Talend, among others.</p>
+
+<p>Apache Beam, version 2.0.0 improves user experience across the project, focusing on seamless portability across execution environments, including engines, operating systems, on-premise clusters, cloud providers, and data storage systems. Other highlights include:</p>
+
+<ul>
+ <li>API stability and future compatibility within this major version.</li>
+ <li>Stateful data processing paradigms that unlock efficient, data-dependent computations.</li>
+ <li>Support for user-extensible file systems, with built-in support for Hadoop Distributed File System, among others.</li>
+ <li>A metrics subsystem for deeper insight into pipeline execution.</li>
+</ul>
+
+<p>Many contributors made this release possible, by participating in different roles: contributing code, writing documentation, testing release candidates, supporting users, or helping in some other way. The following is a partial list of contributors – 76 individuals contributed code to the project since the previous release, assembled from source history:</p>
+
+<ul>
+ <li>Ahmet Altay</li>
+ <li>Eric Anderson</li>
+ <li>Raghu Angadi</li>
+ <li>Sourabh Bajaj</li>
+ <li>Péter Gergő Barna</li>
+ <li>Chen Bin</li>
+ <li>Davor Bonaci</li>
+ <li>Robert Bradshaw</li>
+ <li>Ben Chambers</li>
+ <li>Etienne Chauchot</li>
+ <li>Chang Chen</li>
+ <li>Charles Chen</li>
+ <li>Craig Citro</li>
+ <li>Lukasz Cwik</li>
+ <li>Márton Elek</li>
+ <li>Pablo Estrada</li>
+ <li>Josh Forman-Gornall</li>
+ <li>Maria García Herrero</li>
+ <li>Jins George</li>
+ <li>Damien Gouyette</li>
+ <li>Thomas Groh</li>
+ <li>Dan Halperin</li>
+ <li>Pei He</li>
+ <li>Hadar Hod</li>
+ <li>Chamikara Jayalath</li>
+ <li>Rekha Joshi</li>
+ <li>Uwe Jugel</li>
+ <li>Sung Junyoung</li>
+ <li>Holden Karau</li>
+ <li>Vikas Kedigehalli</li>
+ <li>Eugene Kirpichov</li>
+ <li>Tibor Kiss</li>
+ <li>Kenneth Knowles</li>
+ <li>Vassil Kolarov</li>
+ <li>Chinmay Kolhatkar</li>
+ <li>Aljoscha Krettek</li>
+ <li>Dipti Kulkarni</li>
+ <li>Radhika Kulkarni</li>
+ <li>Jason Kuster</li>
+ <li>Reuven Lax</li>
+ <li>Stas Levin</li>
+ <li>Julien Lhermitte</li>
+ <li>Jingsong Li</li>
+ <li>Neville Li</li>
+ <li>Mark Liu</li>
+ <li>Michael Luckey</li>
+ <li>Andrew Martin</li>
+ <li>Ismaël Mejía</li>
+ <li>Devon Meunier</li>
+ <li>Neda Mirian</li>
+ <li>Anil Muppalla</li>
+ <li>Gergely Novak</li>
+ <li>Jean-Baptiste Onofré</li>
+ <li>Melissa Pashniak</li>
+ <li>peay</li>
+ <li>David Rieber</li>
+ <li>Rahul Sabbineni</li>
+ <li>Kobi Salant</li>
+ <li>Amit Sela</li>
+ <li>Mark Shalda</li>
+ <li>Stephen Sisk</li>
+ <li>Yuya Tajima</li>
+ <li>Wesley Tanaka</li>
+ <li>JiJun Tang</li>
+ <li>Valentyn Tymofieiev</li>
+ <li>David Volquartz</li>
+ <li>Huafeng Wang</li>
+ <li>Thomas Weise</li>
+ <li>Rafal Wojdyla</li>
+ <li>Yangping Wu</li>
+ <li>wyp</li>
+ <li>James Xu</li>
+ <li>Mingmin Xu</li>
+ <li>Ted Yu</li>
+ <li>Borisa Zivkovic</li>
+ <li>Aviem Zur</li>
+</ul>
+
+<p>Apache Beam, version 2.0.0, is making its debut at Apache: Big Data, taking place this week in Miami, FL, with four sessions featuring Apache Beam. Apache Beam will also be highlighted at numerous face-to-face meetups and conferences, including the Future of Data San Jose meetup, Strata Data Conference London, Berlin Buzzwords, and DataWorks Summit San Jose.</p>
+
+<p>We’d like to invite everyone to try out Apache Beam today and consider joining our vibrant community. We welcome feedback, contribution and participation through our mailing lists, issue tracker, pull requests, and events.</p>
+
+ </div>
+
+</article>
+
+ </div>
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<footer class="footer">
+ <div class="footer__contained">
+ <div class="footer__cols">
+ <div class="footer__cols__col">
+ <div class="footer__cols__col__logo">
+ <img src="/images/beam_logo_circle.svg" class="footer__logo" alt="Beam logo">
+ </div>
+ <div class="footer__cols__col__logo">
+ <img src="/images/apache_logo_circle.svg" class="footer__logo" alt="Apache logo">
+ </div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Start</div>
+ <div class="footer__cols__col__link"><a href="/get-started/beam-overview/">Overview</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-java/">Quickstart (Java)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-py/">Quickstart (Python)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-go/">Quickstart (Go)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/downloads/">Downloads</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Docs</div>
+ <div class="footer__cols__col__link"><a href="/documentation/programming-guide/">Concepts</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/pipelines/design-your-pipeline/">Pipelines</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/runners/capability-matrix/">Runners</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Community</div>
+ <div class="footer__cols__col__link"><a href="/contribute/">Contribute</a></div>
+ <div class="footer__cols__col__link"><a href="https://projects.apache.org/committee.html?beam" target="_blank">Team<img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a></div>
+ <div class="footer__cols__col__link"><a href="/contribute/presentation-materials/">Media</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Resources</div>
+ <div class="footer__cols__col__link"><a href="/blog/">Blog</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/support/">Support</a></div>
+ <div class="footer__cols__col__link"><a href="https://github.com/apache/beam">GitHub</a></div>
+ </div>
+ </div>
+ </div>
+ <div class="footer__bottom">
+ ©
+ <a href="http://www.apache.org">The Apache Software Foundation</a>
+ | <a href="/privacy_policy">Privacy Policy</a>
+ | <a href="/feed.xml">RSS Feed</a>
+ <br><br>
+ Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are
+ either registered trademarks or trademarks of The Apache Software
+ Foundation. All other products or name brands are trademarks of their
+ respective holders, including The Apache Software Foundation.
+ </div>
+</footer>
+
+ </body>
+</html>
diff --git a/website/generated-content/blog/2017/08/16/splittable-do-fn.html b/website/generated-content/blog/2017/08/16/splittable-do-fn.html
new file mode 100644
index 0000000..eb5bb0e
--- /dev/null
+++ b/website/generated-content/blog/2017/08/16/splittable-do-fn.html
@@ -0,0 +1,779 @@
+<!DOCTYPE html>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<html lang="en">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <title>Powerful and modular IO connectors with Splittable DoFn in Apache Beam</title>
+ <meta name="description" content="One of the most important parts of the Apache Beam ecosystem is its quicklygrowing set of connectors that allow Beam pipelines to read and write data tovario...">
+ <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
+ <link rel="stylesheet" href="/css/site.css">
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <script src="/js/language-switch.js"></script>
+ <script src="/js/fix-menu.js"></script>
+ <script src="/js/section-nav.js"></script>
+ <script src="/js/page-nav.js"></script>
+ <link rel="canonical" href="https://beam.apache.org/blog/2017/08/16/splittable-do-fn.html" data-proofer-ignore>
+ <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
+ <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+ ga('create', 'UA-73650088-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+</head>
+
+ <body class="body ">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<nav class="header navbar navbar-fixed-top">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+
+ <a href="/" class="navbar-brand" >
+ <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
+ </a>
+ </div>
+
+ <div class="navbar-mask closed"></div>
+
+ <div id="navbar" class="navbar-container closed">
+ <ul class="nav navbar-nav">
+ <li>
+ <a href="/get-started/beam-overview/">Get Started</a>
+ </li>
+ <li>
+ <a href="/documentation/">Documentation</a>
+ </li>
+ <li>
+ <a href="/documentation/sdks/java/">SDKS</a>
+ </li>
+ <li>
+ <a href="/documentation/runners/capability-matrix/">RUNNERS</a>
+ </li>
+ <li>
+ <a href="/contribute/">Contribute</a>
+ </li>
+ <li>
+ <a href="/community/contact-us/">Community</a>
+ </li>
+ <li><a href="/blog">Blog</a></li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:20px;"><span class="caret"></span></a>
+ <ul class="dropdown-menu dropdown-menu-right">
+ <li><a href="http://www.apache.org/">ASF Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">License</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
+ </ul>
+ </li>
+ </ul>
+ </div>
+</nav>
+
+ <div class="body__contained">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+
+
+<article class="post" itemscope itemtype="http://schema.org/BlogPosting">
+
+ <header class="post-header">
+ <h1 class="post-title" itemprop="name headline">Powerful and modular IO connectors with Splittable DoFn in Apache Beam</h1>
+ <p class="post-meta"><time datetime="2017-08-16T01:00:01-07:00" itemprop="datePublished">Aug 16, 2017</time> •
+ Eugene Kirpichov
+
+ </p>
+ </header>
+
+ <div class="post-content" itemprop="articleBody">
+ <!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+<p>One of the most important parts of the Apache Beam ecosystem is its quickly
+growing set of connectors that allow Beam pipelines to read and write data to
+various data storage systems (“IOs”). Currently, Beam ships <a href="/documentation/io/built-in/">over 20 IO
+connectors</a> with many more in
+active development. As user demands for IO connectors grew, our work on
+improving the related Beam APIs (in particular, the Source API) produced an
+unexpected result: a generalization of Beam’s most basic primitive, <code class="highlighter-rouge">DoFn</code>.</p>
+
+<!--more-->
+
+<h2 id="connectors-as-mini-pipelines">Connectors as mini-pipelines</h2>
+
+<p>One of the main reasons for this vibrant IO connector ecosystem is that
+developing a basic IO is relatively straightforward: many connector
+implementations are simply mini-pipelines (composite <code class="highlighter-rouge">PTransform</code>s) made of the
+basic Beam <code class="highlighter-rouge">ParDo</code> and <code class="highlighter-rouge">GroupByKey</code> primitives. For example,
+<code class="highlighter-rouge">ElasticsearchIO.write()</code>
+<a href="https://github.com/apache/beam/blob/f7e8f886c91ea9d0b51e00331eeb4484e2f6e000/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java#L783">expands</a>
+into a single <code class="highlighter-rouge">ParDo</code> with some batching for performance; <code class="highlighter-rouge">JdcbIO.read()</code>
+<a href="https://github.com/apache/beam/blob/f7e8f886c91ea9d0b51e00331eeb4484e2f6e000/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java#L329">expands</a>
+into <code class="highlighter-rouge">Create.of(query)</code>, a reshuffle to <a href="https://cloud.google.com/dataflow/service/dataflow-service-desc#preventing-fusion">prevent
+fusion</a>,
+and <code class="highlighter-rouge">ParDo(execute sub-query)</code>. Some IOs
+<a href="https://github.com/apache/beam/blob/8503adbbc3a590cd0dc2939f6a45d335682a9442/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1139">construct</a>
+considerably more complicated pipelines.</p>
+
+<p><img class="center-block" src="/images/blog/splittable-do-fn/jdbcio-expansion.png" alt="Expansion of the JdbcIO.read() composite transform" width="600" /></p>
+
+<p>This “mini-pipeline” approach is flexible, modular, and generalizes to data
+sources that read from a dynamically computed <code class="highlighter-rouge">PCollection</code> of locations, such
+as
+<a href="https://github.com/apache/beam/blob/f7e8f886c91ea9d0b51e00331eeb4484e2f6e000/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java#L222"><code class="highlighter-rouge">SpannerIO.readAll()</code></a>
+which reads the results of a <code class="highlighter-rouge">PCollection</code> of queries from Cloud Spanner,
+compared to
+<a href="https://github.com/apache/beam/blob/f7e8f886c91ea9d0b51e00331eeb4484e2f6e000/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java#L318"><code class="highlighter-rouge">SpannerIO.read()</code></a>
+which executes a single query. We believe such dynamic data sources are a very
+useful capability, often overlooked by other data processing frameworks.</p>
+
+<h2 id="when-pardo-and-groupbykey-are-not-enough">When ParDo and GroupByKey are not enough</h2>
+
+<p>Despite the flexibility of <code class="highlighter-rouge">ParDo</code>, <code class="highlighter-rouge">GroupByKey</code> and their derivatives, in some
+cases building an efficient IO connector requires extra capabilities.</p>
+
+<p>For example, imagine reading files using the sequence <code class="highlighter-rouge">ParDo(filepattern →
+expand into files)</code>, <code class="highlighter-rouge">ParDo(filename → read records)</code>, or reading a Kafka topic
+using <code class="highlighter-rouge">ParDo(topic → list partitions)</code>, <code class="highlighter-rouge">ParDo(topic, partition → read
+records)</code>. This approach has two big issues:</p>
+
+<ul>
+ <li>
+ <p>In the file example, some files might be much larger than others, so the
+second <code class="highlighter-rouge">ParDo</code> may have very long individual <code class="highlighter-rouge">@ProcessElement</code> calls. As a
+result, the pipeline can suffer from poor performance due to stragglers.</p>
+ </li>
+ <li>
+ <p>In the Kafka example, implementing the second <code class="highlighter-rouge">ParDo</code> is <em>simply impossible</em>
+with a regular <code class="highlighter-rouge">DoFn</code>, because it would need to output an infinite number of
+records per each input element <code class="highlighter-rouge">topic, partition</code> <em>(<a href="/blog/2017/02/13/stateful-processing.html">stateful processing</a> comes close, but it
+has other limitations that make it insufficient for this task</em>).</p>
+ </li>
+</ul>
+
+<h2 id="beam-source-api">Beam Source API</h2>
+
+<p>Apache Beam historically provides a Source API
+(<a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/BoundedSource.html">BoundedSource</a>
+and
+<a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/UnboundedSource.html">UnboundedSource</a>) which does
+not have these limitations and allows development of efficient data sources for
+batch and streaming systems. Pipelines use this API via the
+<a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/Read.html"><code class="highlighter-rouge">Read.from(Source)</code></a> built-in <code class="highlighter-rouge">PTransform</code>.</p>
+
+<p>The Source API is largely similar to that of most other data processing
+frameworks, and allows the system to read data in parallel using multiple
+workers, as well as checkpoint and resume reading from an unbounded data source.
+Additionally, the Beam
+<a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/BoundedSource.html"><code class="highlighter-rouge">BoundedSource</code></a>
+API provides advanced features such as progress reporting and <a href="/blog/2016/05/18/splitAtFraction-method.html">dynamic
+rebalancing</a>
+(which together enable autoscaling), and
+<a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/UnboundedSource.html"><code class="highlighter-rouge">UnboundedSource</code></a> supports
+reporting the source’s watermark and backlog <em>(until SDF, we believed that
+“batch” and “streaming” data sources are fundamentally different and thus
+require fundamentally different APIs)</em>.</p>
+
+<p>Unfortunately, these features come at a price. Coding against the Source API
+involves a lot of boilerplate and is error-prone, and it does not compose well
+with the rest of the Beam model because a <code class="highlighter-rouge">Source</code> can appear only at the root
+of a pipeline. For example:</p>
+
+<ul>
+ <li>
+ <p>Using the Source API, it is not possible to read a <code class="highlighter-rouge">PCollection</code> of
+filepatterns.</p>
+ </li>
+ <li>
+ <p>A <code class="highlighter-rouge">Source</code> can not read a side input, or wait on another pipeline step to
+produce the data.</p>
+ </li>
+ <li>
+ <p>A <code class="highlighter-rouge">Source</code> can not emit an additional output (for example, records that failed to
+parse) and so on.</p>
+ </li>
+</ul>
+
+<p>The Source API is not composable even with itself. For example, suppose Alice
+implements an unbounded <code class="highlighter-rouge">Source</code> that watches a directory for new matching
+files, and Bob implements an unbounded <code class="highlighter-rouge">Source</code> that tails a file. The Source
+API does not let them simply chain the sources together and obtain a <code class="highlighter-rouge">Source</code>
+that returns new records in new log files in a directory (a very common user
+request). Instead, such a source would have to be developed mostly from
+scratch, and our experience shows that a full-featured monolithic
+implementation of such a <code class="highlighter-rouge">Source</code> is incredibly difficult and error-prone.</p>
+
+<p>Another class of issues with the <code class="highlighter-rouge">Source</code> API comes from its strict
+bounded/unbounded dichotomy:</p>
+
+<ul>
+ <li>
+ <p>It is difficult or impossible to reuse code between seemingly very similar
+bounded and unbounded sources, for example, the <code class="highlighter-rouge">BoundedSource</code> that generates
+a sequence <code class="highlighter-rouge">[a, b)</code> and the <code class="highlighter-rouge">UnboundedSource</code> that generates a sequence <code class="highlighter-rouge">[a,
+inf)</code> <a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/CountingSource.java">don’t share any
+code</a>
+in the Beam Java SDK.</p>
+ </li>
+ <li>
+ <p>It is not clear how to classify the ingestion of a very large and
+continuously growing dataset. Ingesting its “already available” part seems to
+require a <code class="highlighter-rouge">BoundedSource</code>: the runner could benefit from knowing its size, and
+could perform dynamic rebalancing. However, ingesting the continuously arriving
+new data seems to require an <code class="highlighter-rouge">UnboundedSource</code> for providing watermarks. From
+this angle, the <code class="highlighter-rouge">Source</code> API has <a href="https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101">the same issues as Lambda
+Architecture</a>.</p>
+ </li>
+</ul>
+
+<p>About two years ago we began thinking about how to address the limitations of
+the Source API, and ended up, surprisingly, addressing the limitations of
+<code class="highlighter-rouge">DoFn</code> instead.</p>
+
+<h2 id="enter-splittable-dofn">Enter Splittable DoFn</h2>
+
+<p><a href="http://s.apache.org/splittable-do-fn">Splittable DoFn</a> (SDF) is a
+generalization of <code class="highlighter-rouge">DoFn</code> that gives it the core capabilities of <code class="highlighter-rouge">Source</code> while
+retaining <code class="highlighter-rouge">DoFn</code>’s syntax, flexibility, modularity, and ease of coding. As a
+result, it becomes possible to develop more powerful IO connectors than before,
+with shorter, simpler, more reusable code.</p>
+
+<p>Note that, unlike <code class="highlighter-rouge">Source</code>, SDF <em>does not</em> have distinct bounded/unbounded APIs,
+just as regular <code class="highlighter-rouge">DoFn</code>s don’t: there is only one API, which covers both of these
+use cases and anything in between. Thus, SDF closes the final gap in the unified
+batch/streaming programming model of Apache Beam.</p>
+
+<p>When reading the explanation of SDF below, keep in mind the running example of a
+<code class="highlighter-rouge">DoFn</code> that takes a filename as input and outputs the records in that file.
+People familiar with the <code class="highlighter-rouge">Source</code> API may find it useful to think of SDF as a
+way to read a <code class="highlighter-rouge">PCollection</code> of sources, treating the source itself as just
+another piece of data in the pipeline <em>(this, in fact, was one of the early
+design iterations among the work that led to creation of SDF)</em>.</p>
+
+<p>The two aspects where <code class="highlighter-rouge">Source</code> has an advantage over a regular <code class="highlighter-rouge">DoFn</code> are:</p>
+
+<ul>
+ <li>
+ <p><strong>Splittability:</strong> applying a <code class="highlighter-rouge">DoFn</code> to a single element is <em>monolithic</em>, but
+reading from a <code class="highlighter-rouge">Source</code> is <em>non-monolithic</em>. The whole <code class="highlighter-rouge">Source</code> doesn’t have to
+be read at once; rather, it is read in parts, called <em>bundles</em>. For example, a
+large file is usually read in several bundles, each reading some sub-range of
+offsets within the file. Likewise, a Kafka topic (which, of course, can never
+be read “fully”) is read over an infinite number of bundles, each reading some
+finite number of elements.</p>
+ </li>
+ <li>
+ <p><strong>Interaction with the runner:</strong> runners apply a <code class="highlighter-rouge">DoFn</code> to a single element as
+a “black box”, but interact quite richly with <code class="highlighter-rouge">Source</code>. <code class="highlighter-rouge">Source</code> provides the
+runner with information such as its estimated size (or its generalization,
+“backlog”), progress through reading the bundle, watermarks etc. The runner
+uses this information to tune the execution and control the breakdown of the
+<code class="highlighter-rouge">Source</code> into bundles. For example, a slowly progressing large bundle of a file
+may be <a href="https://cloud.google.com/blog/big-data/2016/05/no-shard-left-behind-dynamic-work-rebalancing-in-google-cloud-dataflow">dynamically
+split</a>
+by a batch-focused runner before it becomes a straggler, and a latency-focused
+streaming runner may control how many elements it reads from a source in each
+bundle to optimize for latency vs. per-bundle overhead.</p>
+ </li>
+</ul>
+
+<h3 id="non-monolithic-element-processing-with-restrictions">Non-monolithic element processing with restrictions</h3>
+
+<p>Splittable <code class="highlighter-rouge">DoFn</code> supports <code class="highlighter-rouge">Source</code>-like features by allowing the processing of
+a single element to be non-monolithic.</p>
+
+<p>The processing of one element by an SDF is decomposed into a (potentially
+infinite) number of <em>restrictions</em>, each describing some part of the work to be
+done for the whole element. The input to an SDF’s <code class="highlighter-rouge">@ProcessElement</code> call is a
+pair of an element and a restriction (compared to a regular <code class="highlighter-rouge">DoFn</code>, which takes
+just the element).</p>
+
+<p>Processing of every element starts by creating an <em>initial restriction</em> that
+describes the entire work, and the initial restriction is then split further
+into sub-restrictions which must logically add up to the original. For example,
+for a splittable <code class="highlighter-rouge">DoFn</code> called <code class="highlighter-rouge">ReadFn</code> that takes a filename and outputs
+records in the file, the restriction may be a pair of starting and ending byte
+offset, and <code class="highlighter-rouge">ReadFn</code> may interpret it as <em>read records whose starting offsets
+are in the given range</em>.</p>
+
+<p><img class="center-block" src="/images/blog/splittable-do-fn/restrictions.png" alt="Specifying parts of work for an element using restrictions" width="600" /></p>
+
+<p>The idea of restrictions provides non-monolithic execution - the first
+ingredient for parity with <code class="highlighter-rouge">Source</code>. The other ingredient is <em>interaction with
+the runner</em>: the runner has access to the restriction of each active
+<code class="highlighter-rouge">@ProcessElement</code> call of an SDF, can inquire about the progress of the call,
+and most importantly, can <em>split</em> the restriction while it is being processed
+(hence the name <em>Splittable DoFn</em>).</p>
+
+<p>Splitting produces a <em>primary</em> and <em>residual</em> restriction that add up to the
+original restriction being split: the current <code class="highlighter-rouge">@ProcessElement</code> call keeps
+processing the primary, and the residual will be processed by another
+<code class="highlighter-rouge">@ProcessElement</code> call. For example, a runner may schedule the residual to be
+processed in parallel on another worker.</p>
+
+<p>Splitting of a running <code class="highlighter-rouge">@ProcessElement</code> call has two critically important uses:</p>
+
+<ul>
+ <li><strong>Supporting infinite work per element.</strong> A restriction is, in general, not
+required to describe a finite amount of work. For example, reading from a Kafka
+topic starting from offset <em>100</em> can be represented by the
+restriction <em>[100, inf)</em>. A <code class="highlighter-rouge">@ProcessElement</code> call processing this
+entire restriction would, of course, never complete. However, while such a call
+runs, a runner can split the restriction into a <em>finite</em> primary <em>[100, 150)</em>
+(letting the current call complete this part) and an <em>infinite</em> residual <em>[150,
+inf)</em> to be processed later, effectively checkpointing and resuming the call;
+this can be repeated forever.</li>
+</ul>
+
+<p><img class="center-block" src="/images/blog/splittable-do-fn/kafka-splitting.png" alt="Splitting an infinite restriction into a finite primary and infinite residual" width="400" /></p>
+
+<ul>
+ <li><strong>Dynamic rebalancing.</strong> When a (typically batch-focused) runner detects that
+a <code class="highlighter-rouge">@ProcessElement</code> call is going to take too long and become a straggler, it
+can split the restriction in some proportion so that the primary is short enough
+to not be a straggler, and can schedule the residual in parallel on another
+worker. For details, see <a href="https://cloud.google.com/blog/big-data/2016/05/no-shard-left-behind-dynamic-work-rebalancing-in-google-cloud-dataflow">No Shard Left
+Behind</a>.</li>
+</ul>
+
+<p>Logically, the execution of an SDF on an element works according to the
+following diagram, where “magic” stands for the runner-specific ability to split
+the restrictions and schedule processing of residuals.</p>
+
+<p><img class="center-block" src="/images/blog/splittable-do-fn/transform-expansion.png" alt="Execution of an SDF - pairing with a restriction, splitting restrictions, processing element/restriction pairs" width="600" /></p>
+
+<p>This diagram emphasizes that splittability is an implementation detail of the
+particular <code class="highlighter-rouge">DoFn</code>: a splittable <code class="highlighter-rouge">DoFn</code> still looks like a <code class="highlighter-rouge">DoFn<A, B></code> to its
+user, and can be applied via a <code class="highlighter-rouge">ParDo</code> to a <code class="highlighter-rouge">PCollection<A></code> producing a
+<code class="highlighter-rouge">PCollection<B></code>.</p>
+
+<h3 id="which-dofns-need-to-be-splittable">Which DoFns need to be splittable</h3>
+
+<p>Note that decomposition of an element into element/restriction pairs is not
+automatic or “magical”: SDF is a new API for <em>authoring</em> a <code class="highlighter-rouge">DoFn</code>, rather than a
+new way to <em>execute</em> an existing <code class="highlighter-rouge">DoFn</code>. When making a <code class="highlighter-rouge">DoFn</code> splittable, the
+author needs to:</p>
+
+<ul>
+ <li>
+ <p>Consider the structure of the work it does for every element.</p>
+ </li>
+ <li>
+ <p>Come up with a scheme for describing parts of this work using restrictions.</p>
+ </li>
+ <li>
+ <p>Write code for creating the initial restriction, splitting it, and executing
+an element/restriction pair.</p>
+ </li>
+</ul>
+
+<p>An overwhelming majority of <code class="highlighter-rouge">DoFn</code>s found in user pipelines do not need to be
+made splittable: SDF is an advanced, powerful API, primarily targeting authors
+of new IO connectors <em>(though it has interesting non-IO applications as well:
+see <a href="http://s.apache.org/splittable-do-fn#heading=h.5cep9s8k4fxv">Non-IO examples</a>)</em>.</p>
+
+<h3 id="execution-of-a-restriction-and-data-consistency">Execution of a restriction and data consistency</h3>
+
+<p>One of the most important parts of the Splittable <code class="highlighter-rouge">DoFn</code> design is related to
+how it achieves data consistency while splitting. For example, while the runner
+is preparing to split the restriction of an active <code class="highlighter-rouge">@ProcessElement</code> call, how
+can it be sure that the call has not concurrently progressed past the point of
+splitting?</p>
+
+<p>This is achieved by requiring the processing of a restriction to follow a
+certain pattern. We think of a restriction as a sequence of <em>blocks</em> -
+elementary indivisible units of work, identified by a <em>position</em>. A
+<code class="highlighter-rouge">@ProcessElement</code> call processes the blocks one by one, first <em>claiming</em> the
+block’s position to atomically check if it’s still within the range of the
+restriction, until the whole restriction is processed.</p>
+
+<p>The diagram below illustrates this for <code class="highlighter-rouge">ReadFn</code> (a splittable <code class="highlighter-rouge">DoFn</code> that reads
+Avro files) processing the element <code class="highlighter-rouge">foo.avro</code> with restriction <code class="highlighter-rouge">[30, 70)</code>. This
+<code class="highlighter-rouge">@ProcessElement</code> call scans the Avro file for <a href="https://avro.apache.org/docs/current/spec.html#Object+Container+Files">data
+blocks</a>
+starting from offset <code class="highlighter-rouge">30</code> and claims the position of each block in this range.
+If a block is claimed successfully, then the call outputs all records in this
+data block, otherwise, it terminates.</p>
+
+<p><img class="center-block" src="/images/blog/splittable-do-fn/blocks.png" alt="Processing a restriction by claiming blocks inside it" width="400" /></p>
+
+<p>For more details, see <a href="http://s.apache.org/splittable-do-fn#heading=h.vjs7pzbb7kw">Restrictions, blocks and
+positions</a> in the
+design proposal document.</p>
+
+<h3 id="code-example">Code example</h3>
+
+<p>Let us look at some examples of SDF code. The examples use the Beam Java SDK,
+which <a href="https://github.com/apache/beam/blob/f7e8f886c91ea9d0b51e00331eeb4484e2f6e000/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java#L527">represents splittable
+<code class="highlighter-rouge">DoFn</code>s</a>
+as part of the flexible <a href="http://s.apache.org/a-new-dofn">annotation-based
+<code class="highlighter-rouge">DoFn</code></a> machinery, and the <a href="https://s.apache.org/splittable-do-fn-python">proposed SDF syntax
+for Python</a>.</p>
+
+<ul>
+ <li>
+ <p>A splittable <code class="highlighter-rouge">DoFn</code> is a <code class="highlighter-rouge">DoFn</code> - no new base class needed. Any SDF derives
+from the <code class="highlighter-rouge">DoFn</code> class and has a <code class="highlighter-rouge">@ProcessElement</code> method.</p>
+ </li>
+ <li>
+ <p>The <code class="highlighter-rouge">@ProcessElement</code> method takes an additional
+<a href="https://github.com/apache/beam/blob/f7e8f886c91ea9d0b51e00331eeb4484e2f6e000/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/RestrictionTracker.java"><code class="highlighter-rouge">RestrictionTracker</code></a>
+parameter that gives access to the current restriction in addition to the
+current element.</p>
+ </li>
+ <li>
+ <p>An SDF needs to define a <code class="highlighter-rouge">@GetInitialRestriction</code> method that can create a
+restriction describing the complete work for a given element.</p>
+ </li>
+ <li>
+ <p>There are several less important optional methods, such as
+<code class="highlighter-rouge">@SplitRestriction</code> for pre-splitting the initial restriction into several
+smaller restrictions, and a few others.</p>
+ </li>
+</ul>
+
+<p>The “Hello World” of SDF is a counter, which takes pairs <em>(x, N)</em> as input and
+produces pairs <em>(x, 0), (x, 1), …, (x, N-1)</em> as output.</p>
+
+<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">CountFn</span><span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="kd">extends</span> <span class="n">DoFn</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">T</span><span class="o">,</span> <span class="n">Long</span><span class="o">>,</span> <span class="n">KV</span><span clas [...]
+ <span class="nd">@ProcessElement</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">process</span><span class="o">(</span><span class="n">ProcessContext</span> <span class="n">c</span><span class="o">,</span> <span class="n">OffsetRangeTracker</span> <span class="n">tracker</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">for</span> <span class="o">(</span><span class="kt">long</span> <span class="n">i</span> <span class="o">=</span> <span class="n">tracker</span><span class="o">.</span><span class="na">currentRestriction</span><span class="o">().</span><span class="na">getFrom</span><span class="o">();</span> <span class="n">tracker</span><span class="o">.</span><span class="na">tryClaim</span><span class="o">(</span><span class="n">i</span><span class="o">);</span> <span class="o">++ [...]
+ <span class="n">c</span><span class="o">.</span><span class="na">output</span><span class="o">(</span><span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">c</span><span class="o">.</span><span class="na">element</span><span class="o">().</span><span class="na">getKey</span><span class="o">(),</span> <span class="n">i</span><span class="o">));</span>
+ <span class="o">}</span>
+ <span class="o">}</span>
+
+ <span class="nd">@GetInitialRestriction</span>
+ <span class="kd">public</span> <span class="n">OffsetRange</span> <span class="nf">getInitialRange</span><span class="o">(</span><span class="n">KV</span><span class="o"><</span><span class="n">T</span><span class="o">,</span> <span class="n">Long</span><span class="o">></span> <span class="n">element</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="k">new</span> <span class="nf">OffsetRange</span><span class="o">(</span><span class="mi">0L</span><span class="o">,</span> <span class="n">element</span><span class="o">.</span><span class="na">getValue</span><span class="o">());</span>
+ <span class="o">}</span>
+<span class="o">}</span>
+
+<span class="n">PCollection</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">>></span> <span class="n">input</span> <span class="o">=</span> <span class="err">…</span><span class="o">;</span>
+<span class="n">PCollection</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">>></span> <span class="n">output</span> <span class="o">=</span> <span class="n">input</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span>
+ <span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">CountFn</span><span class="o"><</span><span class="n">String</span><span class="o">>());</span>
+</code></pre>
+</div>
+
+<div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="k">class</span> <span class="nc">CountFn</span><span class="p">(</span><span class="n">DoFn</span><span class="p">):</span>
+ <span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="n">element</span><span class="p">,</span> <span class="n">tracker</span><span class="o">=</span><span class="n">DoFn</span><span class="o">.</span><span class="n">RestrictionTrackerParam</span><span class="p">)</span>
+ <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">xrange</span><span class="p">(</span><span class="o">*</span><span class="n">tracker</span><span class="o">.</span><span class="n">current_restriction</span><span class="p">()):</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">tracker</span><span class="o">.</span><span class="n">try_claim</span><span class="p">(</span><span class="n">i</span><span class="p">):</span>
+ <span class="k">return</span>
+ <span class="k">yield</span> <span class="n">element</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">i</span>
+
+ <span class="k">def</span> <span class="nf">get_initial_restriction</span><span class="p">(</span><span class="n">element</span><span class="p">):</span>
+ <span class="k">return</span> <span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">element</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
+</code></pre>
+</div>
+
+<p>This short <code class="highlighter-rouge">DoFn</code> subsumes the functionality of
+<a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/CountingSource.java">CountingSource</a>,
+but is more flexible: <code class="highlighter-rouge">CountingSource</code> generates only one sequence specified at
+pipeline construction time, while this <code class="highlighter-rouge">DoFn</code> can generate a dynamic family of
+sequences, one per element in the input collection (it does not matter whether
+the input collection is bounded or unbounded).</p>
+
+<p>However, the <code class="highlighter-rouge">Source</code>-specific capabilities of <code class="highlighter-rouge">CountingSource</code> are still
+available in <code class="highlighter-rouge">CountFn</code>. For example, if a sequence has a lot of elements, a
+batch-focused runner can still apply dynamic rebalancing to it and generate
+different subranges of the sequence in parallel by splitting the <code class="highlighter-rouge">OffsetRange</code>.
+Likewise, a streaming-focused runner can use the same splitting logic to
+checkpoint and resume the generation of the sequence even if it is, for
+practical purposes, infinite (for example, when applied to a <code class="highlighter-rouge">KV(...,
+Long.MAX_VALUE)</code>).</p>
+
+<p>A slightly more complex example is the <code class="highlighter-rouge">ReadFn</code> considered above, which reads
+data from Avro files and illustrates the idea of <em>blocks</em>: we provide pseudocode
+to illustrate the approach.</p>
+
+<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">ReadFn</span> <span class="kd">extends</span> <span class="n">DoFn</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">AvroRecord</span><span class="o">></span> <span class="o">{</span>
+ <span class="nd">@ProcessElement</span>
+ <span class="kt">void</span> <span class="nf">process</span><span class="o">(</span><span class="n">ProcessContext</span> <span class="n">c</span><span class="o">,</span> <span class="n">OffsetRangeTracker</span> <span class="n">tracker</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">try</span> <span class="o">(</span><span class="n">AvroReader</span> <span class="n">reader</span> <span class="o">=</span> <span class="n">Avro</span><span class="o">.</span><span class="na">open</span><span class="o">(</span><span class="n">filename</span><span class="o">))</span> <span class="o">{</span>
+ <span class="c1">// Seek to the first block starting at or after the start offset.</span>
+ <span class="n">reader</span><span class="o">.</span><span class="na">seek</span><span class="o">(</span><span class="n">tracker</span><span class="o">.</span><span class="na">currentRestriction</span><span class="o">().</span><span class="na">getFrom</span><span class="o">());</span>
+ <span class="k">while</span> <span class="o">(</span><span class="n">reader</span><span class="o">.</span><span class="na">readNextBlock</span><span class="o">())</span> <span class="o">{</span>
+ <span class="c1">// Claim the position of the current Avro block</span>
+ <span class="k">if</span> <span class="o">(!</span><span class="n">tracker</span><span class="o">.</span><span class="na">tryClaim</span><span class="o">(</span><span class="n">reader</span><span class="o">.</span><span class="na">currentBlockOffset</span><span class="o">()))</span> <span class="o">{</span>
+ <span class="c1">// Out of range of the current restriction - we're done.</span>
+ <span class="k">return</span><span class="o">;</span>
+ <span class="o">}</span>
+ <span class="c1">// Emit all records in this block</span>
+ <span class="k">for</span> <span class="o">(</span><span class="n">AvroRecord</span> <span class="n">record</span> <span class="o">:</span> <span class="n">reader</span><span class="o">.</span><span class="na">currentBlock</span><span class="o">())</span> <span class="o">{</span>
+ <span class="n">c</span><span class="o">.</span><span class="na">output</span><span class="o">(</span><span class="n">record</span><span class="o">);</span>
+ <span class="o">}</span>
+ <span class="o">}</span>
+ <span class="o">}</span>
+ <span class="o">}</span>
+
+ <span class="nd">@GetInitialRestriction</span>
+ <span class="n">OffsetRange</span> <span class="nf">getInitialRestriction</span><span class="o">(</span><span class="n">String</span> <span class="n">filename</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="k">new</span> <span class="nf">OffsetRange</span><span class="o">(</span><span class="mi">0</span><span class="o">,</span> <span class="k">new</span> <span class="n">File</span><span class="o">(</span><span class="n">filename</span><span class="o">).</span><span class="na">getSize</span><span class="o">());</span>
+ <span class="o">}</span>
+<span class="o">}</span>
+</code></pre>
+</div>
+
+<div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="k">class</span> <span class="nc">AvroReader</span><span class="p">(</span><span class="n">DoFn</span><span class="p">):</span>
+ <span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="n">tracker</span><span class="o">=</span><span class="n">DoFn</span><span class="o">.</span><span class="n">RestrictionTrackerParam</span><span class="p">)</span>
+ <span class="k">with</span> <span class="n">fileio</span><span class="o">.</span><span class="n">ChannelFactory</span><span class="o">.</span><span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span> <span class="k">as</span> <span class="nb">file</span><span class="p">:</span>
+ <span class="n">start</span><span class="p">,</span> <span class="n">stop</span> <span class="o">=</span> <span class="n">tracker</span><span class="o">.</span><span class="n">current_restriction</span><span class="p">()</span>
+ <span class="c"># Seek to the first block starting at or after the start offset.</span>
+ <span class="nb">file</span><span class="o">.</span><span class="n">seek</span><span class="p">(</span><span class="n">start</span><span class="p">)</span>
+ <span class="n">block</span> <span class="o">=</span> <span class="n">AvroUtils</span><span class="o">.</span><span class="n">get_next_block</span><span class="p">(</span><span class="nb">file</span><span class="p">)</span>
+ <span class="k">while</span> <span class="n">block</span><span class="p">:</span>
+ <span class="c"># Claim the position of the current Avro block</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">tracker</span><span class="o">.</span><span class="n">try_claim</span><span class="p">(</span><span class="n">block</span><span class="o">.</span><span class="n">start</span><span class="p">()):</span>
+ <span class="c"># Out of range of the current restriction - we're done.</span>
+ <span class="k">return</span>
+ <span class="c"># Emit all records in this block</span>
+ <span class="k">for</span> <span class="n">record</span> <span class="ow">in</span> <span class="n">block</span><span class="o">.</span><span class="n">records</span><span class="p">():</span>
+ <span class="k">yield</span> <span class="n">record</span>
+ <span class="n">block</span> <span class="o">=</span> <span class="n">AvroUtils</span><span class="o">.</span><span class="n">get_next_block</span><span class="p">(</span><span class="nb">file</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">get_initial_restriction</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">filename</span><span class="p">):</span>
+ <span class="k">return</span> <span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">fileio</span><span class="o">.</span><span class="n">ChannelFactory</span><span class="o">.</span><span class="n">size_in_bytes</span><span class="p">(</span><span class="n">filename</span><span class="p">))</span>
+</code></pre>
+</div>
+
+<p>This hypothetical <code class="highlighter-rouge">DoFn</code> reads records from a single Avro file. Notably missing
+is the code for expanding a filepattern: it no longer needs to be part of this
+<code class="highlighter-rouge">DoFn</code>! Instead, the SDK includes a
+<a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileIO.java">FileIO.matchAll()</a>
+transform for expanding a filepattern into a <code class="highlighter-rouge">PCollection</code> of filenames, and
+different file format IOs can reuse the same transform, reading the files with
+different <code class="highlighter-rouge">DoFn</code>s.</p>
+
+<p>This example demonstrates the benefits of increased modularity allowed by SDF:
+<code class="highlighter-rouge">FileIO.matchAll()</code> supports continuous ingestion of new files in streaming
+pipelines using <code class="highlighter-rouge">.continuously()</code>, and this functionality becomes automatically
+available to various file format IOs. For example,
+<code class="highlighter-rouge">TextIO.read().watchForNewFiles()</code> <a href="https://github.com/apache/beam/blob/3bd68ecfd7d576d78e02deb0476e549f11e1b5ef/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java#L486">uses <code class="highlighter-rouge">FileIO.matchAll()</code> under the
+hood)</a>.</p>
+
+<h2 id="current-status">Current status</h2>
+
+<p>Splittable <code class="highlighter-rouge">DoFn</code> is a major new API, and its delivery and widespread adoption
+involves a lot of work in different parts of the Apache Beam ecosystem. Some
+of that work is already complete and provides direct benefit to users via new
+IO connectors. However, a large amount of work is in progress or planned.</p>
+
+<p>As of August 2017, SDF is available for use in the Beam Java Direct runner and
+Dataflow Streaming runner, and implementation is in progress in the Flink and
+Apex runners; see <a href="/documentation/runners/capability-matrix/">capability matrix</a> for the current status. Support
+for SDF in the Python SDK is <a href="https://s.apache.org/splittable-do-fn-python">in active
+development</a>.</p>
+
+<p>Several SDF-based transforms and IO connectors are available for Beam users at
+HEAD and will be included in Beam 2.2.0. <code class="highlighter-rouge">TextIO</code> and <code class="highlighter-rouge">AvroIO</code> finally provide
+continuous ingestion of files (one of the most frequently requested features)
+via <code class="highlighter-rouge">.watchForNewFiles()</code> which is backed by the utility transforms
+<code class="highlighter-rouge">FileIO.matchAll().continuously()</code> and the more general
+<a href="https://github.com/apache/beam/blob/f7e8f886c91ea9d0b51e00331eeb4484e2f6e000/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Watch.java"><code class="highlighter-rouge">Watch.growthOf()</code></a>.
+These utility transforms are also independently useful for “power user” use
+cases.</p>
+
+<p>To enable more flexible use cases for IOs currently based on the Source API, we
+will change them to use SDF. This transition is <a href="http://s.apache.org/textio-sdf">pioneered by
+TextIO</a> and involves temporarily <a href="http://s.apache.org/sdf-via-source">executing SDF
+via the Source API</a> to support runners
+lacking the ability to run SDF directly.</p>
+
+<p>In addition to enabling new IOs, work on SDF has influenced our thinking about
+other parts of the Beam programming model:</p>
+
+<ul>
+ <li>
+ <p>SDF unified the final remaining part of the Beam programming model that was
+not batch/streaming agnostic (the <code class="highlighter-rouge">Source</code> API). This led us to consider use
+cases that cannot be described as purely batch or streaming (for example,
+ingesting a large amount of historical data and carrying on with more data
+arriving in real time) and to develop a <a href="http://s.apache.org/beam-fn-api-progress-reporting">unified notion of “progress” and
+“backlog”</a>.</p>
+ </li>
+ <li>
+ <p>The <a href="http://s.apache.org/beam-fn-api">Fn API</a> - the foundation of Beam’s
+future support for cross-language pipelines - uses SDF as <em>the only</em> concept
+representing data ingestion.</p>
+ </li>
+ <li>
+ <p>Implementation of SDF has lead to <a href="https://lists.apache.org/thread.html/86831496a08fe148e3b982cdb904f828f262c0b571543a9fed7b915d@%3Cdev.beam.apache.org%3E">formalizing pipeline termination
+semantics</a>
+and making it consistent between runners.</p>
+ </li>
+ <li>
+ <p>SDF set a new standard for how modular IO connectors can be, inspiring
+creation of similar APIs for some non-SDF-based connectors (for example,
+<code class="highlighter-rouge">SpannerIO.readAll()</code> and the
+<a href="https://issues.apache.org/jira/browse/BEAM-2706">planned</a> <code class="highlighter-rouge">JdbcIO.readAll()</code>).</p>
+ </li>
+</ul>
+
+<h2 id="call-to-action">Call to action</h2>
+
+<p>Apache Beam thrives on having a large community of contributors. Here are some
+ways you can get involved in the SDF effort and help make the Beam IO connector
+ecosystem more modular:</p>
+
+<ul>
+ <li>
+ <p>Use the currently available SDF-based IO connectors, provide feedback, file
+bugs, and suggest or implement improvements.</p>
+ </li>
+ <li>
+ <p>Propose or develop a new IO connector based on SDF.</p>
+ </li>
+ <li>
+ <p>Implement or improve support for SDF in your favorite runner.</p>
+ </li>
+ <li>
+ <p>Subscribe and contribute to the occasional SDF-related discussions on
+<a href="mailto:user@beam.apache.org">user@beam.apache.org</a> (mailing list for Beam
+users) and <a href="mailto:dev@beam.apache.org">dev@beam.apache.org</a> (mailing list for
+Beam developers)!</p>
+ </li>
+</ul>
+
+ </div>
+
+</article>
+
+ </div>
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<footer class="footer">
+ <div class="footer__contained">
+ <div class="footer__cols">
+ <div class="footer__cols__col">
+ <div class="footer__cols__col__logo">
+ <img src="/images/beam_logo_circle.svg" class="footer__logo" alt="Beam logo">
+ </div>
+ <div class="footer__cols__col__logo">
+ <img src="/images/apache_logo_circle.svg" class="footer__logo" alt="Apache logo">
+ </div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Start</div>
+ <div class="footer__cols__col__link"><a href="/get-started/beam-overview/">Overview</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-java/">Quickstart (Java)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-py/">Quickstart (Python)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-go/">Quickstart (Go)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/downloads/">Downloads</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Docs</div>
+ <div class="footer__cols__col__link"><a href="/documentation/programming-guide/">Concepts</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/pipelines/design-your-pipeline/">Pipelines</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/runners/capability-matrix/">Runners</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Community</div>
+ <div class="footer__cols__col__link"><a href="/contribute/">Contribute</a></div>
+ <div class="footer__cols__col__link"><a href="https://projects.apache.org/committee.html?beam" target="_blank">Team<img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a></div>
+ <div class="footer__cols__col__link"><a href="/contribute/presentation-materials/">Media</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Resources</div>
+ <div class="footer__cols__col__link"><a href="/blog/">Blog</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/support/">Support</a></div>
+ <div class="footer__cols__col__link"><a href="https://github.com/apache/beam">GitHub</a></div>
+ </div>
+ </div>
+ </div>
+ <div class="footer__bottom">
+ ©
+ <a href="http://www.apache.org">The Apache Software Foundation</a>
+ | <a href="/privacy_policy">Privacy Policy</a>
+ | <a href="/feed.xml">RSS Feed</a>
+ <br><br>
+ Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are
+ either registered trademarks or trademarks of The Apache Software
+ Foundation. All other products or name brands are trademarks of their
+ respective holders, including The Apache Software Foundation.
+ </div>
+</footer>
+
+ </body>
+</html>
diff --git a/website/generated-content/blog/2017/08/28/timely-processing.html b/website/generated-content/blog/2017/08/28/timely-processing.html
new file mode 100644
index 0000000..34ce597
--- /dev/null
+++ b/website/generated-content/blog/2017/08/28/timely-processing.html
@@ -0,0 +1,736 @@
+<!DOCTYPE html>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<html lang="en">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <title>Timely (and Stateful) Processing with Apache Beam</title>
+ <meta name="description" content="In a prior blogpost, Iintroduced the basics of stateful processing in Apache Beam, focusing on theaddition of state to per-element processing. So-called time...">
+ <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
+ <link rel="stylesheet" href="/css/site.css">
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <script src="/js/language-switch.js"></script>
+ <script src="/js/fix-menu.js"></script>
+ <script src="/js/section-nav.js"></script>
+ <script src="/js/page-nav.js"></script>
+ <link rel="canonical" href="https://beam.apache.org/blog/2017/08/28/timely-processing.html" data-proofer-ignore>
+ <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
+ <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+ ga('create', 'UA-73650088-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+</head>
+
+ <body class="body ">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<nav class="header navbar navbar-fixed-top">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+
+ <a href="/" class="navbar-brand" >
+ <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
+ </a>
+ </div>
+
+ <div class="navbar-mask closed"></div>
+
+ <div id="navbar" class="navbar-container closed">
+ <ul class="nav navbar-nav">
+ <li>
+ <a href="/get-started/beam-overview/">Get Started</a>
+ </li>
+ <li>
+ <a href="/documentation/">Documentation</a>
+ </li>
+ <li>
+ <a href="/documentation/sdks/java/">SDKS</a>
+ </li>
+ <li>
+ <a href="/documentation/runners/capability-matrix/">RUNNERS</a>
+ </li>
+ <li>
+ <a href="/contribute/">Contribute</a>
+ </li>
+ <li>
+ <a href="/community/contact-us/">Community</a>
+ </li>
+ <li><a href="/blog">Blog</a></li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:20px;"><span class="caret"></span></a>
+ <ul class="dropdown-menu dropdown-menu-right">
+ <li><a href="http://www.apache.org/">ASF Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">License</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
+ </ul>
+ </li>
+ </ul>
+ </div>
+</nav>
+
+ <div class="body__contained">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+
+
+<article class="post" itemscope itemtype="http://schema.org/BlogPosting">
+
+ <header class="post-header">
+ <h1 class="post-title" itemprop="name headline">Timely (and Stateful) Processing with Apache Beam</h1>
+ <p class="post-meta"><time datetime="2017-08-28T01:00:01-07:00" itemprop="datePublished">Aug 28, 2017</time> •
+ Kenneth Knowles [<a href="https://twitter.com/KennKnowles">@KennKnowles</a>]
+
+ </p>
+ </header>
+
+ <div class="post-content" itemprop="articleBody">
+ <!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+<p>In a <a href="/blog/2017/02/13/stateful-processing.html">prior blog
+post</a>, I
+introduced the basics of stateful processing in Apache Beam, focusing on the
+addition of state to per-element processing. So-called <em>timely</em> processing
+complements stateful processing in Beam by letting you set timers to request a
+(stateful) callback at some point in the future.</p>
+
+<p>What can you do with timers in Beam? Here are some examples:</p>
+
+<ul>
+ <li>You can output data buffered in state after some amount of processing time.</li>
+ <li>You can take special action when the watermark estimates that you have
+received all data up to a specified point in event time.</li>
+ <li>You can author workflows with timeouts that alter state and emit output in
+response to the absence of additional input for some period of time.</li>
+</ul>
+
+<p>These are just a few possibilities. State and timers together form a powerful
+programming paradigm for fine-grained control to express a huge variety of
+workflows. Stateful and timely processing in Beam is portable across data
+processing engines and integrated with Beam’s unified model of event time
+windowing in both streaming and batch processing.</p>
+
+<!--more-->
+
+<h2 id="what-is-stateful-and-timely-processing">What is stateful and timely processing?</h2>
+
+<p>In my prior post, I developed an understanding of stateful processing largely
+by contrast with associative, commutative combiners. In this post, I’ll
+emphasize a perspective that I had mentioned only briefly: that elementwise
+processing with access to per-key-and-window state and timers represents a
+fundamental pattern for “embarrassingly parallel” computation, distinct from
+the others in Beam.</p>
+
+<p>In fact, stateful and timely computation is the low-level computational pattern
+that underlies the others. Precisely because it is lower level, it allows you
+to really micromanage your computations to unlock new use cases and new
+efficiencies. This incurs the complexity of manually managing your state and
+timers - it isn’t magic! Let’s first look again at the two primary
+computational patterns in Beam.</p>
+
+<h3 id="element-wise-processing-pardo-map-etc">Element-wise processing (ParDo, Map, etc)</h3>
+
+<p>The most elementary embarrassingly parallel pattern is just using a bunch of
+computers to apply the same function to every input element of a massive
+collection. In Beam, per-element processing like this is expressed as a basic
+<code class="highlighter-rouge">ParDo</code> - analogous to “Map” from MapReduce - which is like an enhanced “map”,
+“flatMap”, etc, from functional programming.</p>
+
+<p>The following diagram illustrates per-element processing. Input elements are
+squares, output elements are triangles. The colors of the elements represent
+their key, which will matter later. Each input element maps to the
+corresponding output element(s) completely independently. Processing may be
+distributed across computers in any way, yielding essentially limitless
+parallelism.</p>
+
+<p><img class="center-block" src="/images/blog/timely-processing/ParDo.png" alt="ParDo offers limitless parallelism" width="600" /></p>
+
+<p>This pattern is obvious, exists in all data-parallel paradigms, and has
+a simple stateless implementation. Every input element can be processed
+independently or in arbitrary bundles. Balancing the work between computers is
+actually the hard part, and can be addressed by splitting, progress estimation,
+work-stealing, etc.</p>
+
+<h3 id="per-key-and-window-aggregation-combine-reduce-groupbykey-etc">Per-key (and window) aggregation (Combine, Reduce, GroupByKey, etc.)</h3>
+
+<p>The other embarassingly parallel design pattern at the heart of Beam is per-key
+(and window) aggregation. Elements sharing a key are colocated and then
+combined using some associative and commutative operator. In Beam this is
+expressed as a <code class="highlighter-rouge">GroupByKey</code> or <code class="highlighter-rouge">Combine.perKey</code>, and corresponds to the shuffle
+and “Reduce” from MapReduce. It is sometimes helpful to think of per-key
+<code class="highlighter-rouge">Combine</code> as the fundamental operation, and raw <code class="highlighter-rouge">GroupByKey</code> as a combiner that
+just concatenates input elements. The communication pattern for the input
+elements is the same, modulo some optimizations possible for <code class="highlighter-rouge">Combine</code>.</p>
+
+<p>In the illustration here, recall that the color of each element represents the
+key. So all of the red squares are routed to the same location where they are
+aggregated and the red triangle is the output. Likewise for the yellow and
+green squares, etc. In a real application, you may have millions of keys, so
+the parallelism is still massive.</p>
+
+<p><img class="center-block" src="/images/blog/timely-processing/CombinePerKey.png" alt="Gathering elements per key then combining them" width="600" /></p>
+
+<p>The underlying data processing engine will, at some level of abstraction, use
+state to perform this aggregation across all the elements arriving for a key.
+In particular, in a streaming execution, the aggregation process may need to
+wait for more data to arrive or for the watermark to estimate that all input
+for an event time window is complete. This requires some way to store the
+intermediate aggregation between input elements as well a way to a receive a
+callback when it is time to emit the result. As a result, the <em>execution</em> of
+per key aggregation by a stream processing engine fundamentally involves state
+and timers.</p>
+
+<p>However, <em>your</em> code is just a declarative expression of the aggregation
+operator. The runner can choose a variety of ways to execute your operator.
+I went over this in detail in <a href="/blog/2017/02/13/stateful-processing.html">my prior post focused on state alone</a>. Since you do not
+observe elements in any defined order, nor manipulate mutable state or timers
+directly, I call this neither stateful nor timely processing.</p>
+
+<h3 id="per-key-and-window-stateful-timely-processing">Per-key-and-window stateful, timely processing</h3>
+
+<p>Both <code class="highlighter-rouge">ParDo</code> and <code class="highlighter-rouge">Combine.perKey</code> are standard patterns for parallelism that go
+back decades. When implementing these in a massive-scale distributed data
+processing engine, we can highlight a few characteristics that are particularly
+important.</p>
+
+<p>Let us consider these characteristics of <code class="highlighter-rouge">ParDo</code>:</p>
+
+<ul>
+ <li>You write single-threaded code to process one element.</li>
+ <li>Elements are processed in arbitrary order with no dependencies
+or interaction between processing of elements.</li>
+</ul>
+
+<p>And these characteristics for <code class="highlighter-rouge">Combine.perKey</code>:</p>
+
+<ul>
+ <li>Elements for a common key and window are gathered together.</li>
+ <li>A user-defined operator is applied to those elements.</li>
+</ul>
+
+<p>Combining some of the characteristics of unrestricted parallel mapping and
+per-key-and-window combination, we can discern a megaprimitive from which we
+build stateful and timely processing:</p>
+
+<ul>
+ <li>Elements for a common key and window are gathered together.</li>
+ <li>Elements are processed in arbitrary order.</li>
+ <li>You write single-threaded code to process one element or timer, possibly
+accessing state or setting timers.</li>
+</ul>
+
+<p>In the illustration below, the red squares are gathered and fed one by one to
+the stateful, timely, <code class="highlighter-rouge">DoFn</code>. As each element is processed, the <code class="highlighter-rouge">DoFn</code> has
+access to state (the color-partitioned cylinder on the right) and can set
+timers to receive callbacks (the colorful clocks on the left).</p>
+
+<p><img class="center-block" src="/images/blog/timely-processing/StateAndTimers.png" alt="Gathering elements per key then timely, stateful processing" width="600" /></p>
+
+<p>So that is the abstract notion of per-key-and-window stateful, timely
+processing in Apache Beam. Now let’s see what it looks like to write code that
+accesses state, sets timers, and receives callbacks.</p>
+
+<h2 id="example-batched-rpc">Example: Batched RPC</h2>
+
+<p>To demonstrate stateful and timely processing, let’s work through a concrete
+example, with code.</p>
+
+<p>Suppose you are writing a system to analyze events. You have a ton of data
+coming in and you need to enrich each event by RPC to an external system. You
+can’t just issue an RPC per event. Not only would this be terrible for
+performance, but it would also likely blow your quota with the external system.
+So you’d like to gather a number of events, make one RPC for them all, and then
+output all the enriched events.</p>
+
+<h3 id="state">State</h3>
+
+<p>Let’s set up the state we need to track batches of elements. As each element
+comes in, we will write the element to a buffer while tracking the number of
+elements we have buffered. Here are the state cells in code:</p>
+
+<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="k">new</span> <span class="n">DoFn</span><span class="o"><</span><span class="n">Event</span><span class="o">,</span> <span class="n">EnrichedEvent</span><span class="o">>()</span> <span class="o">{</span>
+
+ <span class="nd">@StateId</span><span class="o">(</span><span class="s">"buffer"</span><span class="o">)</span>
+ <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o"><</span><span class="n">BagState</span><span class="o"><</span><span class="n">Event</span><span class="o">>></span> <span class="n">bufferedEvents</span> <span class="o">=</span> <span class="n">StateSpecs</span><span class="o">.</span><span class="na">bag</span><span class="o">();</span>
+
+ <span class="nd">@StateId</span><span class="o">(</span><span class="s">"count"</span><span class="o">)</span>
+ <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o"><</span><span class="n">ValueState</span><span class="o"><</span><span class="n">Integer</span><span class="o">>></span> <span class="n">countState</span> <span class="o">=</span> <span class="n">StateSpecs</span><span class="o">.</span><span class="na">value</span><span class="o">();</span>
+
+ <span class="err">…</span> <span class="n">TBD</span> <span class="err">…</span>
+<span class="o">}</span>
+</code></pre>
+</div>
+
+<div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="c"># State and timers are not yet supported in Beam's Python SDK.</span>
+<span class="c"># Follow https://issues.apache.org/jira/browse/BEAM-2687 for updates.</span>
+</code></pre>
+</div>
+
+<p>Walking through the code, we have:</p>
+
+<ul>
+ <li>The state cell <code class="highlighter-rouge">"buffer"</code> is an unordered bag of buffered events.</li>
+ <li>The state cell <code class="highlighter-rouge">"count"</code> tracks how many events have been buffered.</li>
+</ul>
+
+<p>Next, as a recap of reading and writing state, let’s write our <code class="highlighter-rouge">@ProcessElement</code>
+method. We will choose a limit on the size of the buffer, <code class="highlighter-rouge">MAX_BUFFER_SIZE</code>. If
+our buffer reaches this size, we will perform a single RPC to enrich all the
+events, and output.</p>
+
+<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="k">new</span> <span class="n">DoFn</span><span class="o"><</span><span class="n">Event</span><span class="o">,</span> <span class="n">EnrichedEvent</span><span class="o">>()</span> <span class="o">{</span>
+
+ <span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="kt">int</span> <span class="n">MAX_BUFFER_SIZE</span> <span class="o">=</span> <span class="mi">500</span><span class="o">;</span>
+
+ <span class="nd">@StateId</span><span class="o">(</span><span class="s">"buffer"</span><span class="o">)</span>
+ <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o"><</span><span class="n">BagState</span><span class="o"><</span><span class="n">Event</span><span class="o">>></span> <span class="n">bufferedEvents</span> <span class="o">=</span> <span class="n">StateSpecs</span><span class="o">.</span><span class="na">bag</span><span class="o">();</span>
+
+ <span class="nd">@StateId</span><span class="o">(</span><span class="s">"count"</span><span class="o">)</span>
+ <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o"><</span><span class="n">ValueState</span><span class="o"><</span><span class="n">Integer</span><span class="o">>></span> <span class="n">countState</span> <span class="o">=</span> <span class="n">StateSpecs</span><span class="o">.</span><span class="na">value</span><span class="o">();</span>
+
+ <span class="nd">@ProcessElement</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">process</span><span class="o">(</span>
+ <span class="n">ProcessContext</span> <span class="n">context</span><span class="o">,</span>
+ <span class="nd">@StateId</span><span class="o">(</span><span class="s">"buffer"</span><span class="o">)</span> <span class="n">BagState</span><span class="o"><</span><span class="n">Event</span><span class="o">></span> <span class="n">bufferState</span><span class="o">,</span>
+ <span class="nd">@StateId</span><span class="o">(</span><span class="s">"count"</span><span class="o">)</span> <span class="n">ValueState</span><span class="o"><</span><span class="n">Integer</span><span class="o">></span> <span class="n">countState</span><span class="o">)</span> <span class="o">{</span>
+
+ <span class="kt">int</span> <span class="n">count</span> <span class="o">=</span> <span class="n">firstNonNull</span><span class="o">(</span><span class="n">countState</span><span class="o">.</span><span class="na">read</span><span class="o">(),</span> <span class="mi">0</span><span class="o">);</span>
+ <span class="n">count</span> <span class="o">=</span> <span class="n">count</span> <span class="o">+</span> <span class="mi">1</span><span class="o">;</span>
+ <span class="n">countState</span><span class="o">.</span><span class="na">write</span><span class="o">(</span><span class="n">count</span><span class="o">);</span>
+ <span class="n">bufferState</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">context</span><span class="o">.</span><span class="na">element</span><span class="o">());</span>
+
+ <span class="k">if</span> <span class="o">(</span><span class="n">count</span> <span class="o">></span> <span class="n">MAX_BUFFER_SIZE</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">for</span> <span class="o">(</span><span class="n">EnrichedEvent</span> <span class="n">enrichedEvent</span> <span class="o">:</span> <span class="n">enrichEvents</span><span class="o">(</span><span class="n">bufferState</span><span class="o">.</span><span class="na">read</span><span class="o">()))</span> <span class="o">{</span>
+ <span class="n">context</span><span class="o">.</span><span class="na">output</span><span class="o">(</span><span class="n">enrichedEvent</span><span class="o">);</span>
+ <span class="o">}</span>
+ <span class="n">bufferState</span><span class="o">.</span><span class="na">clear</span><span class="o">();</span>
+ <span class="n">countState</span><span class="o">.</span><span class="na">clear</span><span class="o">();</span>
+ <span class="o">}</span>
+ <span class="o">}</span>
+
+ <span class="err">…</span> <span class="n">TBD</span> <span class="err">…</span>
+<span class="o">}</span>
+</code></pre>
+</div>
+
+<div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="c"># State and timers are not yet supported in Beam's Python SDK.</span>
+<span class="c"># Follow https://issues.apache.org/jira/browse/BEAM-2687 for updates.</span>
+</code></pre>
+</div>
+
+<p>Here is an illustration to accompany the code:</p>
+
+<p><img class="center-block" src="/images/blog/timely-processing/BatchedRpcState.png" alt="Batching elements in state, then performing RPCs" width="600" /></p>
+
+<ul>
+ <li>The blue box is the <code class="highlighter-rouge">DoFn</code>.</li>
+ <li>The yellow box within it is the <code class="highlighter-rouge">@ProcessElement</code> method.</li>
+ <li>Each input event is a red square - this diagram just shows the activity for
+a single key, represented by the color red. Your <code class="highlighter-rouge">DoFn</code> will run the same
+workflow in parallel for all keys which are perhaps user IDs.</li>
+ <li>Each input event is written to the buffer as a red triangle, representing
+the fact that you might actually buffer more than just the raw input, even
+though this code doesn’t.</li>
+ <li>The external service is drawn as a cloud. When there are enough buffered
+events, the <code class="highlighter-rouge">@ProcessElement</code> method reads the events from state and issues
+a single RPC.</li>
+ <li>Each output enriched event is drawn as a red circle. To consumers of this
+output, it looks just like an element-wise operation.</li>
+</ul>
+
+<p>So far, we have only used state, but not timers. You may have noticed that
+there is a problem - there will usually be data left in the buffer. If no more
+input arrives, that data will never be processed. In Beam, every window has
+some point in event time when any further input for the window is considered
+too late and is discarded. At this point, we say that the window has “expired”.
+Since no further input can arrive to access the state for that window, the
+state is also discarded. For our example, we need to ensure that all leftover
+events are output when the window expires.</p>
+
+<h3 id="event-time-timers">Event Time Timers</h3>
+
+<p>An event time timer requests a call back when the watermark for an input
+<code class="highlighter-rouge">PCollection</code> reaches some threshold. In other words, you can use an event time
+timer to take action at a specific moment in event time - a particular point of
+completeness for a <code class="highlighter-rouge">PCollection</code> - such as when a window expires.</p>
+
+<p>For our example, let us add an event time timer so that when the window expires,
+any events remaining in the buffer are processed.</p>
+
+<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="k">new</span> <span class="n">DoFn</span><span class="o"><</span><span class="n">Event</span><span class="o">,</span> <span class="n">EnrichedEvent</span><span class="o">>()</span> <span class="o">{</span>
+ <span class="err">…</span>
+
+ <span class="nd">@TimerId</span><span class="o">(</span><span class="s">"expiry"</span><span class="o">)</span>
+ <span class="kd">private</span> <span class="kd">final</span> <span class="n">TimerSpec</span> <span class="n">expirySpec</span> <span class="o">=</span> <span class="n">TimerSpecs</span><span class="o">.</span><span class="na">timer</span><span class="o">(</span><span class="n">TimeDomain</span><span class="o">.</span><span class="na">EVENT_TIME</span><span class="o">);</span>
+
+ <span class="nd">@ProcessElement</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">process</span><span class="o">(</span>
+ <span class="n">ProcessContext</span> <span class="n">context</span><span class="o">,</span>
+ <span class="n">BoundedWindow</span> <span class="n">window</span><span class="o">,</span>
+ <span class="nd">@StateId</span><span class="o">(</span><span class="s">"buffer"</span><span class="o">)</span> <span class="n">BagState</span><span class="o"><</span><span class="n">Event</span><span class="o">></span> <span class="n">bufferState</span><span class="o">,</span>
+ <span class="nd">@StateId</span><span class="o">(</span><span class="s">"count"</span><span class="o">)</span> <span class="n">ValueState</span><span class="o"><</span><span class="n">Integer</span><span class="o">></span> <span class="n">countState</span><span class="o">,</span>
+ <span class="nd">@TimerId</span><span class="o">(</span><span class="s">"expiry"</span><span class="o">)</span> <span class="n">Timer</span> <span class="n">expiryTimer</span><span class="o">)</span> <span class="o">{</span>
+
+ <span class="n">expiryTimer</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="n">window</span><span class="o">.</span><span class="na">maxTimestamp</span><span class="o">().</span><span class="na">plus</span><span class="o">(</span><span class="n">allowedLateness</span><span class="o">));</span>
+
+ <span class="err">…</span> <span class="n">same</span> <span class="n">logic</span> <span class="n">as</span> <span class="n">above</span> <span class="err">…</span>
+ <span class="o">}</span>
+
+ <span class="nd">@OnTimer</span><span class="o">(</span><span class="s">"expiry"</span><span class="o">)</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">onExpiry</span><span class="o">(</span>
+ <span class="n">OnTimerContext</span> <span class="n">context</span><span class="o">,</span>
+ <span class="nd">@StateId</span><span class="o">(</span><span class="s">"buffer"</span><span class="o">)</span> <span class="n">BagState</span><span class="o"><</span><span class="n">Event</span><span class="o">></span> <span class="n">bufferState</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">if</span> <span class="o">(!</span><span class="n">bufferState</span><span class="o">.</span><span class="na">isEmpty</span><span class="o">().</span><span class="na">read</span><span class="o">())</span> <span class="o">{</span>
+ <span class="k">for</span> <span class="o">(</span><span class="n">EnrichedEvent</span> <span class="n">enrichedEvent</span> <span class="o">:</span> <span class="n">enrichEvents</span><span class="o">(</span><span class="n">bufferState</span><span class="o">.</span><span class="na">read</span><span class="o">()))</span> <span class="o">{</span>
+ <span class="n">context</span><span class="o">.</span><span class="na">output</span><span class="o">(</span><span class="n">enrichedEvent</span><span class="o">);</span>
+ <span class="o">}</span>
+ <span class="o">}</span>
+ <span class="o">}</span>
+<span class="o">}</span>
+</code></pre>
+</div>
+
+<div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="c"># State and timers are not yet supported in Beam's Python SDK.</span>
+<span class="c"># Follow https://issues.apache.org/jira/browse/BEAM-2687 for updates.</span>
+</code></pre>
+</div>
+
+<p>Let’s unpack the pieces of this snippet:</p>
+
+<ul>
+ <li>
+ <p>We declare an event time timer with <code class="highlighter-rouge">@TimerId("expiry")</code>. We will use the
+identifier <code class="highlighter-rouge">"expiry"</code> to identify the timer for setting the callback time as
+well as receiving the callback.</p>
+ </li>
+ <li>
+ <p>The variable <code class="highlighter-rouge">expiryTimer</code>, annotated with <code class="highlighter-rouge">@TimerId</code>, is set to the value
+<code class="highlighter-rouge">TimerSpecs.timer(TimeDomain.EVENT_TIME)</code>, indicating that we want a
+callback according to the event time watermark of the input elements.</p>
+ </li>
+ <li>
+ <p>In the <code class="highlighter-rouge">@ProcessElement</code> element we annotate a parameter <code class="highlighter-rouge">@TimerId("expiry")
+Timer</code>. The Beam runner automatically provides this <code class="highlighter-rouge">Timer</code> parameter by which
+we can set (and reset) the timer. It is inexpensive to reset a timer
+repeatedly, so we simply set it on every element.</p>
+ </li>
+ <li>
+ <p>We define the <code class="highlighter-rouge">onExpiry</code> method, annotated with <code class="highlighter-rouge">@OnTimer("expiry")</code>, that
+performs a final event enrichment RPC and outputs the result. The Beam runner
+delivers the callback to this method by matching its identifier.</p>
+ </li>
+</ul>
+
+<p>Illustrating this logic, we have the diagram below:</p>
+
+<p><img class="center-block" src="/images/blog/timely-processing/BatchedRpcExpiry.png" alt="Batched RPCs with window expiration" width="600" /></p>
+
+<p>Both the <code class="highlighter-rouge">@ProcessElement</code> and <code class="highlighter-rouge">@OnTimer("expiry")</code> methods perform the same
+access to buffered state, perform the same batched RPC, and output enriched
+elements.</p>
+
+<p>Now, if we are executing this in a streaming real-time manner, we might still
+have unbounded latency for particular buffered data. If the watermark is advancing
+very slowly, or event time windows are chosen to be quite large, then a lot of
+time might pass before output is emitted based either on enough elements or
+window expiration. We can also use timers to limit the amount of wall-clock
+time, aka processing time, before we process buffered elements. We can choose
+some reasonable amount of time so that even though we are issuing RPCs that are
+not as large as they might be, it is still few enough RPCs to avoid blowing our
+quota with the external service.</p>
+
+<h3 id="processing-time-timers">Processing Time Timers</h3>
+
+<p>A timer in processing time (time as it passes while your pipeline is executing)
+is intuitively simple: you want to wait a certain amount of time and then
+receive a call back.</p>
+
+<p>To put the finishing touches on our example, we will set a processing time
+timer as soon as any data is buffered. We track whether or not the timer has
+been set so we don’t continually reset it. When an element arrives, if the
+timer has not been set, then we set it for the current moment plus
+<code class="highlighter-rouge">MAX_BUFFER_DURATION</code>. After the allotted processing time has passed, a
+callback will fire and enrich and emit any buffered elements.</p>
+
+<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="k">new</span> <span class="n">DoFn</span><span class="o"><</span><span class="n">Event</span><span class="o">,</span> <span class="n">EnrichedEvent</span><span class="o">>()</span> <span class="o">{</span>
+ <span class="err">…</span>
+
+ <span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="n">Duration</span> <span class="n">MAX_BUFFER_DURATION</span> <span class="o">=</span> <span class="n">Duration</span><span class="o">.</span><span class="na">standardSeconds</span><span class="o">(</span><span class="mi">1</span><span class="o">);</span>
+
+ <span class="nd">@TimerId</span><span class="o">(</span><span class="s">"stale"</span><span class="o">)</span>
+ <span class="kd">private</span> <span class="kd">final</span> <span class="n">TimerSpec</span> <span class="n">staleSpec</span> <span class="o">=</span> <span class="n">TimerSpecs</span><span class="o">.</span><span class="na">timer</span><span class="o">(</span><span class="n">TimeDomain</span><span class="o">.</span><span class="na">PROCESSING_TIME</span><span class="o">);</span>
+
+ <span class="nd">@ProcessElement</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">process</span><span class="o">(</span>
+ <span class="n">ProcessContext</span> <span class="n">context</span><span class="o">,</span>
+ <span class="n">BoundedWindow</span> <span class="n">window</span><span class="o">,</span>
+ <span class="nd">@StateId</span><span class="o">(</span><span class="s">"count"</span><span class="o">)</span> <span class="n">ValueState</span><span class="o"><</span><span class="n">Integer</span><span class="o">></span> <span class="n">countState</span><span class="o">,</span>
+ <span class="nd">@StateId</span><span class="o">(</span><span class="s">"buffer"</span><span class="o">)</span> <span class="n">BagState</span><span class="o"><</span><span class="n">Event</span><span class="o">></span> <span class="n">bufferState</span><span class="o">,</span>
+ <span class="nd">@TimerId</span><span class="o">(</span><span class="s">"stale"</span><span class="o">)</span> <span class="n">Timer</span> <span class="n">staleTimer</span><span class="o">,</span>
+ <span class="nd">@TimerId</span><span class="o">(</span><span class="s">"expiry"</span><span class="o">)</span> <span class="n">Timer</span> <span class="n">expiryTimer</span><span class="o">)</span> <span class="o">{</span>
+
+ <span class="kt">boolean</span> <span class="n">staleTimerSet</span> <span class="o">=</span> <span class="n">firstNonNull</span><span class="o">(</span><span class="n">staleSetState</span><span class="o">.</span><span class="na">read</span><span class="o">(),</span> <span class="kc">false</span><span class="o">);</span>
+ <span class="k">if</span> <span class="o">(</span><span class="n">firstNonNull</span><span class="o">(</span><span class="n">countState</span><span class="o">.</span><span class="na">read</span><span class="o">(),</span> <span class="mi">0</span><span class="o">)</span> <span class="o">==</span> <span class="mi">0</span><span class="o">)</span> <span class="o">{</span>
+ <span class="n">staleTimer</span><span class="o">.</span><span class="na">offset</span><span class="o">(</span><span class="n">MAX_BUFFER_DURATION</span><span class="o">).</span><span class="na">setRelative</span><span class="o">());</span>
+ <span class="o">}</span>
+
+ <span class="err">…</span> <span class="n">same</span> <span class="n">processing</span> <span class="n">logic</span> <span class="n">as</span> <span class="n">above</span> <span class="err">…</span>
+ <span class="o">}</span>
+
+ <span class="nd">@OnTimer</span><span class="o">(</span><span class="s">"stale"</span><span class="o">)</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">onStale</span><span class="o">(</span>
+ <span class="n">OnTimerContext</span> <span class="n">context</span><span class="o">,</span>
+ <span class="nd">@StateId</span><span class="o">(</span><span class="s">"buffer"</span><span class="o">)</span> <span class="n">BagState</span><span class="o"><</span><span class="n">Event</span><span class="o">></span> <span class="n">bufferState</span><span class="o">,</span>
+ <span class="nd">@StateId</span><span class="o">(</span><span class="s">"count"</span><span class="o">)</span> <span class="n">ValueState</span><span class="o"><</span><span class="n">Integer</span><span class="o">></span> <span class="n">countState</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">if</span> <span class="o">(!</span><span class="n">bufferState</span><span class="o">.</span><span class="na">isEmpty</span><span class="o">().</span><span class="na">read</span><span class="o">())</span> <span class="o">{</span>
+ <span class="k">for</span> <span class="o">(</span><span class="n">EnrichedEvent</span> <span class="n">enrichedEvent</span> <span class="o">:</span> <span class="n">enrichEvents</span><span class="o">(</span><span class="n">bufferState</span><span class="o">.</span><span class="na">read</span><span class="o">()))</span> <span class="o">{</span>
+ <span class="n">context</span><span class="o">.</span><span class="na">output</span><span class="o">(</span><span class="n">enrichedEvent</span><span class="o">);</span>
+ <span class="o">}</span>
+ <span class="n">bufferState</span><span class="o">.</span><span class="na">clear</span><span class="o">();</span>
+ <span class="n">countState</span><span class="o">.</span><span class="na">clear</span><span class="o">();</span>
+ <span class="o">}</span>
+ <span class="o">}</span>
+
+ <span class="err">…</span> <span class="n">same</span> <span class="n">expiry</span> <span class="n">as</span> <span class="n">above</span> <span class="err">…</span>
+<span class="o">}</span>
+</code></pre>
+</div>
+
+<div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="c"># State and timers are not yet supported in Beam's Python SDK.</span>
+<span class="c"># Follow https://issues.apache.org/jira/browse/BEAM-2687 for updates.</span>
+</code></pre>
+</div>
+
+<p>Here is an illustration of the final code:</p>
+
+<p><img class="center-block" src="/images/blog/timely-processing/BatchedRpcStale.png" alt="Batching elements in state, then performing RPCs" width="600" /></p>
+
+<p>Recapping the entirety of the logic:</p>
+
+<ul>
+ <li>As events arrive at <code class="highlighter-rouge">@ProcessElement</code> they are buffered in state.</li>
+ <li>If the size of the buffer exceeds a maximum, the events are enriched and output.</li>
+ <li>If the buffer fills too slowly and the events get stale before the maximum is reached,
+a timer causes a callback which enriches the buffered events and outputs.</li>
+ <li>Finally, as any window is expiring, any events buffered in that window are
+processed and output prior to the state for that window being discarded.</li>
+</ul>
+
+<p>In the end, we have a full example that uses state and timers to explicitly
+manage the low-level details of a performance-sensitive transform in Beam. As
+we added more and more features, our <code class="highlighter-rouge">DoFn</code> actually became pretty large. That
+is a normal characteristic of stateful, timely processing. You are really
+digging in and managing a lot of details that are handled automatically when
+you express your logic using Beam’s higher-level APIs. What you gain from this
+extra effort is an ability to tackle use cases and achieve efficiencies that
+may not have been possible otherwise.</p>
+
+<h2 id="state-and-timers-in-beams-unified-model">State and Timers in Beam’s Unified Model</h2>
+
+<p>Beam’s unified model for event time across streaming and batch processing has
+novel implications for state and timers. Usually, you don’t need to do anything
+for your stateful and timely <code class="highlighter-rouge">DoFn</code> to work well in the Beam model. But it will
+help to be aware of the considerations below, especially if you have used
+similar features before outside of Beam.</p>
+
+<h3 id="event-time-windowing-just-works">Event Time Windowing “Just Works”</h3>
+
+<p>One of the raisons d’etre for Beam is correct processing of out-of-order event
+data, which is almost all event data. Beam’s solution to out-of-order data is
+event time windowing, where windows in event time yield correct results no
+matter what windowing a user chooses or what order the events come in.</p>
+
+<p>If you write a stateful, timely transform, it should work no matter how the
+surrounding pipeline chooses to window event time. If the pipeline chooses
+fixed windows of one hour (sometimes called tumbling windows) or windows of 30
+minutes sliding by 10 minutes, the stateful, timely transform should
+transparently work correctly.</p>
+
+<p><img class="center-block" src="/images/blog/timely-processing/WindowingChoices.png" alt="Two windowing strategies for the same stateful and timely transform" width="600" /></p>
+
+<p>This works in Beam automatically, because state and timers are partitioned per
+key and window. Within each key and window, the stateful, timely processing is
+essentially independent. As an added benefit, the passing of event time (aka
+advancement of the watermark) allows automatic release of unreachable state
+when a window expires, so you often don’t have to worry about evicting old
+state.</p>
+
+<h3 id="unified-real-time-and-historical-processing">Unified real-time and historical processing</h3>
+
+<p>A second tenet of Beam’s semantic model is that processing must be unified
+between batch and streaming. One important use case for this unification
+is the ability to apply the same logic to a stream of events in real time and
+to archived storage of the same events.</p>
+
+<p>A common characteristic of archived data is that it may arrive radically out of
+order. The sharding of archived files often results in a totally different
+ordering for processing than events coming in near-real-time. The data will
+also all be all available and hence delivered instantaneously from the point of
+view of your pipeline. Whether running experiments on past data or reprocessing
+past results to fix a data processing bug, it is critically important that your
+processing logic be applicable to archived events just as easily as incoming
+near-real-time data.</p>
+
+<p><img class="center-block" src="/images/blog/timely-processing/UnifiedModel.png" alt="Unified stateful processing over streams and file archives" width="600" /></p>
+
+<p>It is (deliberately) possible to write a stateful and timely DoFn that delivers
+results that depend on ordering or delivery timing, so in this sense there is
+additional burden on you, the <code class="highlighter-rouge">DoFn</code> author, to ensure that this nondeterminism
+falls within documented allowances.</p>
+
+<h2 id="go-use-it">Go use it!</h2>
+
+<p>I’ll end this post in the same way I ended the last. I hope you will go try out
+Beam with stateful, timely processing. If it opens up new possibilities for
+you, then great! If not, we want to hear about it. Since this is a new feature,
+please check the <a href="/documentation/runners/capability-matrix/">capability matrix</a> to see the level of support for
+your preferred Beam backend(s).</p>
+
+<p>And please do join the Beam community at
+<a href="/get-started/support">user@beam.apache.org</a> and follow
+<a href="https://twitter.com/ApacheBeam">@ApacheBeam</a> on Twitter.</p>
+
+ </div>
+
+</article>
+
+ </div>
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<footer class="footer">
+ <div class="footer__contained">
+ <div class="footer__cols">
+ <div class="footer__cols__col">
+ <div class="footer__cols__col__logo">
+ <img src="/images/beam_logo_circle.svg" class="footer__logo" alt="Beam logo">
+ </div>
+ <div class="footer__cols__col__logo">
+ <img src="/images/apache_logo_circle.svg" class="footer__logo" alt="Apache logo">
+ </div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Start</div>
+ <div class="footer__cols__col__link"><a href="/get-started/beam-overview/">Overview</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-java/">Quickstart (Java)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-py/">Quickstart (Python)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-go/">Quickstart (Go)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/downloads/">Downloads</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Docs</div>
+ <div class="footer__cols__col__link"><a href="/documentation/programming-guide/">Concepts</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/pipelines/design-your-pipeline/">Pipelines</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/runners/capability-matrix/">Runners</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Community</div>
+ <div class="footer__cols__col__link"><a href="/contribute/">Contribute</a></div>
+ <div class="footer__cols__col__link"><a href="https://projects.apache.org/committee.html?beam" target="_blank">Team<img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a></div>
+ <div class="footer__cols__col__link"><a href="/contribute/presentation-materials/">Media</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Resources</div>
+ <div class="footer__cols__col__link"><a href="/blog/">Blog</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/support/">Support</a></div>
+ <div class="footer__cols__col__link"><a href="https://github.com/apache/beam">GitHub</a></div>
+ </div>
+ </div>
+ </div>
+ <div class="footer__bottom">
+ ©
+ <a href="http://www.apache.org">The Apache Software Foundation</a>
+ | <a href="/privacy_policy">Privacy Policy</a>
+ | <a href="/feed.xml">RSS Feed</a>
+ <br><br>
+ Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are
+ either registered trademarks or trademarks of The Apache Software
+ Foundation. All other products or name brands are trademarks of their
+ respective holders, including The Apache Software Foundation.
+ </div>
+</footer>
+
+ </body>
+</html>
diff --git a/website/generated-content/blog/2018/01/09/beam-a-look-back.html b/website/generated-content/blog/2018/01/09/beam-a-look-back.html
new file mode 100644
index 0000000..441b1d4
--- /dev/null
+++ b/website/generated-content/blog/2018/01/09/beam-a-look-back.html
@@ -0,0 +1,362 @@
+<!DOCTYPE html>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<html lang="en">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <title>Apache Beam: A Look Back at 2017</title>
+ <meta name="description" content="On January 10, 2017, Apache Beam got promotedas a Top-Level Apache Software Foundation project. It was an important milestonethat validated the value of the ...">
+ <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
+ <link rel="stylesheet" href="/css/site.css">
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <script src="/js/language-switch.js"></script>
+ <script src="/js/fix-menu.js"></script>
+ <script src="/js/section-nav.js"></script>
+ <script src="/js/page-nav.js"></script>
+ <link rel="canonical" href="https://beam.apache.org/blog/2018/01/09/beam-a-look-back.html" data-proofer-ignore>
+ <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
+ <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+ ga('create', 'UA-73650088-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+</head>
+
+ <body class="body ">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<nav class="header navbar navbar-fixed-top">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+
+ <a href="/" class="navbar-brand" >
+ <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
+ </a>
+ </div>
+
+ <div class="navbar-mask closed"></div>
+
+ <div id="navbar" class="navbar-container closed">
+ <ul class="nav navbar-nav">
+ <li>
+ <a href="/get-started/beam-overview/">Get Started</a>
+ </li>
+ <li>
+ <a href="/documentation/">Documentation</a>
+ </li>
+ <li>
+ <a href="/documentation/sdks/java/">SDKS</a>
+ </li>
+ <li>
+ <a href="/documentation/runners/capability-matrix/">RUNNERS</a>
+ </li>
+ <li>
+ <a href="/contribute/">Contribute</a>
+ </li>
+ <li>
+ <a href="/community/contact-us/">Community</a>
+ </li>
+ <li><a href="/blog">Blog</a></li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:20px;"><span class="caret"></span></a>
+ <ul class="dropdown-menu dropdown-menu-right">
+ <li><a href="http://www.apache.org/">ASF Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">License</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
+ </ul>
+ </li>
+ </ul>
+ </div>
+</nav>
+
+ <div class="body__contained">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+
+
+<article class="post" itemscope itemtype="http://schema.org/BlogPosting">
+
+ <header class="post-header">
+ <h1 class="post-title" itemprop="name headline">Apache Beam: A Look Back at 2017</h1>
+ <p class="post-meta"><time datetime="2018-01-09T00:00:01-08:00" itemprop="datePublished">Jan 9, 2018</time> •
+ Anand Iyer & Jean-Baptiste Onofré [<a href="https://twitter.com/jbonofre">@jbonofre</a>]
+
+ </p>
+ </header>
+
+ <div class="post-content" itemprop="articleBody">
+ <!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+<p>On January 10, 2017, Apache Beam got <a href="/blog/2017/01/10/beam-graduates.html">promoted</a>
+as a Top-Level Apache Software Foundation project. It was an important milestone
+that validated the value of the project, legitimacy of its community, and
+heralded its growing adoption. In the past year, Apache Beam has been on a
+phenomenal growth trajectory, with significant growth in its community and
+feature set. Let us walk you through some of the notable achievements.</p>
+
+<!--more-->
+
+<h2 id="use-cases">Use cases</h2>
+
+<p>First, lets take a glimpse at how Beam was used in 2017. Apache Beam being a
+unified framework for batch and stream processing, enables a very wide spectrum
+of diverse use cases. Here are some use cases that exemplify the versatility of
+Beam.</p>
+
+<p><img class="center-block" src="/images/blog/2017-look-back/timeline.png" alt="Use Cases" width="600" /></p>
+
+<h2 id="community-growth">Community growth</h2>
+
+<p>In 2017, Apache Beam had 174 contributors worldwide, from many different
+organizations. As an Apache project, we are proud to count 18 PMC members and
+31 committers. The community had 7 releases in 2017, each bringing a rich set of
+new features and fixes.</p>
+
+<p>The most obvious and encouraging sign of the growth of Apache Beam’s community,
+and validation of its core value proposition of portability, is the addition of
+significant new <a href="/documentation/runners/capability-matrix/">runners</a>
+(i.e. execution engines). We entered 2017 with Apache Flink, Apache Spark 1.x,
+Google Cloud Dataflow, Apache Apex, and Apache Gearpump. In 2017, the following
+new and updated runners were developed:</p>
+
+<ul>
+ <li>Apache Spark 2.x update</li>
+ <li><a href="https://www.ibm.com/blogs/bluemix/2017/10/streaming-analytics-updates-ibm-streams-runner-apache-beam-2-0/">IBM Streams runner</a></li>
+ <li>MapReduce runner</li>
+ <li><a href="http://jstorm.io/">JStorm runner</a></li>
+</ul>
+
+<p>In addition to runners, Beam added new IO connectors, some notable ones being
+the Cassandra, MQTT, AMQP, HBase/HCatalog, JDBC, Solr, Tika, Redis, and
+ElasticSearch connectors. Beam’s IO connectors make it possible to read from or
+write to data sources/sinks even when they are not natively supported by the
+underlying execution engine. Beam also provides fully pluggable filesystem
+support, allowing us to support and extend our coverage to HDFS, S3, Azure
+Storage, and Google Storage. We continue to add new IO connectors and
+filesystems to extend the Beam use cases.</p>
+
+<p>A particularly telling sign of the maturity of an open source community is when
+it is able to collaborate with multiple other open source communities, and
+mutually improve the state of the art. Over the past few months, the Beam,
+Calcite, and Flink communities have come together to define a robust <a href="https://docs.google.com/document/d/1wrla8mF_mmq-NW9sdJHYVgMyZsgCmHumJJ5f5WUzTiM/edit">spec</a>
+for Streaming SQL, with engineers from over four organizations contributing to
+it. If, like us, you are excited by the prospect of improving the state of
+streaming SQL, please join us!</p>
+
+<p>In addition to SQL, new XML and JSON based declarative DSLs are also in PoC.</p>
+
+<h2 id="continued-innovation">Continued innovation</h2>
+
+<p>Innovation is important to the success on any open source project, and Beam has
+a rich history of bringing innovative new ideas to the open source community.
+Apache Beam was the first to introduce some seminal concepts in the world of
+big-data processing:</p>
+
+<ul>
+ <li>Unified batch and streaming SDK that enables users to author big-data jobs
+without having to learn multiple disparate SDKs/APIs.</li>
+ <li>Cross-Engine Portability: Giving enterprises the confidence that workloads
+authored today will not have to be re-written when open source engines become
+outdated and are supplanted by newer ones.</li>
+ <li><a href="https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101">Semantics</a>
+essential for reasoning about unbounded unordered data, and achieving
+consistent and correct output from a streaming job.</li>
+</ul>
+
+<p>In 2017, the pace of innovation continued. The following capabilities were
+introduced:</p>
+
+<ul>
+ <li>Cross-Language Portability framework, and a <a href="https://golang.org/">Go</a> SDK
+developed with it.</li>
+ <li>Dynamically Shardable IO (SplittableDoFn)</li>
+ <li>Support for schemas in PCollection, allowing us to extend the runner
+capabilities.</li>
+ <li>Extensions addressing new use cases such as machine learning, and new data
+formats.</li>
+</ul>
+
+<h2 id="areas-of-improvement">Areas of improvement</h2>
+
+<p>Any retrospective view of a project is incomplete without an honest assessment
+of areas of improvement. Two aspects stand out:</p>
+
+<ul>
+ <li>Helping runners showcase their individual strengths. After all, portability
+does not imply homogeneity. Different runners have different areas in which
+they excel, and we need to do a better job of helping them highlight their
+strengths.</li>
+ <li>Based on the previous point, helping customers make a more informed decision
+when they select a runner or migrate from one to another.</li>
+</ul>
+
+<p>In 2018, we aim to take proactive steps to improve the above aspects.</p>
+
+<h2 id="ethos-of-the-project-and-its-community">Ethos of the project and its community</h2>
+
+<p>The world of batch and stream big-data processing today is reminiscent of the
+<a href="https://en.wikipedia.org/wiki/Tower_of_Babel">Tower of Babel</a> parable: a
+slowdown of progress because different communities spoke different languages.
+Similarly, today there are multiple disparate big-data SDKs/APIs, each with
+their own distinct terminology to describe similar concepts. The side effect is
+user confusion and slower adoption.</p>
+
+<p>The Apache Beam project aims to provide an industry standard portable SDK that
+will:</p>
+
+<ul>
+ <li>Benefit users by providing <strong><em>innovation with stability</em></strong>: The separation of
+SDK and engine enables healthy competition between runners, without requiring
+users to constantly learn new SDKs/APIs and rewrite their workloads to
+benefit from new innovation.</li>
+ <li>Benefit big-data engines by <strong><em>growing the pie for everyone</em></strong>: Making it
+easier for users to author, maintain, upgrade and migrate their big-data
+workloads will lead to significant growth in the number of production
+big-data deployments.</li>
+</ul>
+
+
+ </div>
+
+</article>
+
+ </div>
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<footer class="footer">
+ <div class="footer__contained">
+ <div class="footer__cols">
+ <div class="footer__cols__col">
+ <div class="footer__cols__col__logo">
+ <img src="/images/beam_logo_circle.svg" class="footer__logo" alt="Beam logo">
+ </div>
+ <div class="footer__cols__col__logo">
+ <img src="/images/apache_logo_circle.svg" class="footer__logo" alt="Apache logo">
+ </div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Start</div>
+ <div class="footer__cols__col__link"><a href="/get-started/beam-overview/">Overview</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-java/">Quickstart (Java)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-py/">Quickstart (Python)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-go/">Quickstart (Go)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/downloads/">Downloads</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Docs</div>
+ <div class="footer__cols__col__link"><a href="/documentation/programming-guide/">Concepts</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/pipelines/design-your-pipeline/">Pipelines</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/runners/capability-matrix/">Runners</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Community</div>
+ <div class="footer__cols__col__link"><a href="/contribute/">Contribute</a></div>
+ <div class="footer__cols__col__link"><a href="https://projects.apache.org/committee.html?beam" target="_blank">Team<img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a></div>
+ <div class="footer__cols__col__link"><a href="/contribute/presentation-materials/">Media</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Resources</div>
+ <div class="footer__cols__col__link"><a href="/blog/">Blog</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/support/">Support</a></div>
+ <div class="footer__cols__col__link"><a href="https://github.com/apache/beam">GitHub</a></div>
+ </div>
+ </div>
+ </div>
+ <div class="footer__bottom">
+ ©
+ <a href="http://www.apache.org">The Apache Software Foundation</a>
+ | <a href="/privacy_policy">Privacy Policy</a>
+ | <a href="/feed.xml">RSS Feed</a>
+ <br><br>
+ Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are
+ either registered trademarks or trademarks of The Apache Software
+ Foundation. All other products or name brands are trademarks of their
+ respective holders, including The Apache Software Foundation.
+ </div>
+</footer>
+
+ </body>
+</html>
diff --git a/website/generated-content/blog/2018/02/19/beam-2.3.0.html b/website/generated-content/blog/2018/02/19/beam-2.3.0.html
new file mode 100644
index 0000000..020c2ff
--- /dev/null
+++ b/website/generated-content/blog/2018/02/19/beam-2.3.0.html
@@ -0,0 +1,319 @@
+<!DOCTYPE html>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<html lang="en">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <title>Apache Beam 2.3.0</title>
+ <meta name="description" content="We are glad to present the new 2.3.0 release of Beam. This release includesmultiple fixes and new functionalities.">
+ <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
+ <link rel="stylesheet" href="/css/site.css">
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
... 78643 lines suppressed ...
[beam] 07/11: [BEAM-5669] Stop empty commits being pushed by
PostCommit_Website_Publish jenkins job
Posted by sc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 938c0223ed1742e9b5ff848d1901aef2875775e7
Author: Alan Myrvold <am...@google.com>
AuthorDate: Fri Oct 5 12:03:10 2018 -0700
[BEAM-5669] Stop empty commits being pushed by PostCommit_Website_Publish jenkins job
---
.gitignore | 82 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 82 insertions(+)
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..f7e691a
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,82 @@
+# NOTE: if you modify this file, you probably need to modify the file set that
+# is an input to 'maven-assembly-plugin' that generates source distribution.
+# This is typically in files named 'src.xml' throughout this repository.
+
+# Ignore any offline repositories the user may have created.
+**/offline-repository/**/*
+
+# Ignore files generated by the Gradle build process.
+**/.gradle/**/*
+**/.gogradle/**/*
+**/gogradle.lock
+**/build/**/*
+sdks/**/vendor/**/*
+runners/**/vendor/**/*
+**/.gradletasknamecache
+
+# Ignore files generated by the Maven build process.
+**/bin/**/*
+**/dependency-reduced-pom.xml
+**/target/**/*
+
+# Ignore generated archetypes
+sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/src/
+sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/src/
+
+# Ignore files generated by the Python build process.
+**/*.pyc
+**/*.pyo
+**/*.pyd
+**/*.egg-info/
+**/.eggs/
+**/nose-*.egg/
+**/.tox/**/*
+**/dist/**/*
+**/distribute-*/**/*
+**/env/**/*
+sdks/python/**/*.c
+sdks/python/**/*.so
+sdks/python/**/*.egg
+sdks/python/LICENSE
+sdks/python/NOTICE
+sdks/python/README.md
+sdks/python/apache_beam/portability/api/*pb2*.*
+sdks/python/nosetests.xml
+
+# Ignore IntelliJ files.
+**/.idea/**/*
+**/*.iml
+**/*.ipr
+**/*.iws
+**/out/**/*
+
+# Ignore Eclipse files.
+**/.classpath
+**/.project
+**/.factorypath
+**/.checkstyle
+**/.fbExcludeFilterFile
+**/.apt_generated/**/*
+**/.settings/**/*
+**/.gitignore
+
+# Ignore Visual Studio Code files.
+**/.vscode/**/*
+
+# Hotspot VM leaves this log in a non-target directory when java crashes
+**/hs_err_pid*.log
+
+# Ignore files that end with '~', since they are most likely auto-save files
+# produced by a text editor.
+**/*~
+
+# Ignore MacOSX files.
+**/.DS_Store/**/*
+**/.DS_Store
+
+# Ignore Jupyter notebook checkpoints.
+**/.ipynb_checkpoints/**/*
+
+# NOTE: if you modify this file, you probably need to modify the file set that
+# is an input to 'maven-assembly-plugin' that generates source distribution.
+# This is typically in files named 'src.xml' throughout this repository.
[beam] 04/11: Publishing website 2018/10/04 20:21:56 at commit
4506080
Posted by sc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 2c342fcdf9f0b22eb5e6e936f5d191c7bffed224
Author: jenkins <bu...@apache.org>
AuthorDate: Thu Oct 4 20:21:57 2018 +0000
Publishing website 2018/10/04 20:21:56 at commit 4506080
---
website/generated-content/.htaccess | 9 ++++++
.../blog/2016/10/20/test-stream.html | 2 +-
.../blog/2017/03/16/python-sdk-release.html | 2 +-
.../blog/2017/08/16/splittable-do-fn.html | 10 +++----
.../08/20/review-input-streaming-connectors.html | 22 +++++++-------
.../contribute/ptransform-style-guide/index.html | 4 +--
.../contribute/release-guide/index.html | 10 +++----
.../contribute/runner-guide/index.html | 16 +++++-----
.../dsls/sql/aggregate-functions/index.html | 4 +--
.../documentation/dsls/sql/create-table/index.html | 6 ++--
.../documentation/dsls/sql/data-types/index.html | 4 +--
.../documentation/dsls/sql/joins/index.html | 4 +--
.../documentation/dsls/sql/lexical/index.html | 4 +--
.../documentation/dsls/sql/overview/index.html | 8 ++---
.../dsls/sql/scalar-functions/index.html | 4 +--
.../documentation/dsls/sql/select/index.html | 4 +--
.../documentation/dsls/sql/set/index.html | 4 +--
.../documentation/dsls/sql/shell/index.html | 4 +--
.../dsls/sql/user-defined-functions/index.html | 4 +--
.../documentation/dsls/sql/walkthrough/index.html | 10 +++----
.../dsls/sql/windowing-and-triggering/index.html | 4 +--
.../pipelines/test-your-pipeline/index.html | 4 +--
.../documentation/programming-guide/index.html | 16 +++++-----
.../documentation/runners/dataflow/index.html | 4 +--
.../documentation/runners/direct/index.html | 12 ++++----
.../documentation/runners/flink/index.html | 2 +-
.../sdks/feature-comparison/index.html | 4 +--
.../documentation/sdks/go/index.html | 4 +--
.../documentation/sdks/java-extensions/index.html | 4 +--
.../documentation/sdks/java-thirdparty/index.html | 4 +--
.../documentation/sdks/java/euphoria/index.html | 4 +--
.../documentation/sdks/java/index.html | 6 ++--
.../documentation/sdks/java/nexmark/index.html | 4 +--
.../documentation/sdks/python-custom-io/index.html | 4 +--
.../sdks/python-pipeline-dependencies/index.html | 4 +--
.../documentation/sdks/python-streaming/index.html | 4 +--
.../sdks/python-type-safety/index.html | 4 +--
.../documentation/sdks/python/index.html | 6 ++--
website/generated-content/feed.xml | 34 +++++++++++-----------
.../get-started/downloads/index.html | 2 +-
.../get-started/quickstart-java/index.html | 2 +-
.../get-started/quickstart-py/index.html | 2 +-
.../get-started/wordcount-example/index.html | 4 +--
43 files changed, 141 insertions(+), 132 deletions(-)
diff --git a/website/generated-content/.htaccess b/website/generated-content/.htaccess
index 06fc74b..77dabf4 100644
--- a/website/generated-content/.htaccess
+++ b/website/generated-content/.htaccess
@@ -13,3 +13,12 @@ RewriteCond %{HTTPS} !on
# * Redirect (R) permanently (301) to https://beam.apache.org/,
# * Stop processing more rules (L).
RewriteRule ^(.*)$ https://beam.apache.org/$1 [L,R=301]
+
+# Javadocs / pydocs are available only on the published website, published from
+# https://github.com/apache/beam-site/tree/release-docs
+# They were previously hosted within this repository, and published at the URL
+# path /documentation/sdks/(javadoc|pydoc)/..
+# The following redirect maintains the previously supported URLs.
+RedirectMatch permanent "/documentation/sdks/(javadoc|pydoc)(.*)" "https://beam.apache.org/documentation/releases/$1$2"
+# Keep this updated to point to the current release.
+RedirectMatch "/releases/([^/]+)/current(.*)" "https://beam.apache.org/documentation/releases/$1/2.6.0$2"
diff --git a/website/generated-content/blog/2016/10/20/test-stream.html b/website/generated-content/blog/2016/10/20/test-stream.html
index 8b0566d..596c845 100644
--- a/website/generated-content/blog/2016/10/20/test-stream.html
+++ b/website/generated-content/blog/2016/10/20/test-stream.html
@@ -210,7 +210,7 @@ be controlled within a test.</p>
<h2 id="writing-deterministic-tests-to-emulate-nondeterminism">Writing Deterministic Tests to Emulate Nondeterminism</h2>
<p>The Beam testing infrastructure provides the
-<a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/testing/PAssert.html">PAssert</a>
+<a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/testing/PAssert.html">PAssert</a>
methods, which assert properties about the contents of a PCollection from within
a pipeline. We have expanded this infrastructure to include
<a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestStream.java">TestStream</a>,
diff --git a/website/generated-content/blog/2017/03/16/python-sdk-release.html b/website/generated-content/blog/2017/03/16/python-sdk-release.html
index a1a3cdc..18f7b55 100644
--- a/website/generated-content/blog/2017/03/16/python-sdk-release.html
+++ b/website/generated-content/blog/2017/03/16/python-sdk-release.html
@@ -168,7 +168,7 @@ limitations under the License.
<h4 id="try-the-apache-beam-python-sdk">Try the Apache Beam Python SDK</h4>
-<p>If you would like to try out the Python SDK, a good place to start is the <a href="/get-started/quickstart-py/">Quickstart</a>. After that, you can take a look at additional <a href="https://github.com/apache/beam/tree/v0.6.0/sdks/python/apache_beam/examples">examples</a>, and deep dive into the <a href="/documentation/sdks/pydoc/">API reference</a>.</p>
+<p>If you would like to try out the Python SDK, a good place to start is the <a href="/get-started/quickstart-py/">Quickstart</a>. After that, you can take a look at additional <a href="https://github.com/apache/beam/tree/v0.6.0/sdks/python/apache_beam/examples">examples</a>, and deep dive into the <a href="https://beam.apache.org/releases/pydoc/">API reference</a>.</p>
<p>Let’s take a look at a quick example together. First, install the <code class="highlighter-rouge">apache-beam</code> package from PyPI and start your Python interpreter.</p>
diff --git a/website/generated-content/blog/2017/08/16/splittable-do-fn.html b/website/generated-content/blog/2017/08/16/splittable-do-fn.html
index eb5bb0e..dd5d359 100644
--- a/website/generated-content/blog/2017/08/16/splittable-do-fn.html
+++ b/website/generated-content/blog/2017/08/16/splittable-do-fn.html
@@ -223,22 +223,22 @@ has other limitations that make it insufficient for this task</em>).</p>
<h2 id="beam-source-api">Beam Source API</h2>
<p>Apache Beam historically provides a Source API
-(<a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/BoundedSource.html">BoundedSource</a>
+(<a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/BoundedSource.html">BoundedSource</a>
and
-<a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/UnboundedSource.html">UnboundedSource</a>) which does
+<a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/UnboundedSource.html">UnboundedSource</a>) which does
not have these limitations and allows development of efficient data sources for
batch and streaming systems. Pipelines use this API via the
-<a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/Read.html"><code class="highlighter-rouge">Read.from(Source)</code></a> built-in <code class="highlighter-rouge">PTransform</code>.</p>
+<a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/Read.html"><code class="highlighter-rouge">Read.from(Source)</code></a> built-in <code class="highlighter-rouge">PTransform</code>.</p>
<p>The Source API is largely similar to that of most other data processing
frameworks, and allows the system to read data in parallel using multiple
workers, as well as checkpoint and resume reading from an unbounded data source.
Additionally, the Beam
-<a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/BoundedSource.html"><code class="highlighter-rouge">BoundedSource</code></a>
+<a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/BoundedSource.html"><code class="highlighter-rouge">BoundedSource</code></a>
API provides advanced features such as progress reporting and <a href="/blog/2016/05/18/splitAtFraction-method.html">dynamic
rebalancing</a>
(which together enable autoscaling), and
-<a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/UnboundedSource.html"><code class="highlighter-rouge">UnboundedSource</code></a> supports
+<a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/UnboundedSource.html"><code class="highlighter-rouge">UnboundedSource</code></a> supports
reporting the source’s watermark and backlog <em>(until SDF, we believed that
“batch” and “streaming” data sources are fundamentally different and thus
require fundamentally different APIs)</em>.</p>
diff --git a/website/generated-content/blog/2018/08/20/review-input-streaming-connectors.html b/website/generated-content/blog/2018/08/20/review-input-streaming-connectors.html
index 2e9c8bb..31384ac 100644
--- a/website/generated-content/blog/2018/08/20/review-input-streaming-connectors.html
+++ b/website/generated-content/blog/2018/08/20/review-input-streaming-connectors.html
@@ -190,7 +190,7 @@ limitations under the License.
</td>
<td>Local<br />(Using the <code>file://</code> URI)
</td>
- <td><a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/TextIO.html">TextIO</a>
+ <td><a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/TextIO.html">TextIO</a>
</td>
<td><a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/StreamingContext.html#textFileStream-java.lang.String-">textFileStream</a><br />(Spark treats most Unix systems as HDFS-compatible, but the location should be accessible from all nodes)
</td>
@@ -198,7 +198,7 @@ limitations under the License.
<tr>
<td>HDFS<br />(Using the <code>hdfs://</code> URI)
</td>
- <td><a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/FileIO.html">FileIO</a> + <a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptions.html">HadoopFileSystemOptions</a>
+ <td><a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/FileIO.html">FileIO</a> + <a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptions.html">HadoopFileSystemOptions</a>
</td>
<td><a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/util/HdfsUtils.html">HdfsUtils</a>
</td>
@@ -208,7 +208,7 @@ limitations under the License.
</td>
<td>Cloud Storage<br />(Using the <code>gs://</code> URI)
</td>
- <td><a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/FileIO.html">FileIO</a> + <a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/extensions/gcp/options/GcsOptions.html">GcsOptions</a>
+ <td><a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/FileIO.html">FileIO</a> + <a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/extensions/gcp/options/GcsOptions.html">GcsOptions</a>
</td>
<td rowspan="2"><a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/SparkContext.html#hadoopConfiguration--">hadoopConfiguration</a>
and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/StreamingContext.html#textFileStream-java.lang.String-">textFileStream</a>
@@ -217,7 +217,7 @@ and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/stre
<tr>
<td>S3<br />(Using the <code>s3://</code> URI)
</td>
- <td><a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/FileIO.html">FileIO</a> + <a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/aws/options/S3Options.html">S3Options</a>
+ <td><a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/FileIO.html">FileIO</a> + <a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/aws/options/S3Options.html">S3Options</a>
</td>
</tr>
<tr>
@@ -225,7 +225,7 @@ and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/stre
</td>
<td>Kafka
</td>
- <td><a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/kafka/KafkaIO.html">KafkaIO</a>
+ <td><a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/kafka/KafkaIO.html">KafkaIO</a>
</td>
<td><a href="https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html">spark-streaming-kafka</a>
</td>
@@ -233,7 +233,7 @@ and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/stre
<tr>
<td>Kinesis
</td>
- <td><a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/kinesis/KinesisIO.html">KinesisIO</a>
+ <td><a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/kinesis/KinesisIO.html">KinesisIO</a>
</td>
<td><a href="https://spark.apache.org/docs/latest/streaming-kinesis-integration.html">spark-streaming-kinesis</a>
</td>
@@ -241,7 +241,7 @@ and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/stre
<tr>
<td>Cloud Pub/Sub
</td>
- <td><a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.html">PubsubIO</a>
+ <td><a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.html">PubsubIO</a>
</td>
<td><a href="https://github.com/apache/bahir/tree/master/streaming-pubsub">spark-streaming-pubsub</a> from <a href="http://bahir.apache.org">Apache Bahir</a>
</td>
@@ -282,7 +282,7 @@ and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/stre
</td>
<td>Local
</td>
- <td><a href="/documentation/sdks/pydoc/2.6.0/apache_beam.io.textio.html">io.textio</a>
+ <td><a href="https://beam.apache.org/releases/pydoc/2.6.0/apache_beam.io.textio.html">io.textio</a>
</td>
<td><a href="http://spark.apache.org/docs/latest/api/python/pyspark.streaming.html#pyspark.streaming.StreamingContext.textFileStream">textFileStream</a>
</td>
@@ -290,7 +290,7 @@ and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/stre
<tr>
<td>HDFS
</td>
- <td><a href="/documentation/sdks/pydoc/2.6.0/apache_beam.io.hadoopfilesystem.html">io.hadoopfilesystem</a>
+ <td><a href="https://beam.apache.org/releases/pydoc/2.6.0/apache_beam.io.hadoopfilesystem.html">io.hadoopfilesystem</a>
</td>
<td><a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/SparkContext.html#hadoopConfiguration--">hadoopConfiguration</a> (Access through <code>sc._jsc</code> with Py4J)
and <a href="http://spark.apache.org/docs/latest/api/python/pyspark.streaming.html#pyspark.streaming.StreamingContext.textFileStream">textFileStream</a>
@@ -301,7 +301,7 @@ and <a href="http://spark.apache.org/docs/latest/api/python/pyspark.streaming.ht
</td>
<td>Google Cloud Storage
</td>
- <td><a href="/documentation/sdks/pydoc/2.6.0/apache_beam.io.gcp.gcsio.html">io.gcp.gcsio</a>
+ <td><a href="https://beam.apache.org/releases/pydoc/2.6.0/apache_beam.io.gcp.gcsio.html">io.gcp.gcsio</a>
</td>
<td rowspan="2"><a href="http://spark.apache.org/docs/latest/api/python/pyspark.streaming.html#pyspark.streaming.StreamingContext.textFileStream">textFileStream</a>
</td>
@@ -333,7 +333,7 @@ and <a href="http://spark.apache.org/docs/latest/api/python/pyspark.streaming.ht
<tr>
<td>Cloud Pub/Sub
</td>
- <td><a href="/documentation/sdks/pydoc/2.6.0/apache_beam.io.gcp.pubsub.html">io.gcp.pubsub</a>
+ <td><a href="https://beam.apache.org/releases/pydoc/2.6.0/apache_beam.io.gcp.pubsub.html">io.gcp.pubsub</a>
</td>
<td>N/A
</td>
diff --git a/website/generated-content/contribute/ptransform-style-guide/index.html b/website/generated-content/contribute/ptransform-style-guide/index.html
index 39c7230..f577420 100644
--- a/website/generated-content/contribute/ptransform-style-guide/index.html
+++ b/website/generated-content/contribute/ptransform-style-guide/index.html
@@ -528,8 +528,8 @@ E.g. when expanding a filepattern into files, log what the filepattern was and h
<ul>
<li>Generally, follow the rules of <a href="http://semver.org/">semantic versioning</a>.</li>
- <li>If the API of the transform is not yet stable, annotate it as <code class="highlighter-rouge">@Experimental</code> (Java) or <code class="highlighter-rouge">@experimental</code> (<a href="/documentation/sdks/pydoc/2.6.0/apache_beam.utils.annotations.html">Python</a>).</li>
- <li>If the API deprecated, annotate it as <code class="highlighter-rouge">@Deprecated</code> (Java) or <code class="highlighter-rouge">@deprecated</code> (<a href="/documentation/sdks/pydoc/2.6.0/apache_beam.utils.annotations.html">Python</a>).</li>
+ <li>If the API of the transform is not yet stable, annotate it as <code class="highlighter-rouge">@Experimental</code> (Java) or <code class="highlighter-rouge">@experimental</code> (<a href="https://beam.apache.org/releases/pydoc/2.6.0/apache_beam.utils.annotations.html">Python</a>).</li>
+ <li>If the API deprecated, annotate it as <code class="highlighter-rouge">@Deprecated</code> (Java) or <code class="highlighter-rouge">@deprecated</code> (<a href="https://beam.apache.org/releases/pydoc/2.6.0/apache_beam.utils.annotations.html">Python</a>).</li>
<li>Pay attention to the stability and versioning of third-party classes exposed by the transform’s API: if they are unstable or improperly versioned (do not obey <a href="http://semver.org/">semantic versioning</a>), it is better to wrap them in your own classes.</li>
</ul>
diff --git a/website/generated-content/contribute/release-guide/index.html b/website/generated-content/contribute/release-guide/index.html
index 2af7e88..5a53b27 100644
--- a/website/generated-content/contribute/release-guide/index.html
+++ b/website/generated-content/contribute/release-guide/index.html
@@ -1089,7 +1089,7 @@ svn commit
website. To update the website, you must unpack this jar file from the release
candidate into the source tree of the website.</p>
-<p>Add the new Javadoc to <a href="/documentation/sdks/javadoc/">SDK API Reference page</a> page, as follows:</p>
+<p>Add the new Javadoc to <a href="https://beam.apache.org/releases/javadoc/">SDK API Reference page</a> page, as follows:</p>
<ul>
<li>Unpack the Maven artifact <code class="highlighter-rouge">org.apache.beam:beam-sdks-java-javadoc</code> into some temporary location. Call this <code class="highlighter-rouge">${JAVADOC_TMP}</code>.</li>
@@ -1099,7 +1099,7 @@ candidate into the source tree of the website.</p>
</ul>
<h5 id="create-pydoc">Create Pydoc</h5>
-<p>Add the new Pydoc to <a href="/documentation/sdks/pydoc/">SDK API Reference page</a> page, as follows:</p>
+<p>Add the new Pydoc to <a href="https://beam.apache.org/releases/pydoc/">SDK API Reference page</a> page, as follows:</p>
<ul>
<li>Copy the generated Pydoc into the website repository: <code class="highlighter-rouge">cp -r ${PYDOC_ROOT} src/documentation/sdks/pydoc/${RELEASE}</code>.</li>
@@ -1122,7 +1122,7 @@ candidate into the source tree of the website.</p>
<ol>
<li>Maven artifacts deployed to the staging repository of <a href="https://repository.apache.org/content/repositories/">repository.apache.org</a></li>
<li>Source distribution deployed to the dev repository of <a href="https://dist.apache.org/repos/dist/dev/beam/">dist.apache.org</a></li>
- <li>Website pull request proposed to list the <a href="/get-started/downloads/">release</a>, publish the <a href="/documentation/sdks/javadoc/">Java API reference manual</a>, and publish the <a href="/documentation/sdks/pydoc/">Python API reference manual</a>.</li>
+ <li>Website pull request proposed to list the <a href="/get-started/downloads/">release</a>, publish the <a href="https://beam.apache.org/releases/javadoc/">Java API reference manual</a>, and publish the <a href="https://beam.apache.org/releases/pydoc/">Python API reference manual</a>.</li>
</ol>
<p>You can (optionally) also do additional verification by:</p>
@@ -1558,7 +1558,7 @@ git push github "$VERSION_TAG"
<h3 id="merge-website-pull-request">Merge website pull request</h3>
-<p>Merge the website pull request to <a href="/get-started/downloads/">list the release</a>, publish the <a href="/documentation/sdks/pydoc/">Python API reference manual</a>, and the <a href="/documentation/sdks/javadoc/">Java API reference manual</a> created earlier.</p>
+<p>Merge the website pull request to <a href="/get-started/downloads/">list the release</a>, publish the <a href="https://beam.apache.org/releases/pydoc/">Python API reference manual</a>, and the <a href="https://beam.apache.org/releases/javadoc/">Java API reference manual</a> created earlier.</p>
<h3 id="mark-the-version-as-released-in-jira">Mark the version as released in JIRA</h3>
@@ -1574,7 +1574,7 @@ git push github "$VERSION_TAG"
<li>Maven artifacts released and indexed in the <a href="https://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.beam%22">Maven Central Repository</a></li>
<li>Source distribution available in the release repository of <a href="https://dist.apache.org/repos/dist/release/beam/">dist.apache.org</a></li>
<li>Source distribution removed from the dev repository of <a href="https://dist.apache.org/repos/dist/dev/beam/">dist.apache.org</a></li>
- <li>Website pull request to <a href="/get-started/downloads/">list the release</a> and publish the <a href="/documentation/sdks/javadoc/">API reference manual</a> merged</li>
+ <li>Website pull request to <a href="/get-started/downloads/">list the release</a> and publish the <a href="https://beam.apache.org/releases/javadoc/">API reference manual</a> merged</li>
<li>Release tagged in the source code repository</li>
<li>Release version finalized in JIRA. (Note: Not all committers have administrator access to JIRA. If you end up getting permissions errors ask on the mailing list for assistance.)</li>
<li>Release version is listed at reporter.apache.org</li>
diff --git a/website/generated-content/contribute/runner-guide/index.html b/website/generated-content/contribute/runner-guide/index.html
index d74b5ae..aff1de0 100644
--- a/website/generated-content/contribute/runner-guide/index.html
+++ b/website/generated-content/contribute/runner-guide/index.html
@@ -683,7 +683,7 @@ via the <a href="#the-fn-api">Fn API</a> may manifest as another implementation
<p><strong>Python</strong></p>
-<p>See the <a href="https://beam.apache.org/documentation/sdks/pydoc/2.0.0/apache_beam.runners.html#apache_beam.runners.common.DoFnRunner">DoFnRunner pydoc</a>.</p>
+<p>See the <a href="https://beam.apache.org/releases/pydoc/2.0.0/apache_beam.runners.html#apache_beam.runners.common.DoFnRunner">DoFnRunner pydoc</a>.</p>
<h4 id="side-inputs">Side Inputs</h4>
@@ -730,7 +730,7 @@ is used to implement this.</p>
<p><strong>Python</strong></p>
-<p>In Python, <a href="https://beam.apache.org/documentation/sdks/pydoc/2.0.0/apache_beam.transforms.html#apache_beam.transforms.sideinputs.SideInputMap"><code class="highlighter-rouge">SideInputMap</code></a> maps
+<p>In Python, <a href="https://beam.apache.org/releases/pydoc/2.0.0/apache_beam.transforms.html#apache_beam.transforms.sideinputs.SideInputMap"><code class="highlighter-rouge">SideInputMap</code></a> maps
windows to side input values. The <code class="highlighter-rouge">WindowMappingFn</code> manifests as a simple
function. See
<a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/sideinputs.py">sideinputs.py</a>.</p>
@@ -786,9 +786,9 @@ have some special knowledge of the types involved.</p>
<p>The elements you are processing will be key-value pairs, and you’ll need to extract
the keys. For this reason, the format of key-value pairs is standardized and
shared across all SDKS. See either
-<a href="https://beam.apache.org/documentation/sdks/javadoc/2.0.0/org/apache/beam/sdk/coders/KvCoder.html"><code class="highlighter-rouge">KvCoder</code></a>
+<a href="https://beam.apache.org/releases/javadoc/2.0.0/org/apache/beam/sdk/coders/KvCoder.html"><code class="highlighter-rouge">KvCoder</code></a>
in Java or
-<a href="https://beam.apache.org/documentation/sdks/pydoc/2.0.0/apache_beam.coders.html#apache_beam.coders.coders.TupleCoder.key_coder"><code class="highlighter-rouge">TupleCoder</code></a>
+<a href="https://beam.apache.org/releases/pydoc/2.0.0/apache_beam.coders.html#apache_beam.coders.coders.TupleCoder.key_coder"><code class="highlighter-rouge">TupleCoder</code></a>
in Python for documentation on the binary format.</p>
<h4 id="window-merging">Window Merging</h4>
@@ -957,9 +957,9 @@ it into primitives for your engine. The general pattern is to write a visitor
that builds a job specification as it walks the graph of <code class="highlighter-rouge">PTransforms</code>.</p>
<p>The entry point for this in Java is
-<a href="https://beam.apache.org/documentation/sdks/javadoc/2.0.0/org/apache/beam/sdk/Pipeline.html#traverseTopologically-org.apache.beam.sdk.Pipeline.PipelineVisitor-"><code class="highlighter-rouge">Pipeline.traverseTopologically</code></a>
+<a href="https://beam.apache.org/releases/javadoc/2.0.0/org/apache/beam/sdk/Pipeline.html#traverseTopologically-org.apache.beam.sdk.Pipeline.PipelineVisitor-"><code class="highlighter-rouge">Pipeline.traverseTopologically</code></a>
and
-<a href="https://beam.apache.org/documentation/sdks/pydoc/2.0.0/apache_beam.html#apache_beam.pipeline.Pipeline.visit"><code class="highlighter-rouge">Pipeline.visit</code></a>
+<a href="https://beam.apache.org/releases/pydoc/2.0.0/apache_beam.html#apache_beam.pipeline.Pipeline.visit"><code class="highlighter-rouge">Pipeline.visit</code></a>
in Python. See the generated documentation for details.</p>
<h3 id="altering-a-pipeline">Altering a pipeline</h3>
@@ -983,7 +983,7 @@ your runner</li>
of work. In Python, support code is still under development.</p>
<p>All pipeline alteration is done via
-<a href="https://beam.apache.org/documentation/sdks/javadoc/2.0.0/org/apache/beam/sdk/Pipeline.html#replaceAll-java.util.List-"><code class="highlighter-rouge">Pipeline.replaceAll(PTransformOverride)</code></a>
+<a href="https://beam.apache.org/releases/javadoc/2.0.0/org/apache/beam/sdk/Pipeline.html#replaceAll-java.util.List-"><code class="highlighter-rouge">Pipeline.replaceAll(PTransformOverride)</code></a>
method. A
<a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/runners/PTransformOverride.java"><code class="highlighter-rouge">PTransformOverride</code></a>
is a pair of a
@@ -1030,7 +1030,7 @@ want the users of that SDK (such as Python) to use it.</p>
<h4 id="allowing-users-to-pass-options-to-your-runner">Allowing users to pass options to your runner</h4>
<p>The mechanism for configuration is
-<a href="https://beam.apache.org/documentation/sdks/javadoc/2.0.0/org/apache/beam/sdk/options/PipelineOptions.html"><code class="highlighter-rouge">PipelineOptions</code></a>,
+<a href="https://beam.apache.org/releases/javadoc/2.0.0/org/apache/beam/sdk/options/PipelineOptions.html"><code class="highlighter-rouge">PipelineOptions</code></a>,
an interface that works completely differently than normal Java objects. Forget
what you know, and follow the rules, and <code class="highlighter-rouge">PipelineOptions</code> will treat you well.</p>
diff --git a/website/generated-content/documentation/dsls/sql/aggregate-functions/index.html b/website/generated-content/documentation/dsls/sql/aggregate-functions/index.html
index ce77ea5..00b3644 100644
--- a/website/generated-content/documentation/dsls/sql/aggregate-functions/index.html
+++ b/website/generated-content/documentation/dsls/sql/aggregate-functions/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="/documentation/sdks/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="/documentation/sdks/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
diff --git a/website/generated-content/documentation/dsls/sql/create-table/index.html b/website/generated-content/documentation/dsls/sql/create-table/index.html
index c40550c..df62158 100644
--- a/website/generated-content/documentation/dsls/sql/create-table/index.html
+++ b/website/generated-content/documentation/dsls/sql/create-table/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="/documentation/sdks/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="/documentation/sdks/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -492,7 +492,7 @@ value if no extra configuration is provided.</li>
<li>A timestamp specified in one of the user-provided message attributes.
The attribute key is configured by the <code class="highlighter-rouge">timestampAttributeKey</code> field of
the <code class="highlighter-rouge">tblProperties</code> blob. The value of the attribute should conform to
-the <a href="https://beam.apache.org/documentation/sdks/javadoc/2.4.0/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.Read.html#withTimestampAttribute-java.lang.String-">requirements of
+the <a href="https://beam.apache.org/releases/javadoc/2.4.0/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.Read.html#withTimestampAttribute-java.lang.String-">requirements of
PubsubIO</a>,
which is either millis since Unix epoch or <a href="https://www.ietf.org/rfc/rfc3339.txt">RFC 339
</a>date string.</li>
diff --git a/website/generated-content/documentation/dsls/sql/data-types/index.html b/website/generated-content/documentation/dsls/sql/data-types/index.html
index c52c50a..58c6319 100644
--- a/website/generated-content/documentation/dsls/sql/data-types/index.html
+++ b/website/generated-content/documentation/dsls/sql/data-types/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="/documentation/sdks/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="/documentation/sdks/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
diff --git a/website/generated-content/documentation/dsls/sql/joins/index.html b/website/generated-content/documentation/dsls/sql/joins/index.html
index 0cd17e2..1ef8463 100644
--- a/website/generated-content/documentation/dsls/sql/joins/index.html
+++ b/website/generated-content/documentation/dsls/sql/joins/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="/documentation/sdks/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="/documentation/sdks/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
diff --git a/website/generated-content/documentation/dsls/sql/lexical/index.html b/website/generated-content/documentation/dsls/sql/lexical/index.html
index 94f3212..779c589 100644
--- a/website/generated-content/documentation/dsls/sql/lexical/index.html
+++ b/website/generated-content/documentation/dsls/sql/lexical/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="/documentation/sdks/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="/documentation/sdks/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
diff --git a/website/generated-content/documentation/dsls/sql/overview/index.html b/website/generated-content/documentation/dsls/sql/overview/index.html
index fcd5879..0cc19ae 100644
--- a/website/generated-content/documentation/dsls/sql/overview/index.html
+++ b/website/generated-content/documentation/dsls/sql/overview/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="/documentation/sdks/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="/documentation/sdks/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -256,9 +256,9 @@ big data processing with some streaming enhancements. Calcite provides the
basic dialect underlying Beam SQL. We have added additional extensions to
make it easy to leverage Beam’s unified batch/streaming model and support
for complex data types.</li>
- <li><a href="/documentation/sdks/javadoc/2.6.0/index.html?org/apache/beam/sdk/extensions/sql/SqlTransform.html">SqlTransform</a>:
+ <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/extensions/sql/SqlTransform.html">SqlTransform</a>:
the interface for creating <code class="highlighter-rouge">PTransforms</code> from SQL queries.</li>
- <li><a href="/documentation/sdks/javadoc/2.6.0/index.html?org/apache/beam/sdk/values/Row.html">Row</a>:
+ <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/values/Row.html">Row</a>:
the type of elements that Beam SQL operates on. A <code class="highlighter-rouge">PCollection<Row></code> plays the role of a table.</li>
</ul>
diff --git a/website/generated-content/documentation/dsls/sql/scalar-functions/index.html b/website/generated-content/documentation/dsls/sql/scalar-functions/index.html
index 781e07c..7fcce88 100644
--- a/website/generated-content/documentation/dsls/sql/scalar-functions/index.html
+++ b/website/generated-content/documentation/dsls/sql/scalar-functions/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="/documentation/sdks/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="/documentation/sdks/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
diff --git a/website/generated-content/documentation/dsls/sql/select/index.html b/website/generated-content/documentation/dsls/sql/select/index.html
index 1d77b77..246ef64 100644
--- a/website/generated-content/documentation/dsls/sql/select/index.html
+++ b/website/generated-content/documentation/dsls/sql/select/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="/documentation/sdks/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="/documentation/sdks/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
diff --git a/website/generated-content/documentation/dsls/sql/set/index.html b/website/generated-content/documentation/dsls/sql/set/index.html
index 144c71a..3de979c 100644
--- a/website/generated-content/documentation/dsls/sql/set/index.html
+++ b/website/generated-content/documentation/dsls/sql/set/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="/documentation/sdks/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="/documentation/sdks/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
diff --git a/website/generated-content/documentation/dsls/sql/shell/index.html b/website/generated-content/documentation/dsls/sql/shell/index.html
index e215b22..4ebe5a2 100644
--- a/website/generated-content/documentation/dsls/sql/shell/index.html
+++ b/website/generated-content/documentation/dsls/sql/shell/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="/documentation/sdks/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="/documentation/sdks/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
diff --git a/website/generated-content/documentation/dsls/sql/user-defined-functions/index.html b/website/generated-content/documentation/dsls/sql/user-defined-functions/index.html
index 135b985..03f754f 100644
--- a/website/generated-content/documentation/dsls/sql/user-defined-functions/index.html
+++ b/website/generated-content/documentation/dsls/sql/user-defined-functions/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="/documentation/sdks/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="/documentation/sdks/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
diff --git a/website/generated-content/documentation/dsls/sql/walkthrough/index.html b/website/generated-content/documentation/dsls/sql/walkthrough/index.html
index 0ba1520..9176acb 100644
--- a/website/generated-content/documentation/dsls/sql/walkthrough/index.html
+++ b/website/generated-content/documentation/dsls/sql/walkthrough/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="/documentation/sdks/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="/documentation/sdks/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -254,8 +254,8 @@ limitations under the License.
<p>Before applying a SQL query to a <code class="highlighter-rouge">PCollection</code>, the data in the collection must
be in <code class="highlighter-rouge">Row</code> format. A <code class="highlighter-rouge">Row</code> represents a single, immutable record in a Beam SQL
<code class="highlighter-rouge">PCollection</code>. The names and types of the fields/columns in the row are defined
-by its associated <a href="/documentation/sdks/javadoc/2.6.0/index.html?org/apache/beam/sdk/schemas/Schema.html">Schema</a>.
-You can use the <a href="/documentation/sdks/javadoc/2.6.0/index.html?org/apache/beam/sdk/schemas/Schema.html">Schema.builder()</a> to create
+by its associated <a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/schemas/Schema.html">Schema</a>.
+You can use the <a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/schemas/Schema.html">Schema.builder()</a> to create
<code class="highlighter-rouge">Schemas</code>. See <a href="/documentation/dsls/sql/data-types">Data
Types</a> for more details on supported primitive data types.</p>
@@ -338,7 +338,7 @@ Types</a> for more details on supported primitive data types.</p>
<h2 id="sqltransform">SqlTransform</h2>
-<p><a href="/documentation/sdks/javadoc/2.6.0/index.html?org/apache/beam/sdk/extensions/sql/SqlTransform.html"><code class="highlighter-rouge">SqlTransform.query(queryString)</code></a> method is the only API to create a <code class="highlighter-rouge">PTransform</code>
+<p><a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/extensions/sql/SqlTransform.html"><code class="highlighter-rouge">SqlTransform.query(queryString)</code></a> method is the only API to create a <code class="highlighter-rouge">PTransform</code>
from a string representation of the SQL query. You can apply this <code class="highlighter-rouge">PTransform</code>
to either a single <code class="highlighter-rouge">PCollection</code> or a <code class="highlighter-rouge">PCollectionTuple</code> which holds multiple
<code class="highlighter-rouge">PCollections</code>:</p>
diff --git a/website/generated-content/documentation/dsls/sql/windowing-and-triggering/index.html b/website/generated-content/documentation/dsls/sql/windowing-and-triggering/index.html
index 6f8d33b..89dfb7f 100644
--- a/website/generated-content/documentation/dsls/sql/windowing-and-triggering/index.html
+++ b/website/generated-content/documentation/dsls/sql/windowing-and-triggering/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="/documentation/sdks/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="/documentation/sdks/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
diff --git a/website/generated-content/documentation/pipelines/test-your-pipeline/index.html b/website/generated-content/documentation/pipelines/test-your-pipeline/index.html
index fad9c5a..72cdc2d 100644
--- a/website/generated-content/documentation/pipelines/test-your-pipeline/index.html
+++ b/website/generated-content/documentation/pipelines/test-your-pipeline/index.html
@@ -472,7 +472,7 @@ for the <code class="highlighter-rouge">DoFnTester</code> as follows:</p>
<p>You can use the <code class="highlighter-rouge">Create</code> transform to create a <code class="highlighter-rouge">PCollection</code> out of a standard in-memory collection class, such as Java <code class="highlighter-rouge">List</code>. See <a href="/documentation/programming-guide/#creating-a-pcollection">Creating a PCollection</a> for more information.</p>
<h3 id="passert">PAssert</h3>
-<p><a href="/documentation/sdks/javadoc/2.6.0/index.html?org/apache/beam/sdk/testing/PAssert.html">PAssert</a> is a class included in the Beam Java SDK that is an assertion on the contents of a <code class="highlighter-rouge">PCollection</code>. You can use <code class="highlighter-rouge">PAssert</code>to verify that a <code class="highlighter-rouge">PCollection</code> contains a specific set of expected elements.</p>
+<p><a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/testing/PAssert.html">PAssert</a> is a class included in the Beam Java SDK that is an assertion on the contents of a <code class="highlighter-rouge">PCollection</code>. You can use <code class="highlighter-rouge">PAssert</code>to verify that a <code class="highlighter-rouge">PCollection</code> contains a specific set of expected elements.</p>
<p>For a given <code class="highlighter-rouge">PCollection</code>, you can use <code class="highlighter-rouge">PAssert</code> to verify the contents as follows:</p>
@@ -498,7 +498,7 @@ for the <code class="highlighter-rouge">DoFnTester</code> as follows:</p>
</code></pre>
</div>
-<p>For more information on how these classes work, see the <a href="/documentation/sdks/javadoc/2.6.0/index.html?org/apache/beam/sdk/testing/package-summary.html">org.apache.beam.sdk.testing</a> package documentation.</p>
+<p>For more information on how these classes work, see the <a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/testing/package-summary.html">org.apache.beam.sdk.testing</a> package documentation.</p>
<h3 id="an-example-test-for-a-composite-transform">An Example Test for a Composite Transform</h3>
diff --git a/website/generated-content/documentation/programming-guide/index.html b/website/generated-content/documentation/programming-guide/index.html
index 4836731..c9b8345 100644
--- a/website/generated-content/documentation/programming-guide/index.html
+++ b/website/generated-content/documentation/programming-guide/index.html
@@ -420,7 +420,7 @@ asynchronous “job” (or equivalent) on that back-end.</p>
<p>The <code class="highlighter-rouge">Pipeline</code> abstraction encapsulates all the data and steps in your data
processing task. Your Beam driver program typically starts by constructing a
-<span class="language-java"><a href="/documentation/sdks/javadoc/2.6.0/index.html?org/apache/beam/sdk/Pipeline.html">Pipeline</a></span>
+<span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/Pipeline.html">Pipeline</a></span>
<span class="language-py"><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/pipeline.py">Pipeline</a></span>
object, and then using that object as the basis for creating the pipeline’s data
sets as <code class="highlighter-rouge">PCollection</code>s and its operations as <code class="highlighter-rouge">Transform</code>s.</p>
@@ -570,7 +570,7 @@ with <code class="highlighter-rouge">PipelineOptionsFactory</code>:</p>
<h2 id="pcollections">3. PCollections</h2>
-<p>The <span class="language-java"><a href="/documentation/sdks/javadoc/2.6.0/index.html?org/apache/beam/sdk/values/PCollection.html">PCollection</a></span>
+<p>The <span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/values/PCollection.html">PCollection</a></span>
<span class="language-py"><code class="highlighter-rouge">PCollection</code></span> abstraction represents a
potentially distributed, multi-element data set. You can think of a
<code class="highlighter-rouge">PCollection</code> as “pipeline” data; Beam transforms use <code class="highlighter-rouge">PCollection</code> objects as
@@ -1380,7 +1380,7 @@ and format data from each collection.</p>
<h4 id="combine">4.2.4. Combine</h4>
-<p><span class="language-java"><a href="/documentation/sdks/javadoc/2.6.0/index.html?org/apache/beam/sdk/transforms/Combine.html"><code class="highlighter-rouge">Combine</code></a></span>
+<p><span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/transforms/Combine.html"><code class="highlighter-rouge">Combine</code></a></span>
<span class="language-py"><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/core.py"><code class="highlighter-rouge">Combine</code></a></span>
is a Beam transform for combining collections of elements or values in your
data. <code class="highlighter-rouge">Combine</code> has variants that work on entire <code class="highlighter-rouge">PCollection</code>s, and some that
@@ -1639,7 +1639,7 @@ PerKey must be an associative reduction function or a subclass of <code class="h
<h4 id="flatten">4.2.5. Flatten</h4>
-<p><span class="language-java"><a href="/documentation/sdks/javadoc/2.6.0/index.html?org/apache/beam/sdk/transforms/Flatten.html"><code class="highlighter-rouge">Flatten</code></a></span>
+<p><span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/transforms/Flatten.html"><code class="highlighter-rouge">Flatten</code></a></span>
<span class="language-py"><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/core.py"><code class="highlighter-rouge">Flatten</code></a></span> and
is a Beam transform for <code class="highlighter-rouge">PCollection</code> objects that store the same data type.
<code class="highlighter-rouge">Flatten</code> merges multiple <code class="highlighter-rouge">PCollection</code> objects into a single logical
@@ -1690,7 +1690,7 @@ pipeline is constructed.</p>
<h4 id="partition">4.2.6. Partition</h4>
-<p><span class="language-java"><a href="/documentation/sdks/javadoc/2.6.0/index.html?org/apache/beam/sdk/transforms/Partition.html"><code class="highlighter-rouge">Partition</code></a></span>
+<p><span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/transforms/Partition.html"><code class="highlighter-rouge">Partition</code></a></span>
<span class="language-py"><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/core.py"><code class="highlighter-rouge">Partition</code></a></span>
is a Beam transform for <code class="highlighter-rouge">PCollection</code> objects that store the same data
type. <code class="highlighter-rouge">Partition</code> splits a single <code class="highlighter-rouge">PCollection</code> into a fixed number of smaller
@@ -2134,8 +2134,8 @@ transform can make your code more modular and easier to understand.</p>
<p>The Beam SDK comes packed with many useful composite transforms. See the API
reference pages for a list of transforms:</p>
<ul>
- <li><a href="/documentation/sdks/javadoc/2.6.0/index.html?org/apache/beam/sdk/transforms/package-summary.html">Pre-written Beam transforms for Java</a></li>
- <li><a href="/documentation/sdks/pydoc/2.6.0/apache_beam.transforms.html">Pre-written Beam transforms for Python</a></li>
+ <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/transforms/package-summary.html">Pre-written Beam transforms for Java</a></li>
+ <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/apache_beam.transforms.html">Pre-written Beam transforms for Python</a></li>
</ul>
<h4 id="composite-transform-example">4.6.1. An example composite transform</h4>
@@ -2722,7 +2722,7 @@ all the elements are by default part of a single, global window.</p>
<p>To use windowing with fixed data sets, you can assign your own timestamps to
each element. To assign timestamps to elements, use a <code class="highlighter-rouge">ParDo</code> transform with a
<code class="highlighter-rouge">DoFn</code> that outputs each element with a new timestamp (for example, the
-<a href="/documentation/sdks/javadoc/2.6.0/index.html?org/apache/beam/sdk/transforms/WithTimestamps.html">WithTimestamps</a>
+<a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/transforms/WithTimestamps.html">WithTimestamps</a>
transform in the Beam SDK for Java).</p>
<p>To illustrate how windowing with a bounded <code class="highlighter-rouge">PCollection</code> can affect how your
diff --git a/website/generated-content/documentation/runners/dataflow/index.html b/website/generated-content/documentation/runners/dataflow/index.html
index 872d9f5..b742b8d 100644
--- a/website/generated-content/documentation/runners/dataflow/index.html
+++ b/website/generated-content/documentation/runners/dataflow/index.html
@@ -383,8 +383,8 @@ Pub/Sub, or Cloud Datastore) if you use them in your pipeline code.</li>
</table>
<p>See the reference documentation for the
-<span class="language-java"><a href="/documentation/sdks/javadoc/2.6.0/index.html?org/apache/beam/runners/dataflow/options/DataflowPipelineOptions.html">DataflowPipelineOptions</a></span>
-<span class="language-py"><a href="/documentation/sdks/pydoc/2.6.0/apache_beam.options.pipeline_options.html#apache_beam.options.pipeline_options.PipelineOptions"><code class="highlighter-rouge">PipelineOptions</code></a></span>
+<span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/runners/dataflow/options/DataflowPipelineOptions.html">DataflowPipelineOptions</a></span>
+<span class="language-py"><a href="https://beam.apache.org/releases/pydoc/2.6.0/apache_beam.options.pipeline_options.html#apache_beam.options.pipeline_options.PipelineOptions"><code class="highlighter-rouge">PipelineOptions</code></a></span>
interface (and any subinterfaces) for additional pipeline configuration options.</p>
<h2 id="additional-info">Additional information and caveats</h2>
diff --git a/website/generated-content/documentation/runners/direct/index.html b/website/generated-content/documentation/runners/direct/index.html
index e25358d..7a6ddc6 100644
--- a/website/generated-content/documentation/runners/direct/index.html
+++ b/website/generated-content/documentation/runners/direct/index.html
@@ -221,11 +221,11 @@ limitations under the License.
<p>Here are some resources with information about how to test your pipelines.</p>
<ul>
<!-- Java specific links -->
- <li class="language-java"><a href="/blog/2016/10/20/test-stream.html">Testing Unbounded Pipelines in Apache Beam</a> talks about the use of Java classes <a href="/documentation/sdks/javadoc/2.6.0/index.html?org/apache/beam/sdk/testing/PAssert.html">PAssert</a> and <a href="/documentation/sdks/javadoc/2.6.0/index.html?org/apache/beam/sdk/testing/TestStream.html">TestStream</a> to test your pipelines.</li>
- <li class="language-java">The <a href="/get-started/wordcount-example/#testing-your-pipeline-with-asserts">Apache Beam WordCount Walkthrough</a> contains an example of logging and testing a pipeline with <a href="/documentation/sdks/javadoc/2.6.0/index.html?org/apache/beam/sdk/testing/PAssert.html">PAssert</a>.</li>
+ <li class="language-java"><a href="/blog/2016/10/20/test-stream.html">Testing Unbounded Pipelines in Apache Beam</a> talks about the use of Java classes <a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/testing/PAssert.html">PAssert</a> and <a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/testing/TestStream.html">TestStream</a> to test your pipelines.</li>
+ <li class="language-java">The <a href="/get-started/wordcount-example/#testing-your-pipeline-with-asserts">Apache Beam WordCount Walkthrough</a> contains an example of logging and testing a pipeline with <a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/testing/PAssert.html">PAssert</a>.</li>
<!-- Python specific links -->
- <li class="language-py">The <a href="/get-started/wordcount-example/#testing-your-pipeline-with-asserts">Apache Beam WordCount Walkthrough</a> contains an example of logging and testing a pipeline with <a href="/documentation/sdks/pydoc/2.6.0/apache_beam.testing.util.html#apache_beam.testing.util.assert_that">assert_that</a>.</li>
+ <li class="language-py">The <a href="/get-started/wordcount-example/#testing-your-pipeline-with-asserts">Apache Beam WordCount Walkthrough</a> contains an example of logging and testing a pipeline with <a href="https://beam.apache.org/releases/pydoc/2.6.0/apache_beam.testing.util.html#apache_beam.testing.util.assert_that">assert_that</a>.</li>
</ul>
<h2 id="direct-runner-prerequisites-and-setup">Direct Runner prerequisites and setup</h2>
@@ -249,15 +249,15 @@ limitations under the License.
<p>When executing your pipeline from the command-line, set <code class="highlighter-rouge">runner</code> to <code class="highlighter-rouge">direct</code> or <code class="highlighter-rouge">DirectRunner</code>. The default values for the other pipeline options are generally sufficient.</p>
<p>See the reference documentation for the
-<span class="language-java"><a href="/documentation/sdks/javadoc/2.6.0/index.html?org/apache/beam/runners/direct/DirectOptions.html"><code class="highlighter-rouge">DirectOptions</code></a></span>
-<span class="language-py"><a href="/documentation/sdks/pydoc/2.6.0/apache_beam.options.pipeline_options.html#apache_beam.options.pipeline_options.DirectOptions"><code class="highlighter-rouge">DirectOptions</code></a></span>
+<span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/runners/direct/DirectOptions.html"><code class="highlighter-rouge">DirectOptions</code></a></span>
+<span class="language-py"><a href="https://beam.apache.org/releases/pydoc/2.6.0/apache_beam.options.pipeline_options.html#apache_beam.options.pipeline_options.DirectOptions"><code class="highlighter-rouge">DirectOptions</code></a></span>
interface for defaults and additional pipeline configuration options.</p>
<h2 id="additional-information-and-caveats">Additional information and caveats</h2>
<h3 id="memory-considerations">Memory considerations</h3>
-<p>Local execution is limited by the memory available in your local environment. It is highly recommended that you run your pipeline with data sets small enough to fit in local memory. You can create a small in-memory data set using a <span class="language-java"><a href="/documentation/sdks/javadoc/2.6.0/index.html?org/apache/beam/sdk/transforms/Create.html"><code class="highlighter-rouge">Create</code></a></span><span class="language-py"><a href="https://github.com/apache/beam/blob/mast [...]
+<p>Local execution is limited by the memory available in your local environment. It is highly recommended that you run your pipeline with data sets small enough to fit in local memory. You can create a small in-memory data set using a <span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/transforms/Create.html"><code class="highlighter-rouge">Create</code></a></span><span class="language-py"><a href="https://github.com/apache/b [...]
<h3 id="streaming-execution">Streaming execution</h3>
diff --git a/website/generated-content/documentation/runners/flink/index.html b/website/generated-content/documentation/runners/flink/index.html
index 84e579b..188c987 100644
--- a/website/generated-content/documentation/runners/flink/index.html
+++ b/website/generated-content/documentation/runners/flink/index.html
@@ -360,7 +360,7 @@ limitations under the License.
</tr>
</table>
-<p>See the reference documentation for the <span class="language-java"><a href="/documentation/sdks/javadoc/2.6.0/index.html?org/apache/beam/runners/flink/FlinkPipelineOptions.html">FlinkPipelineOptions</a></span><span class="language-py"><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py">PipelineOptions</a></span> interface (and its subinterfaces) for the complete list of pipeline configuration options.</p>
+<p>See the reference documentation for the <span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/runners/flink/FlinkPipelineOptions.html">FlinkPipelineOptions</a></span><span class="language-py"><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py">PipelineOptions</a></span> interface (and its subinterfaces) for the complete list of pipeline configuration options.</p>
<h2 id="additional-information-and-caveats">Additional information and caveats</h2>
diff --git a/website/generated-content/documentation/sdks/feature-comparison/index.html b/website/generated-content/documentation/sdks/feature-comparison/index.html
index 41a4dd0..c6bbcee 100644
--- a/website/generated-content/documentation/sdks/feature-comparison/index.html
+++ b/website/generated-content/documentation/sdks/feature-comparison/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="/documentation/sdks/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="/documentation/sdks/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
diff --git a/website/generated-content/documentation/sdks/go/index.html b/website/generated-content/documentation/sdks/go/index.html
index e8c5bad..00c0d1f 100644
--- a/website/generated-content/documentation/sdks/go/index.html
+++ b/website/generated-content/documentation/sdks/go/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="/documentation/sdks/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="/documentation/sdks/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
diff --git a/website/generated-content/documentation/sdks/java-extensions/index.html b/website/generated-content/documentation/sdks/java-extensions/index.html
index 904a9b0..4ee5f3f 100644
--- a/website/generated-content/documentation/sdks/java-extensions/index.html
+++ b/website/generated-content/documentation/sdks/java-extensions/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="/documentation/sdks/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="/documentation/sdks/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
diff --git a/website/generated-content/documentation/sdks/java-thirdparty/index.html b/website/generated-content/documentation/sdks/java-thirdparty/index.html
index a4e0afb..3a4884c 100644
--- a/website/generated-content/documentation/sdks/java-thirdparty/index.html
+++ b/website/generated-content/documentation/sdks/java-thirdparty/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="/documentation/sdks/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="/documentation/sdks/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
diff --git a/website/generated-content/documentation/sdks/java/euphoria/index.html b/website/generated-content/documentation/sdks/java/euphoria/index.html
index a0c0393..af6ed2a 100644
--- a/website/generated-content/documentation/sdks/java/euphoria/index.html
+++ b/website/generated-content/documentation/sdks/java/euphoria/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="/documentation/sdks/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="/documentation/sdks/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
diff --git a/website/generated-content/documentation/sdks/java/index.html b/website/generated-content/documentation/sdks/java/index.html
index 322fa99..6cd38f5 100644
--- a/website/generated-content/documentation/sdks/java/index.html
+++ b/website/generated-content/documentation/sdks/java/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="/documentation/sdks/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="/documentation/sdks/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -254,7 +254,7 @@ limitations under the License.
<p>Get started with the <a href="/documentation/programming-guide/">Beam Programming Model</a> to learn the basic concepts that apply to all SDKs in Beam.</p>
-<p>See the <a href="/documentation/sdks/javadoc/">Java API Reference</a> for more information on individual APIs.</p>
+<p>See the <a href="https://beam.apache.org/releases/javadoc/">Java API Reference</a> for more information on individual APIs.</p>
<h2 id="supported-features">Supported Features</h2>
diff --git a/website/generated-content/documentation/sdks/java/nexmark/index.html b/website/generated-content/documentation/sdks/java/nexmark/index.html
index f1b80fc..1676235 100644
--- a/website/generated-content/documentation/sdks/java/nexmark/index.html
+++ b/website/generated-content/documentation/sdks/java/nexmark/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="/documentation/sdks/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="/documentation/sdks/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
diff --git a/website/generated-content/documentation/sdks/python-custom-io/index.html b/website/generated-content/documentation/sdks/python-custom-io/index.html
index 3402e0b..9b20418 100644
--- a/website/generated-content/documentation/sdks/python-custom-io/index.html
+++ b/website/generated-content/documentation/sdks/python-custom-io/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="/documentation/sdks/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="/documentation/sdks/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
diff --git a/website/generated-content/documentation/sdks/python-pipeline-dependencies/index.html b/website/generated-content/documentation/sdks/python-pipeline-dependencies/index.html
index 9dfba4c..b80898b 100644
--- a/website/generated-content/documentation/sdks/python-pipeline-dependencies/index.html
+++ b/website/generated-content/documentation/sdks/python-pipeline-dependencies/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="/documentation/sdks/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="/documentation/sdks/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
diff --git a/website/generated-content/documentation/sdks/python-streaming/index.html b/website/generated-content/documentation/sdks/python-streaming/index.html
index f521c16..3681102 100644
--- a/website/generated-content/documentation/sdks/python-streaming/index.html
+++ b/website/generated-content/documentation/sdks/python-streaming/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="/documentation/sdks/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="/documentation/sdks/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
diff --git a/website/generated-content/documentation/sdks/python-type-safety/index.html b/website/generated-content/documentation/sdks/python-type-safety/index.html
index b2c071c..f45c0cb 100644
--- a/website/generated-content/documentation/sdks/python-type-safety/index.html
+++ b/website/generated-content/documentation/sdks/python-type-safety/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="/documentation/sdks/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="/documentation/sdks/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
diff --git a/website/generated-content/documentation/sdks/python/index.html b/website/generated-content/documentation/sdks/python/index.html
index c5a9b17..68265f7 100644
--- a/website/generated-content/documentation/sdks/python/index.html
+++ b/website/generated-content/documentation/sdks/python/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="/documentation/sdks/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="/documentation/sdks/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -255,7 +255,7 @@ limitations under the License.
<p>Get started with the <a href="/get-started/quickstart-py">Beam Python SDK quickstart</a> to set up your Python development environment, get the Beam SDK for Python, and run an example pipeline. Then, read through the <a href="/documentation/programming-guide">Beam programming guide</a> to learn the basic concepts that apply to all SDKs in Beam.</p>
-<p>See the <a href="/documentation/sdks/pydoc/">Python API reference</a> for more information on individual APIs.</p>
+<p>See the <a href="https://beam.apache.org/releases/pydoc/">Python API reference</a> for more information on individual APIs.</p>
<h2 id="python-streaming-pipelines">Python streaming pipelines</h2>
diff --git a/website/generated-content/feed.xml b/website/generated-content/feed.xml
index 1d4e692..b51ea98 100644
--- a/website/generated-content/feed.xml
+++ b/website/generated-content/feed.xml
@@ -125,7 +125,7 @@ limitations under the License.
</td>
<td>Local<br />(Using the <code>file://</code> URI)
</td>
- <td><a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/TextIO.html">TextIO</a>
+ <td><a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/TextIO.html">TextIO</a>
</td>
<td><a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/StreamingContext.html#textFileStream-java.lang.String-">textFileStream</a><br />(Spark treats most Unix systems as HDFS-compatible, but the location should be accessible from all nodes)
</td>
@@ -133,7 +133,7 @@ limitations under the License.
<tr>
<td>HDFS<br />(Using the <code>hdfs://</code> URI)
</td>
- <td><a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/FileIO.html">FileIO</a> + <a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptions.html">HadoopFileSystemOptions</a>
+ <td><a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/FileIO.html">FileIO</a> + <a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptions.html">HadoopFileSystemOptions</a>
</td>
<td><a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/util/HdfsUtils.html">HdfsUtils</a>
</td>
@@ -143,7 +143,7 @@ limitations under the License.
</td>
<td>Cloud Storage<br />(Using the <code>gs://</code> URI)
</td>
- <td><a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/FileIO.html">FileIO</a> + <a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/extensions/gcp/options/GcsOptions.html">GcsOptions</a>
+ <td><a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/FileIO.html">FileIO</a> + <a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/extensions/gcp/options/GcsOptions.html">GcsOptions</a>
</td>
<td rowspan="2"><a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/SparkContext.html#hadoopConfiguration--">hadoopConfiguration</a>
and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/StreamingContext.html#textFileStream-java.lang.String-">textFileStream</a>
@@ -152,7 +152,7 @@ and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/sp
<tr>
<td>S3<br />(Using the <code>s3://</code> URI)
</td>
- <td><a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/FileIO.html">FileIO</a> + <a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/aws/options/S3Options.html">S3Options</a>
+ <td><a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/FileIO.html">FileIO</a> + <a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/aws/options/S3Options.html">S3Options</a>
</td>
</tr>
<tr>
@@ -160,7 +160,7 @@ and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/sp
</td>
<td>Kafka
</td>
- <td><a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/kafka/KafkaIO.html">KafkaIO</a>
+ <td><a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/kafka/KafkaIO.html">KafkaIO</a>
</td>
<td><a href="https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html">spark-streaming-kafka</a>
</td>
@@ -168,7 +168,7 @@ and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/sp
<tr>
<td>Kinesis
</td>
- <td><a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/kinesis/KinesisIO.html">KinesisIO</a>
+ <td><a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/kinesis/KinesisIO.html">KinesisIO</a>
</td>
<td><a href="https://spark.apache.org/docs/latest/streaming-kinesis-integration.html">spark-streaming-kinesis</a>
</td>
@@ -176,7 +176,7 @@ and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/sp
<tr>
<td>Cloud Pub/Sub
</td>
- <td><a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.html">PubsubIO</a>
+ <td><a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.html">PubsubIO</a>
</td>
<td><a href="https://github.com/apache/bahir/tree/master/streaming-pubsub">spark-streaming-pubsub</a> from <a href="http://bahir.apache.org">Apache Bahir</a>
</td>
@@ -217,7 +217,7 @@ and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/sp
</td>
<td>Local
</td>
- <td><a href="/documentation/sdks/pydoc/2.6.0/apache_beam.io.textio.html">io.textio</a>
+ <td><a href="https://beam.apache.org/releases/pydoc/2.6.0/apache_beam.io.textio.html">io.textio</a>
</td>
<td><a href="http://spark.apache.org/docs/latest/api/python/pyspark.streaming.html#pyspark.streaming.StreamingContext.textFileStream">textFileStream</a>
</td>
@@ -225,7 +225,7 @@ and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/sp
<tr>
<td>HDFS
</td>
- <td><a href="/documentation/sdks/pydoc/2.6.0/apache_beam.io.hadoopfilesystem.html">io.hadoopfilesystem</a>
+ <td><a href="https://beam.apache.org/releases/pydoc/2.6.0/apache_beam.io.hadoopfilesystem.html">io.hadoopfilesystem</a>
</td>
<td><a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/SparkContext.html#hadoopConfiguration--">hadoopConfiguration</a> (Access through <code>sc._jsc</code> with Py4J)
and <a href="http://spark.apache.org/docs/latest/api/python/pyspark.streaming.html#pyspark.streaming.StreamingContext.textFileStream">textFileStream</a>
@@ -236,7 +236,7 @@ and <a href="http://spark.apache.org/docs/latest/api/python/pyspark.stre
</td>
<td>Google Cloud Storage
</td>
- <td><a href="/documentation/sdks/pydoc/2.6.0/apache_beam.io.gcp.gcsio.html">io.gcp.gcsio</a>
+ <td><a href="https://beam.apache.org/releases/pydoc/2.6.0/apache_beam.io.gcp.gcsio.html">io.gcp.gcsio</a>
</td>
<td rowspan="2"><a href="http://spark.apache.org/docs/latest/api/python/pyspark.streaming.html#pyspark.streaming.StreamingContext.textFileStream">textFileStream</a>
</td>
@@ -268,7 +268,7 @@ and <a href="http://spark.apache.org/docs/latest/api/python/pyspark.stre
<tr>
<td>Cloud Pub/Sub
</td>
- <td><a href="/documentation/sdks/pydoc/2.6.0/apache_beam.io.gcp.pubsub.html">io.gcp.pubsub</a>
+ <td><a href="https://beam.apache.org/releases/pydoc/2.6.0/apache_beam.io.gcp.pubsub.html">io.gcp.pubsub</a>
</td>
<td>N/A
</td>
@@ -1408,22 +1408,22 @@ has other limitations that make it insufficient for this task</em>).</p
<h2 id="beam-source-api">Beam Source API</h2>
<p>Apache Beam historically provides a Source API
-(<a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/BoundedSource.html">BoundedSource</a>
+(<a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/BoundedSource.html">BoundedSource</a>
and
-<a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/UnboundedSource.html">UnboundedSource</a>) which does
+<a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/UnboundedSource.html">UnboundedSource</a>) which does
not have these limitations and allows development of efficient data sources for
batch and streaming systems. Pipelines use this API via the
-<a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/Read.html"><code class="highlighter-rouge">Read.from(Source)</code></a> built-in <code class="highlighter-rouge">PTransform</code>.</p>
+<a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/Read.html"><code class="highlighter-rouge">Read.from(Source)</code></a> built-in <code class="highlighter-rouge">PTransform</code>.</p>
<p>The Source API is largely similar to that of most other data processing
frameworks, and allows the system to read data in parallel using multiple
workers, as well as checkpoint and resume reading from an unbounded data source.
Additionally, the Beam
-<a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/BoundedSource.html"><code class="highlighter-rouge">BoundedSource</code></a>
+<a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/BoundedSource.html"><code class="highlighter-rouge">BoundedSource</code></a>
API provides advanced features such as progress reporting and <a href="/blog/2016/05/18/splitAtFraction-method.html">dynamic
rebalancing</a>
(which together enable autoscaling), and
-<a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/UnboundedSource.html"><code class="highlighter-rouge">UnboundedSource</code></a> supports
+<a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/UnboundedSource.html"><code class="highlighter-rouge">UnboundedSource</code></a> supports
reporting the source’s watermark and backlog <em>(until SDF, we believed that
“batch” and “streaming” data sources are fundamentally different and thus
require fundamentally different APIs)</em>.</p>
@@ -2051,7 +2051,7 @@ limitations under the License.
<h4 id="try-the-apache-beam-python-sdk">Try the Apache Beam Python SDK</h4>
-<p>If you would like to try out the Python SDK, a good place to start is the <a href="/get-started/quickstart-py/">Quickstart</a>. After that, you can take a look at additional <a href="https://github.com/apache/beam/tree/v0.6.0/sdks/python/apache_beam/examples">examples</a>, and deep dive into the <a href="/documentation/sdks/pydoc/">API reference</a>.</p>
+<p>If you would like to try out the Python SDK, a good place to start is the <a href="/get-started/quickstart-py/">Quickstart</a>. After that, you can take a look at additional <a href="https://github.com/apache/beam/tree/v0.6.0/sdks/python/apache_beam/examples">examples</a>, and deep dive into the <a href="https://beam.apache.org/releases/pydoc/">API reference</a>.</p>
<p>Let’s take a look at a quick example together. First, install the <code class="highlighter-rouge">apache-beam</code> package from PyPI and start your Python interpreter.</p>
diff --git a/website/generated-content/get-started/downloads/index.html b/website/generated-content/get-started/downloads/index.html
index ef1648f..a1342c7 100644
--- a/website/generated-content/get-started/downloads/index.html
+++ b/website/generated-content/get-started/downloads/index.html
@@ -267,7 +267,7 @@ the form <code class="highlighter-rouge">major.minor.incremental</code> and are
<li>incremental version for forward-compatible bug fixes</li>
</ul>
-<p>Please note that APIs marked <a href="/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/annotations/Experimental.html"><code class="highlighter-rouge">@Experimental</code></a>
+<p>Please note that APIs marked <a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/annotations/Experimental.html"><code class="highlighter-rouge">@Experimental</code></a>
may change at any point and are not guaranteed to remain compatible across versions.</p>
<p>Additionally, any API may change before the first stable release, i.e., between
diff --git a/website/generated-content/get-started/quickstart-java/index.html b/website/generated-content/get-started/quickstart-java/index.html
index 34c5416..b3fb7b5 100644
--- a/website/generated-content/get-started/quickstart-java/index.html
+++ b/website/generated-content/get-started/quickstart-java/index.html
@@ -523,7 +523,7 @@ has: 2
<ul>
<li>Learn more about the <a href="/documentation/sdks/java/">Beam SDK for Java</a>
-and look through the <a href="/documentation/sdks/javadoc">Java SDK API reference</a>.</li>
+and look through the <a href="https://beam.apache.org/releases/javadoc">Java SDK API reference</a>.</li>
<li>Walk through these WordCount examples in the <a href="/get-started/wordcount-example">WordCount Example Walkthrough</a>.</li>
<li>Dive in to some of our favorite <a href="/documentation/resources">articles and presentations</a>.</li>
<li>Join the Beam <a href="/community/contact-us">users@</a> mailing list.</li>
diff --git a/website/generated-content/get-started/quickstart-py/index.html b/website/generated-content/get-started/quickstart-py/index.html
index f761339..17fc112 100644
--- a/website/generated-content/get-started/quickstart-py/index.html
+++ b/website/generated-content/get-started/quickstart-py/index.html
@@ -413,7 +413,7 @@ sequentially in the format <code class="highlighter-rouge">counts-0000-of-0001</
<ul>
<li>Learn more about the <a href="/documentation/sdks/python/">Beam SDK for Python</a>
-and look through the <a href="/documentation/sdks/pydoc">Python SDK API reference</a>.</li>
+and look through the <a href="https://beam.apache.org/releases/pydoc">Python SDK API reference</a>.</li>
<li>Walk through these WordCount examples in the <a href="/get-started/wordcount-example">WordCount Example Walkthrough</a>.</li>
<li>Dive in to some of our favorite <a href="/documentation/resources">articles and presentations</a>.</li>
<li>Join the Beam <a href="/community/contact-us">users@</a> mailing list.</li>
diff --git a/website/generated-content/get-started/wordcount-example/index.html b/website/generated-content/get-started/wordcount-example/index.html
index 7878541..eb45e10 100644
--- a/website/generated-content/get-started/wordcount-example/index.html
+++ b/website/generated-content/get-started/wordcount-example/index.html
@@ -1614,7 +1614,7 @@ python -m apache_beam.examples.streaming_wordcount \
<p>This example uses an unbounded dataset as input. The code reads Pub/Sub
messages from a Pub/Sub subscription or topic using
-<a href="/documentation/sdks/pydoc/2.6.0/apache_beam.io.gcp.pubsub.html#apache_beam.io.gcp.pubsub.ReadStringsFromPubSub"><code class="highlighter-rouge">beam.io.ReadStringsFromPubSub</code></a>.</p>
+<a href="https://beam.apache.org/releases/pydoc/2.6.0/apache_beam.io.gcp.pubsub.html#apache_beam.io.gcp.pubsub.ReadStringsFromPubSub"><code class="highlighter-rouge">beam.io.ReadStringsFromPubSub</code></a>.</p>
<div class="language-java highlighter-rouge"><pre class="highlight"><code> <span class="c1">// This example is not currently available for the Beam SDK for Java.</span>
</code></pre>
@@ -1640,7 +1640,7 @@ outputs.</p>
<p>This example uses an unbounded <code class="highlighter-rouge">PCollection</code> and streams the results to
Google Pub/Sub. The code formats the results and writes them to a Pub/Sub topic
-using <a href="/documentation/sdks/pydoc/2.6.0/apache_beam.io.gcp.pubsub.html#apache_beam.io.gcp.pubsub.WriteStringsToPubSub"><code class="highlighter-rouge">beam.io.WriteStringsToPubSub</code></a>.</p>
+using <a href="https://beam.apache.org/releases/pydoc/2.6.0/apache_beam.io.gcp.pubsub.html#apache_beam.io.gcp.pubsub.WriteStringsToPubSub"><code class="highlighter-rouge">beam.io.WriteStringsToPubSub</code></a>.</p>
<div class="language-java highlighter-rouge"><pre class="highlight"><code> <span class="c1">// This example is not currently available for the Beam SDK for Java.</span>
</code></pre>
[beam] 09/11: Publishing website 2018/10/05 23:02:55 at commit
9ac7ad4
Posted by sc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 7e2bb0d94a135cfea1cff5a99396124470e8428a
Author: jenkins <bu...@apache.org>
AuthorDate: Fri Oct 5 23:02:56 2018 +0000
Publishing website 2018/10/05 23:02:55 at commit 9ac7ad4
---
.../blog/2016/10/20/test-stream.html | 2 +-
.../blog/2017/08/16/splittable-do-fn.html | 10 +-
.../blog/2018/06/26/beam-2.5.0.html | 2 +-
.../08/20/review-input-streaming-connectors.html | 22 +-
.../26/beam-2.5.0.html => 10/03/beam-2.7.0.html} | 123 ++-
website/generated-content/blog/index.html | 32 +
.../contribute/dependencies/index.html | 24 +-
website/generated-content/contribute/index.html | 2 +-
.../contribute/postcommits-policies/index.html | 3 +-
.../contribute/ptransform-style-guide/index.html | 4 +-
.../contribute/release-guide/index.html | 5 +-
.../contribute/runner-guide/index.html | 2 +-
.../contribute/testing/index.html | 6 +-
.../dsls/sql/aggregate-functions/index.html | 6 +-
.../index.html | 44 +-
.../documentation/dsls/sql/create-table/index.html | 743 +-----------------
.../documentation/dsls/sql/data-types/index.html | 6 +-
.../documentation/dsls/sql/joins/index.html | 6 +-
.../documentation/dsls/sql/lexical/index.html | 6 +-
.../documentation/dsls/sql/overview/index.html | 10 +-
.../dsls/sql/scalar-functions/index.html | 6 +-
.../documentation/dsls/sql/select/index.html | 831 +++++++++++++++++++--
.../documentation/dsls/sql/set/index.html | 6 +-
.../documentation/dsls/sql/shell/index.html | 14 +-
.../dsls/sql/statements/create-table/index.html | 8 +-
.../dsls/sql/user-defined-functions/index.html | 6 +-
.../documentation/dsls/sql/walkthrough/index.html | 12 +-
.../dsls/sql/windowing-and-triggering/index.html | 6 +-
.../documentation/io/testing/index.html | 6 +-
.../pipelines/test-your-pipeline/index.html | 4 +-
.../documentation/programming-guide/index.html | 16 +-
.../documentation/runners/dataflow/index.html | 6 +-
.../documentation/runners/direct/index.html | 14 +-
.../documentation/runners/flink/index.html | 4 +-
.../documentation/runners/gearpump/index.html | 4 +-
.../documentation/runners/jstorm/index.html | 2 +-
.../documentation/runners/mapreduce/index.html | 2 +-
.../documentation/runners/samza/index.html | 2 +-
.../documentation/runners/spark/index.html | 2 +-
.../sdks/feature-comparison/index.html | 6 +-
.../documentation/sdks/go/index.html | 6 +-
.../documentation/sdks/java-extensions/index.html | 6 +-
.../documentation/sdks/java-thirdparty/index.html | 6 +-
.../documentation/sdks/java/euphoria/index.html | 6 +-
.../documentation/sdks/java/index.html | 6 +-
.../documentation/sdks/java/nexmark/index.html | 10 +-
.../documentation/sdks/python-custom-io/index.html | 6 +-
.../sdks/python-pipeline-dependencies/index.html | 6 +-
.../documentation/sdks/python-streaming/index.html | 6 +-
.../sdks/python-type-safety/index.html | 6 +-
.../documentation/sdks/python/index.html | 6 +-
website/generated-content/feed.xml | 208 +++---
.../get-started/downloads/index.html | 18 +-
.../get-started/quickstart-java/index.html | 8 +-
.../get-started/quickstart-py/index.html | 2 +-
.../get-started/wordcount-example/index.html | 6 +-
website/generated-content/index.html | 14 +-
57 files changed, 1185 insertions(+), 1156 deletions(-)
diff --git a/website/generated-content/blog/2016/10/20/test-stream.html b/website/generated-content/blog/2016/10/20/test-stream.html
index 596c845..7b306cd 100644
--- a/website/generated-content/blog/2016/10/20/test-stream.html
+++ b/website/generated-content/blog/2016/10/20/test-stream.html
@@ -210,7 +210,7 @@ be controlled within a test.</p>
<h2 id="writing-deterministic-tests-to-emulate-nondeterminism">Writing Deterministic Tests to Emulate Nondeterminism</h2>
<p>The Beam testing infrastructure provides the
-<a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/testing/PAssert.html">PAssert</a>
+<a href="https://beam.apache.org/releases/javadoc/2.7.0/org/apache/beam/sdk/testing/PAssert.html">PAssert</a>
methods, which assert properties about the contents of a PCollection from within
a pipeline. We have expanded this infrastructure to include
<a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestStream.java">TestStream</a>,
diff --git a/website/generated-content/blog/2017/08/16/splittable-do-fn.html b/website/generated-content/blog/2017/08/16/splittable-do-fn.html
index dd5d359..6a95ba5 100644
--- a/website/generated-content/blog/2017/08/16/splittable-do-fn.html
+++ b/website/generated-content/blog/2017/08/16/splittable-do-fn.html
@@ -223,22 +223,22 @@ has other limitations that make it insufficient for this task</em>).</p>
<h2 id="beam-source-api">Beam Source API</h2>
<p>Apache Beam historically provides a Source API
-(<a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/BoundedSource.html">BoundedSource</a>
+(<a href="https://beam.apache.org/releases/javadoc/2.7.0/org/apache/beam/sdk/io/BoundedSource.html">BoundedSource</a>
and
-<a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/UnboundedSource.html">UnboundedSource</a>) which does
+<a href="https://beam.apache.org/releases/javadoc/2.7.0/org/apache/beam/sdk/io/UnboundedSource.html">UnboundedSource</a>) which does
not have these limitations and allows development of efficient data sources for
batch and streaming systems. Pipelines use this API via the
-<a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/Read.html"><code class="highlighter-rouge">Read.from(Source)</code></a> built-in <code class="highlighter-rouge">PTransform</code>.</p>
+<a href="https://beam.apache.org/releases/javadoc/2.7.0/org/apache/beam/sdk/io/Read.html"><code class="highlighter-rouge">Read.from(Source)</code></a> built-in <code class="highlighter-rouge">PTransform</code>.</p>
<p>The Source API is largely similar to that of most other data processing
frameworks, and allows the system to read data in parallel using multiple
workers, as well as checkpoint and resume reading from an unbounded data source.
Additionally, the Beam
-<a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/BoundedSource.html"><code class="highlighter-rouge">BoundedSource</code></a>
+<a href="https://beam.apache.org/releases/javadoc/2.7.0/org/apache/beam/sdk/io/BoundedSource.html"><code class="highlighter-rouge">BoundedSource</code></a>
API provides advanced features such as progress reporting and <a href="/blog/2016/05/18/splitAtFraction-method.html">dynamic
rebalancing</a>
(which together enable autoscaling), and
-<a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/UnboundedSource.html"><code class="highlighter-rouge">UnboundedSource</code></a> supports
+<a href="https://beam.apache.org/releases/javadoc/2.7.0/org/apache/beam/sdk/io/UnboundedSource.html"><code class="highlighter-rouge">UnboundedSource</code></a> supports
reporting the source’s watermark and backlog <em>(until SDF, we believed that
“batch” and “streaming” data sources are fundamentally different and thus
require fundamentally different APIs)</em>.</p>
diff --git a/website/generated-content/blog/2018/06/26/beam-2.5.0.html b/website/generated-content/blog/2018/06/26/beam-2.5.0.html
index e054124..32a2531 100644
--- a/website/generated-content/blog/2018/06/26/beam-2.5.0.html
+++ b/website/generated-content/blog/2018/06/26/beam-2.5.0.html
@@ -165,7 +165,7 @@ please check the detailed release notes.</p>
<h1 id="new-features--improvements">New Features / Improvements</h1>
<h2 id="go-sdk-support">Go SDK support</h2>
-<p>The Go SDK has been officially accepted into the project, after an incubation period and community effort. Go pipelines run on Dataflow runner. More details are <a href="https://beam.apache.org/documentation/sdks/go/">here</a>.</p>
+<p>The Go SDK has been officially accepted into the project, after an incubation period and community effort. Go pipelines run on Dataflow runner. More details are <a href="/documentation/sdks/go/">here</a>.</p>
<h2 id="parquet-support">Parquet support</h2>
<p>Support for Apache Parquet format was added. It uses Parquet 1.10 release which, thanks to AvroParquerWriter’s API changes, allows FileIO.Sink implementation.</p>
diff --git a/website/generated-content/blog/2018/08/20/review-input-streaming-connectors.html b/website/generated-content/blog/2018/08/20/review-input-streaming-connectors.html
index 31384ac..58ed34c 100644
--- a/website/generated-content/blog/2018/08/20/review-input-streaming-connectors.html
+++ b/website/generated-content/blog/2018/08/20/review-input-streaming-connectors.html
@@ -190,7 +190,7 @@ limitations under the License.
</td>
<td>Local<br />(Using the <code>file://</code> URI)
</td>
- <td><a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/TextIO.html">TextIO</a>
+ <td><a href="https://beam.apache.org/releases/javadoc/2.7.0/org/apache/beam/sdk/io/TextIO.html">TextIO</a>
</td>
<td><a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/StreamingContext.html#textFileStream-java.lang.String-">textFileStream</a><br />(Spark treats most Unix systems as HDFS-compatible, but the location should be accessible from all nodes)
</td>
@@ -198,7 +198,7 @@ limitations under the License.
<tr>
<td>HDFS<br />(Using the <code>hdfs://</code> URI)
</td>
- <td><a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/FileIO.html">FileIO</a> + <a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptions.html">HadoopFileSystemOptions</a>
+ <td><a href="https://beam.apache.org/releases/javadoc/2.7.0/org/apache/beam/sdk/io/FileIO.html">FileIO</a> + <a href="https://beam.apache.org/releases/javadoc/2.7.0/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptions.html">HadoopFileSystemOptions</a>
</td>
<td><a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/util/HdfsUtils.html">HdfsUtils</a>
</td>
@@ -208,7 +208,7 @@ limitations under the License.
</td>
<td>Cloud Storage<br />(Using the <code>gs://</code> URI)
</td>
- <td><a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/FileIO.html">FileIO</a> + <a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/extensions/gcp/options/GcsOptions.html">GcsOptions</a>
+ <td><a href="https://beam.apache.org/releases/javadoc/2.7.0/org/apache/beam/sdk/io/FileIO.html">FileIO</a> + <a href="https://beam.apache.org/releases/javadoc/2.7.0/org/apache/beam/sdk/extensions/gcp/options/GcsOptions.html">GcsOptions</a>
</td>
<td rowspan="2"><a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/SparkContext.html#hadoopConfiguration--">hadoopConfiguration</a>
and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/StreamingContext.html#textFileStream-java.lang.String-">textFileStream</a>
@@ -217,7 +217,7 @@ and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/stre
<tr>
<td>S3<br />(Using the <code>s3://</code> URI)
</td>
- <td><a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/FileIO.html">FileIO</a> + <a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/aws/options/S3Options.html">S3Options</a>
+ <td><a href="https://beam.apache.org/releases/javadoc/2.7.0/org/apache/beam/sdk/io/FileIO.html">FileIO</a> + <a href="https://beam.apache.org/releases/javadoc/2.7.0/org/apache/beam/sdk/io/aws/options/S3Options.html">S3Options</a>
</td>
</tr>
<tr>
@@ -225,7 +225,7 @@ and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/stre
</td>
<td>Kafka
</td>
- <td><a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/kafka/KafkaIO.html">KafkaIO</a>
+ <td><a href="https://beam.apache.org/releases/javadoc/2.7.0/org/apache/beam/sdk/io/kafka/KafkaIO.html">KafkaIO</a>
</td>
<td><a href="https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html">spark-streaming-kafka</a>
</td>
@@ -233,7 +233,7 @@ and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/stre
<tr>
<td>Kinesis
</td>
- <td><a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/kinesis/KinesisIO.html">KinesisIO</a>
+ <td><a href="https://beam.apache.org/releases/javadoc/2.7.0/org/apache/beam/sdk/io/kinesis/KinesisIO.html">KinesisIO</a>
</td>
<td><a href="https://spark.apache.org/docs/latest/streaming-kinesis-integration.html">spark-streaming-kinesis</a>
</td>
@@ -241,7 +241,7 @@ and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/stre
<tr>
<td>Cloud Pub/Sub
</td>
- <td><a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.html">PubsubIO</a>
+ <td><a href="https://beam.apache.org/releases/javadoc/2.7.0/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.html">PubsubIO</a>
</td>
<td><a href="https://github.com/apache/bahir/tree/master/streaming-pubsub">spark-streaming-pubsub</a> from <a href="http://bahir.apache.org">Apache Bahir</a>
</td>
@@ -282,7 +282,7 @@ and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/stre
</td>
<td>Local
</td>
- <td><a href="https://beam.apache.org/releases/pydoc/2.6.0/apache_beam.io.textio.html">io.textio</a>
+ <td><a href="https://beam.apache.org/releases/pydoc/2.7.0/apache_beam.io.textio.html">io.textio</a>
</td>
<td><a href="http://spark.apache.org/docs/latest/api/python/pyspark.streaming.html#pyspark.streaming.StreamingContext.textFileStream">textFileStream</a>
</td>
@@ -290,7 +290,7 @@ and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/stre
<tr>
<td>HDFS
</td>
- <td><a href="https://beam.apache.org/releases/pydoc/2.6.0/apache_beam.io.hadoopfilesystem.html">io.hadoopfilesystem</a>
+ <td><a href="https://beam.apache.org/releases/pydoc/2.7.0/apache_beam.io.hadoopfilesystem.html">io.hadoopfilesystem</a>
</td>
<td><a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/SparkContext.html#hadoopConfiguration--">hadoopConfiguration</a> (Access through <code>sc._jsc</code> with Py4J)
and <a href="http://spark.apache.org/docs/latest/api/python/pyspark.streaming.html#pyspark.streaming.StreamingContext.textFileStream">textFileStream</a>
@@ -301,7 +301,7 @@ and <a href="http://spark.apache.org/docs/latest/api/python/pyspark.streaming.ht
</td>
<td>Google Cloud Storage
</td>
- <td><a href="https://beam.apache.org/releases/pydoc/2.6.0/apache_beam.io.gcp.gcsio.html">io.gcp.gcsio</a>
+ <td><a href="https://beam.apache.org/releases/pydoc/2.7.0/apache_beam.io.gcp.gcsio.html">io.gcp.gcsio</a>
</td>
<td rowspan="2"><a href="http://spark.apache.org/docs/latest/api/python/pyspark.streaming.html#pyspark.streaming.StreamingContext.textFileStream">textFileStream</a>
</td>
@@ -333,7 +333,7 @@ and <a href="http://spark.apache.org/docs/latest/api/python/pyspark.streaming.ht
<tr>
<td>Cloud Pub/Sub
</td>
- <td><a href="https://beam.apache.org/releases/pydoc/2.6.0/apache_beam.io.gcp.pubsub.html">io.gcp.pubsub</a>
+ <td><a href="https://beam.apache.org/releases/pydoc/2.7.0/apache_beam.io.gcp.pubsub.html">io.gcp.pubsub</a>
</td>
<td>N/A
</td>
diff --git a/website/generated-content/blog/2018/06/26/beam-2.5.0.html b/website/generated-content/blog/2018/10/03/beam-2.7.0.html
similarity index 65%
copy from website/generated-content/blog/2018/06/26/beam-2.5.0.html
copy to website/generated-content/blog/2018/10/03/beam-2.7.0.html
index e054124..9476ed0 100644
--- a/website/generated-content/blog/2018/06/26/beam-2.5.0.html
+++ b/website/generated-content/blog/2018/10/03/beam-2.7.0.html
@@ -28,8 +28,8 @@
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
- <title>Apache Beam 2.5.0</title>
- <meta name="description" content="We are glad to present the new 2.5.0 release of Beam. This release includesmultiple fixes and new functionalities.">
+ <title>Apache Beam 2.7.0</title>
+ <meta name="description" content="We are happy to present the new 2.7.0 release of Beam. This release includes both improvements and new functionality.See the download page for this release.">
<link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
<link rel="stylesheet" href="/css/site.css">
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
@@ -38,7 +38,7 @@
<script src="/js/fix-menu.js"></script>
<script src="/js/section-nav.js"></script>
<script src="/js/page-nav.js"></script>
- <link rel="canonical" href="https://beam.apache.org/blog/2018/06/26/beam-2.5.0.html" data-proofer-ignore>
+ <link rel="canonical" href="https://beam.apache.org/blog/2018/10/03/beam-2.7.0.html" data-proofer-ignore>
<link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
<link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
<script>
@@ -136,9 +136,9 @@
<article class="post" itemscope itemtype="http://schema.org/BlogPosting">
<header class="post-header">
- <h1 class="post-title" itemprop="name headline">Apache Beam 2.5.0</h1>
- <p class="post-meta"><time datetime="2018-06-26T01:00:01-07:00" itemprop="datePublished">Jun 26, 2018</time> •
- Alexey Romanenko [<a href="https://twitter.com/alexromdev">@alexromdev</a>]
+ <h1 class="post-title" itemprop="name headline">Apache Beam 2.7.0</h1>
+ <p class="post-meta"><time datetime="2018-10-03T01:00:01-07:00" itemprop="datePublished">Oct 3, 2018</time> •
+ Charles Chen
</p>
</header>
@@ -158,95 +158,66 @@ See the License for the specific language governing permissions and
limitations under the License.
-->
-<p>We are glad to present the new 2.5.0 release of Beam. This release includes
-multiple fixes and new functionalities. <!--more--> For more information
-please check the detailed release notes.</p>
+<p>We are happy to present the new 2.7.0 release of Beam. This release includes both improvements and new functionality.
+See the <a href="/get-started/downloads/#270-2018-10-02">download page</a> for this release.<!--more-->
+For more information on changes in 2.7.0, check out the
+<a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654">detailed release notes</a>.</p>
-<h1 id="new-features--improvements">New Features / Improvements</h1>
+<h2 id="new-features--improvements">New Features / Improvements</h2>
-<h2 id="go-sdk-support">Go SDK support</h2>
-<p>The Go SDK has been officially accepted into the project, after an incubation period and community effort. Go pipelines run on Dataflow runner. More details are <a href="https://beam.apache.org/documentation/sdks/go/">here</a>.</p>
+<h3 id="new-ios">New I/Os</h3>
-<h2 id="parquet-support">Parquet support</h2>
-<p>Support for Apache Parquet format was added. It uses Parquet 1.10 release which, thanks to AvroParquerWriter’s API changes, allows FileIO.Sink implementation.</p>
-
-<h2 id="performanceintegration-tests">Performance/Integration Tests</h2>
<ul>
- <li>Added new integration tests - HCatalogIOIT (Hive), HBaseIOIT, ParquetIOIT (with the IO itself, local filesystem, HDFS)</li>
- <li>Multinode (3 data node) HDFS cluster is used for running tests on HDFS.</li>
- <li>Several improvements on performance tests running and results analysis.</li>
- <li>Scaled up Kubernetes cluster from 1 to 3 nodes.</li>
- <li>Added metrics in Spark streaming.</li>
+ <li>KuduIO</li>
+ <li>Amazon SNS sink</li>
+ <li>Amazon SqsIO</li>
</ul>
-<h2 id="internal-build-system-migrated-to-gradle">Internal Build System: Migrated to Gradle</h2>
-<p>After a months-long community effort, the internal Beam build has been migrated from Maven to Gradle. The new build system was chosen because of dependency-driven build support, incremental build/test, and support for non-Java languages.</p>
+<h3 id="dependency-upgrades">Dependency Upgrades</h3>
-<h2 id="nexmark-improvements">Nexmark Improvements</h2>
<ul>
- <li>Kafka support as a source/sink for events and results.</li>
- <li>Translation of some queries to Beam SQL.</li>
+ <li>Apache Calcite dependency upgraded to 1.17.0</li>
+ <li>Apache Derby dependency upgraded to 10.14.2.0</li>
+ <li>Apache HTTP components upgraded (see release notes).</li>
</ul>
-<h2 id="beam-sql">Beam SQL</h2>
-<ul>
- <li>Support for MAP, ROW, ARRAY data types</li>
- <li>Support UNNEST on array fields</li>
- <li>Improved optimizations</li>
- <li>Upgrade Calcite to 1.16</li>
- <li>Support SQL on POJOs via automatic conversion</li>
- <li>Schema moved into core Beam</li>
- <li>UDAFs can be indirect suclasses of CombineFn</li>
- <li>Many other small bugfixes</li>
-</ul>
+<h3 id="portability">Portability</h3>
-<h2 id="portability">Portability</h2>
<ul>
- <li>Common shared code related to supporting portable execution for runners.</li>
- <li>Python SDK supporting side inputs over the portability APIs.</li>
+ <li>Experimental support for Python on local Flink runner for simple
+examples, see latest information here:
+/contribute/portability/#status.</li>
</ul>
-<h2 id="extract-metrics-in-a-runner-agnostic-way">Extract metrics in a runner agnostic way</h2>
-<p>Metrics are pushed by the runners to configurable sinks (Http REST sink available). It is already enabled in Flink and Spark runner, work is in progress for Dataflow.</p>
+<h2 id="miscellaneous-fixes">Miscellaneous Fixes</h2>
-<h1 id="miscellaneous-fixes">Miscellaneous Fixes</h1>
-
-<h2 id="sdks">SDKs</h2>
+<h3 id="ios">I/Os</h3>
<ul>
- <li>Implemented HDFS FileSystem for Python SDK.</li>
- <li>Python SDK adds support for side inputs for streaming execution.</li>
+ <li>KinesisIO, fixed dependency issue</li>
</ul>
-<h2 id="runners">Runners</h2>
-<ul>
- <li>Updated Spark runner to Spark version 2.3.1</li>
- <li>Fixed issue with late elements windowed into expired fixed windows get dropped in Directrunner.</li>
-</ul>
-
-<h2 id="ios">IOs</h2>
-
-<ul>
- <li>CassandraIO gained a better split algorithm based on overlapping regions.</li>
- <li>ElasticsearchIO supports partial updates.</li>
- <li>ElasticsearchIO allows to pass id, type and index per document.</li>
- <li>SolrIO supports a more robust retry on write strategy.</li>
- <li>S3 FileSystem supports encryption (SSE-S3, SSE-C and SSE-KMS).</li>
- <li>Improved connection management in JdbcIO.</li>
- <li>Added support the element timestamps while publishing to Kafka.</li>
-</ul>
-
-<h2 id="other">Other</h2>
-
-<ul>
- <li>Use Java ErrorProne for static analysis.</li>
-</ul>
-
-<h1 id="list-of-contributors">List of Contributors</h1>
-
-<p>According to git shortlog, the following 84 people contributed to the 2.5.0 release. Thank you to all contributors!</p>
-
-<p>Ahmet Altay, Alan Myrvold, Alex Amato, Alex Van Boxel, Alexander Dejanovski, Alexey Romanenko, Aljoscha Krettek, ananvay, Andreas Ehrencrona, Andrew Pilloud, Ankur Goenka, Anton Kedin, arkash, Austin Bennett, Axel Magnuson, Ben Chambers, Ben Sidhom, Bill Neubauer, Boyuan Zhang, Braden Bassingthwaite, Cade Markegard, cclauss, Chamikara Jayalath, Charles Chen, Chuan Yu Foo, Cody Schroeder, Colm O hEigeartaigh, Daniel Oliveira, Dariusz Aniszewski, David Cavazos, Dawid Wysakowicz, Eric Ro [...]
+<h2 id="list-of-contributors">List of Contributors</h2>
+
+<p>According to git shortlog, the following 72 people contributed
+to the 2.7.0 release. Thank you to all contributors!</p>
+
+<p>Ahmet Altay, Alan Myrvold, Alexey Romanenko, Aljoscha Krettek,
+Andrew Pilloud, Ankit Jhalaria, Ankur Goenka, Anton Kedin, Boyuan
+Zhang, Carl McGraw, Carlos Alonso, cclauss, Chamikara Jayalath,
+Charles Chen, Cory Brzycki, Daniel Oliveira, Dariusz Aniszewski,
+devinduan, Eric Beach, Etienne Chauchot, Eugene Kirpichov, Garrett
+Jones, Gene Peters, Gleb Kanterov, Henning Rohde, Henry Suryawirawan,
+Holden Karau, Huygaa Batsaikhan, Ismaël Mejía, Jason Kuster, Jean-
+Baptiste Onofré, Joachim van der Herten, Jozef Vilcek, jxlewis, Kai
+Jiang, Katarzyna Kucharczyk, Kenn Knowles, Krzysztof Trubalski, Kyle
+Winkelman, Leen Toelen, Luis Enrique Ortíz Ramirez, Lukasz Cwik,
+Łukasz Gajowy, Luke Cwik, Mark Liu, Matthias Feys, Maximilian Michels,
+Melissa Pashniak, Mikhail Gryzykhin, Mikhail Sokolov, mingmxu, Norbert
+Chen, Pablo Estrada, Prateek Chanda, Raghu Angadi, Ravi Pathak, Reuven
+Lax, Robert Bradshaw, Robert Burke, Rui Wang, Ryan Williams, Sindy Li,
+Thomas Weise, Tim Robertson, Tormod Haavi, Udi Meiri, Vaclav Plajt,
+Valentyn Tymofieiev, xiliu, XuMingmin, Yifan Zou, Yueyang Qiu.</p>
</div>
diff --git a/website/generated-content/blog/index.html b/website/generated-content/blog/index.html
index f3c8774..24639b1 100644
--- a/website/generated-content/blog/index.html
+++ b/website/generated-content/blog/index.html
@@ -139,6 +139,38 @@ limitations under the License.
<p>This is the blog for the Apache Beam project. This blog contains news and updates
for the project.</p>
+<h3 id="apache-beam-270"><a class="post-link" href="/blog/2018/10/03/beam-2.7.0.html">Apache Beam 2.7.0</a></h3>
+<p><i>Oct 3, 2018 •
+ Charles Chen
+</i></p>
+
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+<p>We are happy to present the new 2.7.0 release of Beam. This release includes both improvements and new functionality.
+See the <a href="/get-started/downloads/#270-2018-10-02">download page</a> for this release.</p>
+
+<!-- Render a "read more" button if the post is longer than the excerpt -->
+
+<p>
+<a class="btn btn-default btn-sm" href="/blog/2018/10/03/beam-2.7.0.html" role="button">
+Read more <span class="glyphicon glyphicon-menu-right" aria-hidden="true"></span>
+</a>
+</p>
+
+<hr />
+
<h3 id="beam-summit-europe-2018"><a class="post-link" href="/blog/2018/08/21/beam-summit-europe.html">Beam Summit Europe 2018</a></h3>
<p><i>Aug 21, 2018 •
Matthias Baetens [<a href="https://twitter.com/matthiasbaetens">@matthiasbaetens</a>]
diff --git a/website/generated-content/contribute/dependencies/index.html b/website/generated-content/contribute/dependencies/index.html
index a1f750c..9239370 100644
--- a/website/generated-content/contribute/dependencies/index.html
+++ b/website/generated-content/contribute/dependencies/index.html
@@ -200,7 +200,7 @@
<ul class="nav">
<li><a href="#identifying-outdated-dependencies">Identifying outdated dependencies</a></li>
- <li><a href="#jira-automation">JIRA Automation</a></li>
+ <li><a href="#jira-issue-automation">JIRA Issue Automation</a></li>
<li><a href="#upgrading-identified-outdated-dependencies">Upgrading identified outdated dependencies</a></li>
<li><a href="#dependency-updates-and-backwards-compatibility">Dependency updates and backwards compatibility</a></li>
</ul>
@@ -259,17 +259,21 @@ limitations under the License.
<p>These kind of urgently required upgrades might not get automatically picked up by the Jenkins job for few months. So Beam community has to act to identify such issues and perform upgrades early.</p>
-<h2 id="jira-automation">JIRA Automation</h2>
+<h2 id="jira-issue-automation">JIRA Issue Automation</h2>
<p>In order to track the dependency upgrade process, JIRA tickets will be created per significant outdated dependency based on the report. A bot named <em>Beam Jira Bot</em> was created for managing JIRA issues. Beam community agrees on the following policies that creates and updates issues.</p>
<ul>
- <li>Issues will be named as “Beam Dependency Update Request: <dep_name> <dep_newest_version>".</dep_newest_version></dep_name></li>
- <li>Issues will be created under the component <em>“dependencies”</em></li>
- <li>Issues will be assigned to the primary owner of the dependencies, who are mentioned in the dependency ownership files. (<a href="https://github.com/apache/beam/blob/master/ownership/JAVA_DEPENDENCY_OWNERS.yaml">Java Dependency Owners</a> and <a href="https://github.com/apache/beam/blob/master/ownership/PYTHON_DEPENDENCY_OWNERS.yaml">Python Dependency Owners</a>)</li>
- <li>If more than one owners found for a dependency, the first owner will be picked as the primary owner, the others will be pinged in the issue’s description.</li>
- <li>If no owners found, leave the assignee empty. The component lead is responsible for triaging the issue.</li>
- <li>Avoid creating duplicate issues. Updating the descriptions of the open issues created by the previous dependency check.</li>
- <li>The dependency sometimes is not able to be upgraded, the issue should be closed as <em>“won’t fix”</em>. And, the bot should avoid recreating issues with “won’t fix”.</li>
+ <li>Title (summary) of the issues will be in the format “Beam Dependency Update Request: <dep_name>" where <dep_name> is the dependency artifact name.</dep_name></dep_name></li>
+ <li>Issues will be created under the component <em>“dependencies”</em>.</li>
+ <li>Owners of dependencies will be notified by tagging the corresponding JIRA IDs mentioned in the ownership files in the issue description. See <a href="https://github.com/apache/beam/blob/master/ownership/JAVA_DEPENDENCY_OWNERS.yaml">Java Dependency Owners</a> and <a href="https://github.com/apache/beam/blob/master/ownership/PYTHON_DEPENDENCY_OWNERS.yaml">Python Dependency Owners</a> for current owners for Java SDK and Python SDK dependencies respectively.</li>
+ <li>Automated tool will not create duplicate issues for the same dependency. Instead the tool will look for an existing JIRA when one has to be created for a given dependency and description of the JIRA will be updated with latest information, for example, current version of the dependency.</li>
+ <li>If a Beam community member determines that a given dependency should not be upgraded the corresponding JIRA issue can be closed with a fix version specified.</li>
+ <li>Automated tool will reopen a JIRA for a given dependency when one of following conditions is met:
+ <ul>
+ <li>Next SDK release is for a fix version mentioned in the JIRA.</li>
+ <li>Six months <strong>and</strong> three or more minor releases have passed since the JIRA was closed.</li>
+ </ul>
+ </li>
</ul>
<h2 id="upgrading-identified-outdated-dependencies">Upgrading identified outdated dependencies</h2>
@@ -300,7 +304,7 @@ limitations under the License.
<h2 id="dependency-updates-and-backwards-compatibility">Dependency updates and backwards compatibility</h2>
-<p>Beam releases <a href="https://beam.apache.org/get-started/downloads/">adhere to</a> semantic versioning. Hence, community members should take care when updating dependencies. Minor version updates to dependencies should be backwards compatible in most cases. Some updates to dependencies though may result in backwards incompatible API or functionality changes to Beam. PR reviewers and committers should take care to detect any dependency updates that could potentially introduce backwar [...]
+<p>Beam releases <a href="/get-started/downloads/">adhere to</a> semantic versioning. Hence, community members should take care when updating dependencies. Minor version updates to dependencies should be backwards compatible in most cases. Some updates to dependencies though may result in backwards incompatible API or functionality changes to Beam. PR reviewers and committers should take care to detect any dependency updates that could potentially introduce backwards incompatible changes [...]
</div>
</div>
diff --git a/website/generated-content/contribute/index.html b/website/generated-content/contribute/index.html
index 094b4c0..6259dea 100644
--- a/website/generated-content/contribute/index.html
+++ b/website/generated-content/contribute/index.html
@@ -631,7 +631,7 @@ When submitting a new PR, please tag <a href="https://github.com/robbesneyders">
<li>improving existing Performance Testing Framework and it’s documentation</li>
</ul>
-<p>See the <a href="https://beam.apache.org/documentation/io/testing/#i-o-transform-integration-tests">documentation</a> and the <a href="https://docs.google.com/document/d/1dA-5s6OHiP_cz-NRAbwapoKF5MEC1wKps4A5tFbIPKE/edit?usp=sharing">initial proposal</a>(for file based tests).</p>
+<p>See the <a href="/documentation/io/testing/#i-o-transform-integration-tests">documentation</a> and the <a href="https://docs.google.com/document/d/1dA-5s6OHiP_cz-NRAbwapoKF5MEC1wKps4A5tFbIPKE/edit?usp=sharing">initial proposal</a>(for file based tests).</p>
<p>If you’re willing to help in this area, tag the following people in PRs: <a href="https://github.com/chamikaramj">@chamikaramj</a>, <a href="https://github.com/dariuszaniszewski">@DariuszAniszewski</a>, <a href="https://github.com/lgajowy">@lgajowy</a>, <a href="https://github.com/szewi">@szewi</a>, <a href="https://github.com/kkucharc">@kkucharc</a></p>
diff --git a/website/generated-content/contribute/postcommits-policies/index.html b/website/generated-content/contribute/postcommits-policies/index.html
index af94c5b..90bb9e3 100644
--- a/website/generated-content/contribute/postcommits-policies/index.html
+++ b/website/generated-content/contribute/postcommits-policies/index.html
@@ -261,8 +261,7 @@ community follows these post-commit test policies:</p>
<h3 id="found-failing-test">I found a test failure</h3>
<ol>
- <li>Create a <a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20component%20%3D%20test-failures">JIRA issue</a>
-and assign it to yourself.</li>
+ <li>Create a <a href="https://s.apache.org/beam-test-failure">JIRA issue</a> and assign it to yourself.</li>
<li>Do high level triage of the failure.</li>
<li><a href="/contribute/postcommits-guides/index.html#find_specialist">Assign the JIRA issue to a relevant person</a>.</li>
</ol>
diff --git a/website/generated-content/contribute/ptransform-style-guide/index.html b/website/generated-content/contribute/ptransform-style-guide/index.html
index 0474d13..7359f26 100644
--- a/website/generated-content/contribute/ptransform-style-guide/index.html
+++ b/website/generated-content/contribute/ptransform-style-guide/index.html
@@ -528,8 +528,8 @@ E.g. when expanding a filepattern into files, log what the filepattern was and h
<ul>
<li>Generally, follow the rules of <a href="http://semver.org/">semantic versioning</a>.</li>
- <li>If the API of the transform is not yet stable, annotate it as <code class="highlighter-rouge">@Experimental</code> (Java) or <code class="highlighter-rouge">@experimental</code> (<a href="https://beam.apache.org/releases/pydoc/2.6.0/apache_beam.utils.annotations.html">Python</a>).</li>
- <li>If the API deprecated, annotate it as <code class="highlighter-rouge">@Deprecated</code> (Java) or <code class="highlighter-rouge">@deprecated</code> (<a href="https://beam.apache.org/releases/pydoc/2.6.0/apache_beam.utils.annotations.html">Python</a>).</li>
+ <li>If the API of the transform is not yet stable, annotate it as <code class="highlighter-rouge">@Experimental</code> (Java) or <code class="highlighter-rouge">@experimental</code> (<a href="https://beam.apache.org/releases/pydoc/2.7.0/apache_beam.utils.annotations.html">Python</a>).</li>
+ <li>If the API deprecated, annotate it as <code class="highlighter-rouge">@Deprecated</code> (Java) or <code class="highlighter-rouge">@deprecated</code> (<a href="https://beam.apache.org/releases/pydoc/2.7.0/apache_beam.utils.annotations.html">Python</a>).</li>
<li>Pay attention to the stability and versioning of third-party classes exposed by the transform’s API: if they are unstable or improperly versioned (do not obey <a href="http://semver.org/">semantic versioning</a>), it is better to wrap them in your own classes.</li>
</ul>
diff --git a/website/generated-content/contribute/release-guide/index.html b/website/generated-content/contribute/release-guide/index.html
index 965fdb4..d291075 100644
--- a/website/generated-content/contribute/release-guide/index.html
+++ b/website/generated-content/contribute/release-guide/index.html
@@ -585,8 +585,9 @@ Automation Reliability</a>.</p>
<p><strong>Attention</strong>: Only PMC has permission to perform this. If you are not a PMC, please ask for help in dev@ mailing list.</p>
<ol>
- <li>In JIRA, navigate to the <a href="https://issues.apache.org/jira/plugins/servlet/project-config/BEAM/versions"><code class="highlighter-rouge">Beam > Administration > Versions</code></a>.</li>
- <li>Add a new release: choose the next minor version number compared to the one currently underway, select today’s date as the <code class="highlighter-rouge">Start Date</code>, and choose <code class="highlighter-rouge">Add</code>.</li>
+ <li>In JIRA, navigate to <a href="https://issues.apache.org/jira/plugins/servlet/project-config/BEAM/versions"><code class="highlighter-rouge">Beam > Administration > Versions</code></a>.</li>
+ <li>Add a new release. Choose the next minor version number after the version currently underway, select the release cut date (today’s date) as the <code class="highlighter-rouge">Start Date</code>, and choose <code class="highlighter-rouge">Add</code>.</li>
+ <li>At the end of the release, go to the same page and mark the recently released version as released. Use the <code class="highlighter-rouge">...</code> menu and choose <code class="highlighter-rouge">Release</code>.</li>
</ol>
<h3 id="triage-release-blocking-issues-in-jira">Triage release-blocking issues in JIRA</h3>
diff --git a/website/generated-content/contribute/runner-guide/index.html b/website/generated-content/contribute/runner-guide/index.html
index 03f55e3..86eb847 100644
--- a/website/generated-content/contribute/runner-guide/index.html
+++ b/website/generated-content/contribute/runner-guide/index.html
@@ -911,7 +911,7 @@ collection of log files, or a database table. The capabilities are:</p>
<li><code class="highlighter-rouge">createReader(...)</code> - call this to start reading elements; it is an enhanced iterator, with also:</li>
<li>timestamps to associate with each element read</li>
<li><code class="highlighter-rouge">splitAtFraction</code> for dynamic splitting to enable work stealing, and other
-methods to support it - see the <a href="https://beam.apache.org/blog/2016/05/18/splitAtFraction-method.html">Beam blog post on dynamic work
+methods to support it - see the <a href="/blog/2016/05/18/splitAtFraction-method.html">Beam blog post on dynamic work
rebalancing</a></li>
</ul>
diff --git a/website/generated-content/contribute/testing/index.html b/website/generated-content/contribute/testing/index.html
index c3761cf..8ce9d05 100644
--- a/website/generated-content/contribute/testing/index.html
+++ b/website/generated-content/contribute/testing/index.html
@@ -336,13 +336,13 @@ desired configurations.</p>
<p>NeedsRunner is a category of tests that require a Beam runner. To run
NeedsRunner tests:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>$ ./gradlew :runners:direct-java:needsRunnerTests
+<div class="highlighter-rouge"><pre class="highlight"><code>$ ./gradlew beam-runners-direct-java:needsRunnerTests
</code></pre>
</div>
<p>To run a single NeedsRunner test use the <code class="highlighter-rouge">test</code> property, e.g.</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>$ ./gradlew :runners:direct-java:needsRunnerTests --tests org.apache.beam.sdk.transforms.MapElementsTest.testMapBasic
+<div class="highlighter-rouge"><pre class="highlight"><code>$ ./gradlew beam-runners-direct-java:needsRunnerTests --tests org.apache.beam.sdk.transforms.MapElementsTest.testMapBasic
</code></pre>
</div>
@@ -352,7 +352,7 @@ NeedsRunner tests:</p>
<code class="highlighter-rouge">sdks/java/io/google-cloud-platform</code>) can be executed with the <code class="highlighter-rouge">gradle test</code>
command:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>$ ./gradlew sdks:java:io:google-cloud-platform:test --tests org.apache.beam.sdk.io.gcp.spanner.SpannerIOWriteTest
+<div class="highlighter-rouge"><pre class="highlight"><code>$ ./gradlew beam-sdks-java-io-google-cloud-platform:test --tests org.apache.beam.sdk.io.gcp.spanner.SpannerIOWriteTest
</code></pre>
</div>
diff --git a/website/generated-content/documentation/dsls/sql/aggregate-functions/index.html b/website/generated-content/documentation/dsls/sql/aggregate-functions/index.html
index 00b3644..ce4480d 100644
--- a/website/generated-content/documentation/dsls/sql/aggregate-functions/index.html
+++ b/website/generated-content/documentation/dsls/sql/aggregate-functions/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.7.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.7.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -190,7 +190,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/data-types/">Data types</a></li>
<li><a href="/documentation/dsls/sql/lexical/">Lexical structure</a></li>
- <li><a href="/documentation/dsls/sql/create-table/">CREATE TABLE</a></li>
+ <li><a href="/documentation/dsls/sql/create-external-table/">CREATE EXTERNAL TABLE</a></li>
<li><a href="/documentation/dsls/sql/select/">SELECT</a></li>
<li><a href="/documentation/dsls/sql/windowing-and-triggering/">Windowing & Triggering</a></li>
<li><a href="/documentation/dsls/sql/joins/">Joins</a></li>
diff --git a/website/generated-content/documentation/dsls/sql/create-table/index.html b/website/generated-content/documentation/dsls/sql/create-external-table/index.html
similarity index 94%
copy from website/generated-content/documentation/dsls/sql/create-table/index.html
copy to website/generated-content/documentation/dsls/sql/create-external-table/index.html
index df62158..1d143e8 100644
--- a/website/generated-content/documentation/dsls/sql/create-table/index.html
+++ b/website/generated-content/documentation/dsls/sql/create-external-table/index.html
@@ -28,7 +28,7 @@
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
- <title>Beam SQL: CREATE TABLE Statement</title>
+ <title>Beam SQL: CREATE EXTERNAL TABLE Statement</title>
<meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow [...]
">
<link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
@@ -39,7 +39,7 @@
<script src="/js/fix-menu.js"></script>
<script src="/js/section-nav.js"></script>
<script src="/js/page-nav.js"></script>
- <link rel="canonical" href="https://beam.apache.org/documentation/dsls/sql/create-table/" data-proofer-ignore>
+ <link rel="canonical" href="https://beam.apache.org/documentation/dsls/sql/create-external-table/" data-proofer-ignore>
<link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
<link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
<script>
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.7.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.7.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -190,7 +190,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/data-types/">Data types</a></li>
<li><a href="/documentation/dsls/sql/lexical/">Lexical structure</a></li>
- <li><a href="/documentation/dsls/sql/create-table/">CREATE TABLE</a></li>
+ <li><a href="/documentation/dsls/sql/create-external-table/">CREATE EXTERNAL TABLE</a></li>
<li><a href="/documentation/dsls/sql/select/">SELECT</a></li>
<li><a href="/documentation/dsls/sql/windowing-and-triggering/">Windowing & Triggering</a></li>
<li><a href="/documentation/dsls/sql/joins/">Joins</a></li>
@@ -282,19 +282,19 @@ See the License for the specific language governing permissions and
limitations under the License.
-->
-<h1 id="create-table">CREATE TABLE</h1>
+<h1 id="create-external-table">CREATE EXTERNAL TABLE</h1>
-<p>Beam SQL’s <code class="highlighter-rouge">CREATE TABLE</code> statement registers a virtual table that maps to an
-<a href="https://beam.apache.org/documentation/io/built-in/">external storage system</a>.
-For some storage systems, <code class="highlighter-rouge">CREATE TABLE</code> does not create a physical table until
+<p>Beam SQL’s <code class="highlighter-rouge">CREATE EXTERNAL TABLE</code> statement registers a virtual table that maps to an
+<a href="/documentation/io/built-in/">external storage system</a>.
+For some storage systems, <code class="highlighter-rouge">CREATE EXTERNAL TABLE</code> does not create a physical table until
a write occurs. After the physical table exists, you can access the table with
the <code class="highlighter-rouge">SELECT</code>, <code class="highlighter-rouge">JOIN</code>, and <code class="highlighter-rouge">INSERT INTO</code> statements.</p>
-<p>The <code class="highlighter-rouge">CREATE TABLE</code> statement includes a schema and extended clauses.</p>
+<p>The <code class="highlighter-rouge">CREATE EXTERNAL TABLE</code> statement includes a schema and extended clauses.</p>
<h2 id="syntax">Syntax</h2>
-<div class="highlighter-rouge"><pre class="highlight"><code>CREATE TABLE [ IF NOT EXISTS ] tableName (tableElement [, tableElement ]*)
+<div class="highlighter-rouge"><pre class="highlight"><code>CREATE EXTERNAL TABLE [ IF NOT EXISTS ] tableName (tableElement [, tableElement ]*)
TYPE type
[LOCATION location]
[TBLPROPERTIES tblProperties]
@@ -312,7 +312,7 @@ tableElement: columnName fieldType [ NOT NULL ]
ignores the statement instead of returning an error.</li>
<li><code class="highlighter-rouge">tableName</code>: The case sensitive name of the table to create and register,
specified as an
-<a href="https://beam.apache.org/documentation/dsls/sql/lexical/#identifiers">Identifier</a>.
+<a href="/documentation/dsls/sql/lexical/#identifiers">Identifier</a>.
The table name does not need to match the name in the underlying data
storage system.</li>
<li><code class="highlighter-rouge">tableElement</code>: <code class="highlighter-rouge">columnName</code> <code class="highlighter-rouge">fieldType</code> <code class="highlighter-rouge">[ NOT NULL ]</code>
@@ -333,7 +333,7 @@ backtick_quoted_expression.</li>
</ul>
</li>
<li><code class="highlighter-rouge">type</code>: The I/O transform that backs the virtual table, specified as an
-<a href="https://beam.apache.org/documentation/dsls/sql/lexical/#identifiers">Identifier</a>
+<a href="/documentation/dsls/sql/lexical/#identifiers">Identifier</a>
with one of the following values:
<ul>
<li><code class="highlighter-rouge">bigquery</code></li>
@@ -343,11 +343,11 @@ with one of the following values:
</ul>
</li>
<li><code class="highlighter-rouge">location</code>: The I/O specific location of the underlying table, specified as
-a <a href="https://beam.apache.org/documentation/dsls/sql/lexical/#string-literals">String
+a <a href="/documentation/dsls/sql/lexical/#string-literals">String
Literal</a>.
See the I/O specific sections for <code class="highlighter-rouge">location</code> format requirements.</li>
<li><code class="highlighter-rouge">tblProperties</code>: The I/O specific quoted key value JSON object with extra
-configuration, specified as a <a href="https://beam.apache.org/documentation/dsls/sql/lexical/#string-literals">String
+configuration, specified as a <a href="/documentation/dsls/sql/lexical/#string-literals">String
Literal</a>.
See the I/O specific sections for <code class="highlighter-rouge">tblProperties</code> format requirements.</li>
</ul>
@@ -356,7 +356,7 @@ See the I/O specific sections for <code class="highlighter-rouge">tblProperties<
<h3 id="syntax-1">Syntax</h3>
-<div class="highlighter-rouge"><pre class="highlight"><code>CREATE TABLE [ IF NOT EXISTS ] tableName (tableElement [, tableElement ]*)
+<div class="highlighter-rouge"><pre class="highlight"><code>CREATE EXTERNAL TABLE [ IF NOT EXISTS ] tableName (tableElement [, tableElement ]*)
TYPE bigquery
LOCATION '[PROJECT_ID]:[DATASET].[TABLE]'
</code></pre>
@@ -461,7 +461,7 @@ as follows:</p>
<h3 id="example">Example</h3>
-<div class="highlighter-rouge"><pre class="highlight"><code>CREATE TABLE users (id INTEGER, username VARCHAR)
+<div class="highlighter-rouge"><pre class="highlight"><code>CREATE EXTERNAL TABLE users (id INTEGER, username VARCHAR)
TYPE bigquery
LOCATION 'testing-integration:apache.users'
</code></pre>
@@ -471,7 +471,7 @@ LOCATION 'testing-integration:apache.users'
<h3 id="syntax-2">Syntax</h3>
-<div class="highlighter-rouge"><pre class="highlight"><code>CREATE TABLE [ IF NOT EXISTS ] tableName
+<div class="highlighter-rouge"><pre class="highlight"><code>CREATE EXTERNAL TABLE [ IF NOT EXISTS ] tableName
(
event_timestamp TIMESTAMP,
attributes MAP<VARCHAR, VARCHAR>,
@@ -557,7 +557,7 @@ objects. Beam attempts to parse JSON to match the schema of the
<h3 id="example-1">Example</h3>
-<div class="highlighter-rouge"><pre class="highlight"><code>CREATE TABLE locations (event_timestamp TIMESTAMP, attributes MAP<VARCHAR, VARCHAR>, payload ROW<id INTEGER, location VARCHAR>)
+<div class="highlighter-rouge"><pre class="highlight"><code>CREATE EXTERNAL TABLE locations (event_timestamp TIMESTAMP, attributes MAP<VARCHAR, VARCHAR>, payload ROW<id INTEGER, location VARCHAR>)
TYPE pubsub
LOCATION 'projects/testing-integration/topics/user-location'
</code></pre>
@@ -569,7 +569,7 @@ LOCATION 'projects/testing-integration/topics/user-location'
<h3 id="syntax-3">Syntax</h3>
-<div class="highlighter-rouge"><pre class="highlight"><code>CREATE TABLE [ IF NOT EXISTS ] tableName (tableElement [, tableElement ]*)
+<div class="highlighter-rouge"><pre class="highlight"><code>CREATE EXTERNAL TABLE [ IF NOT EXISTS ] tableName (tableElement [, tableElement ]*)
TYPE kafka
LOCATION 'kafka://localhost:2181/brokers'
TBLPROPERTIES '{"bootstrap.servers":"localhost:9092", "topics": ["topic1", "topic2"]}'
@@ -617,7 +617,7 @@ access the same underlying data.</p>
<h3 id="syntax-4">Syntax</h3>
-<div class="highlighter-rouge"><pre class="highlight"><code>CREATE TABLE [ IF NOT EXISTS ] tableName (tableElement [, tableElement ]*)
+<div class="highlighter-rouge"><pre class="highlight"><code>CREATE EXTERNAL TABLE [ IF NOT EXISTS ] tableName (tableElement [, tableElement ]*)
TYPE text
LOCATION '/home/admin/orders'
TBLPROPERTIES '{"format: "Excel"}'
@@ -659,7 +659,7 @@ types specified in the schema using org.apache.commons.csv.</li>
<h3 id="example-2">Example</h3>
-<div class="highlighter-rouge"><pre class="highlight"><code>CREATE TABLE orders (id INTEGER, price INTEGER)
+<div class="highlighter-rouge"><pre class="highlight"><code>CREATE EXTERNAL TABLE orders (id INTEGER, price INTEGER)
TYPE text
LOCATION '/home/admin/orders'
</code></pre>
diff --git a/website/generated-content/documentation/dsls/sql/create-table/index.html b/website/generated-content/documentation/dsls/sql/create-table/index.html
index df62158..d1ba0da 100644
--- a/website/generated-content/documentation/dsls/sql/create-table/index.html
+++ b/website/generated-content/documentation/dsls/sql/create-table/index.html
@@ -1,737 +1,10 @@
<!DOCTYPE html>
-<!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<html lang="en">
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <title>Beam SQL: CREATE TABLE Statement</title>
- <meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow [...]
-">
- <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
- <link rel="stylesheet" href="/css/site.css">
- <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
- <script src="/js/bootstrap.min.js"></script>
- <script src="/js/language-switch.js"></script>
- <script src="/js/fix-menu.js"></script>
- <script src="/js/section-nav.js"></script>
- <script src="/js/page-nav.js"></script>
- <link rel="canonical" href="https://beam.apache.org/documentation/dsls/sql/create-table/" data-proofer-ignore>
- <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
- <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
- <script>
- (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
- (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
- m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
- })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
- ga('create', 'UA-73650088-1', 'auto');
- ga('send', 'pageview');
- </script>
-</head>
-
- <body class="body" data-spy="scroll" data-target=".page-nav" data-offset="0">
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<nav class="header navbar navbar-fixed-top">
- <div class="navbar-header">
- <button type="button" class="navbar-toggle" aria-expanded="false" aria-controls="navbar">
- <span class="sr-only">Toggle navigation</span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- </button>
-
- <a href="/" class="navbar-brand" >
- <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
- </a>
- </div>
-
- <div class="navbar-mask closed"></div>
-
- <div id="navbar" class="navbar-container closed">
- <ul class="nav navbar-nav">
- <li>
- <a href="/get-started/beam-overview/">Get Started</a>
- </li>
- <li>
- <a href="/documentation/">Documentation</a>
- </li>
- <li>
- <a href="/documentation/sdks/java/">SDKS</a>
- </li>
- <li>
- <a href="/documentation/runners/capability-matrix/">RUNNERS</a>
- </li>
- <li>
- <a href="/contribute/">Contribute</a>
- </li>
- <li>
- <a href="/community/contact-us/">Community</a>
- </li>
- <li><a href="/blog">Blog</a></li>
- </ul>
- <ul class="nav navbar-nav navbar-right">
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:20px;"><span class="caret"></span></a>
- <ul class="dropdown-menu dropdown-menu-right">
- <li><a href="http://www.apache.org/">ASF Homepage</a></li>
- <li><a href="http://www.apache.org/licenses/">License</a></li>
- <li><a href="http://www.apache.org/security/">Security</a></li>
- <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
- <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
- <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
- </ul>
- </li>
- </ul>
- </div>
-</nav>
-
- <div class="clearfix container-main-content">
- <div class="section-nav closed" data-offset-top="90" data-offset-bottom="500">
- <span class="section-nav-back glyphicon glyphicon-menu-left"></span>
- <nav>
- <ul class="section-nav-list" data-section-nav>
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<li><span class="section-nav-list-main-title">SDKS</span></li>
-
-<li>
- <span class="section-nav-list-title">Java</span>
- <ul class="section-nav-list">
- <li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
- width="14" height="14"
- alt="External link."></a>
- </li>
- <li><a href="/documentation/sdks/java-extensions/">Java SDK extensions</a></li>
- <li><a href="/documentation/sdks/java-thirdparty/">Java 3rd party extensions</a></li>
- <li><a href="/documentation/sdks/java/nexmark/">Nexmark benchmark suite</a></li>
- </ul>
-</li>
-
-<li>
- <span class="section-nav-list-title">Python</span>
- <ul class="section-nav-list">
- <li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
- width="14" height="14"
- alt="External link."></a>
- </li>
- <li><a href="/documentation/sdks/python-streaming/">Python streaming pipelines</a></li>
- <li><a href="/documentation/sdks/python-type-safety/">Ensuring Python type safety</a></li>
- <li><a href="/documentation/sdks/python-pipeline-dependencies/">Managing pipeline dependencies</a></li>
- <li><a href="/documentation/sdks/python-custom-io/">Creating new sources and sinks</a></li>
- </ul>
-</li>
-
-<li>
- <span class="section-nav-list-title">Go SDK</span>
- <ul class="section-nav-list">
- <li><a href="/documentation/sdks/go/">Go SDK overview</a></li>
- <li><a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam" target="_blank">Go SDK API reference <img src="/images/external-link-icon.png"
- width="14" height="14"
- alt="External link."></a>
- </li>
- </ul>
-</li>
-
-<li>
- <span class="section-nav-list-title">SQL</span>
- <ul class="section-nav-list">
- <li><a href="/documentation/dsls/sql/overview/">Overview</a></li>
- <li><a href="/documentation/dsls/sql/walkthrough/">Walkthrough</a></li>
- <li><a href="/documentation/dsls/sql/shell/">Shell</a></li>
- <li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">SQL Reference</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/dsls/sql/data-types/">Data types</a></li>
- <li><a href="/documentation/dsls/sql/lexical/">Lexical structure</a></li>
- <li><a href="/documentation/dsls/sql/create-table/">CREATE TABLE</a></li>
- <li><a href="/documentation/dsls/sql/select/">SELECT</a></li>
- <li><a href="/documentation/dsls/sql/windowing-and-triggering/">Windowing & Triggering</a></li>
- <li><a href="/documentation/dsls/sql/joins/">Joins</a></li>
- <li><a href="/documentation/dsls/sql/scalar-functions/">Scalar functions</a></li>
- <li><a href="/documentation/dsls/sql/aggregate-functions/">Aggregate functions</a></li>
- <li><a href="/documentation/dsls/sql/user-defined-functions/">User-defined functions</a></li>
- <li><a href="/documentation/dsls/sql/set/">SET Pipeline Options</a></li>
- </ul>
- </li>
- </ul>
-</li>
-
- </ul>
- </nav>
- </div>
-
- <nav class="page-nav clearfix" data-offset-top="90" data-offset-bottom="500">
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-
-
-<ul class="nav">
- <li><a href="#syntax">Syntax</a></li>
- <li><a href="#bigquery">BigQuery</a>
- <ul>
- <li><a href="#syntax-1">Syntax</a></li>
- <li><a href="#read-mode">Read Mode</a></li>
- <li><a href="#write-mode">Write Mode</a></li>
- <li><a href="#schema">Schema</a></li>
- <li><a href="#example">Example</a></li>
- </ul>
- </li>
- <li><a href="#pubsub">Pub/Sub</a>
- <ul>
- <li><a href="#syntax-2">Syntax</a></li>
- <li><a href="#read-mode-1">Read Mode</a></li>
- <li><a href="#write-mode-1">Write Mode</a></li>
- <li><a href="#schema-1">Schema</a></li>
- <li><a href="#supported-payload">Supported Payload</a></li>
- <li><a href="#example-1">Example</a></li>
- </ul>
- </li>
- <li><a href="#kafka">Kafka</a>
- <ul>
- <li><a href="#syntax-3">Syntax</a></li>
- <li><a href="#read-mode-2">Read Mode</a></li>
- <li><a href="#write-mode-2">Write Mode</a></li>
- <li><a href="#supported-payload-1">Supported Payload</a></li>
- <li><a href="#schema-2">Schema</a></li>
- </ul>
- </li>
- <li><a href="#text">Text</a>
- <ul>
- <li><a href="#syntax-4">Syntax</a></li>
- <li><a href="#read-mode-3">Read Mode</a></li>
- <li><a href="#write-mode-3">Write Mode</a></li>
- <li><a href="#supported-payload-2">Supported Payload</a></li>
- <li><a href="#schema-3">Schema</a></li>
- <li><a href="#example-2">Example</a></li>
- </ul>
- </li>
-</ul>
-
-
- </nav>
-
- <div class="body__contained body__section-nav">
- <!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-<h1 id="create-table">CREATE TABLE</h1>
-
-<p>Beam SQL’s <code class="highlighter-rouge">CREATE TABLE</code> statement registers a virtual table that maps to an
-<a href="https://beam.apache.org/documentation/io/built-in/">external storage system</a>.
-For some storage systems, <code class="highlighter-rouge">CREATE TABLE</code> does not create a physical table until
-a write occurs. After the physical table exists, you can access the table with
-the <code class="highlighter-rouge">SELECT</code>, <code class="highlighter-rouge">JOIN</code>, and <code class="highlighter-rouge">INSERT INTO</code> statements.</p>
-
-<p>The <code class="highlighter-rouge">CREATE TABLE</code> statement includes a schema and extended clauses.</p>
-
-<h2 id="syntax">Syntax</h2>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>CREATE TABLE [ IF NOT EXISTS ] tableName (tableElement [, tableElement ]*)
-TYPE type
-[LOCATION location]
-[TBLPROPERTIES tblProperties]
-
-simpleType: TINYINT | SMALLINT | INTEGER | BIGINT | FLOAT | DOUBLE | DECIMAL | BOOLEAN | DATE | TIME | TIMESTAMP | CHAR | VARCHAR
-
-fieldType: simpleType | MAP<simpleType, fieldType> | ARRAY<fieldType> | ROW<tableElement [, tableElement ]*>
-
-tableElement: columnName fieldType [ NOT NULL ]
-</code></pre>
-</div>
-
-<ul>
- <li><code class="highlighter-rouge">IF NOT EXISTS</code>: Optional. If the table is already registered, Beam SQL
-ignores the statement instead of returning an error.</li>
- <li><code class="highlighter-rouge">tableName</code>: The case sensitive name of the table to create and register,
-specified as an
-<a href="https://beam.apache.org/documentation/dsls/sql/lexical/#identifiers">Identifier</a>.
-The table name does not need to match the name in the underlying data
-storage system.</li>
- <li><code class="highlighter-rouge">tableElement</code>: <code class="highlighter-rouge">columnName</code> <code class="highlighter-rouge">fieldType</code> <code class="highlighter-rouge">[ NOT NULL ]</code>
- <ul>
- <li><code class="highlighter-rouge">columnName</code>: The case sensitive name of the column, specified as a
-backtick_quoted_expression.</li>
- <li><code class="highlighter-rouge">fieldType</code>: The field’s type, specified as one of the following types:
- <ul>
- <li><code class="highlighter-rouge">simpleType</code>: <code class="highlighter-rouge">TINYINT</code>, <code class="highlighter-rouge">SMALLINT</code>, <code class="highlighter-rouge">INTEGER</code>, <code class="highlighter-rouge">BIGINT</code>, <code class="highlighter-rouge">FLOAT</code>,
-<code class="highlighter-rouge">DOUBLE</code>, <code class="highlighter-rouge">DECIMAL</code>, <code class="highlighter-rouge">BOOLEAN</code>, <code class="highlighter-rouge">DATE</code>, <code class="highlighter-rouge">TIME</code>, <code class="highlighter-rouge">TIMESTAMP</code>, <code class="highlighter-rouge">CHAR</code>,
-<code class="highlighter-rouge">VARCHAR</code></li>
- <li><code class="highlighter-rouge">MAP<simpleType, fieldType></code></li>
- <li><code class="highlighter-rouge">ARRAY<fieldType></code></li>
- <li><code class="highlighter-rouge">ROW<tableElement [, tableElement ]*></code></li>
- </ul>
- </li>
- <li><code class="highlighter-rouge">NOT NULL</code>: Optional. Indicates that the column is not nullable.</li>
- </ul>
- </li>
- <li><code class="highlighter-rouge">type</code>: The I/O transform that backs the virtual table, specified as an
-<a href="https://beam.apache.org/documentation/dsls/sql/lexical/#identifiers">Identifier</a>
-with one of the following values:
- <ul>
- <li><code class="highlighter-rouge">bigquery</code></li>
- <li><code class="highlighter-rouge">pubsub</code></li>
- <li><code class="highlighter-rouge">kafka</code></li>
- <li><code class="highlighter-rouge">text</code></li>
- </ul>
- </li>
- <li><code class="highlighter-rouge">location</code>: The I/O specific location of the underlying table, specified as
-a <a href="https://beam.apache.org/documentation/dsls/sql/lexical/#string-literals">String
-Literal</a>.
-See the I/O specific sections for <code class="highlighter-rouge">location</code> format requirements.</li>
- <li><code class="highlighter-rouge">tblProperties</code>: The I/O specific quoted key value JSON object with extra
-configuration, specified as a <a href="https://beam.apache.org/documentation/dsls/sql/lexical/#string-literals">String
-Literal</a>.
-See the I/O specific sections for <code class="highlighter-rouge">tblProperties</code> format requirements.</li>
-</ul>
-
-<h2 id="bigquery">BigQuery</h2>
-
-<h3 id="syntax-1">Syntax</h3>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>CREATE TABLE [ IF NOT EXISTS ] tableName (tableElement [, tableElement ]*)
-TYPE bigquery
-LOCATION '[PROJECT_ID]:[DATASET].[TABLE]'
-</code></pre>
-</div>
-
-<ul>
- <li><code class="highlighter-rouge">LOCATION:</code>Location of the table in the BigQuery CLI format.
- <ul>
- <li><code class="highlighter-rouge">PROJECT_ID</code>: ID of the Google Cloud Project</li>
- <li><code class="highlighter-rouge">DATASET</code>: BigQuery Dataset ID</li>
- <li><code class="highlighter-rouge">TABLE</code>: BigQuery Table ID within the Dataset</li>
- </ul>
- </li>
-</ul>
-
-<h3 id="read-mode">Read Mode</h3>
-
-<p>Beam SQL supports reading columns with simple types (<code class="highlighter-rouge">simpleType</code>) and arrays of simple
-types (<code class="highlighter-rouge">ARRAY<simpleType></code>).</p>
-
-<h3 id="write-mode">Write Mode</h3>
-
-<p>if the table does not exist, Beam creates the table specified in location when
-the first record is written. If the table does exist, the specified columns must
-match the existing table.</p>
-
-<h3 id="schema">Schema</h3>
-
-<p>Schema-related errors will cause the pipeline to crash. The Map type is not
-supported. Beam SQL types map to <a href="https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types">BigQuery Standard SQL
-types</a>
-as follows:</p>
-
-<table>
- <tr>
- <td>Beam SQL Type
- </td>
- <td>BigQuery Standard SQL Type
- </td>
- </tr>
- <tr>
- <td>TINYINT, SMALLINT, INTEGER, BIGINT
- </td>
- <td>INT64
- </td>
- </tr>
- <tr>
- <td>FLOAT, DOUBLE, DECIMAL
- </td>
- <td>FLOAT64
- </td>
- </tr>
- <tr>
- <td>BOOLEAN
- </td>
- <td>BOOL
- </td>
- </tr>
- <tr>
- <td>DATE
- </td>
- <td>DATE
- </td>
- </tr>
- <tr>
- <td>TIME
- </td>
- <td>TIME
- </td>
- </tr>
- <tr>
- <td>TIMESTAMP
- </td>
- <td>TIMESTAMP
- </td>
- </tr>
- <tr>
- <td>CHAR, VARCHAR
- </td>
- <td>STRING
- </td>
- </tr>
- <tr>
- <td>MAP
- </td>
- <td>(not supported)
- </td>
- </tr>
- <tr>
- <td>ARRAY
- </td>
- <td>ARRAY
- </td>
- </tr>
- <tr>
- <td>ROW
- </td>
- <td>STRUCT
- </td>
- </tr>
-</table>
-
-<h3 id="example">Example</h3>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>CREATE TABLE users (id INTEGER, username VARCHAR)
-TYPE bigquery
-LOCATION 'testing-integration:apache.users'
-</code></pre>
-</div>
-
-<h2 id="pubsub">Pub/Sub</h2>
-
-<h3 id="syntax-2">Syntax</h3>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>CREATE TABLE [ IF NOT EXISTS ] tableName
- (
- event_timestamp TIMESTAMP,
- attributes MAP<VARCHAR, VARCHAR>,
- payload ROW<tableElement [, tableElement ]*>
- )
-TYPE pubsub
-LOCATION 'projects/[PROJECT]/topics/[TOPIC]'
-TBLPROPERTIES '{"timestampAttributeKey": "key", "deadLetterQueue": "projects/[PROJECT]/topics/[TOPIC]"}'
-</code></pre>
-</div>
-
-<ul>
- <li><code class="highlighter-rouge">event_timestamp</code>: The event timestamp associated with the Pub/Sub message
-by PubsubIO. It can be one of the following:
- <ul>
- <li>Message publish time, which is provided by Pub/Sub. This is the default
-value if no extra configuration is provided.</li>
- <li>A timestamp specified in one of the user-provided message attributes.
-The attribute key is configured by the <code class="highlighter-rouge">timestampAttributeKey</code> field of
-the <code class="highlighter-rouge">tblProperties</code> blob. The value of the attribute should conform to
-the <a href="https://beam.apache.org/releases/javadoc/2.4.0/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.Read.html#withTimestampAttribute-java.lang.String-">requirements of
-PubsubIO</a>,
-which is either millis since Unix epoch or <a href="https://www.ietf.org/rfc/rfc3339.txt">RFC 339
-</a>date string.</li>
- </ul>
- </li>
- <li><code class="highlighter-rouge">attributes</code>: The user-provided attributes map from the Pub/Sub message;</li>
- <li><code class="highlighter-rouge">payload</code>: The schema of the JSON payload of the Pub/Sub message. No other
-payload formats are currently supported by Beam SQL. If a record can’t be
-unmarshalled, the record is written to the topic specified in the
-<code class="highlighter-rouge">deadLeaderQueue</code> field of the <code class="highlighter-rouge">tblProperties</code> blob. If no dead-letter queue
-is specified in this case, an exception is thrown and the pipeline will
-crash.</li>
- <li><code class="highlighter-rouge">LOCATION</code>:
- <ul>
- <li><code class="highlighter-rouge">PROJECT</code>: ID of the Google Cloud Project</li>
- <li><code class="highlighter-rouge">TOPIC</code>: The Pub/Sub topic name. A subscription will be created
-automatically, but the subscription is not cleaned up automatically.
-Specifying an existing subscription is not supported.</li>
- </ul>
- </li>
- <li><code class="highlighter-rouge">TBLPROPERTIES</code>:
- <ul>
- <li><code class="highlighter-rouge">timestampAttributeKey</code>: Optional. The key which contains the event
-timestamp associated with the Pub/Sub message. If not specified, the
-message publish timestamp is used as an event timestamp for
-windowing/watermarking.</li>
- <li><code class="highlighter-rouge">deadLetterQueue</code>: The topic into which messages are written if the
-payload was not parsed. If not specified, an exception is thrown for
-parsing failures.</li>
- </ul>
- </li>
-</ul>
-
-<h3 id="read-mode-1">Read Mode</h3>
-
-<p>PubsubIO is currently limited to read access only.</p>
-
-<h3 id="write-mode-1">Write Mode</h3>
-
-<p>Not supported. PubSubIO is currently limited to read access only in Beam SQL.</p>
-
-<h3 id="schema-1">Schema</h3>
-
-<p>Pub/Sub messages have metadata associated with them, and you can reference this
-metadata in your queries. For each message, Pub/Sub exposes its publish time and
-a map of user-provided attributes in addition to the payload (unstructured in
-the general case). This information must be preserved and accessible from the
-SQL statements. Currently, this means that PubsubIO tables require you to
-declare a special set of columns, as shown below.</p>
-
-<h3 id="supported-payload">Supported Payload</h3>
-
-<ul>
- <li>JSON Objects
- <ul>
- <li>Beam only supports querying messages with payload containing JSON
-objects. Beam attempts to parse JSON to match the schema of the
-<code class="highlighter-rouge">payload</code> field.</li>
- </ul>
- </li>
-</ul>
-
-<h3 id="example-1">Example</h3>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>CREATE TABLE locations (event_timestamp TIMESTAMP, attributes MAP<VARCHAR, VARCHAR>, payload ROW<id INTEGER, location VARCHAR>)
-TYPE pubsub
-LOCATION 'projects/testing-integration/topics/user-location'
-</code></pre>
-</div>
-
-<h2 id="kafka">Kafka</h2>
-
-<p>KafkaIO is experimental in Beam SQL.</p>
-
-<h3 id="syntax-3">Syntax</h3>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>CREATE TABLE [ IF NOT EXISTS ] tableName (tableElement [, tableElement ]*)
-TYPE kafka
-LOCATION 'kafka://localhost:2181/brokers'
-TBLPROPERTIES '{"bootstrap.servers":"localhost:9092", "topics": ["topic1", "topic2"]}'
-</code></pre>
-</div>
-
-<ul>
- <li><code class="highlighter-rouge">LOCATION</code>: The Kafka topic URL.</li>
- <li><code class="highlighter-rouge">TBLPROPERTIES</code>:
- <ul>
- <li><code class="highlighter-rouge">bootstrap.servers</code>: Optional. Allows you to specify the bootstrap
-server.</li>
- <li><code class="highlighter-rouge">topics</code>: Optional. Allows you to specify specific topics.</li>
- </ul>
- </li>
-</ul>
-
-<h3 id="read-mode-2">Read Mode</h3>
-
-<p>Read Mode supports reading from a topic.</p>
-
-<h3 id="write-mode-2">Write Mode</h3>
-
-<p>Write Mode supports writing to a topic.</p>
-
-<h3 id="supported-payload-1">Supported Payload</h3>
-
-<ul>
- <li>CSV
- <ul>
- <li>Beam parses the messages, attempting to parse fields according to the
-types specified in the schema.</li>
- </ul>
- </li>
-</ul>
-
-<h3 id="schema-2">Schema</h3>
-
-<p>Only simple types are supported.</p>
-
-<h2 id="text">Text</h2>
-
-<p>TextIO is experimental in Beam SQL. Read Mode and Write Mode do not currently
-access the same underlying data.</p>
-
-<h3 id="syntax-4">Syntax</h3>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>CREATE TABLE [ IF NOT EXISTS ] tableName (tableElement [, tableElement ]*)
-TYPE text
-LOCATION '/home/admin/orders'
-TBLPROPERTIES '{"format: "Excel"}'
-</code></pre>
-</div>
-
-<ul>
- <li><code class="highlighter-rouge">LOCATION</code>: The path to the file for Read Mode. The prefix for Write Mode.</li>
- <li><code class="highlighter-rouge">TBLPROPERTIES</code>:
- <ul>
- <li><code class="highlighter-rouge">format</code>: Optional. Allows you to specify the
-<a href="https://commons.apache.org/proper/commons-csv/archives/1.5/apidocs/org/apache/commons/csv/CSVFormat.Predefined.html">CSVFormat</a>.</li>
- </ul>
- </li>
-</ul>
-
-<h3 id="read-mode-3">Read Mode</h3>
-
-<p>Read Mode supports reading from a file.</p>
-
-<h3 id="write-mode-3">Write Mode</h3>
-
-<p>Write Mode supports writing to a set of files. TextIO creates file on writes.</p>
-
-<h3 id="supported-payload-2">Supported Payload</h3>
-
-<ul>
- <li>CSV
- <ul>
- <li>Beam parses the messages, attempting to parse fields according to the
-types specified in the schema using org.apache.commons.csv.</li>
- </ul>
- </li>
-</ul>
-
-<h3 id="schema-3">Schema</h3>
-
-<p>Only simple types are supported.</p>
-
-<h3 id="example-2">Example</h3>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>CREATE TABLE orders (id INTEGER, price INTEGER)
-TYPE text
-LOCATION '/home/admin/orders'
-</code></pre>
-</div>
-
- </div>
- </div>
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<footer class="footer">
- <div class="footer__contained">
- <div class="footer__cols">
- <div class="footer__cols__col">
- <div class="footer__cols__col__logo">
- <img src="/images/beam_logo_circle.svg" class="footer__logo" alt="Beam logo">
- </div>
- <div class="footer__cols__col__logo">
- <img src="/images/apache_logo_circle.svg" class="footer__logo" alt="Apache logo">
- </div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Start</div>
- <div class="footer__cols__col__link"><a href="/get-started/beam-overview/">Overview</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/quickstart-java/">Quickstart (Java)</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/quickstart-py/">Quickstart (Python)</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/quickstart-go/">Quickstart (Go)</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/downloads/">Downloads</a></div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Docs</div>
- <div class="footer__cols__col__link"><a href="/documentation/programming-guide/">Concepts</a></div>
- <div class="footer__cols__col__link"><a href="/documentation/pipelines/design-your-pipeline/">Pipelines</a></div>
- <div class="footer__cols__col__link"><a href="/documentation/runners/capability-matrix/">Runners</a></div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Community</div>
- <div class="footer__cols__col__link"><a href="/contribute/">Contribute</a></div>
- <div class="footer__cols__col__link"><a href="https://projects.apache.org/committee.html?beam" target="_blank">Team<img src="/images/external-link-icon.png"
- width="14" height="14"
- alt="External link."></a></div>
- <div class="footer__cols__col__link"><a href="/contribute/presentation-materials/">Media</a></div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Resources</div>
- <div class="footer__cols__col__link"><a href="/blog/">Blog</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/support/">Support</a></div>
- <div class="footer__cols__col__link"><a href="https://github.com/apache/beam">GitHub</a></div>
- </div>
- </div>
- </div>
- <div class="footer__bottom">
- ©
- <a href="http://www.apache.org">The Apache Software Foundation</a>
- | <a href="/privacy_policy">Privacy Policy</a>
- | <a href="/feed.xml">RSS Feed</a>
- <br><br>
- Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are
- either registered trademarks or trademarks of The Apache Software
- Foundation. All other products or name brands are trademarks of their
- respective holders, including The Apache Software Foundation.
- </div>
-</footer>
-
- </body>
+<html lang="en-US">
+<meta charset="utf-8">
+<title>Redirecting…</title>
+<link rel="canonical" href="https://beam.apache.org/documentation/dsls/sql/create-external-table/">
+<meta http-equiv="refresh" content="0; url=https://beam.apache.org/documentation/dsls/sql/create-external-table/">
+<h1>Redirecting…</h1>
+<a href="https://beam.apache.org/documentation/dsls/sql/create-external-table/">Click here if you are not redirected.</a>
+<script>location="https://beam.apache.org/documentation/dsls/sql/create-external-table/"</script>
</html>
diff --git a/website/generated-content/documentation/dsls/sql/data-types/index.html b/website/generated-content/documentation/dsls/sql/data-types/index.html
index 58c6319..7017594 100644
--- a/website/generated-content/documentation/dsls/sql/data-types/index.html
+++ b/website/generated-content/documentation/dsls/sql/data-types/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.7.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.7.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -190,7 +190,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/data-types/">Data types</a></li>
<li><a href="/documentation/dsls/sql/lexical/">Lexical structure</a></li>
- <li><a href="/documentation/dsls/sql/create-table/">CREATE TABLE</a></li>
+ <li><a href="/documentation/dsls/sql/create-external-table/">CREATE EXTERNAL TABLE</a></li>
<li><a href="/documentation/dsls/sql/select/">SELECT</a></li>
<li><a href="/documentation/dsls/sql/windowing-and-triggering/">Windowing & Triggering</a></li>
<li><a href="/documentation/dsls/sql/joins/">Joins</a></li>
diff --git a/website/generated-content/documentation/dsls/sql/joins/index.html b/website/generated-content/documentation/dsls/sql/joins/index.html
index 1ef8463..d9f216e 100644
--- a/website/generated-content/documentation/dsls/sql/joins/index.html
+++ b/website/generated-content/documentation/dsls/sql/joins/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.7.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.7.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -190,7 +190,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/data-types/">Data types</a></li>
<li><a href="/documentation/dsls/sql/lexical/">Lexical structure</a></li>
- <li><a href="/documentation/dsls/sql/create-table/">CREATE TABLE</a></li>
+ <li><a href="/documentation/dsls/sql/create-external-table/">CREATE EXTERNAL TABLE</a></li>
<li><a href="/documentation/dsls/sql/select/">SELECT</a></li>
<li><a href="/documentation/dsls/sql/windowing-and-triggering/">Windowing & Triggering</a></li>
<li><a href="/documentation/dsls/sql/joins/">Joins</a></li>
diff --git a/website/generated-content/documentation/dsls/sql/lexical/index.html b/website/generated-content/documentation/dsls/sql/lexical/index.html
index 779c589..5d2de69 100644
--- a/website/generated-content/documentation/dsls/sql/lexical/index.html
+++ b/website/generated-content/documentation/dsls/sql/lexical/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.7.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.7.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -190,7 +190,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/data-types/">Data types</a></li>
<li><a href="/documentation/dsls/sql/lexical/">Lexical structure</a></li>
- <li><a href="/documentation/dsls/sql/create-table/">CREATE TABLE</a></li>
+ <li><a href="/documentation/dsls/sql/create-external-table/">CREATE EXTERNAL TABLE</a></li>
<li><a href="/documentation/dsls/sql/select/">SELECT</a></li>
<li><a href="/documentation/dsls/sql/windowing-and-triggering/">Windowing & Triggering</a></li>
<li><a href="/documentation/dsls/sql/joins/">Joins</a></li>
diff --git a/website/generated-content/documentation/dsls/sql/overview/index.html b/website/generated-content/documentation/dsls/sql/overview/index.html
index 0cc19ae..05ea2f1 100644
--- a/website/generated-content/documentation/dsls/sql/overview/index.html
+++ b/website/generated-content/documentation/dsls/sql/overview/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.7.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.7.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -190,7 +190,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/data-types/">Data types</a></li>
<li><a href="/documentation/dsls/sql/lexical/">Lexical structure</a></li>
- <li><a href="/documentation/dsls/sql/create-table/">CREATE TABLE</a></li>
+ <li><a href="/documentation/dsls/sql/create-external-table/">CREATE EXTERNAL TABLE</a></li>
<li><a href="/documentation/dsls/sql/select/">SELECT</a></li>
<li><a href="/documentation/dsls/sql/windowing-and-triggering/">Windowing & Triggering</a></li>
<li><a href="/documentation/dsls/sql/joins/">Joins</a></li>
@@ -256,9 +256,9 @@ big data processing with some streaming enhancements. Calcite provides the
basic dialect underlying Beam SQL. We have added additional extensions to
make it easy to leverage Beam’s unified batch/streaming model and support
for complex data types.</li>
- <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/extensions/sql/SqlTransform.html">SqlTransform</a>:
+ <li><a href="https://beam.apache.org/releases/javadoc/2.7.0/index.html?org/apache/beam/sdk/extensions/sql/SqlTransform.html">SqlTransform</a>:
the interface for creating <code class="highlighter-rouge">PTransforms</code> from SQL queries.</li>
- <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/values/Row.html">Row</a>:
+ <li><a href="https://beam.apache.org/releases/javadoc/2.7.0/index.html?org/apache/beam/sdk/values/Row.html">Row</a>:
the type of elements that Beam SQL operates on. A <code class="highlighter-rouge">PCollection<Row></code> plays the role of a table.</li>
</ul>
diff --git a/website/generated-content/documentation/dsls/sql/scalar-functions/index.html b/website/generated-content/documentation/dsls/sql/scalar-functions/index.html
index 7fcce88..244e3a2 100644
--- a/website/generated-content/documentation/dsls/sql/scalar-functions/index.html
+++ b/website/generated-content/documentation/dsls/sql/scalar-functions/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.7.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.7.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -190,7 +190,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/data-types/">Data types</a></li>
<li><a href="/documentation/dsls/sql/lexical/">Lexical structure</a></li>
- <li><a href="/documentation/dsls/sql/create-table/">CREATE TABLE</a></li>
+ <li><a href="/documentation/dsls/sql/create-external-table/">CREATE EXTERNAL TABLE</a></li>
<li><a href="/documentation/dsls/sql/select/">SELECT</a></li>
<li><a href="/documentation/dsls/sql/windowing-and-triggering/">Windowing & Triggering</a></li>
<li><a href="/documentation/dsls/sql/joins/">Joins</a></li>
diff --git a/website/generated-content/documentation/dsls/sql/select/index.html b/website/generated-content/documentation/dsls/sql/select/index.html
index 246ef64..c9226f5 100644
--- a/website/generated-content/documentation/dsls/sql/select/index.html
+++ b/website/generated-content/documentation/dsls/sql/select/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.7.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.7.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -190,7 +190,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/data-types/">Data types</a></li>
<li><a href="/documentation/dsls/sql/lexical/">Lexical structure</a></li>
- <li><a href="/documentation/dsls/sql/create-table/">CREATE TABLE</a></li>
+ <li><a href="/documentation/dsls/sql/create-external-table/">CREATE EXTERNAL TABLE</a></li>
<li><a href="/documentation/dsls/sql/select/">SELECT</a></li>
<li><a href="/documentation/dsls/sql/windowing-and-triggering/">Windowing & Triggering</a></li>
<li><a href="/documentation/dsls/sql/joins/">Joins</a></li>
@@ -222,6 +222,75 @@
+<ul class="nav">
+ <li><a href="#sql-syntax">SQL Syntax</a></li>
+ <li><a href="#select-list">SELECT list</a>
+ <ul>
+ <li><a href="#select-">SELECT *</a></li>
+ <li><a href="#select-expression">SELECT <code class="highlighter-rouge">expression</code></a></li>
+ <li><a href="#select-expression_1">SELECT <code class="highlighter-rouge">expression.*</code></a></li>
+ <li><a href="#select-modifiers">SELECT modifiers</a></li>
+ <li><a href="#aliases">Aliases</a></li>
+ </ul>
+ </li>
+ <li><a href="#from-clause">FROM clause</a>
+ <ul>
+ <li><a href="#syntax">Syntax</a></li>
+ <li><a href="#subqueries">Subqueries</a></li>
+ <li><a href="#aliases_1">Aliases</a></li>
+ </ul>
+ </li>
+ <li><a href="#join-types">JOIN types</a>
+ <ul>
+ <li><a href="#syntax_1">Syntax</a></li>
+ <li><a href="#inner-join">[INNER] JOIN</a></li>
+ <li><a href="#cross-join">CROSS JOIN</a></li>
+ <li><a href="#full-outer-join">FULL [OUTER] JOIN</a></li>
+ <li><a href="#left-outer-join">LEFT [OUTER] JOIN</a></li>
+ <li><a href="#right-outer-join">RIGHT [OUTER] JOIN</a></li>
+ <li><a href="#on-clause">ON clause</a></li>
+ <li><a href="#using-clause">USING clause</a></li>
+ <li><a href="#sequences-of-joins">Sequences of JOINs</a></li>
+ </ul>
+ </li>
+ <li><a href="#where-clause">WHERE clause</a>
+ <ul>
+ <li><a href="#syntax_2">Syntax</a></li>
+ </ul>
+ </li>
+ <li><a href="#group-by-clause">GROUP BY clause</a>
+ <ul>
+ <li><a href="#syntax_3">Syntax</a></li>
+ </ul>
+ </li>
+ <li><a href="#having-clause">HAVING clause</a>
+ <ul>
+ <li><a href="#syntax_4">Syntax</a></li>
+ </ul>
+ </li>
+ <li><a href="#set-operators">Set operators</a>
+ <ul>
+ <li><a href="#syntax_6">Syntax</a></li>
+ <li><a href="#union">UNION</a></li>
+ <li><a href="#intersect">INTERSECT</a></li>
+ <li><a href="#except">EXCEPT</a></li>
+ </ul>
+ </li>
+ <li><a href="#limit-clause-and-offset-clause">LIMIT clause and OFFSET clause</a>
+ <ul>
+ <li><a href="#syntax_7">Syntax</a></li>
+ </ul>
+ </li>
+ <li><a href="#with-clause">WITH clause</a></li>
+ <li><a href="#aliases_2">Aliases</a>
+ <ul>
+ <li><a href="#explicit-alias-syntax">Explicit alias syntax</a></li>
+ <li><a href="#explicit-alias-visibility">Explicit alias visibility</a></li>
+ <li><a href="#ambiguous-aliases">Ambiguous aliases</a></li>
+ <li><a href="#implicit-aliases">Implicit aliases</a></li>
+ </ul>
+ </li>
+</ul>
</nav>
@@ -256,62 +325,736 @@ batch/streaming model:</p>
<li><a href="/documentation/dsls/sql/windowing-and-triggering/">Windowing & Triggering</a></li>
</ul>
-<p>Below is a curated grammar of the supported syntax in Beam SQL</p>
+<p>Query statements scan one or more tables or expressions and return the computed
+result rows. This topic describes the syntax for SQL queries in Beam.</p>
+
+<h2 id="sql-syntax">SQL Syntax</h2>
-<div class="highlighter-rouge"><pre class="highlight"><code>query:
- {
- select
- | query UNION [ ALL ] query
- | query MINUS [ ALL ] query
- | query INTERSECT [ ALL ] query
- }
- [ ORDER BY orderItem [, orderItem ]* LIMIT count [OFFSET offset] ]
+<div class="highlighter-rouge"><pre class="highlight"><code>query_statement:
+ [ WITH with_query_name AS ( query_expr ) [, ...] ]
+ query_expr
-orderItem:
- expression [ ASC | DESC ]
+query_expr:
+ { select | ( query_expr ) | query_expr set_op query_expr }
+ [ LIMIT count [ OFFSET skip_rows ] ]
select:
- SELECT
- { * | projectItem [, projectItem ]* }
- FROM tableExpression
- [ WHERE booleanExpression ]
- [ GROUP BY { groupItem [, groupItem ]* } ]
- [ HAVING booleanExpression ]
+ SELECT [{ ALL | DISTINCT }]
+ { [ expression. ]* [ EXCEPT ( column_name [, ...] ) ]
+ [ REPLACE ( expression [ AS ] column_name [, ...] ) ]
+ | expression [ [ AS ] alias ] } [, ...]
+ [ FROM from_item [, ...] ]
+ [ WHERE bool_expression ]
+ [ GROUP BY { expression [, ...] | ROLLUP ( expression [, ...] ) } ]
+ [ HAVING bool_expression ]
+
+set_op:
+ UNION { ALL | DISTINCT } | INTERSECT DISTINCT | EXCEPT DISTINCT
+
+from_item: {
+ table_name [ [ AS ] alias ] |
+ join |
+ ( query_expr ) [ [ AS ] alias ]
+ with_query_name [ [ AS ] alias ]
+}
+
+join:
+ from_item [ join_type ] JOIN from_item
+ [ { ON bool_expression | USING ( join_column [, ...] ) } ]
+
+join_type:
+ { INNER | CROSS | FULL [OUTER] | LEFT [OUTER] | RIGHT [OUTER] }
+</code></pre>
+</div>
+
+<p>Notation:</p>
+
+<ul>
+ <li>Square brackets “[ ]” indicate optional clauses.</li>
+ <li>Parentheses “( )” indicate literal parentheses.</li>
+ <li>
+ <table>
+ <tbody>
+ <tr>
+ <td>The vertical bar “</td>
+ <td>” indicates a logical OR.</td>
+ </tr>
+ </tbody>
+ </table>
+ </li>
+ <li>Curly braces “{ }” enclose a set of options.</li>
+ <li>A comma followed by an ellipsis within square brackets “[, … ]”
+indicates that the preceding item can repeat in a comma-separated list.</li>
+</ul>
+
+<h2 id="select-list">SELECT list</h2>
+
+<p>Syntax:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>SELECT [{ ALL | DISTINCT }]
+ { [ expression. ]*
+ | expression [ [ AS ] alias ] } [, ...]
+</code></pre>
+</div>
+
+<p>The <code class="highlighter-rouge">SELECT</code> list defines the columns that the query will return. Expressions in
+the <code class="highlighter-rouge">SELECT</code> list can refer to columns in any of the <code class="highlighter-rouge">from_item</code>s in its
+corresponding <code class="highlighter-rouge">FROM</code> clause.</p>
+
+<p>Each item in the <code class="highlighter-rouge">SELECT</code> list is one of:</p>
+
+<ul>
+ <li>*</li>
+ <li><code class="highlighter-rouge">expression</code></li>
+ <li><code class="highlighter-rouge">expression.*</code></li>
+</ul>
+
+<h3 id="select-">SELECT *</h3>
+
+<p><code class="highlighter-rouge">SELECT *</code>, often referred to as <em>select star</em>, produces one output column for
+each column that is visible after executing the full query.</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>SELECT * FROM (SELECT 'apple' AS fruit, 'carrot' AS vegetable);
+
++-------+-----------+
+| fruit | vegetable |
++-------+-----------+
+| apple | carrot |
++-------+-----------+
+</code></pre>
+</div>
+
+<h3 id="select-expression">SELECT <code class="highlighter-rouge">expression</code></h3>
+
+<p>Items in a <code class="highlighter-rouge">SELECT</code> list can be expressions. These expressions evaluate to a
+single value and produce one output column, with an optional explicit <code class="highlighter-rouge">alias</code>.</p>
+
+<p>If the expression does not have an explicit alias, it receives an implicit alias
+according to the rules for <a href="#implicit-aliases">implicit aliases</a>, if possible.
+Otherwise, the column is anonymous and you cannot refer to it by name elsewhere
+in the query.</p>
+
+<h3 id="select-expression_1">SELECT <code class="highlighter-rouge">expression.*</code></h3>
+
+<p>An item in a <code class="highlighter-rouge">SELECT</code> list can also take the form of <code class="highlighter-rouge">expression.*</code>. This
+produces one output column for each column or top-level field of <code class="highlighter-rouge">expression</code>.
+The expression must be a table alias.</p>
+
+<p>The following query produces one output column for each column in the table
+<code class="highlighter-rouge">groceries</code>, aliased as <code class="highlighter-rouge">g</code>.</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>WITH groceries AS
+ (SELECT 'milk' AS dairy,
+ 'eggs' AS protein,
+ 'bread' AS grain)
+SELECT g.*
+FROM groceries AS g;
+
++-------+---------+-------+
+| dairy | protein | grain |
++-------+---------+-------+
+| milk | eggs | bread |
++-------+---------+-------+
+</code></pre>
+</div>
+
+<h3 id="select-modifiers">SELECT modifiers</h3>
+
+<p>You can modify the results returned from a <code class="highlighter-rouge">SELECT</code> query, as follows.</p>
+
+<h4 id="select-distinct">SELECT DISTINCT</h4>
+
+<p>A <code class="highlighter-rouge">SELECT DISTINCT</code> statement discards duplicate rows and returns only the
+remaining rows. <code class="highlighter-rouge">SELECT DISTINCT</code> cannot return columns of the following types:</p>
+
+<ul>
+ <li>STRUCT</li>
+ <li>ARRAY</li>
+</ul>
+
+<h4 id="select-all">SELECT ALL</h4>
+
+<p>A <code class="highlighter-rouge">SELECT ALL</code> statement returns all rows, including duplicate rows. <code class="highlighter-rouge">SELECT
+ALL</code> is the default behavior of <code class="highlighter-rouge">SELECT</code>.</p>
+
+<h3 id="aliases">Aliases</h3>
+
+<p>See <a href="#aliases_2">Aliases</a> for information on syntax and visibility for
+<code class="highlighter-rouge">SELECT</code> list aliases.</p>
+
+<h2 id="from-clause">FROM clause</h2>
+
+<p>The <code class="highlighter-rouge">FROM</code> clause indicates the table or tables from which to retrieve rows, and
+specifies how to join those rows together to produce a single stream of rows for
+processing in the rest of the query.</p>
+
+<h3 id="syntax">Syntax</h3>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>from_item: {
+ table_name [ [ AS ] alias ] |
+ join |
+ ( query_expr ) [ [ AS ] alias ] |
+ with_query_name [ [ AS ] alias ]
+}
+</code></pre>
+</div>
+
+<h4 id="table_name">table_name</h4>
+
+<p>The name (optionally qualified) of an existing table.</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>SELECT * FROM Roster;
+SELECT * FROM beam.Roster;
+</code></pre>
+</div>
+
+<h4 id="join">join</h4>
+
+<p>See <a href="#join-types">JOIN Types</a> below and <a href="/documentation/dsls/sql/joins">Joins</a>.</p>
+
+<h4 id="select_1">select</h4>
+
+<p><code class="highlighter-rouge">( select ) [ [ AS ] alias ]</code> is a table <a href="#subqueries">subquery</a>.</p>
+
+<h4 id="with_query_name">with_query_name</h4>
+
+<p>The query names in a <code class="highlighter-rouge">WITH</code> clause (see <a href="#with-clause">WITH Clause</a>) act like
+names of temporary tables that you can reference anywhere in the <code class="highlighter-rouge">FROM</code> clause.
+In the example below, <code class="highlighter-rouge">subQ1</code> and <code class="highlighter-rouge">subQ2</code> are <code class="highlighter-rouge">with_query_names</code>.</p>
+
+<p>Example:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>WITH
+ subQ1 AS (SELECT * FROM Roster WHERE SchoolID = 52),
+ subQ2 AS (SELECT SchoolID FROM subQ1)
+SELECT DISTINCT * FROM subQ2;
+</code></pre>
+</div>
+
+<p>The <code class="highlighter-rouge">WITH</code> clause hides any permanent tables with the same name for the duration
+of the query, unless you qualify the table name, e.g. <code class="highlighter-rouge">beam.Roster</code>.</p>
+
+<h3 id="subqueries">Subqueries</h3>
+
+<p>A subquery is a query that appears inside another statement, and is written
+inside parentheses. These are also referred to as “sub-SELECTs” or “nested
+SELECTs”. The full <code class="highlighter-rouge">SELECT</code> syntax is valid in subqueries.</p>
+
+<p>There are two types of subquery:</p>
+
+<ul>
+ <li>Expression Subqueries
+which you can use in a query wherever expressions are valid. Expression
+subqueries return a single value.</li>
+ <li>Table subqueries, which you can use only in a <code class="highlighter-rouge">FROM</code> clause. The outer query
+treats the result of the subquery as a table.</li>
+</ul>
+
+<p>Note that there must be parentheses around both types of subqueries.</p>
+
+<p>Example:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>SELECT AVG ( PointsScored )
+FROM
+( SELECT PointsScored
+ FROM Stats
+ WHERE SchoolID = 77 )
+</code></pre>
+</div>
+
+<p>Optionally, a table subquery can have an alias.</p>
+
+<p>Example:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>SELECT r.LastName
+FROM
+( SELECT * FROM Roster) AS r;
+</code></pre>
+</div>
+
+<h3 id="aliases_1">Aliases</h3>
+
+<p>See <a href="#aliases_2">Aliases</a> for information on syntax and visibility for
+<code class="highlighter-rouge">FROM</code> clause aliases.</p>
+
+<h2 id="join-types">JOIN types</h2>
+
+<p>Also see <a href="/documentation/dsls/sql/joins">Joins</a>.</p>
+
+<h3 id="syntax_1">Syntax</h3>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>join:
+ from_item [ join_type ] JOIN from_item
+ [ ON bool_expression | USING ( join_column [, ...] ) ]
+
+join_type:
+ { INNER | CROSS | FULL [OUTER] | LEFT [OUTER] | RIGHT [OUTER] }
+</code></pre>
+</div>
+
+<p>The <code class="highlighter-rouge">JOIN</code> clause merges two <code class="highlighter-rouge">from_item</code>s so that the <code class="highlighter-rouge">SELECT</code> clause can query
+them as one source. The <code class="highlighter-rouge">join_type</code> and <code class="highlighter-rouge">ON</code> or <code class="highlighter-rouge">USING</code> clause (a “join
+condition”) specify how to combine and discard rows from the two <code class="highlighter-rouge">from_item</code>s to
+form a single source.</p>
+
+<p>All <code class="highlighter-rouge">JOIN</code> clauses require a <code class="highlighter-rouge">join_type</code>.</p>
+
+<p>A <code class="highlighter-rouge">JOIN</code> clause requires a join condition unless one of the following conditions
+is true:</p>
+
+<ul>
+ <li><code class="highlighter-rouge">join_type</code> is <code class="highlighter-rouge">CROSS</code>.</li>
+ <li>One or both of the <code class="highlighter-rouge">from_item</code>s is not a table, e.g. an <code class="highlighter-rouge">array_path</code> or
+<code class="highlighter-rouge">field_path</code>.</li>
+</ul>
-projectItem:
- expression [ [ AS ] columnAlias ]
- | tableAlias . *
+<h3 id="inner-join">[INNER] JOIN</h3>
-tableExpression:
- tableReference [, tableReference ]*
- | tableExpression [ ( LEFT | RIGHT ) [ OUTER ] ] JOIN tableExpression [ joinCondition ]
+<p>An <code class="highlighter-rouge">INNER JOIN</code>, or simply <code class="highlighter-rouge">JOIN</code>, effectively calculates the Cartesian product
+of the two <code class="highlighter-rouge">from_item</code>s and discards all rows that do not meet the join
+condition. “Effectively” means that it is possible to implement an <code class="highlighter-rouge">INNER JOIN</code>
+without actually calculating the Cartesian product.</p>
-booleanExpression:
- expression [ IS NULL | IS NOT NULL ]
- | expression [ > | >= | = | < | <= | <> ] expression
- | booleanExpression [ AND | OR ] booleanExpression
- | NOT booleanExpression
- | '(' booleanExpression ')'
+<h3 id="cross-join">CROSS JOIN</h3>
-joinCondition:
- ON booleanExpression
+<p><code class="highlighter-rouge">CROSS JOIN</code> is generally not yet supported.</p>
-tableReference:
- tableName [ [ AS ] alias ]
+<h3 id="full-outer-join">FULL [OUTER] JOIN</h3>
-values:
- VALUES expression [, expression ]*
+<p>A <code class="highlighter-rouge">FULL OUTER JOIN</code> (or simply <code class="highlighter-rouge">FULL JOIN</code>) returns all fields for all rows in
+both <code class="highlighter-rouge">from_item</code>s that meet the join condition.</p>
-groupItem:
- expression
- | '(' expression [, expression ]* ')'
- | HOP '(' expression [, expression ]* ')'
- | TUMBLE '(' expression [, expression ]* ')'
- | SESSION '(' expression [, expression ]* ')'
+<p><code class="highlighter-rouge">FULL</code> indicates that <em>all rows</em> from both <code class="highlighter-rouge">from_item</code>s are returned, even if
+they do not meet the join condition. For streaming jobs, all rows that are
+not late according to default trigger and belonging to the same window
+if there’s non-global window applied.</p>
+<p><code class="highlighter-rouge">OUTER</code> indicates that if a given row from one <code class="highlighter-rouge">from_item</code> does not join to any
+row in the other <code class="highlighter-rouge">from_item</code>, the row will return with NULLs for all columns
+from the other <code class="highlighter-rouge">from_item</code>.</p>
+
+<p>Also see <a href="/documentation/dsls/sql/joins">Joins</a>.</p>
+
+<h3 id="left-outer-join">LEFT [OUTER] JOIN</h3>
+
+<p>The result of a <code class="highlighter-rouge">LEFT OUTER JOIN</code> (or simply <code class="highlighter-rouge">LEFT JOIN</code>) for two <code class="highlighter-rouge">from_item</code>s
+always retains all rows of the left <code class="highlighter-rouge">from_item</code> in the <code class="highlighter-rouge">JOIN</code> clause, even if no
+rows in the right <code class="highlighter-rouge">from_item</code> satisfy the join predicate.</p>
+
+<p><code class="highlighter-rouge">LEFT</code> indicates that all rows from the <em>left</em> <code class="highlighter-rouge">from_item</code> are returned; if a
+given row from the left <code class="highlighter-rouge">from_item</code> does not join to any row in the <em>right</em>
+<code class="highlighter-rouge">from_item</code>, the row will return with NULLs for all columns from the right
+<code class="highlighter-rouge">from_item</code>. Rows from the right <code class="highlighter-rouge">from_item</code> that do not join to any row in the
+left <code class="highlighter-rouge">from_item</code> are discarded.</p>
+
+<h3 id="right-outer-join">RIGHT [OUTER] JOIN</h3>
+
+<p>The result of a <code class="highlighter-rouge">RIGHT OUTER JOIN</code> (or simply <code class="highlighter-rouge">RIGHT JOIN</code>) is similar and
+symmetric to that of <code class="highlighter-rouge">LEFT OUTER JOIN</code>.</p>
+
+<h3 id="on-clause">ON clause</h3>
+
+<p>The <code class="highlighter-rouge">ON</code> clause contains a <code class="highlighter-rouge">bool_expression</code>. A combined row (the result of
+joining two rows) meets the join condition if <code class="highlighter-rouge">bool_expression</code> returns TRUE.</p>
+
+<p>Example:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>SELECT * FROM Roster INNER JOIN PlayerStats
+ON Roster.LastName = PlayerStats.LastName;
+</code></pre>
+</div>
+
+<h3 id="using-clause">USING clause</h3>
+
+<p>The <code class="highlighter-rouge">USING</code> clause requires a <code class="highlighter-rouge">column_list</code> of one or more columns which occur
+in both input tables. It performs an equality comparison on that column, and the
+rows meet the join condition if the equality comparison returns TRUE.</p>
+
+<p>In most cases, a statement with the <code class="highlighter-rouge">USING</code> keyword is equivalent to using the
+<code class="highlighter-rouge">ON</code> keyword. For example, the statement:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>SELECT FirstName
+FROM Roster INNER JOIN PlayerStats
+USING (LastName);
+</code></pre>
+</div>
+
+<p>is equivalent to:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>SELECT FirstName
+FROM Roster INNER JOIN PlayerStats
+ON Roster.LastName = PlayerStats.LastName;
</code></pre>
</div>
+<p>The results from queries with <code class="highlighter-rouge">USING</code> do differ from queries that use <code class="highlighter-rouge">ON</code> when
+you use <code class="highlighter-rouge">SELECT *</code>. To illustrate this, consider the query:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>SELECT * FROM Roster INNER JOIN PlayerStats
+USING (LastName);
+</code></pre>
+</div>
+
+<p>This statement returns the rows from <code class="highlighter-rouge">Roster</code> and <code class="highlighter-rouge">PlayerStats</code> where
+<code class="highlighter-rouge">Roster.LastName</code> is the same as <code class="highlighter-rouge">PlayerStats.LastName</code>. The results include a
+single <code class="highlighter-rouge">LastName</code> column.</p>
+
+<p>By contrast, consider the following query:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>SELECT * FROM Roster INNER JOIN PlayerStats
+ON Roster.LastName = PlayerStats.LastName;
+</code></pre>
+</div>
+
+<p>This statement returns the rows from <code class="highlighter-rouge">Roster</code> and <code class="highlighter-rouge">PlayerStats</code> where
+<code class="highlighter-rouge">Roster.LastName</code> is the same as <code class="highlighter-rouge">PlayerStats.LastName</code>. The results include two
+<code class="highlighter-rouge">LastName</code> columns; one from <code class="highlighter-rouge">Roster</code> and one from <code class="highlighter-rouge">PlayerStats</code>.</p>
+
+<h3 id="sequences-of-joins">Sequences of JOINs</h3>
+
+<p>The <code class="highlighter-rouge">FROM</code> clause can contain multiple <code class="highlighter-rouge">JOIN</code> clauses in sequence.</p>
+
+<p>Example:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>SELECT * FROM a LEFT JOIN b ON TRUE LEFT JOIN c ON TRUE;
+</code></pre>
+</div>
+
+<p>where <code class="highlighter-rouge">a</code>, <code class="highlighter-rouge">b</code>, and <code class="highlighter-rouge">c</code> are any <code class="highlighter-rouge">from_item</code>s. JOINs are bound from left to
+right, but you can insert parentheses to group them in a different order.</p>
+
+<h2 id="where-clause">WHERE clause</h2>
+
+<h3 id="syntax_2">Syntax</h3>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>WHERE bool_expression
+</code></pre>
+</div>
+
+<p>The <code class="highlighter-rouge">WHERE</code> clause filters out rows by evaluating each row against
+<code class="highlighter-rouge">bool_expression</code>, and discards all rows that do not return TRUE (that is, rows
+that return FALSE or NULL).</p>
+
+<p>Example:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>SELECT * FROM Roster
+WHERE SchoolID = 52;
+</code></pre>
+</div>
+
+<p>The <code class="highlighter-rouge">bool_expression</code> can contain multiple sub-conditions.</p>
+
+<p>Example:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>SELECT * FROM Roster
+WHERE LastName LIKE 'Mc%' OR LastName LIKE 'Mac%';
+</code></pre>
+</div>
+
+<p>You cannot reference column aliases from the <code class="highlighter-rouge">SELECT</code> list in the <code class="highlighter-rouge">WHERE</code>
+clause.</p>
+
+<p>Expressions in an <code class="highlighter-rouge">INNER JOIN</code> have an equivalent expression in the <code class="highlighter-rouge">WHERE</code>
+clause. For example, a query using <code class="highlighter-rouge">INNER</code> <code class="highlighter-rouge">JOIN</code> and <code class="highlighter-rouge">ON</code> has an equivalent
+expression using <code class="highlighter-rouge">CROSS JOIN</code> and <code class="highlighter-rouge">WHERE</code>.</p>
+
+<p>Example - this query:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>SELECT * FROM Roster INNER JOIN TeamMascot
+ON Roster.SchoolID = TeamMascot.SchoolID;
+</code></pre>
+</div>
+
+<p>is equivalent to:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>SELECT * FROM Roster CROSS JOIN TeamMascot
+WHERE Roster.SchoolID = TeamMascot.SchoolID;
+</code></pre>
+</div>
+
+<h2 id="group-by-clause">GROUP BY clause</h2>
+
+<p>Also see <a href="/documentation/dsls/sql/windowing-and-triggering/">Windowing & Triggering</a></p>
+
+<h3 id="syntax_3">Syntax</h3>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>GROUP BY { expression [, ...] | ROLLUP ( expression [, ...] ) }
+</code></pre>
+</div>
+
+<p>The <code class="highlighter-rouge">GROUP BY</code> clause groups together rows in a table with non-distinct values
+for the <code class="highlighter-rouge">expression</code> in the <code class="highlighter-rouge">GROUP BY</code> clause. For multiple rows in the source
+table with non-distinct values for <code class="highlighter-rouge">expression</code>, the <code class="highlighter-rouge">GROUP BY</code> clause produces
+a single combined row. <code class="highlighter-rouge">GROUP BY</code> is commonly used when aggregate functions are
+present in the <code class="highlighter-rouge">SELECT</code> list, or to eliminate redundancy in the output.</p>
+
+<p>Example:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>SELECT SUM(PointsScored), LastName
+FROM PlayerStats
+GROUP BY LastName;
+</code></pre>
+</div>
+
+<h2 id="having-clause">HAVING clause</h2>
+
+<h3 id="syntax_4">Syntax</h3>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>HAVING bool_expression
+</code></pre>
+</div>
+
+<p>The <code class="highlighter-rouge">HAVING</code> clause is similar to the <code class="highlighter-rouge">WHERE</code> clause: it filters out rows that
+do not return TRUE when they are evaluated against the <code class="highlighter-rouge">bool_expression</code>.</p>
+
+<p>As with the <code class="highlighter-rouge">WHERE</code> clause, the <code class="highlighter-rouge">bool_expression</code> can be any expression that
+returns a boolean, and can contain multiple sub-conditions.</p>
+
+<p>The <code class="highlighter-rouge">HAVING</code> clause differs from the <code class="highlighter-rouge">WHERE</code> clause in that:</p>
+
+<ul>
+ <li>The <code class="highlighter-rouge">HAVING</code> clause requires <code class="highlighter-rouge">GROUP BY</code> or aggregation to be present in the
+query.</li>
+ <li>The <code class="highlighter-rouge">HAVING</code> clause occurs after <code class="highlighter-rouge">GROUP BY</code> and aggregation.
+This means that the <code class="highlighter-rouge">HAVING</code> clause is evaluated once for every
+aggregated row in the result set. This differs from the <code class="highlighter-rouge">WHERE</code> clause,
+which is evaluated before <code class="highlighter-rouge">GROUP BY</code> and aggregation.</li>
+</ul>
+
+<p>The <code class="highlighter-rouge">HAVING</code> clause can reference columns available via the <code class="highlighter-rouge">FROM</code> clause, as
+well as <code class="highlighter-rouge">SELECT</code> list aliases. Expressions referenced in the <code class="highlighter-rouge">HAVING</code> clause
+must either appear in the <code class="highlighter-rouge">GROUP BY</code> clause or they must be the result of an
+aggregate function:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>SELECT LastName
+FROM Roster
+GROUP BY LastName
+HAVING SUM(PointsScored) > 15;
+</code></pre>
+</div>
+
+<h2 id="set-operators">Set operators</h2>
+
+<h3 id="syntax_6">Syntax</h3>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>UNION { ALL | DISTINCT } | INTERSECT DISTINCT | EXCEPT DISTINCT
+</code></pre>
+</div>
+
+<p>Set operators combine results from two or more input queries into a single
+result set. You must specify <code class="highlighter-rouge">ALL</code> or <code class="highlighter-rouge">DISTINCT</code>; if you specify <code class="highlighter-rouge">ALL</code>, then all
+rows are retained. If <code class="highlighter-rouge">DISTINCT</code> is specified, duplicate rows are discarded.</p>
+
+<p>If a given row R appears exactly m times in the first input query and n times in
+the second input query (m >= 0, n >= 0):</p>
+
+<ul>
+ <li>For <code class="highlighter-rouge">UNION ALL</code>, R appears exactly m + n times in the result.</li>
+ <li>For <code class="highlighter-rouge">UNION DISTINCT</code>, the <code class="highlighter-rouge">DISTINCT</code> is computed after the <code class="highlighter-rouge">UNION</code> is
+computed, so R appears exactly one time.</li>
+ <li>For <code class="highlighter-rouge">INTERSECT DISTINCT</code>, the <code class="highlighter-rouge">DISTINCT</code> is computed after the result above
+is computed.</li>
+ <li>For <code class="highlighter-rouge">EXCEPT DISTINCT</code>, row R appears once in the output if m > 0 and
+n = 0.</li>
+ <li>If there are more than two input queries, the above operations generalize
+and the output is the same as if the inputs were combined incrementally from
+left to right.</li>
+</ul>
+
+<p>The following rules apply:</p>
+
+<ul>
+ <li>For set operations other than <code class="highlighter-rouge">UNION ALL</code>, all column types must support
+equality comparison.</li>
+ <li>The input queries on each side of the operator must return the same number
+of columns.</li>
+ <li>The operators pair the columns returned by each input query according to the
+columns’ positions in their respective <code class="highlighter-rouge">SELECT</code> lists. That is, the first
+column in the first input query is paired with the first column in the
+second input query.</li>
+ <li>The result set always uses the column names from the first input query.</li>
+ <li>The result set always uses the supertypes of input types in corresponding
+columns, so paired columns must also have either the same data type or a
+common supertype.</li>
+ <li>You must use parentheses to separate different set operations; for this
+purpose, set operations such as <code class="highlighter-rouge">UNION ALL</code> and <code class="highlighter-rouge">UNION DISTINCT</code> are
+different. If the statement only repeats the same set operation, parentheses
+are not necessary.</li>
+</ul>
+
+<p>Examples:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>query1 UNION ALL (query2 UNION DISTINCT query3)
+query1 UNION ALL query2 UNION ALL query3
+</code></pre>
+</div>
+
+<p>Invalid:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>query1 UNION ALL query2 UNION DISTINCT query3
+query1 UNION ALL query2 INTERSECT ALL query3; // INVALID.
+</code></pre>
+</div>
+
+<h3 id="union">UNION</h3>
+
+<p>The <code class="highlighter-rouge">UNION</code> operator combines the result sets of two or more input queries by
+pairing columns from the result set of each query and vertically concatenating
+them.</p>
+
+<h3 id="intersect">INTERSECT</h3>
+
+<p>The <code class="highlighter-rouge">INTERSECT</code> operator returns rows that are found in the result sets of both
+the left and right input queries. Unlike <code class="highlighter-rouge">EXCEPT</code>, the positioning of the input
+queries (to the left vs. right of the <code class="highlighter-rouge">INTERSECT</code> operator) does not matter.</p>
+
+<h3 id="except">EXCEPT</h3>
+
+<p>The <code class="highlighter-rouge">EXCEPT</code> operator returns rows from the left input query that are not
+present in the right input query.</p>
+
+<h2 id="limit-clause-and-offset-clause">LIMIT clause and OFFSET clause</h2>
+
+<h3 id="syntax_7">Syntax</h3>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>LIMIT count [ OFFSET skip_rows ]
+</code></pre>
+</div>
+
+<p><code class="highlighter-rouge">LIMIT</code> specifies a non-negative <code class="highlighter-rouge">count</code> of type INTEGER, and no more than <code class="highlighter-rouge">count</code>
+rows will be returned. <code class="highlighter-rouge">LIMIT</code> <code class="highlighter-rouge">0</code> returns 0 rows. If there is a set operation,
+<code class="highlighter-rouge">LIMIT</code> is applied after the set operation is evaluated.</p>
+
+<p><code class="highlighter-rouge">OFFSET</code> specifies a non-negative <code class="highlighter-rouge">skip_rows</code> of type INTEGER, and only rows from
+that offset in the table will be considered.</p>
+
+<p>These clauses accept only literal or parameter values.</p>
+
+<p>The rows that are returned by <code class="highlighter-rouge">LIMIT</code> and <code class="highlighter-rouge">OFFSET</code> is unspecified.</p>
+
+<h2 id="with-clause">WITH clause</h2>
+
+<p>The <code class="highlighter-rouge">WITH</code> clause contains one or more named subqueries which execute every time
+a subsequent <code class="highlighter-rouge">SELECT</code> statement references them. Any clause or subquery can
+reference subqueries you define in the <code class="highlighter-rouge">WITH</code> clause. This includes any <code class="highlighter-rouge">SELECT</code>
+statements on either side of a set operator, such as <code class="highlighter-rouge">UNION</code>.</p>
+
+<p>Example:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>WITH subQ1 AS (SELECT SchoolID FROM Roster),
+ subQ2 AS (SELECT OpponentID FROM PlayerStats)
+SELECT * FROM subQ1
+UNION ALL
+SELECT * FROM subQ2;
+</code></pre>
+</div>
+
+<h2 id="aliases_2">Aliases</h2>
+
+<p>An alias is a temporary name given to a table, column, or expression present in
+a query. You can introduce explicit aliases in the <code class="highlighter-rouge">SELECT</code> list or <code class="highlighter-rouge">FROM</code>
+clause, or Beam will infer an implicit alias for some expressions.
+Expressions with neither an explicit nor implicit alias are anonymous and the
+query cannot reference them by name.</p>
+
+<h3 id="explicit-alias-syntax">Explicit alias syntax</h3>
+
+<p>You can introduce explicit aliases in either the <code class="highlighter-rouge">FROM</code> clause or the <code class="highlighter-rouge">SELECT</code>
+list.</p>
+
+<p>In a <code class="highlighter-rouge">FROM</code> clause, you can introduce explicit aliases for any item, including
+tables, arrays, subqueries, and <code class="highlighter-rouge">UNNEST</code> clauses, using <code class="highlighter-rouge">[AS] alias</code>. The <code class="highlighter-rouge">AS</code>
+keyword is optional.</p>
+
+<p>Example:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>SELECT s.FirstName, s2.SongName
+FROM Singers AS s JOIN Songs AS s2 ON s.SingerID = s2.SingerID;
+</code></pre>
+</div>
+
+<p>You can introduce explicit aliases for any expression in the <code class="highlighter-rouge">SELECT</code> list using
+<code class="highlighter-rouge">[AS] alias</code>. The <code class="highlighter-rouge">AS</code> keyword is optional.</p>
+
+<p>Example:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>SELECT s.FirstName AS name, LOWER(s.FirstName) AS lname
+FROM Singers s;
+</code></pre>
+</div>
+
+<h3 id="explicit-alias-visibility">Explicit alias visibility</h3>
+
+<p>After you introduce an explicit alias in a query, there are restrictions on
+where else in the query you can reference that alias. These restrictions on
+alias visibility are the result of Beam’s name scoping rules.</p>
+
+<h4 id="from-clause-aliases">FROM clause aliases</h4>
+
+<p>Beam processes aliases in a <code class="highlighter-rouge">FROM</code> clause from left to right, and aliases
+are visible only to subsequent <code class="highlighter-rouge">JOIN</code> clauses.</p>
+
+<h3 id="ambiguous-aliases">Ambiguous aliases</h3>
+
+<p>Beam provides an error if a name is ambiguous, meaning it can resolve to
+more than one unique object.</p>
+
+<p>Examples:</p>
+
+<p>This query contains column names that conflict between tables, since both
+<code class="highlighter-rouge">Singers</code> and <code class="highlighter-rouge">Songs</code> have a column named <code class="highlighter-rouge">SingerID</code>:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>SELECT SingerID
+FROM Singers, Songs;
+</code></pre>
+</div>
+
+<h3 id="implicit-aliases">Implicit aliases</h3>
+
+<p>In the <code class="highlighter-rouge">SELECT</code> list, if there is an expression that does not have an explicit
+alias, Beam assigns an implicit alias according to the following rules.
+There can be multiple columns with the same alias in the <code class="highlighter-rouge">SELECT</code> list.</p>
+
+<ul>
+ <li>For identifiers, the alias is the identifier. For example, <code class="highlighter-rouge">SELECT abc</code>
+implies <code class="highlighter-rouge">AS abc</code>.</li>
+ <li>For path expressions, the alias is the last identifier in the path. For
+example, <code class="highlighter-rouge">SELECT abc.def.ghi</code> implies <code class="highlighter-rouge">AS ghi</code>.</li>
+ <li>For field access using the “dot” member field access operator, the alias is
+the field name. For example, <code class="highlighter-rouge">SELECT (struct_function()).fname</code> implies <code class="highlighter-rouge">AS
+fname</code>.</li>
+</ul>
+
+<p>In all other cases, there is no implicit alias, so the column is anonymous and
+cannot be referenced by name. The data from that column will still be returned
+and the displayed query results may have a generated label for that column, but
+the label cannot be used like an alias.</p>
+
+<p>In a <code class="highlighter-rouge">FROM</code> clause, <code class="highlighter-rouge">from_item</code>s are not required to have an alias. The
+following rules apply:</p>
+
+<p>If there is an expression that does not have an explicit alias, Beam assigns
+an implicit alias in these cases:</p>
+
+<ul>
+ <li>For identifiers, the alias is the identifier. For example, <code class="highlighter-rouge">FROM abc</code>
+implies <code class="highlighter-rouge">AS abc</code>.</li>
+ <li>For path expressions, the alias is the last identifier in the path. For
+example, <code class="highlighter-rouge">FROM abc.def.ghi</code> implies <code class="highlighter-rouge">AS ghi</code></li>
+</ul>
+
+<p>Table subqueries do not have implicit aliases.</p>
+
+<p><code class="highlighter-rouge">FROM UNNEST(x)</code> does not have an implicit alias.</p>
+
+<blockquote>
+ <p>Portions of this page are modifications based on
+<a href="https://cloud.google.com/bigquery/docs/reference/standard-sql/query-syntax">work</a>
+created and
+<a href="https://developers.google.com/terms/site-policies">shared by Google</a>
+and used according to terms described in the <a href="http://creativecommons.org/licenses/by/3.0/">Creative Commons 3.0
+Attribution License</a>.</p>
+</blockquote>
</div>
</div>
diff --git a/website/generated-content/documentation/dsls/sql/set/index.html b/website/generated-content/documentation/dsls/sql/set/index.html
index 3de979c..bf8dccf 100644
--- a/website/generated-content/documentation/dsls/sql/set/index.html
+++ b/website/generated-content/documentation/dsls/sql/set/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.7.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.7.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -190,7 +190,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/data-types/">Data types</a></li>
<li><a href="/documentation/dsls/sql/lexical/">Lexical structure</a></li>
- <li><a href="/documentation/dsls/sql/create-table/">CREATE TABLE</a></li>
+ <li><a href="/documentation/dsls/sql/create-external-table/">CREATE EXTERNAL TABLE</a></li>
<li><a href="/documentation/dsls/sql/select/">SELECT</a></li>
<li><a href="/documentation/dsls/sql/windowing-and-triggering/">Windowing & Triggering</a></li>
<li><a href="/documentation/dsls/sql/joins/">Joins</a></li>
diff --git a/website/generated-content/documentation/dsls/sql/shell/index.html b/website/generated-content/documentation/dsls/sql/shell/index.html
index 4ebe5a2..3b16e16 100644
--- a/website/generated-content/documentation/dsls/sql/shell/index.html
+++ b/website/generated-content/documentation/dsls/sql/shell/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.7.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.7.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -190,7 +190,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/data-types/">Data types</a></li>
<li><a href="/documentation/dsls/sql/lexical/">Lexical structure</a></li>
- <li><a href="/documentation/dsls/sql/create-table/">CREATE TABLE</a></li>
+ <li><a href="/documentation/dsls/sql/create-external-table/">CREATE EXTERNAL TABLE</a></li>
<li><a href="/documentation/dsls/sql/select/">SELECT</a></li>
<li><a href="/documentation/dsls/sql/windowing-and-triggering/">Windowing & Triggering</a></li>
<li><a href="/documentation/dsls/sql/joins/">Joins</a></li>
@@ -292,17 +292,17 @@ limitations under the License.
<h2 id="declaring-tables">Declaring Tables</h2>
-<p>Before reading data from a source or writing data to a destination, you must declare a virtual table using the <code class="highlighter-rouge">CREATE TABLE</code> statement. For example, if you have a local CSV file <code class="highlighter-rouge">"test-file.csv"</code> in the current folder, you can create a table with the following statement:</p>
+<p>Before reading data from a source or writing data to a destination, you must declare a virtual table using the <code class="highlighter-rouge">CREATE EXTERNAL TABLE</code> statement. For example, if you have a local CSV file <code class="highlighter-rouge">"test-file.csv"</code> in the current folder, you can create a table with the following statement:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>0: BeamSQL> CREATE TABLE csv_file (field1 VARCHAR, field2 INTEGER) TYPE text LOCATION 'test-file.csv';
+<div class="highlighter-rouge"><pre class="highlight"><code>0: BeamSQL> CREATE EXTERNAL TABLE csv_file (field1 VARCHAR, field2 INTEGER) TYPE text LOCATION 'test-file.csv';
No rows affected (0.042 seconds)
</code></pre>
</div>
-<p>The <code class="highlighter-rouge">CREATE TABLE</code> statement registers the CSV file as a table in Beam SQL and specifies the table’s schema. This statement does not directly create a persistent physical table; it only describes the source/sink to Beam SQL so that you can use the table in the queries that read data and write data.</p>
+<p>The <code class="highlighter-rouge">CREATE EXTERNAL TABLE</code> statement registers the CSV file as a table in Beam SQL and specifies the table’s schema. This statement does not directly create a persistent physical table; it only describes the source/sink to Beam SQL so that you can use the table in the queries that read data and write data.</p>
-<p><em>For more information about <code class="highlighter-rouge">CREATE TABLE</code> syntax and supported table types, see the <a href="/documentation/dsls/sql/create-table/">CREATE TABLE reference page</a>.</em></p>
+<p><em>For more information about <code class="highlighter-rouge">CREATE EXTERNAL TABLE</code> syntax and supported table types, see the <a href="/documentation/dsls/sql/create-external-table/">CREATE EXTERNAL TABLE reference page</a>.</em></p>
<h2 id="reading-and-writing-data">Reading and Writing Data</h2>
diff --git a/website/generated-content/documentation/dsls/sql/statements/create-table/index.html b/website/generated-content/documentation/dsls/sql/statements/create-table/index.html
index e603c02..d1ba0da 100644
--- a/website/generated-content/documentation/dsls/sql/statements/create-table/index.html
+++ b/website/generated-content/documentation/dsls/sql/statements/create-table/index.html
@@ -2,9 +2,9 @@
<html lang="en-US">
<meta charset="utf-8">
<title>Redirecting…</title>
-<link rel="canonical" href="https://beam.apache.org/documentation/dsls/sql/create-table/">
-<meta http-equiv="refresh" content="0; url=https://beam.apache.org/documentation/dsls/sql/create-table/">
+<link rel="canonical" href="https://beam.apache.org/documentation/dsls/sql/create-external-table/">
+<meta http-equiv="refresh" content="0; url=https://beam.apache.org/documentation/dsls/sql/create-external-table/">
<h1>Redirecting…</h1>
-<a href="https://beam.apache.org/documentation/dsls/sql/create-table/">Click here if you are not redirected.</a>
-<script>location="https://beam.apache.org/documentation/dsls/sql/create-table/"</script>
+<a href="https://beam.apache.org/documentation/dsls/sql/create-external-table/">Click here if you are not redirected.</a>
+<script>location="https://beam.apache.org/documentation/dsls/sql/create-external-table/"</script>
</html>
diff --git a/website/generated-content/documentation/dsls/sql/user-defined-functions/index.html b/website/generated-content/documentation/dsls/sql/user-defined-functions/index.html
index 03f754f..9e92328 100644
--- a/website/generated-content/documentation/dsls/sql/user-defined-functions/index.html
+++ b/website/generated-content/documentation/dsls/sql/user-defined-functions/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.7.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.7.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -190,7 +190,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/data-types/">Data types</a></li>
<li><a href="/documentation/dsls/sql/lexical/">Lexical structure</a></li>
- <li><a href="/documentation/dsls/sql/create-table/">CREATE TABLE</a></li>
+ <li><a href="/documentation/dsls/sql/create-external-table/">CREATE EXTERNAL TABLE</a></li>
<li><a href="/documentation/dsls/sql/select/">SELECT</a></li>
<li><a href="/documentation/dsls/sql/windowing-and-triggering/">Windowing & Triggering</a></li>
<li><a href="/documentation/dsls/sql/joins/">Joins</a></li>
diff --git a/website/generated-content/documentation/dsls/sql/walkthrough/index.html b/website/generated-content/documentation/dsls/sql/walkthrough/index.html
index 9176acb..06251a7 100644
--- a/website/generated-content/documentation/dsls/sql/walkthrough/index.html
+++ b/website/generated-content/documentation/dsls/sql/walkthrough/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.7.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.7.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -190,7 +190,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/data-types/">Data types</a></li>
<li><a href="/documentation/dsls/sql/lexical/">Lexical structure</a></li>
- <li><a href="/documentation/dsls/sql/create-table/">CREATE TABLE</a></li>
+ <li><a href="/documentation/dsls/sql/create-external-table/">CREATE EXTERNAL TABLE</a></li>
<li><a href="/documentation/dsls/sql/select/">SELECT</a></li>
<li><a href="/documentation/dsls/sql/windowing-and-triggering/">Windowing & Triggering</a></li>
<li><a href="/documentation/dsls/sql/joins/">Joins</a></li>
@@ -254,8 +254,8 @@ limitations under the License.
<p>Before applying a SQL query to a <code class="highlighter-rouge">PCollection</code>, the data in the collection must
be in <code class="highlighter-rouge">Row</code> format. A <code class="highlighter-rouge">Row</code> represents a single, immutable record in a Beam SQL
<code class="highlighter-rouge">PCollection</code>. The names and types of the fields/columns in the row are defined
-by its associated <a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/schemas/Schema.html">Schema</a>.
-You can use the <a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/schemas/Schema.html">Schema.builder()</a> to create
+by its associated <a href="https://beam.apache.org/releases/javadoc/2.7.0/index.html?org/apache/beam/sdk/schemas/Schema.html">Schema</a>.
+You can use the <a href="https://beam.apache.org/releases/javadoc/2.7.0/index.html?org/apache/beam/sdk/schemas/Schema.html">Schema.builder()</a> to create
<code class="highlighter-rouge">Schemas</code>. See <a href="/documentation/dsls/sql/data-types">Data
Types</a> for more details on supported primitive data types.</p>
@@ -338,7 +338,7 @@ Types</a> for more details on supported primitive data types.</p>
<h2 id="sqltransform">SqlTransform</h2>
-<p><a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/extensions/sql/SqlTransform.html"><code class="highlighter-rouge">SqlTransform.query(queryString)</code></a> method is the only API to create a <code class="highlighter-rouge">PTransform</code>
+<p><a href="https://beam.apache.org/releases/javadoc/2.7.0/index.html?org/apache/beam/sdk/extensions/sql/SqlTransform.html"><code class="highlighter-rouge">SqlTransform.query(queryString)</code></a> method is the only API to create a <code class="highlighter-rouge">PTransform</code>
from a string representation of the SQL query. You can apply this <code class="highlighter-rouge">PTransform</code>
to either a single <code class="highlighter-rouge">PCollection</code> or a <code class="highlighter-rouge">PCollectionTuple</code> which holds multiple
<code class="highlighter-rouge">PCollections</code>:</p>
diff --git a/website/generated-content/documentation/dsls/sql/windowing-and-triggering/index.html b/website/generated-content/documentation/dsls/sql/windowing-and-triggering/index.html
index 89dfb7f..1c925c1 100644
--- a/website/generated-content/documentation/dsls/sql/windowing-and-triggering/index.html
+++ b/website/generated-content/documentation/dsls/sql/windowing-and-triggering/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.7.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.7.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -190,7 +190,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/data-types/">Data types</a></li>
<li><a href="/documentation/dsls/sql/lexical/">Lexical structure</a></li>
- <li><a href="/documentation/dsls/sql/create-table/">CREATE TABLE</a></li>
+ <li><a href="/documentation/dsls/sql/create-external-table/">CREATE EXTERNAL TABLE</a></li>
<li><a href="/documentation/dsls/sql/select/">SELECT</a></li>
<li><a href="/documentation/dsls/sql/windowing-and-triggering/">Windowing & Triggering</a></li>
<li><a href="/documentation/dsls/sql/joins/">Joins</a></li>
diff --git a/website/generated-content/documentation/io/testing/index.html b/website/generated-content/documentation/io/testing/index.html
index 78a9192..947002d 100644
--- a/website/generated-content/documentation/io/testing/index.html
+++ b/website/generated-content/documentation/io/testing/index.html
@@ -385,7 +385,7 @@ limitations under the License.
<h3 id="implementing-unit-tests">Implementing unit tests</h3>
-<p>A general guide to writing Unit Tests for all transforms can be found in the <a href="https://beam.apache.org/contribute/ptransform-style-guide/#testing">PTransform Style Guide</a>. We have expanded on a few important points below.</p>
+<p>A general guide to writing Unit Tests for all transforms can be found in the <a href="/contribute/ptransform-style-guide/#testing">PTransform Style Guide</a>. We have expanded on a few important points below.</p>
<p>If you are using the <code class="highlighter-rouge">Source</code> API, make sure to exhaustively unit-test your code. A minor implementation error can lead to data corruption or data loss (such as skipping or duplicating records) that can be hard for your users to detect. Also look into using <span class="language-java"><code class="highlighter-rouge">SourceTestUtils</code></span><span class="language-py"><code class="highlighter-rouge">source_test_utils</code></span> - it is a key p [...]
@@ -453,12 +453,12 @@ limitations under the License.
<p>You won’t need to invoke PerfKit Benchmarker directly. Run <code class="highlighter-rouge">./gradlew performanceTest</code> task in project’s root directory, passing kubernetes scripts of your choice (located in .test_infra/kubernetes directory). It will setup PerfKitBenchmarker for you.</p>
-<p>Example run with the <a href="https://beam.apache.org/documentation/runners/direct/">Direct</a> runner:</p>
+<p>Example run with the <a href="/documentation/runners/direct/">Direct</a> runner:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>./gradlew performanceTest -DpkbLocation="/Users/me/PerfKitBenchmarker/pkb.py" -DintegrationTestPipelineOptions='["--numberOfRecords=1000"]' -DitModule=sdks/java/io/jdbc/ -DintegrationTest=org.apache.beam.sdk.io.jdbc.JdbcIOIT -DkubernetesScripts="/Users/me/beam/.test-infra/kubernetes/postgres/postgres-service-for-local-dev.yml" -DbeamITOptions="/Users/me/beam/.test-infra/kubernetes/postgres/pkb-config-local.yml" -DintegrationTest [...]
</code></pre>
</div>
-<p>Example run with the <a href="https://beam.apache.org/documentation/runners/dataflow/">Google Cloud Dataflow</a> runner:</p>
+<p>Example run with the <a href="/documentation/runners/dataflow/">Google Cloud Dataflow</a> runner:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>./gradlew performanceTest -DpkbLocation="/Users/me/PerfKitBenchmarker/pkb.py" -DintegrationTestPipelineOptions='["--numberOfRecords=1000", "--project=GOOGLE_CLOUD_PROJECT", "--tempRoot=GOOGLE_STORAGE_BUCKET"]' -DitModule=sdks/java/io/jdbc/ -DintegrationTest=org.apache.beam.sdk.io.jdbc.JdbcIOIT -DkubernetesScripts="/Users/me/beam/.test-infra/kubernetes/postgres/postgres-service-for-local-dev.yml" -DbeamITOptions="/Users/me/beam/. [...]
</code></pre>
</div>
diff --git a/website/generated-content/documentation/pipelines/test-your-pipeline/index.html b/website/generated-content/documentation/pipelines/test-your-pipeline/index.html
index 72cdc2d..80a98ed 100644
--- a/website/generated-content/documentation/pipelines/test-your-pipeline/index.html
+++ b/website/generated-content/documentation/pipelines/test-your-pipeline/index.html
@@ -472,7 +472,7 @@ for the <code class="highlighter-rouge">DoFnTester</code> as follows:</p>
<p>You can use the <code class="highlighter-rouge">Create</code> transform to create a <code class="highlighter-rouge">PCollection</code> out of a standard in-memory collection class, such as Java <code class="highlighter-rouge">List</code>. See <a href="/documentation/programming-guide/#creating-a-pcollection">Creating a PCollection</a> for more information.</p>
<h3 id="passert">PAssert</h3>
-<p><a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/testing/PAssert.html">PAssert</a> is a class included in the Beam Java SDK that is an assertion on the contents of a <code class="highlighter-rouge">PCollection</code>. You can use <code class="highlighter-rouge">PAssert</code>to verify that a <code class="highlighter-rouge">PCollection</code> contains a specific set of expected elements.</p>
+<p><a href="https://beam.apache.org/releases/javadoc/2.7.0/index.html?org/apache/beam/sdk/testing/PAssert.html">PAssert</a> is a class included in the Beam Java SDK that is an assertion on the contents of a <code class="highlighter-rouge">PCollection</code>. You can use <code class="highlighter-rouge">PAssert</code>to verify that a <code class="highlighter-rouge">PCollection</code> contains a specific set of expected elements.</p>
<p>For a given <code class="highlighter-rouge">PCollection</code>, you can use <code class="highlighter-rouge">PAssert</code> to verify the contents as follows:</p>
@@ -498,7 +498,7 @@ for the <code class="highlighter-rouge">DoFnTester</code> as follows:</p>
</code></pre>
</div>
-<p>For more information on how these classes work, see the <a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/testing/package-summary.html">org.apache.beam.sdk.testing</a> package documentation.</p>
+<p>For more information on how these classes work, see the <a href="https://beam.apache.org/releases/javadoc/2.7.0/index.html?org/apache/beam/sdk/testing/package-summary.html">org.apache.beam.sdk.testing</a> package documentation.</p>
<h3 id="an-example-test-for-a-composite-transform">An Example Test for a Composite Transform</h3>
diff --git a/website/generated-content/documentation/programming-guide/index.html b/website/generated-content/documentation/programming-guide/index.html
index c9b8345..ed302bc 100644
--- a/website/generated-content/documentation/programming-guide/index.html
+++ b/website/generated-content/documentation/programming-guide/index.html
@@ -420,7 +420,7 @@ asynchronous “job” (or equivalent) on that back-end.</p>
<p>The <code class="highlighter-rouge">Pipeline</code> abstraction encapsulates all the data and steps in your data
processing task. Your Beam driver program typically starts by constructing a
-<span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/Pipeline.html">Pipeline</a></span>
+<span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.7.0/index.html?org/apache/beam/sdk/Pipeline.html">Pipeline</a></span>
<span class="language-py"><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/pipeline.py">Pipeline</a></span>
object, and then using that object as the basis for creating the pipeline’s data
sets as <code class="highlighter-rouge">PCollection</code>s and its operations as <code class="highlighter-rouge">Transform</code>s.</p>
@@ -570,7 +570,7 @@ with <code class="highlighter-rouge">PipelineOptionsFactory</code>:</p>
<h2 id="pcollections">3. PCollections</h2>
-<p>The <span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/values/PCollection.html">PCollection</a></span>
+<p>The <span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.7.0/index.html?org/apache/beam/sdk/values/PCollection.html">PCollection</a></span>
<span class="language-py"><code class="highlighter-rouge">PCollection</code></span> abstraction represents a
potentially distributed, multi-element data set. You can think of a
<code class="highlighter-rouge">PCollection</code> as “pipeline” data; Beam transforms use <code class="highlighter-rouge">PCollection</code> objects as
@@ -1380,7 +1380,7 @@ and format data from each collection.</p>
<h4 id="combine">4.2.4. Combine</h4>
-<p><span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/transforms/Combine.html"><code class="highlighter-rouge">Combine</code></a></span>
+<p><span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.7.0/index.html?org/apache/beam/sdk/transforms/Combine.html"><code class="highlighter-rouge">Combine</code></a></span>
<span class="language-py"><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/core.py"><code class="highlighter-rouge">Combine</code></a></span>
is a Beam transform for combining collections of elements or values in your
data. <code class="highlighter-rouge">Combine</code> has variants that work on entire <code class="highlighter-rouge">PCollection</code>s, and some that
@@ -1639,7 +1639,7 @@ PerKey must be an associative reduction function or a subclass of <code class="h
<h4 id="flatten">4.2.5. Flatten</h4>
-<p><span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/transforms/Flatten.html"><code class="highlighter-rouge">Flatten</code></a></span>
+<p><span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.7.0/index.html?org/apache/beam/sdk/transforms/Flatten.html"><code class="highlighter-rouge">Flatten</code></a></span>
<span class="language-py"><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/core.py"><code class="highlighter-rouge">Flatten</code></a></span> and
is a Beam transform for <code class="highlighter-rouge">PCollection</code> objects that store the same data type.
<code class="highlighter-rouge">Flatten</code> merges multiple <code class="highlighter-rouge">PCollection</code> objects into a single logical
@@ -1690,7 +1690,7 @@ pipeline is constructed.</p>
<h4 id="partition">4.2.6. Partition</h4>
-<p><span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/transforms/Partition.html"><code class="highlighter-rouge">Partition</code></a></span>
+<p><span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.7.0/index.html?org/apache/beam/sdk/transforms/Partition.html"><code class="highlighter-rouge">Partition</code></a></span>
<span class="language-py"><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/core.py"><code class="highlighter-rouge">Partition</code></a></span>
is a Beam transform for <code class="highlighter-rouge">PCollection</code> objects that store the same data
type. <code class="highlighter-rouge">Partition</code> splits a single <code class="highlighter-rouge">PCollection</code> into a fixed number of smaller
@@ -2134,8 +2134,8 @@ transform can make your code more modular and easier to understand.</p>
<p>The Beam SDK comes packed with many useful composite transforms. See the API
reference pages for a list of transforms:</p>
<ul>
- <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/transforms/package-summary.html">Pre-written Beam transforms for Java</a></li>
- <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/apache_beam.transforms.html">Pre-written Beam transforms for Python</a></li>
+ <li><a href="https://beam.apache.org/releases/javadoc/2.7.0/index.html?org/apache/beam/sdk/transforms/package-summary.html">Pre-written Beam transforms for Java</a></li>
+ <li><a href="https://beam.apache.org/releases/pydoc/2.7.0/apache_beam.transforms.html">Pre-written Beam transforms for Python</a></li>
</ul>
<h4 id="composite-transform-example">4.6.1. An example composite transform</h4>
@@ -2722,7 +2722,7 @@ all the elements are by default part of a single, global window.</p>
<p>To use windowing with fixed data sets, you can assign your own timestamps to
each element. To assign timestamps to elements, use a <code class="highlighter-rouge">ParDo</code> transform with a
<code class="highlighter-rouge">DoFn</code> that outputs each element with a new timestamp (for example, the
-<a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/transforms/WithTimestamps.html">WithTimestamps</a>
+<a href="https://beam.apache.org/releases/javadoc/2.7.0/index.html?org/apache/beam/sdk/transforms/WithTimestamps.html">WithTimestamps</a>
transform in the Beam SDK for Java).</p>
<p>To illustrate how windowing with a bounded <code class="highlighter-rouge">PCollection</code> can affect how your
diff --git a/website/generated-content/documentation/runners/dataflow/index.html b/website/generated-content/documentation/runners/dataflow/index.html
index b742b8d..6b49d31 100644
--- a/website/generated-content/documentation/runners/dataflow/index.html
+++ b/website/generated-content/documentation/runners/dataflow/index.html
@@ -245,7 +245,7 @@ Pub/Sub, or Cloud Datastore) if you use them in your pipeline code.</li>
<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="o"><</span><span class="n">dependency</span><span class="o">></span>
<span class="o"><</span><span class="n">groupId</span><span class="o">></span><span class="n">org</span><span class="o">.</span><span class="na">apache</span><span class="o">.</span><span class="na">beam</span><span class="o"></</span><span class="n">groupId</span><span class="o">></span>
<span class="o"><</span><span class="n">artifactId</span><span class="o">></span><span class="n">beam</span><span class="o">-</span><span class="n">runners</span><span class="o">-</span><span class="n">google</span><span class="o">-</span><span class="n">cloud</span><span class="o">-</span><span class="n">dataflow</span><span class="o">-</span><span class="n">java</span><span class="o"></</span><span class="n">artifactId</span><span class="o">></span>
- <span class="o"><</span><span class="n">version</span><span class="o">></span><span class="mf">2.6</span><span class="o">.</span><span class="mi">0</span><span class="o"></</span><span class="n">version</span><span class="o">></span>
+ <span class="o"><</span><span class="n">version</span><span class="o">></span><span class="mf">2.7</span><span class="o">.</span><span class="mi">0</span><span class="o"></</span><span class="n">version</span><span class="o">></span>
<span class="o"><</span><span class="n">scope</span><span class="o">></span><span class="n">runtime</span><span class="o"></</span><span class="n">scope</span><span class="o">></span>
<span class="o"></</span><span class="n">dependency</span><span class="o">></span>
</code></pre>
@@ -383,8 +383,8 @@ Pub/Sub, or Cloud Datastore) if you use them in your pipeline code.</li>
</table>
<p>See the reference documentation for the
-<span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/runners/dataflow/options/DataflowPipelineOptions.html">DataflowPipelineOptions</a></span>
-<span class="language-py"><a href="https://beam.apache.org/releases/pydoc/2.6.0/apache_beam.options.pipeline_options.html#apache_beam.options.pipeline_options.PipelineOptions"><code class="highlighter-rouge">PipelineOptions</code></a></span>
+<span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.7.0/index.html?org/apache/beam/runners/dataflow/options/DataflowPipelineOptions.html">DataflowPipelineOptions</a></span>
+<span class="language-py"><a href="https://beam.apache.org/releases/pydoc/2.7.0/apache_beam.options.pipeline_options.html#apache_beam.options.pipeline_options.PipelineOptions"><code class="highlighter-rouge">PipelineOptions</code></a></span>
interface (and any subinterfaces) for additional pipeline configuration options.</p>
<h2 id="additional-info">Additional information and caveats</h2>
diff --git a/website/generated-content/documentation/runners/direct/index.html b/website/generated-content/documentation/runners/direct/index.html
index 7a6ddc6..c552933 100644
--- a/website/generated-content/documentation/runners/direct/index.html
+++ b/website/generated-content/documentation/runners/direct/index.html
@@ -221,11 +221,11 @@ limitations under the License.
<p>Here are some resources with information about how to test your pipelines.</p>
<ul>
<!-- Java specific links -->
- <li class="language-java"><a href="/blog/2016/10/20/test-stream.html">Testing Unbounded Pipelines in Apache Beam</a> talks about the use of Java classes <a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/testing/PAssert.html">PAssert</a> and <a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/testing/TestStream.html">TestStream</a> to test your pipelines.</li>
- <li class="language-java">The <a href="/get-started/wordcount-example/#testing-your-pipeline-with-asserts">Apache Beam WordCount Walkthrough</a> contains an example of logging and testing a pipeline with <a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/testing/PAssert.html">PAssert</a>.</li>
+ <li class="language-java"><a href="/blog/2016/10/20/test-stream.html">Testing Unbounded Pipelines in Apache Beam</a> talks about the use of Java classes <a href="https://beam.apache.org/releases/javadoc/2.7.0/index.html?org/apache/beam/sdk/testing/PAssert.html">PAssert</a> and <a href="https://beam.apache.org/releases/javadoc/2.7.0/index.html?org/apache/beam/sdk/testing/TestStream.html">TestStream</a> to test your pipelines.</li>
+ <li class="language-java">The <a href="/get-started/wordcount-example/#testing-your-pipeline-with-asserts">Apache Beam WordCount Walkthrough</a> contains an example of logging and testing a pipeline with <a href="https://beam.apache.org/releases/javadoc/2.7.0/index.html?org/apache/beam/sdk/testing/PAssert.html">PAssert</a>.</li>
<!-- Python specific links -->
- <li class="language-py">The <a href="/get-started/wordcount-example/#testing-your-pipeline-with-asserts">Apache Beam WordCount Walkthrough</a> contains an example of logging and testing a pipeline with <a href="https://beam.apache.org/releases/pydoc/2.6.0/apache_beam.testing.util.html#apache_beam.testing.util.assert_that">assert_that</a>.</li>
+ <li class="language-py">The <a href="/get-started/wordcount-example/#testing-your-pipeline-with-asserts">Apache Beam WordCount Walkthrough</a> contains an example of logging and testing a pipeline with <a href="https://beam.apache.org/releases/pydoc/2.7.0/apache_beam.testing.util.html#apache_beam.testing.util.assert_that">assert_that</a>.</li>
</ul>
<h2 id="direct-runner-prerequisites-and-setup">Direct Runner prerequisites and setup</h2>
@@ -236,7 +236,7 @@ limitations under the License.
<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="o"><</span><span class="n">dependency</span><span class="o">></span>
<span class="o"><</span><span class="n">groupId</span><span class="o">></span><span class="n">org</span><span class="o">.</span><span class="na">apache</span><span class="o">.</span><span class="na">beam</span><span class="o"></</span><span class="n">groupId</span><span class="o">></span>
<span class="o"><</span><span class="n">artifactId</span><span class="o">></span><span class="n">beam</span><span class="o">-</span><span class="n">runners</span><span class="o">-</span><span class="n">direct</span><span class="o">-</span><span class="n">java</span><span class="o"></</span><span class="n">artifactId</span><span class="o">></span>
- <span class="o"><</span><span class="n">version</span><span class="o">></span><span class="mf">2.6</span><span class="o">.</span><span class="mi">0</span><span class="o"></</span><span class="n">version</span><span class="o">></span>
+ <span class="o"><</span><span class="n">version</span><span class="o">></span><span class="mf">2.7</span><span class="o">.</span><span class="mi">0</span><span class="o"></</span><span class="n">version</span><span class="o">></span>
<span class="o"><</span><span class="n">scope</span><span class="o">></span><span class="n">runtime</span><span class="o"></</span><span class="n">scope</span><span class="o">></span>
<span class="o"></</span><span class="n">dependency</span><span class="o">></span>
</code></pre>
@@ -249,15 +249,15 @@ limitations under the License.
<p>When executing your pipeline from the command-line, set <code class="highlighter-rouge">runner</code> to <code class="highlighter-rouge">direct</code> or <code class="highlighter-rouge">DirectRunner</code>. The default values for the other pipeline options are generally sufficient.</p>
<p>See the reference documentation for the
-<span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/runners/direct/DirectOptions.html"><code class="highlighter-rouge">DirectOptions</code></a></span>
-<span class="language-py"><a href="https://beam.apache.org/releases/pydoc/2.6.0/apache_beam.options.pipeline_options.html#apache_beam.options.pipeline_options.DirectOptions"><code class="highlighter-rouge">DirectOptions</code></a></span>
+<span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.7.0/index.html?org/apache/beam/runners/direct/DirectOptions.html"><code class="highlighter-rouge">DirectOptions</code></a></span>
+<span class="language-py"><a href="https://beam.apache.org/releases/pydoc/2.7.0/apache_beam.options.pipeline_options.html#apache_beam.options.pipeline_options.DirectOptions"><code class="highlighter-rouge">DirectOptions</code></a></span>
interface for defaults and additional pipeline configuration options.</p>
<h2 id="additional-information-and-caveats">Additional information and caveats</h2>
<h3 id="memory-considerations">Memory considerations</h3>
-<p>Local execution is limited by the memory available in your local environment. It is highly recommended that you run your pipeline with data sets small enough to fit in local memory. You can create a small in-memory data set using a <span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/sdk/transforms/Create.html"><code class="highlighter-rouge">Create</code></a></span><span class="language-py"><a href="https://github.com/apache/b [...]
+<p>Local execution is limited by the memory available in your local environment. It is highly recommended that you run your pipeline with data sets small enough to fit in local memory. You can create a small in-memory data set using a <span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.7.0/index.html?org/apache/beam/sdk/transforms/Create.html"><code class="highlighter-rouge">Create</code></a></span><span class="language-py"><a href="https://github.com/apache/b [...]
<h3 id="streaming-execution">Streaming execution</h3>
diff --git a/website/generated-content/documentation/runners/flink/index.html b/website/generated-content/documentation/runners/flink/index.html
index 188c987..4e4b355 100644
--- a/website/generated-content/documentation/runners/flink/index.html
+++ b/website/generated-content/documentation/runners/flink/index.html
@@ -273,7 +273,7 @@ limitations under the License.
<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="o"><</span><span class="n">dependency</span><span class="o">></span>
<span class="o"><</span><span class="n">groupId</span><span class="o">></span><span class="n">org</span><span class="o">.</span><span class="na">apache</span><span class="o">.</span><span class="na">beam</span><span class="o"></</span><span class="n">groupId</span><span class="o">></span>
<span class="o"><</span><span class="n">artifactId</span><span class="o">></span><span class="n">beam</span><span class="o">-</span><span class="n">runners</span><span class="o">-</span><span class="n">flink_2</span><span class="o">.</span><span class="mi">11</span><span class="o"></</span><span class="n">artifactId</span><span class="o">></span>
- <span class="o"><</span><span class="n">version</span><span class="o">></span><span class="mf">2.6</span><span class="o">.</span><span class="mi">0</span><span class="o"></</span><span class="n">version</span><span class="o">></span>
+ <span class="o"><</span><span class="n">version</span><span class="o">></span><span class="mf">2.7</span><span class="o">.</span><span class="mi">0</span><span class="o"></</span><span class="n">version</span><span class="o">></span>
<span class="o"></</span><span class="n">dependency</span><span class="o">></span>
</code></pre>
</div>
@@ -360,7 +360,7 @@ limitations under the License.
</tr>
</table>
-<p>See the reference documentation for the <span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.6.0/index.html?org/apache/beam/runners/flink/FlinkPipelineOptions.html">FlinkPipelineOptions</a></span><span class="language-py"><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py">PipelineOptions</a></span> interface (and its subinterfaces) for the complete list of pipeline configuration options.</p>
+<p>See the reference documentation for the <span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.7.0/index.html?org/apache/beam/runners/flink/FlinkPipelineOptions.html">FlinkPipelineOptions</a></span><span class="language-py"><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py">PipelineOptions</a></span> interface (and its subinterfaces) for the complete list of pipeline configuration options.</p>
<h2 id="additional-information-and-caveats">Additional information and caveats</h2>
diff --git a/website/generated-content/documentation/runners/gearpump/index.html b/website/generated-content/documentation/runners/gearpump/index.html
index 5b9f9c1..60a85b7 100644
--- a/website/generated-content/documentation/runners/gearpump/index.html
+++ b/website/generated-content/documentation/runners/gearpump/index.html
@@ -214,7 +214,7 @@ And your Beam application should also pack Beam SDK explicitly and here is a sni
<span class="o"><</span><span class="n">dependency</span><span class="o">></span>
<span class="o"><</span><span class="n">groupId</span><span class="o">></span><span class="n">org</span><span class="o">.</span><span class="na">apache</span><span class="o">.</span><span class="na">beam</span><span class="o"></</span><span class="n">groupId</span><span class="o">></span>
<span class="o"><</span><span class="n">artifactId</span><span class="o">></span><span class="n">beam</span><span class="o">-</span><span class="n">runners</span><span class="o">-</span><span class="n">gearpump</span><span class="o"></</span><span class="n">artifactId</span><span class="o">></span>
- <span class="o"><</span><span class="n">version</span><span class="o">></span><span class="mf">2.6</span><span class="o">.</span><span class="mi">0</span><span class="o"></</span><span class="n">version</span><span class="o">></span>
+ <span class="o"><</span><span class="n">version</span><span class="o">></span><span class="mf">2.7</span><span class="o">.</span><span class="mi">0</span><span class="o"></</span><span class="n">version</span><span class="o">></span>
<span class="o"></</span><span class="n">dependency</span><span class="o">></span>
<span class="o"><</span><span class="n">dependency</span><span class="o">></span>
@@ -234,7 +234,7 @@ And your Beam application should also pack Beam SDK explicitly and here is a sni
<span class="o"><</span><span class="n">dependency</span><span class="o">></span>
<span class="o"><</span><span class="n">groupId</span><span class="o">></span><span class="n">org</span><span class="o">.</span><span class="na">apache</span><span class="o">.</span><span class="na">beam</span><span class="o"></</span><span class="n">groupId</span><span class="o">></span>
<span class="o"><</span><span class="n">artifactId</span><span class="o">></span><span class="n">beam</span><span class="o">-</span><span class="n">sdks</span><span class="o">-</span><span class="n">java</span><span class="o">-</span><span class="n">core</span><span class="o"></</span><span class="n">artifactId</span><span class="o">></span>
- <span class="o"><</span><span class="n">version</span><span class="o">></span><span class="mf">2.6</span><span class="o">.</span><span class="mi">0</span><span class="o"></</span><span class="n">version</span><span class="o">></span>
+ <span class="o"><</span><span class="n">version</span><span class="o">></span><span class="mf">2.7</span><span class="o">.</span><span class="mi">0</span><span class="o"></</span><span class="n">version</span><span class="o">></span>
<span class="o"></</span><span class="n">dependency</span><span class="o">></span>
<span class="o"></</span><span class="n">dependencies</span><span class="o">></span>
diff --git a/website/generated-content/documentation/runners/jstorm/index.html b/website/generated-content/documentation/runners/jstorm/index.html
index 74ccbe7..53410d9 100644
--- a/website/generated-content/documentation/runners/jstorm/index.html
+++ b/website/generated-content/documentation/runners/jstorm/index.html
@@ -303,7 +303,7 @@ limitations under the License.
<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="o"><</span><span class="n">dependency</span><span class="o">></span>
<span class="o"><</span><span class="n">groupId</span><span class="o">></span><span class="n">org</span><span class="o">.</span><span class="na">apache</span><span class="o">.</span><span class="na">beam</span><span class="o"></</span><span class="n">groupId</span><span class="o">></span>
<span class="o"><</span><span class="n">artifactId</span><span class="o">></span><span class="n">beam</span><span class="o">-</span><span class="n">runners</span><span class="o">-</span><span class="n">jstorm</span><span class="o"></</span><span class="n">artifactId</span><span class="o">></span>
- <span class="o"><</span><span class="n">version</span><span class="o">></span><span class="mf">2.6</span><span class="o">.</span><span class="mi">0</span><span class="o"></</span><span class="n">version</span><span class="o">></span>
+ <span class="o"><</span><span class="n">version</span><span class="o">></span><span class="mf">2.7</span><span class="o">.</span><span class="mi">0</span><span class="o"></</span><span class="n">version</span><span class="o">></span>
<span class="o"></</span><span class="n">dependency</span><span class="o">></span>
</code></pre>
</div>
diff --git a/website/generated-content/documentation/runners/mapreduce/index.html b/website/generated-content/documentation/runners/mapreduce/index.html
index 763a2ac..406211f 100644
--- a/website/generated-content/documentation/runners/mapreduce/index.html
+++ b/website/generated-content/documentation/runners/mapreduce/index.html
@@ -203,7 +203,7 @@ limitations under the License.
<div class="highlighter-rouge"><pre class="highlight"><code><dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-mapreduce</artifactId>
- <version>2.6.0</version>
+ <version>2.7.0</version>
</dependency>
</code></pre>
</div>
diff --git a/website/generated-content/documentation/runners/samza/index.html b/website/generated-content/documentation/runners/samza/index.html
index 6662d7c..ce73dff 100644
--- a/website/generated-content/documentation/runners/samza/index.html
+++ b/website/generated-content/documentation/runners/samza/index.html
@@ -220,7 +220,7 @@ limitations under the License.
<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="o"><</span><span class="n">dependency</span><span class="o">></span>
<span class="o"><</span><span class="n">groupId</span><span class="o">></span><span class="n">org</span><span class="o">.</span><span class="na">apache</span><span class="o">.</span><span class="na">beam</span><span class="o"></</span><span class="n">groupId</span><span class="o">></span>
<span class="o"><</span><span class="n">artifactId</span><span class="o">></span><span class="n">beam</span><span class="o">-</span><span class="n">runners</span><span class="o">-</span><span class="n">samza_2</span><span class="o">.</span><span class="mi">11</span><span class="o"></</span><span class="n">artifactId</span><span class="o">></span>
- <span class="o"><</span><span class="n">version</span><span class="o">></span><span class="mf">2.6</span><span class="o">.</span><span class="mi">0</span><span class="o"></</span><span class="n">version</span><span class="o">></span>
+ <span class="o"><</span><span class="n">version</span><span class="o">></span><span class="mf">2.7</span><span class="o">.</span><span class="mi">0</span><span class="o"></</span><span class="n">version</span><span class="o">></span>
<span class="o"><</span><span class="n">scope</span><span class="o">></span><span class="n">runtime</span><span class="o"></</span><span class="n">scope</span><span class="o">></span>
<span class="o"></</span><span class="n">dependency</span><span class="o">></span>
diff --git a/website/generated-content/documentation/runners/spark/index.html b/website/generated-content/documentation/runners/spark/index.html
index c0ceffb..9000d2a 100644
--- a/website/generated-content/documentation/runners/spark/index.html
+++ b/website/generated-content/documentation/runners/spark/index.html
@@ -227,7 +227,7 @@ The Spark Runner can execute Spark pipelines just like a native Spark applicatio
<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="o"><</span><span class="n">dependency</span><span class="o">></span>
<span class="o"><</span><span class="n">groupId</span><span class="o">></span><span class="n">org</span><span class="o">.</span><span class="na">apache</span><span class="o">.</span><span class="na">beam</span><span class="o"></</span><span class="n">groupId</span><span class="o">></span>
<span class="o"><</span><span class="n">artifactId</span><span class="o">></span><span class="n">beam</span><span class="o">-</span><span class="n">runners</span><span class="o">-</span><span class="n">spark</span><span class="o"></</span><span class="n">artifactId</span><span class="o">></span>
- <span class="o"><</span><span class="n">version</span><span class="o">></span><span class="mf">2.6</span><span class="o">.</span><span class="mi">0</span><span class="o"></</span><span class="n">version</span><span class="o">></span>
+ <span class="o"><</span><span class="n">version</span><span class="o">></span><span class="mf">2.7</span><span class="o">.</span><span class="mi">0</span><span class="o"></</span><span class="n">version</span><span class="o">></span>
<span class="o"></</span><span class="n">dependency</span><span class="o">></span>
</code></pre>
</div>
diff --git a/website/generated-content/documentation/sdks/feature-comparison/index.html b/website/generated-content/documentation/sdks/feature-comparison/index.html
index c6bbcee..e260beb 100644
--- a/website/generated-content/documentation/sdks/feature-comparison/index.html
+++ b/website/generated-content/documentation/sdks/feature-comparison/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.7.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.7.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -190,7 +190,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/data-types/">Data types</a></li>
<li><a href="/documentation/dsls/sql/lexical/">Lexical structure</a></li>
- <li><a href="/documentation/dsls/sql/create-table/">CREATE TABLE</a></li>
+ <li><a href="/documentation/dsls/sql/create-external-table/">CREATE EXTERNAL TABLE</a></li>
<li><a href="/documentation/dsls/sql/select/">SELECT</a></li>
<li><a href="/documentation/dsls/sql/windowing-and-triggering/">Windowing & Triggering</a></li>
<li><a href="/documentation/dsls/sql/joins/">Joins</a></li>
diff --git a/website/generated-content/documentation/sdks/go/index.html b/website/generated-content/documentation/sdks/go/index.html
index 00c0d1f..367eac0 100644
--- a/website/generated-content/documentation/sdks/go/index.html
+++ b/website/generated-content/documentation/sdks/go/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.7.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.7.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -190,7 +190,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/data-types/">Data types</a></li>
<li><a href="/documentation/dsls/sql/lexical/">Lexical structure</a></li>
- <li><a href="/documentation/dsls/sql/create-table/">CREATE TABLE</a></li>
+ <li><a href="/documentation/dsls/sql/create-external-table/">CREATE EXTERNAL TABLE</a></li>
<li><a href="/documentation/dsls/sql/select/">SELECT</a></li>
<li><a href="/documentation/dsls/sql/windowing-and-triggering/">Windowing & Triggering</a></li>
<li><a href="/documentation/dsls/sql/joins/">Joins</a></li>
diff --git a/website/generated-content/documentation/sdks/java-extensions/index.html b/website/generated-content/documentation/sdks/java-extensions/index.html
index 4ee5f3f..b5be5b2 100644
--- a/website/generated-content/documentation/sdks/java-extensions/index.html
+++ b/website/generated-content/documentation/sdks/java-extensions/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.7.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.7.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -190,7 +190,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/data-types/">Data types</a></li>
<li><a href="/documentation/dsls/sql/lexical/">Lexical structure</a></li>
- <li><a href="/documentation/dsls/sql/create-table/">CREATE TABLE</a></li>
+ <li><a href="/documentation/dsls/sql/create-external-table/">CREATE EXTERNAL TABLE</a></li>
<li><a href="/documentation/dsls/sql/select/">SELECT</a></li>
<li><a href="/documentation/dsls/sql/windowing-and-triggering/">Windowing & Triggering</a></li>
<li><a href="/documentation/dsls/sql/joins/">Joins</a></li>
diff --git a/website/generated-content/documentation/sdks/java-thirdparty/index.html b/website/generated-content/documentation/sdks/java-thirdparty/index.html
index 3a4884c..382dc20 100644
--- a/website/generated-content/documentation/sdks/java-thirdparty/index.html
+++ b/website/generated-content/documentation/sdks/java-thirdparty/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.7.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.7.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -190,7 +190,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/data-types/">Data types</a></li>
<li><a href="/documentation/dsls/sql/lexical/">Lexical structure</a></li>
- <li><a href="/documentation/dsls/sql/create-table/">CREATE TABLE</a></li>
+ <li><a href="/documentation/dsls/sql/create-external-table/">CREATE EXTERNAL TABLE</a></li>
<li><a href="/documentation/dsls/sql/select/">SELECT</a></li>
<li><a href="/documentation/dsls/sql/windowing-and-triggering/">Windowing & Triggering</a></li>
<li><a href="/documentation/dsls/sql/joins/">Joins</a></li>
diff --git a/website/generated-content/documentation/sdks/java/euphoria/index.html b/website/generated-content/documentation/sdks/java/euphoria/index.html
index af6ed2a..493f44a 100644
--- a/website/generated-content/documentation/sdks/java/euphoria/index.html
+++ b/website/generated-content/documentation/sdks/java/euphoria/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.7.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.7.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -190,7 +190,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/data-types/">Data types</a></li>
<li><a href="/documentation/dsls/sql/lexical/">Lexical structure</a></li>
- <li><a href="/documentation/dsls/sql/create-table/">CREATE TABLE</a></li>
+ <li><a href="/documentation/dsls/sql/create-external-table/">CREATE EXTERNAL TABLE</a></li>
<li><a href="/documentation/dsls/sql/select/">SELECT</a></li>
<li><a href="/documentation/dsls/sql/windowing-and-triggering/">Windowing & Triggering</a></li>
<li><a href="/documentation/dsls/sql/joins/">Joins</a></li>
diff --git a/website/generated-content/documentation/sdks/java/index.html b/website/generated-content/documentation/sdks/java/index.html
index 6cd38f5..88bf9fb 100644
--- a/website/generated-content/documentation/sdks/java/index.html
+++ b/website/generated-content/documentation/sdks/java/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.7.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.7.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -190,7 +190,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/data-types/">Data types</a></li>
<li><a href="/documentation/dsls/sql/lexical/">Lexical structure</a></li>
- <li><a href="/documentation/dsls/sql/create-table/">CREATE TABLE</a></li>
+ <li><a href="/documentation/dsls/sql/create-external-table/">CREATE EXTERNAL TABLE</a></li>
<li><a href="/documentation/dsls/sql/select/">SELECT</a></li>
<li><a href="/documentation/dsls/sql/windowing-and-triggering/">Windowing & Triggering</a></li>
<li><a href="/documentation/dsls/sql/joins/">Joins</a></li>
diff --git a/website/generated-content/documentation/sdks/java/nexmark/index.html b/website/generated-content/documentation/sdks/java/nexmark/index.html
index 1676235..63beb70 100644
--- a/website/generated-content/documentation/sdks/java/nexmark/index.html
+++ b/website/generated-content/documentation/sdks/java/nexmark/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.7.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.7.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -190,7 +190,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/data-types/">Data types</a></li>
<li><a href="/documentation/dsls/sql/lexical/">Lexical structure</a></li>
- <li><a href="/documentation/dsls/sql/create-table/">CREATE TABLE</a></li>
+ <li><a href="/documentation/dsls/sql/create-external-table/">CREATE EXTERNAL TABLE</a></li>
<li><a href="/documentation/dsls/sql/select/">SELECT</a></li>
<li><a href="/documentation/dsls/sql/windowing-and-triggering/">Windowing & Triggering</a></li>
<li><a href="/documentation/dsls/sql/joins/">Joins</a></li>
@@ -503,7 +503,7 @@ or may be published to Pub/Sub or Kafka.</p>
--stagingLocation=gs://<a gs path for staging> \
--runner=DataflowRunner \
--tempLocation=gs://<a gs path for temporary files> \
---filesToStage=target/beam-sdks-java-nexmark-2.6.0.jar
+--filesToStage=target/beam-sdks-java-nexmark-2.7.0.jar
</code></pre>
</div>
@@ -987,7 +987,7 @@ PUBSUB_TOPCI=<existing pubsub topic>
--driver-memory 512m \
--executor-memory 512m \
--executor-cores 1 \
- sdks/java/nexmark/build/libs/beam-sdks-java-nexmark-2.6.0-spark.jar \
+ sdks/java/nexmark/build/libs/beam-sdks-java-nexmark-2.7.0-spark.jar \
--runner=SparkRunner \
--query=0 \
--streamTimeout=60 \
diff --git a/website/generated-content/documentation/sdks/python-custom-io/index.html b/website/generated-content/documentation/sdks/python-custom-io/index.html
index 9b20418..7d7e68f 100644
--- a/website/generated-content/documentation/sdks/python-custom-io/index.html
+++ b/website/generated-content/documentation/sdks/python-custom-io/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.7.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.7.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -190,7 +190,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/data-types/">Data types</a></li>
<li><a href="/documentation/dsls/sql/lexical/">Lexical structure</a></li>
- <li><a href="/documentation/dsls/sql/create-table/">CREATE TABLE</a></li>
+ <li><a href="/documentation/dsls/sql/create-external-table/">CREATE EXTERNAL TABLE</a></li>
<li><a href="/documentation/dsls/sql/select/">SELECT</a></li>
<li><a href="/documentation/dsls/sql/windowing-and-triggering/">Windowing & Triggering</a></li>
<li><a href="/documentation/dsls/sql/joins/">Joins</a></li>
diff --git a/website/generated-content/documentation/sdks/python-pipeline-dependencies/index.html b/website/generated-content/documentation/sdks/python-pipeline-dependencies/index.html
index b80898b..7e3f88d 100644
--- a/website/generated-content/documentation/sdks/python-pipeline-dependencies/index.html
+++ b/website/generated-content/documentation/sdks/python-pipeline-dependencies/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.7.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.7.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -190,7 +190,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/data-types/">Data types</a></li>
<li><a href="/documentation/dsls/sql/lexical/">Lexical structure</a></li>
- <li><a href="/documentation/dsls/sql/create-table/">CREATE TABLE</a></li>
+ <li><a href="/documentation/dsls/sql/create-external-table/">CREATE EXTERNAL TABLE</a></li>
<li><a href="/documentation/dsls/sql/select/">SELECT</a></li>
<li><a href="/documentation/dsls/sql/windowing-and-triggering/">Windowing & Triggering</a></li>
<li><a href="/documentation/dsls/sql/joins/">Joins</a></li>
diff --git a/website/generated-content/documentation/sdks/python-streaming/index.html b/website/generated-content/documentation/sdks/python-streaming/index.html
index 3681102..c2582f6 100644
--- a/website/generated-content/documentation/sdks/python-streaming/index.html
+++ b/website/generated-content/documentation/sdks/python-streaming/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.7.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.7.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -190,7 +190,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/data-types/">Data types</a></li>
<li><a href="/documentation/dsls/sql/lexical/">Lexical structure</a></li>
- <li><a href="/documentation/dsls/sql/create-table/">CREATE TABLE</a></li>
+ <li><a href="/documentation/dsls/sql/create-external-table/">CREATE EXTERNAL TABLE</a></li>
<li><a href="/documentation/dsls/sql/select/">SELECT</a></li>
<li><a href="/documentation/dsls/sql/windowing-and-triggering/">Windowing & Triggering</a></li>
<li><a href="/documentation/dsls/sql/joins/">Joins</a></li>
diff --git a/website/generated-content/documentation/sdks/python-type-safety/index.html b/website/generated-content/documentation/sdks/python-type-safety/index.html
index f45c0cb..fe8bd4a 100644
--- a/website/generated-content/documentation/sdks/python-type-safety/index.html
+++ b/website/generated-content/documentation/sdks/python-type-safety/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.7.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.7.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -190,7 +190,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/data-types/">Data types</a></li>
<li><a href="/documentation/dsls/sql/lexical/">Lexical structure</a></li>
- <li><a href="/documentation/dsls/sql/create-table/">CREATE TABLE</a></li>
+ <li><a href="/documentation/dsls/sql/create-external-table/">CREATE EXTERNAL TABLE</a></li>
<li><a href="/documentation/dsls/sql/select/">SELECT</a></li>
<li><a href="/documentation/dsls/sql/windowing-and-triggering/">Windowing & Triggering</a></li>
<li><a href="/documentation/dsls/sql/joins/">Joins</a></li>
diff --git a/website/generated-content/documentation/sdks/python/index.html b/website/generated-content/documentation/sdks/python/index.html
index 68265f7..3342edd 100644
--- a/website/generated-content/documentation/sdks/python/index.html
+++ b/website/generated-content/documentation/sdks/python/index.html
@@ -142,7 +142,7 @@
<span class="section-nav-list-title">Java</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/javadoc/2.6.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/javadoc/2.7.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -156,7 +156,7 @@
<span class="section-nav-list-title">Python</span>
<ul class="section-nav-list">
<li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/pydoc/2.6.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
+ <li><a href="https://beam.apache.org/releases/pydoc/2.7.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
width="14" height="14"
alt="External link."></a>
</li>
@@ -190,7 +190,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/data-types/">Data types</a></li>
<li><a href="/documentation/dsls/sql/lexical/">Lexical structure</a></li>
- <li><a href="/documentation/dsls/sql/create-table/">CREATE TABLE</a></li>
+ <li><a href="/documentation/dsls/sql/create-external-table/">CREATE EXTERNAL TABLE</a></li>
<li><a href="/documentation/dsls/sql/select/">SELECT</a></li>
<li><a href="/documentation/dsls/sql/windowing-and-triggering/">Windowing & Triggering</a></li>
<li><a href="/documentation/dsls/sql/joins/">Joins</a></li>
diff --git a/website/generated-content/feed.xml b/website/generated-content/feed.xml
index b51ea98..e2329e4 100644
--- a/website/generated-content/feed.xml
+++ b/website/generated-content/feed.xml
@@ -20,6 +20,92 @@
<generator>Jekyll v3.2.0</generator>
<item>
+ <title>Apache Beam 2.7.0</title>
+ <description><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+<p>We are happy to present the new 2.7.0 release of Beam. This release includes both improvements and new functionality.
+See the <a href="/get-started/downloads/#270-2018-10-02">download page</a> for this release.<!--more-->
+For more information on changes in 2.7.0, check out the
+<a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12343654">detailed release notes</a>.</p>
+
+<h2 id="new-features--improvements">New Features / Improvements</h2>
+
+<h3 id="new-ios">New I/Os</h3>
+
+<ul>
+ <li>KuduIO</li>
+ <li>Amazon SNS sink</li>
+ <li>Amazon SqsIO</li>
+</ul>
+
+<h3 id="dependency-upgrades">Dependency Upgrades</h3>
+
+<ul>
+ <li>Apache Calcite dependency upgraded to 1.17.0</li>
+ <li>Apache Derby dependency upgraded to 10.14.2.0</li>
+ <li>Apache HTTP components upgraded (see release notes).</li>
+</ul>
+
+<h3 id="portability">Portability</h3>
+
+<ul>
+ <li>Experimental support for Python on local Flink runner for simple
+examples, see latest information here:
+/contribute/portability/#status.</li>
+</ul>
+
+<h2 id="miscellaneous-fixes">Miscellaneous Fixes</h2>
+
+<h3 id="ios">I/Os</h3>
+
+<ul>
+ <li>KinesisIO, fixed dependency issue</li>
+</ul>
+
+<h2 id="list-of-contributors">List of Contributors</h2>
+
+<p>According to git shortlog, the following 72 people contributed
+to the 2.7.0 release. Thank you to all contributors!</p>
+
+<p>Ahmet Altay, Alan Myrvold, Alexey Romanenko, Aljoscha Krettek,
+Andrew Pilloud, Ankit Jhalaria, Ankur Goenka, Anton Kedin, Boyuan
+Zhang, Carl McGraw, Carlos Alonso, cclauss, Chamikara Jayalath,
+Charles Chen, Cory Brzycki, Daniel Oliveira, Dariusz Aniszewski,
+devinduan, Eric Beach, Etienne Chauchot, Eugene Kirpichov, Garrett
+Jones, Gene Peters, Gleb Kanterov, Henning Rohde, Henry Suryawirawan,
+Holden Karau, Huygaa Batsaikhan, Ismaël Mejía, Jason Kuster, Jean-
+Baptiste Onofré, Joachim van der Herten, Jozef Vilcek, jxlewis, Kai
+Jiang, Katarzyna Kucharczyk, Kenn Knowles, Krzysztof Trubalski, Kyle
+Winkelman, Leen Toelen, Luis Enrique Ortíz Ramirez, Lukasz Cwik,
+Łukasz Gajowy, Luke Cwik, Mark Liu, Matthias Feys, Maximilian Michels,
+Melissa Pashniak, Mikhail Gryzykhin, Mikhail Sokolov, mingmxu, Norbert
+Chen, Pablo Estrada, Prateek Chanda, Raghu Angadi, Ravi Pathak, Reuven
+Lax, Robert Bradshaw, Robert Burke, Rui Wang, Ryan Williams, Sindy Li,
+Thomas Weise, Tim Robertson, Tormod Haavi, Udi Meiri, Vaclav Plajt,
+Valentyn Tymofieiev, xiliu, XuMingmin, Yifan Zou, Yueyang Qiu.</p>
+</description>
+ <pubDate>Wed, 03 Oct 2018 01:00:01 -0700</pubDate>
+ <link>https://beam.apache.org/blog/2018/10/03/beam-2.7.0.html</link>
+ <guid isPermaLink="true">https://beam.apache.org/blog/2018/10/03/beam-2.7.0.html</guid>
+
+
+ <category>blog</category>
+
+ </item>
+
+ <item>
<title>Beam Summit Europe 2018</title>
<description><!--
Licensed under the Apache License, Version 2.0 (the "License");
@@ -125,7 +211,7 @@ limitations under the License.
</td>
<td>Local<br />(Using the <code>file://</code> URI)
</td>
- <td><a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/TextIO.html">TextIO</a>
+ <td><a href="https://beam.apache.org/releases/javadoc/2.7.0/org/apache/beam/sdk/io/TextIO.html">TextIO</a>
</td>
<td><a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/StreamingContext.html#textFileStream-java.lang.String-">textFileStream</a><br />(Spark treats most Unix systems as HDFS-compatible, but the location should be accessible from all nodes)
</td>
@@ -133,7 +219,7 @@ limitations under the License.
<tr>
<td>HDFS<br />(Using the <code>hdfs://</code> URI)
</td>
- <td><a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/FileIO.html">FileIO</a> + <a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptions.html">HadoopFileSystemOptions</a>
+ <td><a href="https://beam.apache.org/releases/javadoc/2.7.0/org/apache/beam/sdk/io/FileIO.html">FileIO</a> + <a href="https://beam.apache.org/releases/javadoc/2.7.0/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptions.html">HadoopFileSystemOptions</a>
</td>
<td><a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/util/HdfsUtils.html">HdfsUtils</a>
</td>
@@ -143,7 +229,7 @@ limitations under the License.
</td>
<td>Cloud Storage<br />(Using the <code>gs://</code> URI)
</td>
- <td><a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/FileIO.html">FileIO</a> + <a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/extensions/gcp/options/GcsOptions.html">GcsOptions</a>
+ <td><a href="https://beam.apache.org/releases/javadoc/2.7.0/org/apache/beam/sdk/io/FileIO.html">FileIO</a> + <a href="https://beam.apache.org/releases/javadoc/2.7.0/org/apache/beam/sdk/extensions/gcp/options/GcsOptions.html">GcsOptions</a>
</td>
<td rowspan="2"><a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/SparkContext.html#hadoopConfiguration--">hadoopConfiguration</a>
and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/StreamingContext.html#textFileStream-java.lang.String-">textFileStream</a>
@@ -152,7 +238,7 @@ and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/sp
<tr>
<td>S3<br />(Using the <code>s3://</code> URI)
</td>
- <td><a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/FileIO.html">FileIO</a> + <a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/aws/options/S3Options.html">S3Options</a>
+ <td><a href="https://beam.apache.org/releases/javadoc/2.7.0/org/apache/beam/sdk/io/FileIO.html">FileIO</a> + <a href="https://beam.apache.org/releases/javadoc/2.7.0/org/apache/beam/sdk/io/aws/options/S3Options.html">S3Options</a>
</td>
</tr>
<tr>
@@ -160,7 +246,7 @@ and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/sp
</td>
<td>Kafka
</td>
- <td><a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/kafka/KafkaIO.html">KafkaIO</a>
+ <td><a href="https://beam.apache.org/releases/javadoc/2.7.0/org/apache/beam/sdk/io/kafka/KafkaIO.html">KafkaIO</a>
</td>
<td><a href="https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html">spark-streaming-kafka</a>
</td>
@@ -168,7 +254,7 @@ and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/sp
<tr>
<td>Kinesis
</td>
- <td><a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/kinesis/KinesisIO.html">KinesisIO</a>
+ <td><a href="https://beam.apache.org/releases/javadoc/2.7.0/org/apache/beam/sdk/io/kinesis/KinesisIO.html">KinesisIO</a>
</td>
<td><a href="https://spark.apache.org/docs/latest/streaming-kinesis-integration.html">spark-streaming-kinesis</a>
</td>
@@ -176,7 +262,7 @@ and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/sp
<tr>
<td>Cloud Pub/Sub
</td>
- <td><a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.html">PubsubIO</a>
+ <td><a href="https://beam.apache.org/releases/javadoc/2.7.0/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.html">PubsubIO</a>
</td>
<td><a href="https://github.com/apache/bahir/tree/master/streaming-pubsub">spark-streaming-pubsub</a> from <a href="http://bahir.apache.org">Apache Bahir</a>
</td>
@@ -217,7 +303,7 @@ and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/sp
</td>
<td>Local
</td>
- <td><a href="https://beam.apache.org/releases/pydoc/2.6.0/apache_beam.io.textio.html">io.textio</a>
+ <td><a href="https://beam.apache.org/releases/pydoc/2.7.0/apache_beam.io.textio.html">io.textio</a>
</td>
<td><a href="http://spark.apache.org/docs/latest/api/python/pyspark.streaming.html#pyspark.streaming.StreamingContext.textFileStream">textFileStream</a>
</td>
@@ -225,7 +311,7 @@ and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/sp
<tr>
<td>HDFS
</td>
- <td><a href="https://beam.apache.org/releases/pydoc/2.6.0/apache_beam.io.hadoopfilesystem.html">io.hadoopfilesystem</a>
+ <td><a href="https://beam.apache.org/releases/pydoc/2.7.0/apache_beam.io.hadoopfilesystem.html">io.hadoopfilesystem</a>
</td>
<td><a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/SparkContext.html#hadoopConfiguration--">hadoopConfiguration</a> (Access through <code>sc._jsc</code> with Py4J)
and <a href="http://spark.apache.org/docs/latest/api/python/pyspark.streaming.html#pyspark.streaming.StreamingContext.textFileStream">textFileStream</a>
@@ -236,7 +322,7 @@ and <a href="http://spark.apache.org/docs/latest/api/python/pyspark.stre
</td>
<td>Google Cloud Storage
</td>
- <td><a href="https://beam.apache.org/releases/pydoc/2.6.0/apache_beam.io.gcp.gcsio.html">io.gcp.gcsio</a>
+ <td><a href="https://beam.apache.org/releases/pydoc/2.7.0/apache_beam.io.gcp.gcsio.html">io.gcp.gcsio</a>
</td>
<td rowspan="2"><a href="http://spark.apache.org/docs/latest/api/python/pyspark.streaming.html#pyspark.streaming.StreamingContext.textFileStream">textFileStream</a>
</td>
@@ -268,7 +354,7 @@ and <a href="http://spark.apache.org/docs/latest/api/python/pyspark.stre
<tr>
<td>Cloud Pub/Sub
</td>
- <td><a href="https://beam.apache.org/releases/pydoc/2.6.0/apache_beam.io.gcp.pubsub.html">io.gcp.pubsub</a>
+ <td><a href="https://beam.apache.org/releases/pydoc/2.7.0/apache_beam.io.gcp.pubsub.html">io.gcp.pubsub</a>
</td>
<td>N/A
</td>
@@ -442,7 +528,7 @@ please check the detailed release notes.</p>
<h1 id="new-features--improvements">New Features / Improvements</h1>
<h2 id="go-sdk-support">Go SDK support</h2>
-<p>The Go SDK has been officially accepted into the project, after an incubation period and community effort. Go pipelines run on Dataflow runner. More details are <a href="https://beam.apache.org/documentation/sdks/go/">here</a>.</p>
+<p>The Go SDK has been officially accepted into the project, after an incubation period and community effort. Go pipelines run on Dataflow runner. More details are <a href="/documentation/sdks/go/">here</a>.</p>
<h2 id="parquet-support">Parquet support</h2>
<p>Support for Apache Parquet format was added. It uses Parquet 1.10 release which, thanks to AvroParquerWriter’s API changes, allows FileIO.Sink implementation.</p>
@@ -1408,22 +1494,22 @@ has other limitations that make it insufficient for this task</em>).</p
<h2 id="beam-source-api">Beam Source API</h2>
<p>Apache Beam historically provides a Source API
-(<a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/BoundedSource.html">BoundedSource</a>
+(<a href="https://beam.apache.org/releases/javadoc/2.7.0/org/apache/beam/sdk/io/BoundedSource.html">BoundedSource</a>
and
-<a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/UnboundedSource.html">UnboundedSource</a>) which does
+<a href="https://beam.apache.org/releases/javadoc/2.7.0/org/apache/beam/sdk/io/UnboundedSource.html">UnboundedSource</a>) which does
not have these limitations and allows development of efficient data sources for
batch and streaming systems. Pipelines use this API via the
-<a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/Read.html"><code class="highlighter-rouge">Read.from(Source)</code></a> built-in <code class="highlighter-rouge">PTransform</code>.</p>
+<a href="https://beam.apache.org/releases/javadoc/2.7.0/org/apache/beam/sdk/io/Read.html"><code class="highlighter-rouge">Read.from(Source)</code></a> built-in <code class="highlighter-rouge">PTransform</code>.</p>
<p>The Source API is largely similar to that of most other data processing
frameworks, and allows the system to read data in parallel using multiple
workers, as well as checkpoint and resume reading from an unbounded data source.
Additionally, the Beam
-<a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/BoundedSource.html"><code class="highlighter-rouge">BoundedSource</code></a>
+<a href="https://beam.apache.org/releases/javadoc/2.7.0/org/apache/beam/sdk/io/BoundedSource.html"><code class="highlighter-rouge">BoundedSource</code></a>
API provides advanced features such as progress reporting and <a href="/blog/2016/05/18/splitAtFraction-method.html">dynamic
rebalancing</a>
(which together enable autoscaling), and
-<a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/io/UnboundedSource.html"><code class="highlighter-rouge">UnboundedSource</code></a> supports
+<a href="https://beam.apache.org/releases/javadoc/2.7.0/org/apache/beam/sdk/io/UnboundedSource.html"><code class="highlighter-rouge">UnboundedSource</code></a> supports
reporting the source’s watermark and backlog <em>(until SDF, we believed that
“batch” and “streaming” data sources are fundamentally different and thus
require fundamentally different APIs)</em>.</p>
@@ -2025,93 +2111,5 @@ limitations under the License.
</item>
- <item>
- <title>Python SDK released in Apache Beam 0.6.0</title>
- <description><!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-<p>Apache Beam’s latest release, version <a href="/get-started/downloads/">0.6.0</a>, introduces a new SDK – this time, for the Python programming language. The Python SDK joins the Java SDK as the second implementation of the Beam programming model.</p>
-
-<!--more-->
-
-<p>The Python SDK incorporates all of the main concepts of the Beam model, including ParDo, GroupByKey, Windowing, and others. It features extensible IO APIs for writing bounded sources and sinks, and provides built-in implementation for reading and writing Text, Avro, and TensorFlow record files, as well as connectors to Google BigQuery and Google Cloud Datastore.</p>
-
-<p>There are two runners capable of executing pipelines written with the Python SDK today: <a href="/documentation/runners/direct/">Direct Runner</a> and <a href="/documentation/runners/dataflow/">Dataflow Runner</a>, both of which are currently limited to batch execution only. Upcoming features will shortly bring the benefits of the Python SDK to additional runners.</p>
-
-<h4 id="try-the-apache-beam-python-sdk">Try the Apache Beam Python SDK</h4>
-
-<p>If you would like to try out the Python SDK, a good place to start is the <a href="/get-started/quickstart-py/">Quickstart</a>. After that, you can take a look at additional <a href="https://github.com/apache/beam/tree/v0.6.0/sdks/python/apache_beam/examples">examples</a>, and deep dive into the <a href="https://beam.apache.org/releases/pydoc/">API reference</a>.</p>
-
-<p>Let’s take a look at a quick example together. First, install the <code class="highlighter-rouge">apache-beam</code> package from PyPI and start your Python interpreter.</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ pip install apache-beam
-$ python
-</code></pre>
-</div>
-
-<p>We will harness the power of Apache Beam to estimate Pi in honor of the recently passed Pi Day.</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>import random
-import apache_beam as beam
-
-def run_trials(count):
- """Throw darts into unit square and count how many fall into unit circle."""
- inside = 0
- for _ in xrange(count):
- x, y = random.uniform(0, 1), random.uniform(0, 1)
- inside += 1 if x*x + y*y &lt;= 1.0 else 0
- return count, inside
-
-def combine_results(results):
- """Given all the trial results, estimate pi."""
- total, inside = sum(r[0] for r in results), sum(r[1] for r in results)
- return total, inside, 4 * float(inside) / total if total &gt; 0 else 0
-
-p = beam.Pipeline()
-(p | beam.Create([500] * 10) # Create 10 experiments with 500 samples each.
- | beam.Map(run_trials) # Run experiments in parallel.
- | beam.CombineGlobally(combine_results) # Combine the results.
- | beam.io.WriteToText('./pi_estimate.txt')) # Write PI estimate to a file.
-
-p.run()
-</code></pre>
-</div>
-
-<p>This example estimates Pi by throwing random darts into the unit square and keeping track of the fraction of those darts that fell into the unit circle (see the full <a href="https://github.com/apache/beam/blob/v0.6.0/sdks/python/apache_beam/examples/complete/estimate_pi.py">example</a> for details). If you are curious, you can check the result of our estimation by looking at the output file.</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ cat pi_estimate.txt*
-</code></pre>
-</div>
-
-<h4 id="roadmap">Roadmap</h4>
-
-<p>The first thing on the Python SDK’s roadmap is to address two of its limitations. First, the existing runners are currently limited to bounded PCollections, and we are looking forward to extending the SDK to support unbounded PCollections (“streaming”). Additionally, we are working on extending support to more Apache Beam runners, and the upcoming Fn API will do the heavy lifting.</p>
-
-<p>Both of these improvements will enable the Python SDK to fulfill the mission of Apache Beam: a unified programming model for batch and streaming data processing that can run on any execution engine.</p>
-
-<h4 id="join-us">Join us!</h4>
-
-<p>Please consider joining us, whether as a user or a contributor, as we work towards our first release with API stability. If you’d like to try out Apache Beam today, check out the latest <a href="/get-started/downloads/">0.6.0</a> release. We welcome contributions and participation from anyone through our mailing lists, issue tracker, pull requests, and events.</p>
-</description>
- <pubDate>Thu, 16 Mar 2017 01:00:01 -0700</pubDate>
- <link>https://beam.apache.org/blog/2017/03/16/python-sdk-release.html</link>
- <guid isPermaLink="true">https://beam.apache.org/blog/2017/03/16/python-sdk-release.html</guid>
-
-
- <category>blog</category>
-
- </item>
-
</channel>
</rss>
diff --git a/website/generated-content/get-started/downloads/index.html b/website/generated-content/get-started/downloads/index.html
index a1342c7..e1a5be7 100644
--- a/website/generated-content/get-started/downloads/index.html
+++ b/website/generated-content/get-started/downloads/index.html
@@ -182,6 +182,7 @@
<li><a href="#api-stability">API stability</a></li>
<li><a href="#releases">Releases</a>
<ul>
+ <li><a href="#270-2018-10-02">2.7.0 (2018-10-02)</a></li>
<li><a href="#260-2018-08-08">2.6.0 (2018-08-08)</a></li>
<li><a href="#250-2018-06-06">2.5.0 (2018-06-06)</a></li>
<li><a href="#240-2018-03-20">2.4.0 (2018-03-20)</a></li>
@@ -214,7 +215,7 @@ limitations under the License.
<h1 id="apache-beam-downloads">Apache Beam™ Downloads</h1>
<blockquote>
- <p>Beam SDK 2.6.0 is the latest released version.</p>
+ <p>Beam SDK 2.7.0 is the latest released version.</p>
</blockquote>
<h2 id="using-a-central-repository">Using a central repository</h2>
@@ -229,12 +230,12 @@ with the <code class="highlighter-rouge">DirectRunner</code>, add the following
<div class="highlighter-rouge"><pre class="highlight"><code><dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-core</artifactId>
- <version>2.6.0</version>
+ <version>2.7.0</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-direct-java</artifactId>
- <version>2.6.0</version>
+ <version>2.7.0</version>
<scope>runtime</scope>
</dependency>
</code></pre>
@@ -243,7 +244,7 @@ with the <code class="highlighter-rouge">DirectRunner</code>, add the following
<p>Similarly in Python, if you are using PyPI and want to use the SDK for Python
with <code class="highlighter-rouge">DirectRunner</code>, add the following requirement to your <code class="highlighter-rouge">setup.py</code> file:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>apache-beam==2.6.0
+<div class="highlighter-rouge"><pre class="highlight"><code>apache-beam==2.7.0
</code></pre>
</div>
@@ -267,7 +268,7 @@ the form <code class="highlighter-rouge">major.minor.incremental</code> and are
<li>incremental version for forward-compatible bug fixes</li>
</ul>
-<p>Please note that APIs marked <a href="https://beam.apache.org/releases/javadoc/2.6.0/org/apache/beam/sdk/annotations/Experimental.html"><code class="highlighter-rouge">@Experimental</code></a>
+<p>Please note that APIs marked <a href="https://beam.apache.org/releases/javadoc/2.7.0/org/apache/beam/sdk/annotations/Experimental.html"><code class="highlighter-rouge">@Experimental</code></a>
may change at any point and are not guaranteed to remain compatible across versions.</p>
<p>Additionally, any API may change before the first stable release, i.e., between
@@ -275,6 +276,13 @@ versions denoted <code class="highlighter-rouge">0.x.y</code>.</p>
<h2 id="releases">Releases</h2>
+<h3 id="270-2018-10-02">2.7.0 (2018-10-02)</h3>
+<p>Official <a href="https://dist.apache.org/repos/dist/release/beam/2.7.0/apache-beam-2.7.0-source-release.zip">source code download</a>
+<a href="https://dist.apache.org/repos/dist/release/beam/2.7.0/apache-beam-2.7.0-source-release.zip.sha512">SHA-512</a>
+<a href="https://dist.apache.org/repos/dist/release/beam/2.7.0/apache-beam-2.7.0-source-release.zip.asc">signature</a>.</p>
+
+<p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654">Release notes</a>.</p>
+
<h3 id="260-2018-08-08">2.6.0 (2018-08-08)</h3>
<p>Official <a href="https://archive.apache.org/dist/beam/2.6.0/apache-beam-2.6.0-source-release.zip">source code download</a>
<a href="https://archive.apache.org/dist/beam/2.6.0/apache-beam-2.6.0-source-release.zip.sha512">SHA-512</a>
diff --git a/website/generated-content/get-started/quickstart-java/index.html b/website/generated-content/get-started/quickstart-java/index.html
index b3fb7b5..fa05923 100644
--- a/website/generated-content/get-started/quickstart-java/index.html
+++ b/website/generated-content/get-started/quickstart-java/index.html
@@ -232,7 +232,7 @@ limitations under the License.
<div class="shell-unix highlighter-rouge"><pre class="highlight"><code>$ mvn archetype:generate \
-DarchetypeGroupId=org.apache.beam \
-DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
- -DarchetypeVersion=2.6.0 \
+ -DarchetypeVersion=2.7.0 \
-DgroupId=org.example \
-DartifactId=word-count-beam \
-Dversion="0.1" \
@@ -244,7 +244,7 @@ limitations under the License.
<div class="shell-PowerShell highlighter-rouge"><pre class="highlight"><code>PS> mvn archetype:generate `
-D archetypeGroupId=org.apache.beam `
-D archetypeArtifactId=beam-sdks-java-maven-archetypes-examples `
- -D archetypeVersion=2.6.0 `
+ -D archetypeVersion=2.7.0 `
-D groupId=org.example `
-D artifactId=word-count-beam `
-D version="0.1" `
@@ -342,7 +342,7 @@ You can monitor the running job by visiting the Flink dashboard at http://<fl
</code></pre>
</div>
-<div class="runner-dataflow highlighter-rouge"><pre class="highlight"><code>Make sure you complete the setup steps at https://beam.apache.org/documentation/runners/dataflow/#setup
+<div class="runner-dataflow highlighter-rouge"><pre class="highlight"><code>Make sure you complete the setup steps at /documentation/runners/dataflow/#setup
$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
-Dexec.args="--runner=DataflowRunner --project=<your-gcp-project> \
@@ -386,7 +386,7 @@ You can monitor the running job by visiting the Flink dashboard at http://<fl
</code></pre>
</div>
-<div class="runner-dataflow highlighter-rouge"><pre class="highlight"><code>Make sure you complete the setup steps at https://beam.apache.org/documentation/runners/dataflow/#setup
+<div class="runner-dataflow highlighter-rouge"><pre class="highlight"><code>Make sure you complete the setup steps at /documentation/runners/dataflow/#setup
PS> mvn compile exec:java -D exec.mainClass=org.apache.beam.examples.WordCount `
-D exec.args="--runner=DataflowRunner --project=<your-gcp-project> `
diff --git a/website/generated-content/get-started/quickstart-py/index.html b/website/generated-content/get-started/quickstart-py/index.html
index 17fc112..0eca7b7 100644
--- a/website/generated-content/get-started/quickstart-py/index.html
+++ b/website/generated-content/get-started/quickstart-py/index.html
@@ -394,7 +394,7 @@ environment’s directories.</p>
</div>
<div class="runner-dataflow highlighter-rouge"><pre class="highlight"><code># As part of the initial setup, install Google Cloud Platform specific extra components. Make sure you
-# complete the setup steps at https://beam.apache.org/documentation/runners/dataflow/#setup
+# complete the setup steps at /documentation/runners/dataflow/#setup
pip install apache-beam[gcp]
python -m apache_beam.examples.wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \
--output gs://<your-gcs-bucket>/counts \
diff --git a/website/generated-content/get-started/wordcount-example/index.html b/website/generated-content/get-started/wordcount-example/index.html
index eb45e10..1e4dbac 100644
--- a/website/generated-content/get-started/wordcount-example/index.html
+++ b/website/generated-content/get-started/wordcount-example/index.html
@@ -1375,7 +1375,7 @@ unbounded datasets. If your dataset has a fixed number of elements, it is a boun
dataset and all of the data can be processed together. For bounded datasets,
the question to ask is “Do I have all of the data?” If data continuously
arrives (such as an endless stream of game scores in the
-<a href="https://beam.apache.org/get-started/mobile-gaming-example/">Mobile gaming example</a>,
+<a href="/get-started/mobile-gaming-example/">Mobile gaming example</a>,
it is an unbounded dataset. An unbounded dataset is never available for
processing at any one time, so the data must be processed using a streaming
pipeline that runs continuously. The dataset will only be complete up to a
@@ -1614,7 +1614,7 @@ python -m apache_beam.examples.streaming_wordcount \
<p>This example uses an unbounded dataset as input. The code reads Pub/Sub
messages from a Pub/Sub subscription or topic using
-<a href="https://beam.apache.org/releases/pydoc/2.6.0/apache_beam.io.gcp.pubsub.html#apache_beam.io.gcp.pubsub.ReadStringsFromPubSub"><code class="highlighter-rouge">beam.io.ReadStringsFromPubSub</code></a>.</p>
+<a href="https://beam.apache.org/releases/pydoc/2.7.0/apache_beam.io.gcp.pubsub.html#apache_beam.io.gcp.pubsub.ReadStringsFromPubSub"><code class="highlighter-rouge">beam.io.ReadStringsFromPubSub</code></a>.</p>
<div class="language-java highlighter-rouge"><pre class="highlight"><code> <span class="c1">// This example is not currently available for the Beam SDK for Java.</span>
</code></pre>
@@ -1640,7 +1640,7 @@ outputs.</p>
<p>This example uses an unbounded <code class="highlighter-rouge">PCollection</code> and streams the results to
Google Pub/Sub. The code formats the results and writes them to a Pub/Sub topic
-using <a href="https://beam.apache.org/releases/pydoc/2.6.0/apache_beam.io.gcp.pubsub.html#apache_beam.io.gcp.pubsub.WriteStringsToPubSub"><code class="highlighter-rouge">beam.io.WriteStringsToPubSub</code></a>.</p>
+using <a href="https://beam.apache.org/releases/pydoc/2.7.0/apache_beam.io.gcp.pubsub.html#apache_beam.io.gcp.pubsub.WriteStringsToPubSub"><code class="highlighter-rouge">beam.io.WriteStringsToPubSub</code></a>.</p>
<div class="language-java highlighter-rouge"><pre class="highlight"><code> <span class="c1">// This example is not currently available for the Beam SDK for Java.</span>
</code></pre>
diff --git a/website/generated-content/index.html b/website/generated-content/index.html
index dbdaf3d..8f3af7f 100644
--- a/website/generated-content/index.html
+++ b/website/generated-content/index.html
@@ -146,7 +146,7 @@ limitations under the License.
</div>
<div class="hero__ctas hero__ctas--first">
<a class="button button--primary" href="/get-started/beam-overview/">Learn more</a>
- <a class="button button--primary" href="/get-started/downloads/">Download Beam SDK 2.6.0</a>
+ <a class="button button--primary" href="/get-started/downloads/">Download Beam SDK 2.7.0</a>
</div>
<div class="hero__ctas">
<a class="button" href="/get-started/quickstart-java/">Java Quickstart</a>
@@ -162,6 +162,11 @@ limitations under the License.
</div>
<div class="hero__blog__cards">
+ <a class="hero__blog__cards__card" href="/blog/2018/10/03/beam-2.7.0.html">
+ <div class="hero__blog__cards__card__title">Apache Beam 2.7.0</div>
+ <div class="hero__blog__cards__card__date">Oct 3, 2018</div>
+ </a>
+
<a class="hero__blog__cards__card" href="/blog/2018/08/21/beam-summit-europe.html">
<div class="hero__blog__cards__card__title">Beam Summit Europe 2018</div>
<div class="hero__blog__cards__card__date">Aug 21, 2018</div>
@@ -172,11 +177,6 @@ limitations under the License.
<div class="hero__blog__cards__card__date">Aug 20, 2018</div>
</a>
- <a class="hero__blog__cards__card" href="/blog/2018/08/10/beam-2.6.0.html">
- <div class="hero__blog__cards__card__title">Apache Beam 2.6.0</div>
- <div class="hero__blog__cards__card__date">Aug 10, 2018</div>
- </a>
-
</div>
</div>
</div>
@@ -325,7 +325,7 @@ limitations under the License.
</div>
<div class="ctas__ctas ctas__ctas--top">
<a class="button button--primary" href="/get-started/beam-overview/">Learn more</a>
- <a class="button button--primary" href="/get-started/downloads/">Download Beam SDK 2.6.0</a>
+ <a class="button button--primary" href="/get-started/downloads/">Download Beam SDK 2.7.0</a>
</div>
<div class="ctas__ctas">
<a class="button" href="/get-started/quickstart-java/">Java Quickstart</a>
[beam] 05/11: Publishing website 2018/10/04 22:23:47 at commit
3de49ec
Posted by sc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 316b5ca2f87648151a0fb54bb12c9e4235fb208c
Author: jenkins <bu...@apache.org>
AuthorDate: Thu Oct 4 22:23:48 2018 +0000
Publishing website 2018/10/04 22:23:47 at commit 3de49ec
---
website/generated-content/.htaccess | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/generated-content/.htaccess b/website/generated-content/.htaccess
index 77dabf4..b36da9e 100644
--- a/website/generated-content/.htaccess
+++ b/website/generated-content/.htaccess
@@ -21,4 +21,4 @@ RewriteRule ^(.*)$ https://beam.apache.org/$1 [L,R=301]
# The following redirect maintains the previously supported URLs.
RedirectMatch permanent "/documentation/sdks/(javadoc|pydoc)(.*)" "https://beam.apache.org/documentation/releases/$1$2"
# Keep this updated to point to the current release.
-RedirectMatch "/releases/([^/]+)/current(.*)" "https://beam.apache.org/documentation/releases/$1/2.6.0$2"
+RedirectMatch "/releases/([^/]+)/current(.*)" "https://beam.apache.org/releases/$1/2.6.0$2"
[beam] 11/11: Publishing website 2018/10/10 19:58:02 at commit
9be4fb9
Posted by sc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 56bfddf46f6a85d521eae97a0dc04823481203e5
Author: jenkins <bu...@apache.org>
AuthorDate: Wed Oct 10 19:58:03 2018 +0000
Publishing website 2018/10/10 19:58:02 at commit 9be4fb9
---
website/generated-content/get-started/wordcount-example/index.html | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/generated-content/get-started/wordcount-example/index.html b/website/generated-content/get-started/wordcount-example/index.html
index 1e4dbac..bde4edc 100644
--- a/website/generated-content/get-started/wordcount-example/index.html
+++ b/website/generated-content/get-started/wordcount-example/index.html
@@ -335,7 +335,7 @@ show other best practices.</p>
<strong><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount_minimal.py">wordcount_minimal.py</a>.</strong></p>
<p class="language-go">To view the full code in Go, see
-<strong><a href="https://github.com/apache/beam/blob/master/sdks/go/examples/minimal_wordcount/minimal_wordcount.go">wordcount_minimal.go</a>.</strong></p>
+<strong><a href="https://github.com/apache/beam/blob/master/sdks/go/examples/minimal_wordcount/minimal_wordcount.go">minimal_wordcount.go</a>.</strong></p>
<p><strong>Key Concepts:</strong></p>
[beam] 10/11: Publishing website 2018/10/09 18:41:21 at commit
ddbca9d
Posted by sc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 167811e6e8cc54a78b84f9af841221834192843b
Author: jenkins <bu...@apache.org>
AuthorDate: Tue Oct 9 18:41:22 2018 +0000
Publishing website 2018/10/09 18:41:21 at commit ddbca9d
---
website/generated-content/contribute/index.html | 3 +++
1 file changed, 3 insertions(+)
diff --git a/website/generated-content/contribute/index.html b/website/generated-content/contribute/index.html
index 6259dea..54e6b99 100644
--- a/website/generated-content/contribute/index.html
+++ b/website/generated-content/contribute/index.html
@@ -290,6 +290,9 @@ you want to be assigned issues, a PMC member can add you to the project contribu
group. Email the <a href="/community/contact-us">dev@ mailing list</a>
to ask to be added as a contributor in the Beam issue tracker.</p>
+<p>Anyone can browse the <a href="https://cwiki.apache.org/confluence/display/BEAM/Apache+Beam">Beam Wiki Space</a>.
+If you wish to contribute changes, please request edit access on the <a href="/community/contact-us">dev@ mailing list</a>.</p>
+
<h2 id="contributing-code">Contributing code</h2>
<p>Discussons about contributing code to beam happens on the <a href="/community/contact-us/">dev@ mailing list</a>. Introduce yourself!</p>