You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by me...@apache.org on 2018/08/23 19:16:26 UTC
[beam-site] branch asf-site updated (d3a72da -> 423f9e8)
This is an automated email from the ASF dual-hosted git repository.
mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.
from d3a72da Prepare repository for deployment.
add e5d0c8f Beam Summit blogpost
add d8070cb This closes #542
new 423f9e8 Prepare repository for deployment.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../2018/08/21/beam-summit-europe.html} | 61 +-
content/blog/index.html | 19 +
content/feed.xml | 639 ++-------------------
content/images/blog/Facebook-AD.png | Bin 0 -> 1275709 bytes
content/index.html | 10 +-
src/_data/authors.yml | 4 +
src/_posts/2018-08-21-beam-summit-europe.md | 40 ++
src/images/blog/Facebook-AD.png | Bin 0 -> 1275709 bytes
8 files changed, 141 insertions(+), 632 deletions(-)
copy content/{beam/capability/2016/04/03/presentation-materials.html => blog/2018/08/21/beam-summit-europe.html} (74%)
create mode 100644 content/images/blog/Facebook-AD.png
create mode 100644 src/_posts/2018-08-21-beam-summit-europe.md
create mode 100644 src/images/blog/Facebook-AD.png
[beam-site] 01/01: Prepare repository for deployment.
Posted by me...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git
commit 423f9e8e70fca762385f58e85743a62b87ffb3d0
Author: Mergebot <me...@apache.org>
AuthorDate: Thu Aug 23 19:16:24 2018 +0000
Prepare repository for deployment.
---
.../2018/08/21/beam-summit-europe.html} | 265 ++-------
content/blog/index.html | 19 +
content/feed.xml | 639 ++-------------------
content/images/blog/Facebook-AD.png | Bin 0 -> 1275709 bytes
content/index.html | 10 +-
5 files changed, 124 insertions(+), 809 deletions(-)
diff --git a/content/index.html b/content/blog/2018/08/21/beam-summit-europe.html
similarity index 53%
copy from content/index.html
copy to content/blog/2018/08/21/beam-summit-europe.html
index 33e68cf..db5f28c 100644
--- a/content/index.html
+++ b/content/blog/2018/08/21/beam-summit-europe.html
@@ -28,9 +28,8 @@
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
- <title>Apache Beam</title>
- <meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow [...]
-">
+ <title>Beam Summit Europe 2018</title>
+ <meta name="description" content="With a growing community of contributors and users, the Apache Beam project is organising the first European Beam Summit.We are happy to invite you to this e...">
<link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
<link rel="stylesheet" href="/css/site.css">
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
@@ -39,7 +38,7 @@
<script src="/js/fix-menu.js"></script>
<script src="/js/section-nav.js"></script>
<script src="/js/page-nav.js"></script>
- <link rel="canonical" href="https://beam.apache.org/" data-proofer-ignore>
+ <link rel="canonical" href="https://beam.apache.org/blog/2018/08/21/beam-summit-europe.html" data-proofer-ignore>
<link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
<link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
<script>
@@ -52,7 +51,7 @@
</script>
</head>
- <body class="body body--index">
+ <body class="body ">
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -121,218 +120,66 @@
<div class="body__contained">
<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
-->
-<div class="hero-bg">
- <div class="hero section">
- <div class="hero__cols">
- <div class="hero__cols__col">
- <div class="hero__cols__col__content">
- <div class="hero__title">
- Apache Beam: An advanced unified programming model
- </div>
- <div class="hero__subtitle">
- Implement batch and streaming data processing jobs that run on any execution engine.
- </div>
- <div class="hero__ctas hero__ctas--first">
- <a class="button button--primary" href="/get-started/beam-overview/">Learn more</a>
- <a class="button button--primary" href="/get-started/downloads/">Download Beam SDK 2.6.0</a>
- </div>
- <div class="hero__ctas">
- <a class="button" href="/get-started/quickstart-java/">Java Quickstart</a>
- <a class="button" href="/get-started/quickstart-py/">Python Quickstart</a>
- <a class="button" href="/get-started/quickstart-go/">Go Quickstart</a>
- </div>
- </div>
- </div>
- <div class="hero__cols__col">
- <div class="hero__blog">
- <div class="hero__blog__title">
- The latest from the blog
- </div>
- <div class="hero__blog__cards">
-
- <a class="hero__blog__cards__card" href="/blog/2018/08/20/review-input-streaming-connectors.html">
- <div class="hero__blog__cards__card__title">A review of input streaming connectors</div>
- <div class="hero__blog__cards__card__date">Aug 20, 2018</div>
- </a>
-
- <a class="hero__blog__cards__card" href="/blog/2018/08/10/beam-2.6.0.html">
- <div class="hero__blog__cards__card__title">Apache Beam 2.6.0</div>
- <div class="hero__blog__cards__card__date">Aug 10, 2018</div>
- </a>
-
- <a class="hero__blog__cards__card" href="/blog/2018/06/26/beam-2.5.0.html">
- <div class="hero__blog__cards__card__title">Apache Beam 2.5.0</div>
- <div class="hero__blog__cards__card__date">Jun 26, 2018</div>
- </a>
-
- </div>
- </div>
- </div>
- </div>
- </div>
-</div>
-<div class="pillars section">
- <div class="pillars__title">
- All about Apache Beam
- </div>
- <div class="pillars__cols">
-
- <div class="pillars__cols__col">
- <div class="pillars__cols__col__title">
- Unified
- </div>
- <div class="pillars__cols__col__body">
- Use a single programming model for both batch and streaming use cases.
- </div>
- </div>
-
- <div class="pillars__cols__col">
- <div class="pillars__cols__col__title">
- Portable
- </div>
- <div class="pillars__cols__col__body">
- Execute pipelines on multiple execution environments.
- </div>
- </div>
-
- <div class="pillars__cols__col">
- <div class="pillars__cols__col__title">
- Extensible
- </div>
- <div class="pillars__cols__col__body">
- Write and share new SDKs, IO connectors, and transformation libraries.
- </div>
- </div>
-
- </div>
-</div>
-<div class="graphic section">
-<div class="graphic__image">
-<img src="/images/beam_architecture.png" alt="Beam architecture" />
-</div>
-</div>
-<div class="logos section">
- <div class="logos__title">
- Works with
- </div>
- <div class="logos__logos">
-
- <div class="logos__logos__logo">
- <a href="http://apex.apache.org"><img src="/images/logo_apex.png" alt="APEX" /></a>
- </div>
-
- <div class="logos__logos__logo">
- <a href="http://flink.apache.org"><img src="/images/logo_flink.png" alt="Flink" /></a>
- </div>
-
- <div class="logos__logos__logo">
- <a href="http://spark.apache.org/"><img src="/images/logo_spark.png" alt="Spark" /></a>
- </div>
-
- <div class="logos__logos__logo">
- <a href="https://cloud.google.com/dataflow/"><img src="/images/logo_google_cloud.png" alt="Google Cloud Dataflow" /></a>
- </div>
-
- <div class="logos__logos__logo">
- <a href="http://gearpump.apache.org/"><img src="/images/logo_gearpump.png" alt="Gearpump" /></a>
- </div>
-
- <div class="logos__logos__logo">
- <a href="http://samza.apache.org/"><img src="/images/logo_samza.png" alt="Samza" /></a>
- </div>
-
- </div>
-</div>
+<article class="post" itemscope itemtype="http://schema.org/BlogPosting">
-<div class="cards section section--wide">
- <div class="section__contained">
- <div class="cards__title">
- Testimonials
- </div>
- <div class="cards__cards">
+ <header class="post-header">
+ <h1 class="post-title" itemprop="name headline">Beam Summit Europe 2018</h1>
+ <p class="post-meta"><time datetime="2018-08-21T01:00:01-07:00" itemprop="datePublished">Aug 21, 2018</time> •
+ Matthias Baetens [<a href="https://twitter.com/matthiasbaetens">@matthiasbaetens</a>]
- <div class="cards__cards__card">
- <div class="cards__cards__card__body">
- A framework that delivers the flexibility and advanced functionality our customers need.
- </div>
- <div class="cards__cards__card__user">
- <!-- TODO: Implement icons.
- <div class="cards__cards__card__user__icon">
- </div>
- -->
- <div class="cards__cards__card__user__name">
- –Talend
- </div>
- </div>
- </div>
-
- <div class="cards__cards__card">
- <div class="cards__cards__card__body">
- Apache Beam has powerful semantics that solve real-world challenges of stream processing.
- </div>
- <div class="cards__cards__card__user">
- <!-- TODO: Implement icons.
- <div class="cards__cards__card__user__icon">
- </div>
- -->
- <div class="cards__cards__card__user__name">
- –PayPal
- </div>
- </div>
- </div>
-
- <div class="cards__cards__card">
- <div class="cards__cards__card__body">
- Apache Beam represents a principled approach for analyzing data streams.
- </div>
- <div class="cards__cards__card__user">
- <!-- TODO: Implement icons.
- <div class="cards__cards__card__user__icon">
- </div>
- -->
- <div class="cards__cards__card__user__name">
- –data Artisans
- </div>
- </div>
- </div>
-
- </div>
- <div class="cards__body">
- Beam is an open source community and contributions are greatly appreciated!
- If you’d like to contribute, please see the <a href="/contribute/">Contribute</a> section.
- </div>
- </div>
-</div>
+ </p>
+ </header>
+
+ <div class="post-content" itemprop="articleBody">
+ <p>With a growing community of contributors and users, the Apache Beam project is organising the first European Beam Summit.</p>
+
+<p>We are happy to invite you to this event, which will take place in <strong>London</strong> on <strong>October 1st and 2nd of 2018</strong>. <!--more--></p>
+
+<p><img src="/images/blog/Facebook-AD.png" alt="Beam Summit Europe 2018 flyer" height="360" width="640" /></p>
+
+<h3 id="what-is-the-beam-summit-2018">What is the Beam Summit 2018?</h3>
+<p>The summit is a 2 day, multi-track event.</p>
+
+<p>During the first day we’ll host sessions to share use cases from companies using Apache Beam, community driven talks, and a session to discuss the project’s roadmap (from the main partners in the project as well as all users planning to contribute to the project and wanting to share their plans). We’ll also have break-out sessions that will allow cross team collaboration in multiple sub-topics.</p>
+
+<p>The second day will be a “hands-on” day. We will offer an introductory session to Apache Beam. Additionally, we’ll host an advanced track for more advanced users with open-table discussions about more complex and newer Apache Beam features.</p>
+
+<p>The agenda will grow and be communicated in the coming month, keep an eye on the page.</p>
+
+<h3 id="event-details">Event Details</h3>
+<ul>
+ <li><strong>Venue</strong>: <a href="https://goo.gl/maps/LAC4haDzSzR2">Level39, One Canada Square, Canary Wharf, London E14 5AB</a></li>
+ <li><strong>Dates</strong>: 1-2 October 2018</li>
+</ul>
+
+<h3 id="how-do-i-register">How do I register?</h3>
+<p>You can register for free on the <a href="https://www.eventbrite.com/e/beam-summit-london-2018-tickets-49100625292#tickets">Eventbrite registration page</a>.</p>
+
+<h3 id="i-am-interested-in-speaking-how-do-i-propose-my-session">I am interested in speaking, how do I propose my session?</h3>
+<p>With this we are also launching a Call for Papers in case you want to secure a slot for one of the sessions. Please fill out the <a href="https://goo.gl/forms/nrZOCC1JwEfLtKfA2">CfP form</a>.</p>
+
+<h3 id="id-love-to-get-involved-as-a-volunteer-or-sponsor">I’d love to get involved as a volunteer or sponsor</h3>
+<p>Furthermore, in order to keep this event free, we are looking for people to help out at and/or sponsor some parts of the conference. If you (or your company) are interested to help out, please reach out to: <a href="mailto:baetensmatthias@gmail.com">baetensmatthias@gmail.com</a> or <a href="mailto:alex@vanboxel.be">alex@vanboxel.be</a>. You can find more info in the <a href="https://drive.google.com/file/d/1RnZ52rGaB6BR-EKneBcabdMcg9Pl7z9M">sponsor booklet</a></p>
+
+<p>Thanks, and we hope to see you at the event!
+The Events & Meetups Group</p>
-<div class="ctas section">
- <div class="ctas__title">
- Get started
- </div>
- <div class="ctas__ctas ctas__ctas--top">
- <a class="button button--primary" href="/get-started/beam-overview/">Learn more</a>
- <a class="button button--primary" href="/get-started/downloads/">Download Beam SDK 2.6.0</a>
- </div>
- <div class="ctas__ctas">
- <a class="button" href="/get-started/quickstart-java/">Java Quickstart</a>
- <a class="button" href="/get-started/quickstart-py/">Python Quickstart</a>
- <a class="button" href="/get-started/quickstart-go/">Go Quickstart</a>
</div>
-</div>
+
+</article>
</div>
<!--
diff --git a/content/blog/index.html b/content/blog/index.html
index 08b2478..23508c8 100644
--- a/content/blog/index.html
+++ b/content/blog/index.html
@@ -139,6 +139,25 @@ limitations under the License.
<p>This is the blog for the Apache Beam project. This blog contains news and updates
for the project.</p>
+<h3 id="beam-summit-europe-2018"><a class="post-link" href="/blog/2018/08/21/beam-summit-europe.html">Beam Summit Europe 2018</a></h3>
+<p><i>Aug 21, 2018 •
+ Matthias Baetens [<a href="https://twitter.com/matthiasbaetens">@matthiasbaetens</a>]
+</i></p>
+
+<p>With a growing community of contributors and users, the Apache Beam project is organising the first European Beam Summit.</p>
+
+<p>We are happy to invite you to this event, which will take place in <strong>London</strong> on <strong>October 1st and 2nd of 2018</strong>.</p>
+
+<!-- Render a "read more" button if the post is longer than the excerpt -->
+
+<p>
+<a class="btn btn-default btn-sm" href="/blog/2018/08/21/beam-summit-europe.html" role="button">
+Read more <span class="glyphicon glyphicon-menu-right" aria-hidden="true"></span>
+</a>
+</p>
+
+<hr />
+
<h3 id="a-review-of-input-streaming-connectors"><a class="post-link" href="/blog/2018/08/20/review-input-streaming-connectors.html">A review of input streaming connectors</a></h3>
<p><i>Aug 20, 2018 •
Leonid Kuligin [<a href="https://twitter.com/lkulighin">@lkulighin</a>] & Julien Phalip [<a href="https://twitter.com/julienphalip">@julienphalip</a>]
diff --git a/content/feed.xml b/content/feed.xml
index 4fa53fd..cb30250 100644
--- a/content/feed.xml
+++ b/content/feed.xml
@@ -20,6 +20,50 @@
<generator>Jekyll v3.2.0</generator>
<item>
+ <title>Beam Summit Europe 2018</title>
+ <description><p>With a growing community of contributors and users, the Apache Beam project is organising the first European Beam Summit.</p>
+
+<p>We are happy to invite you to this event, which will take place in <strong>London</strong> on <strong>October 1st and 2nd of 2018</strong>. <!--more--></p>
+
+<p><img src="/images/blog/Facebook-AD.png" alt="Beam Summit Europe 2018 flyer" height="360" width="640" /></p>
+
+<h3 id="what-is-the-beam-summit-2018">What is the Beam Summit 2018?</h3>
+<p>The summit is a 2 day, multi-track event.</p>
+
+<p>During the first day we’ll host sessions to share use cases from companies using Apache Beam, community driven talks, and a session to discuss the project’s roadmap (from the main partners in the project as well as all users planning to contribute to the project and wanting to share their plans). We’ll also have break-out sessions that will allow cross team collaboration in multiple sub-topics.</p>
+
+<p>The second day will be a “hands-on” day. We will offer an introductory session to Apache Beam. Additionally, we’ll host an advanced track for more advanced users with open-table discussions about more complex and newer Apache Beam features.</p>
+
+<p>The agenda will grow and be communicated in the coming month, keep an eye on the page.</p>
+
+<h3 id="event-details">Event Details</h3>
+<ul>
+ <li><strong>Venue</strong>: <a href="https://goo.gl/maps/LAC4haDzSzR2">Level39, One Canada Square, Canary Wharf, London E14 5AB</a></li>
+ <li><strong>Dates</strong>: 1-2 October 2018</li>
+</ul>
+
+<h3 id="how-do-i-register">How do I register?</h3>
+<p>You can register for free on the <a href="https://www.eventbrite.com/e/beam-summit-london-2018-tickets-49100625292#tickets">Eventbrite registration page</a>.</p>
+
+<h3 id="i-am-interested-in-speaking-how-do-i-propose-my-session">I am interested in speaking, how do I propose my session?</h3>
+<p>With this we are also launching a Call for Papers in case you want to secure a slot for one of the sessions. Please fill out the <a href="https://goo.gl/forms/nrZOCC1JwEfLtKfA2">CfP form</a>.</p>
+
+<h3 id="id-love-to-get-involved-as-a-volunteer-or-sponsor">I’d love to get involved as a volunteer or sponsor</h3>
+<p>Furthermore, in order to keep this event free, we are looking for people to help out at and/or sponsor some parts of the conference. If you (or your company) are interested to help out, please reach out to: <a href="mailto:baetensmatthias@gmail.com">baetensmatthias@gmail.com</a> or <a href="mailto:alex@vanboxel.be">alex@vanboxel.be</a>. You can find more info in the <a href="https://drive.google.com/file/d/1RnZ52rGaB6BR-EKneBca [...]
+
+<p>Thanks, and we hope to see you at the event!
+The Events &amp; Meetups Group</p>
+</description>
+ <pubDate>Tue, 21 Aug 2018 01:00:01 -0700</pubDate>
+ <link>https://beam.apache.org/blog/2018/08/21/beam-summit-europe.html</link>
+ <guid isPermaLink="true">https://beam.apache.org/blog/2018/08/21/beam-summit-europe.html</guid>
+
+
+ <category>blog</category>
+
+ </item>
+
+ <item>
<title>A review of input streaming connectors</title>
<description><p>In this post, you’ll learn about the current state of support for input streaming connectors in <a href="/">Apache Beam</a>. For more context, you’ll also learn about the corresponding state of support in <a href="https://spark.apache.org/">Apache Spark</a>.<!--more--></p>
@@ -2013,600 +2057,5 @@ p.run()
</item>
- <item>
- <title>Stateful processing with Apache Beam</title>
- <description><!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-<p>Beam lets you process unbounded, out-of-order, global-scale data with portable
-high-level pipelines. Stateful processing is a new feature of the Beam model
-that expands the capabilities of Beam, unlocking new use cases and new
-efficiencies. In this post, I will guide you through stateful processing in
-Beam: how it works, how it fits in with the other features of the Beam model,
-what you might use it for, and what it looks like in code.</p>
-
-<!--more-->
-
-<blockquote>
- <p><strong>Warning: new features ahead!</strong>: This is a very new aspect of the Beam
-model. Runners are still adding support. You can try it out today on multiple
-runners, but do check the <a href="/documentation/runners/capability-matrix/">runner capability
-matrix</a> for
-the current status in each runner.</p>
-</blockquote>
-
-<p>First, a quick recap: In Beam, a big data processing <em>pipeline</em> is a directed,
-acyclic graph of parallel operations called <em><code class="highlighter-rouge">PTransforms</code></em> processing data
-from <em><code class="highlighter-rouge">PCollections</code></em>. I’ll expand on that by walking through this illustration:</p>
-
-<p><img class="center-block" src="/images/blog/stateful-processing/pipeline.png" alt="A Beam Pipeline - PTransforms are boxes - PCollections are arrows" width="300" /></p>
-
-<p>The boxes are <code class="highlighter-rouge">PTransforms</code> and the edges represent the data in <code class="highlighter-rouge">PCollections</code>
-flowing from one <code class="highlighter-rouge">PTransform</code> to the next. A <code class="highlighter-rouge">PCollection</code> may be <em>bounded</em> (which
-means it is finite and you know it) or <em>unbounded</em> (which means you don’t know if
-it is finite or not - basically, it is like an incoming stream of data that may
-or may not ever terminate). The cylinders are the data sources and sinks at the
-edges of your pipeline, such as bounded collections of log files or unbounded
-data streaming over a Kafka topic. This blog post isn’t about sources or sinks,
-but about what happens in between - your data processing.</p>
-
-<p>There are two main building blocks for processing your data in Beam: <em><code class="highlighter-rouge">ParDo</code></em>,
-for performing an operation in parallel across all elements, and <em><code class="highlighter-rouge">GroupByKey</code></em>
-(and the closely related <code class="highlighter-rouge">CombinePerKey</code> that I will talk about quite soon)
-for aggregating elements to which you have assigned the same key. In the
-picture below (featured in many of our presentations) the color indicates the
-key of the element. Thus the <code class="highlighter-rouge">GroupByKey</code>/<code class="highlighter-rouge">CombinePerKey</code> transform gathers all the
-green squares to produce a single output element.</p>
-
-<p><img class="center-block" src="/images/blog/stateful-processing/pardo-and-gbk.png" alt="ParDo and GroupByKey/CombinePerKey: Elementwise versus aggregating computations" width="400" /></p>
-
-<p>But not all use cases are easily expressed as pipelines of simple <code class="highlighter-rouge">ParDo</code>/<code class="highlighter-rouge">Map</code> and
-<code class="highlighter-rouge">GroupByKey</code>/<code class="highlighter-rouge">CombinePerKey</code> transforms. The topic of this blog post is a new
-extension to the Beam programming model: <strong>per-element operation augmented with
-mutable state</strong>.</p>
-
-<p><img class="center-block" src="/images/blog/stateful-processing/stateful-pardo.png" alt="Stateful ParDo - sequential per-key processing with persistent state" width="300" /></p>
-
-<p>In the illustration above, ParDo now has a bit of durable, consistent state on
-the side, which can be read and written during the processing of each element.
-The state is partitioned by key, so it is drawn as having disjoint sections for
-each color. It is also partitioned per window, but I thought plaid
-<img src="/images/blog/stateful-processing/plaid.png" alt="A plaid storage cylinder" width="20" />
-would be a bit much :-). I’ll talk about
-why state is partitioned this way a bit later, via my first example.</p>
-
-<p>For the rest of this post, I will describe this new feature of Beam in detail -
-how it works at a high level, how it differs from existing features, how to
-make sure it is still massively scalable. After that introduction at the model
-level, I’ll walk through a simple example of how you use it in the Beam Java
-SDK.</p>
-
-<h2 id="how-does-stateful-processing-in-beam-work">How does stateful processing in Beam work?</h2>
-
-<p>The processing logic of your <code class="highlighter-rouge">ParDo</code> transform is expressed through the <code class="highlighter-rouge">DoFn</code>
-that it applies to each element. Without stateful augmentations, a <code class="highlighter-rouge">DoFn</code> is a
-mostly-pure function from inputs to one or more outputs, corresponding to the
-Mapper in a MapReduce. With state, a <code class="highlighter-rouge">DoFn</code> has the ability to access
-persistent mutable state while processing each input element. Consider this
-illustration:</p>
-
-<p><img class="center-block" src="/images/blog/stateful-processing/stateful-dofn.png" alt="Stateful DoFn - the runner controls input but the DoFn controls storage and output" width="300" /></p>
-
-<p>The first thing to note is that all the data - the little squares, circles, and
-triangles - are red. This is to illustrate that stateful processing occurs in
-the context of a single key - all of the elements are key-value pairs with the
-same key. Calls from your chosen Beam runner to the <code class="highlighter-rouge">DoFn</code> are colored in
-yellow, while calls from the <code class="highlighter-rouge">DoFn</code> to the runner are in purple:</p>
-
-<ul>
- <li>The runner invokes the <code class="highlighter-rouge">DoFn</code>’s <code class="highlighter-rouge">@ProcessElement</code> method on each element for a
-key+window.</li>
- <li>The <code class="highlighter-rouge">DoFn</code> reads and writes state - the curved arrows to/from the storage on
-the side.</li>
- <li>The <code class="highlighter-rouge">DoFn</code> emits output (or side output) to the runner as usual via
-<code class="highlighter-rouge">ProcessContext.output</code> (resp. <code class="highlighter-rouge">ProcessContext.sideOutput</code>).</li>
-</ul>
-
-<p>At this very high level, it is pretty intuitive: In your programming
-experience, you have probably at some point written a loop over elements that
-updates some mutable variables while performing other actions. The interesting
-question is how does this fit into the Beam model: how does it relate with
-other features? How does it scale, since state implies some synchronization?
-When should it be used versus other features?</p>
-
-<h2 id="how-does-stateful-processing-fit-into-the-beam-model">How does stateful processing fit into the Beam model?</h2>
-
-<p>To see where stateful processing fits in the Beam model, consider another
-way that you can keep some “state” while processing many elements: CombineFn. In
-Beam, you can write <code class="highlighter-rouge">Combine.perKey(CombineFn)</code> in Java or Python to apply an
-associative, commutative accumulating operation across all the elements with a
-common key (and window).</p>
-
-<p>Here is a diagram illustrating the basics of a <code class="highlighter-rouge">CombineFn</code>, the simplest way
-that a runner might invoke it on a per-key basis to build an accumulator and
-extract an output from the final accumulator:</p>
-
-<p><img class="center-block" src="/images/blog/stateful-processing/combinefn.png" alt="CombineFn - the runner controls input, storage, and output" width="300" /></p>
-
-<p>As with the illustration of stateful <code class="highlighter-rouge">DoFn</code>, all the data is colored red, since
-this is the processing of Combine for a single key. The illustrated method
-calls are colored yellow, since they are all controlled by the runner: The
-runner invokes <code class="highlighter-rouge">addInput</code> on each method to add it to the current accumulator.</p>
-
-<ul>
- <li>The runner persists the accumulator when it chooses.</li>
- <li>The runner calls <code class="highlighter-rouge">extractOutput</code> when ready to emit an output element.</li>
-</ul>
-
-<p>At this point, the diagram for <code class="highlighter-rouge">CombineFn</code> looks a whole lot like the diagram
-for stateful <code class="highlighter-rouge">DoFn</code>. In practice, the flow of data is, indeed, quite similar.
-But there are important differences, even so:</p>
-
-<ul>
- <li>The runner controls all invocations and storage here. You do not decide when
-or how state is persisted, when an accumulator is discarded (based on
-triggering) or when output is extracted from an accumulator.</li>
- <li>You can only have one piece of state - the accumulator. In a stateful DoFn
-you can read only what you need to know and write only what has changed.</li>
- <li>You don’t have the extended features of <code class="highlighter-rouge">DoFn</code>, such as multiple outputs per
-input or side outputs. (These could be simulated by a sufficient complex
-accumulator, but it would not be natural or efficient. Some other features of
-<code class="highlighter-rouge">DoFn</code> such as side inputs and access to the window make perfect sense for
-<code class="highlighter-rouge">CombineFn</code>)</li>
-</ul>
-
-<p>But the main thing that <code class="highlighter-rouge">CombineFn</code> allows a runner to do is to
-<code class="highlighter-rouge">mergeAccumulators</code>, the concrete expression of the <code class="highlighter-rouge">CombineFn</code>’s associativity.
-This unlocks some huge optimizations: the runner can invoke multiple instances
-of a <code class="highlighter-rouge">CombineFn</code> on a number of inputs and later combine them in a classic
-divide-and-conquer architecture, as in this picture:</p>
-
-<p><img class="center-block" src="/images/blog/stateful-processing/combiner-lifting.png" alt="Divide-and-conquer aggregation with a CombineFn" width="600" /></p>
-
-<p>The contract of a <code class="highlighter-rouge">CombineFn</code> is that the result should be exactly the same,
-whether or not the runner decides to actually do such a thing, or even more
-complex trees with hot-key fanout, etc.</p>
-
-<p>This merge operation is not (necessarily) provided by a stateful <code class="highlighter-rouge">DoFn</code>: the
-runner cannot freely branch its execution and recombine the states. Note that
-the input elements are still received in an arbitrary order, so the <code class="highlighter-rouge">DoFn</code> should
-be insensitive to ordering and bundling but it doesn’t mean the output must be
-exactly equal. (fun and easy fact: if the outputs are actually always equal,
-then the <code class="highlighter-rouge">DoFn</code> is an associative and commutative operator)</p>
-
-<p>So now you can see how a stateful <code class="highlighter-rouge">DoFn</code> differs from <code class="highlighter-rouge">CombineFn</code>, but I want to
-step back and extrapolate this to a high level picture of how state in Beam
-relates to using other features to achieve the same or similar goals: In a lot
-of cases, what stateful processing represents is a chance to “get under the
-hood” of the highly abstract mostly-deterministic functional paradigm of Beam
-and do potentially-nondeterministic imperative-style programming that is hard
-to express any other way.</p>
-
-<h2 id="example-arbitrary-but-consistent-index-assignment">Example: arbitrary-but-consistent index assignment</h2>
-
-<p>Suppose that you want to give an index to every incoming element for a
-key-and-window. You don’t care what the indices are, just as long as they are
-unique and consistent. Before diving into the code for how to do this in a Beam
-SDK, I’ll go over this example from the level of the model. In pictures, you
-want to write a transform that maps input to output like this:</p>
-
-<p><img class="center-block" src="/images/blog/stateful-processing/assign-indices.png" alt="Assigning arbitrary but unique indices to each element" width="180" /></p>
-
-<p>The order of the elements A, B, C, D, E is arbitrary, hence their assigned
-indices are arbitrary, but downstream transforms just need to be OK with this.
-There is no associativity or commutativity as far as the actual values are
-concerned. The order-insensitivity of this transform only extends to the point
-of ensuring the necessary properties of the output: no duplicated indices, no
-gaps, and every element gets an index.</p>
-
-<p>Conceptually expressing this as a stateful loop is as trivial as you can
-imagine: The state you should store is the next index.</p>
-
-<ul>
- <li>As an element comes in, output it along with the next index.</li>
- <li>Increment the index.</li>
-</ul>
-
-<p>This presents a good opportunity to talk about big data and parallelism,
-because the algorithm in those bullet points is not parallelizable at all! If
-you wanted to apply this logic over an entire <code class="highlighter-rouge">PCollection</code>, you would have to
-process each element of the <code class="highlighter-rouge">PCollection</code> one-at-a-time… this is obvious a
-bad idea. State in Beam is tightly scoped so that most of the time a stateful
-<code class="highlighter-rouge">ParDo</code> transform should still be possible for a runner to execute in parallel,
-though you still have to be thoughtful about it.</p>
-
-<p>A state cell in Beam is scoped to a key+window pair. When your DoFn reads or
-writes state by the name of <code class="highlighter-rouge">"index"</code>, it is actually accessing a mutable cell
-specified by <code class="highlighter-rouge">"index"</code> <em>along with</em> the key and window currently being
-processed. So, when thinking about a state cell, it may be helpful to consider
-the full state of your transform as a table, where the rows are named according
-to names you use in your program, like <code class="highlighter-rouge">"index"</code>, and the columns are
-key+window pairs, like this:</p>
-
-<table class="table">
- <thead>
- <tr>
- <th> </th>
- <th>(key, window)<sub>1</sub></th>
- <th>(key, window)<sub>2</sub></th>
- <th>(key, window)<sub>3</sub></th>
- <th>…</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td><code class="highlighter-rouge">"index"</code></td>
- <td><code class="highlighter-rouge">3</code></td>
- <td><code class="highlighter-rouge">7</code></td>
- <td><code class="highlighter-rouge">15</code></td>
- <td>…</td>
- </tr>
- <tr>
- <td><code class="highlighter-rouge">"fizzOrBuzz?"</code></td>
- <td><code class="highlighter-rouge">"fizz"</code></td>
- <td><code class="highlighter-rouge">"7"</code></td>
- <td><code class="highlighter-rouge">"fizzbuzz"</code></td>
- <td>…</td>
- </tr>
- <tr>
- <td>…</td>
- <td>…</td>
- <td>…</td>
- <td>…</td>
- <td>…</td>
- </tr>
- </tbody>
-</table>
-
-<p>(if you have a superb spatial sense, feel free to imagine this as a cube where
-keys and windows are independent dimensions)</p>
-
-<p>You can provide the opportunity for parallelism by making sure that table has
-enough columns. You might have many keys and many windows, or you might have
-many of just one or the other:</p>
-
-<ul>
- <li>Many keys in few windows, for example a globally windowed stateful computation
-keyed by user ID.</li>
- <li>Many windows over few keys, for example a fixed windowed stateful computation
-over a global key.</li>
-</ul>
-
-<p>Caveat: all Beam runners today parallelize only over the key.</p>
-
-<p>Most often your mental model of state can be focused on only a single column of
-the table, a single key+window pair. Cross-column interactions do not occur
-directly, by design.</p>
-
-<h2 id="state-in-beams-java-sdk">State in Beam’s Java SDK</h2>
-
-<p>Now that I have talked a bit about stateful processing in the Beam model and
-worked through an abstract example, I’d like to show you what it looks like to
-write stateful processing code using Beam’s Java SDK. Here is the code for a
-stateful <code class="highlighter-rouge">DoFn</code> that assigns an arbitrary-but-consistent index to each element
-on a per key-and-window basis:</p>
-
-<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">MyKey</span><span class="o">,</span> <span class="n"> [...]
-
- <span class="c1">// A state cell holding a single Integer per key+window</span>
- <span class="nd">@StateId</span><span class="o">(</span><span class="s">"index"</span><span class="o">)</span>
- <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o">&lt;</span><span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;&gt;</span> <span class="n">indexSpec</span> [...]
- <span class="n">StateSpecs</span><span class="o">.</span><span class="na">value</span><span class="o">(</span><span class="n">VarIntCoder</span><span class="o">.</span><span class="na">of</span><span class="o">());</span>
-
- <span class="nd">@ProcessElement</span>
- <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span>
- <span class="n">ProcessContext</span> <span class="n">context</span><span class="o">,</span>
- <span class="nd">@StateId</span><span class="o">(</span><span class="s">"index"</span><span class="o">)</span> <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="n">index</span><span clas [...]
- <span class="kt">int</span> <span class="n">current</span> <span class="o">=</span> <span class="n">firstNonNull</span><span class="o">(</span><span class="n">index</span><span class="o">.</span><span class="na">read</span><span class="o">(),</span> <span class="mi">0</s [...]
- <span class="n">context</span><span class="o">.</span><span class="na">output</span><span class="o">(</span><span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">current</span><span class="o">,</span> <sp [...]
- <span class="n">index</span><span class="o">.</span><span class="na">write</span><span class="o">(</span><span class="n">current</span><span class="o">+</span><span class="mi">1</span><span class="o">);</span>
- <span class="o">}</span>
-<span class="o">}</span>
-</code></pre>
-</div>
-
-<div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="c"># State and timers are not yet supported in Beam's Python SDK.</span>
-<span class="c"># Watch this space!</span>
-</code></pre>
-</div>
-
-<p>Let’s dissect this:</p>
-
-<ul>
- <li>The first thing to look at is the presence of a couple of <code class="highlighter-rouge">@StateId("index")</code>
-annotations. This calls out that you are using a mutable state cell named
-“index” in this <code class="highlighter-rouge">DoFn</code>. The Beam Java SDK, and from there your chosen runner,
-will also note these annotations and use them to wire up your DoFn correctly.</li>
- <li>The first <code class="highlighter-rouge">@StateId("index")</code> is annotated on a field of type <code class="highlighter-rouge">StateSpec</code> (for
-“state specification”). This declares and configures the state cell. The
-type parameter <code class="highlighter-rouge">ValueState</code> describes the kind of state you can get out of this
-cell - <code class="highlighter-rouge">ValueState</code> stores just a single value. Note that the spec itself is not
-a usable state cell - you need the runner to provide that during pipeline
-execution.</li>
- <li>To fully specify a <code class="highlighter-rouge">ValueState</code> cell, you need to provide the coder
-that the runner will use (as necessary) to serialize the value
-you will be storing. This is the invocation <code class="highlighter-rouge">StateSpecs.value(VarIntCoder.of())</code>.</li>
- <li>The second <code class="highlighter-rouge">@StateId("index")</code> annotation is on a parameter to your
-<code class="highlighter-rouge">@ProcessElement</code> method. This indicates access to the ValueState cell that
-was specified earlier.</li>
- <li>The state is accessed in the simplest way: <code class="highlighter-rouge">read()</code> to read it, and
-<code class="highlighter-rouge">write(newvalue)</code> to write it.</li>
- <li>The other features of <code class="highlighter-rouge">DoFn</code> are available in the usual way - such as
-<code class="highlighter-rouge">context.output(...)</code>. You can also use side inputs, side outputs, gain access
-to the window, etc.</li>
-</ul>
-
-<p>A few notes on how the SDK and runners see this DoFn:</p>
-
-<ul>
- <li>Your state cells are all explicitly declared so a Beam SDK or runner can
-reason about them, for example to clear them out when a window expires.</li>
- <li>If you declare a state cell and then use it with the wrong type, the Beam
-Java SDK will catch that error for you.</li>
- <li>If you declare two state cells with the same ID, the SDK will catch that,
-too.</li>
- <li>The runner knows that this is a stateful <code class="highlighter-rouge">DoFn</code> and may run it quite
-differently, for example by additional data shuffling and synchronization in
-order to avoid concurrent access to state cells.</li>
-</ul>
-
-<p>Let’s look at one more example of how to use this API, this time a bit more real-world.</p>
-
-<h2 id="example-anomaly-detection">Example: anomaly detection</h2>
-
-<p>Suppose you are feeding a stream of actions by your user into some complex
-model to predict some quantitative expression of the sorts of actions they
-take, for example to detect fraudulent activity. You will build up the model
-from events, and also compare incoming events against the latest model to
-determine if something has changed.</p>
-
-<p>If you try to express the building of your model as a <code class="highlighter-rouge">CombineFn</code>, you may have
-trouble with <code class="highlighter-rouge">mergeAccumulators</code>. Assuming you could express that, it might
-look something like this:</p>
-
-<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">ModelFromEventsFn</span> <span class="kd">extends</span> <span class="n">CombineFn</span><span class="o">&lt;</span><span class="n">Event</span><span class="o">,</span> <sp [...]
- <span class="nd">@Override</span>
- <span class="kd">public</span> <span class="kd">abstract</span> <span class="n">Model</span> <span class="nf">createAccumulator</span><span class="o">()</span> <span class="o">{</span>
- <span class="k">return</span> <span class="n">Model</span><span class="o">.</span><span class="na">empty</span><span class="o">();</span>
- <span class="o">}</span>
-
- <span class="nd">@Override</span>
- <span class="kd">public</span> <span class="kd">abstract</span> <span class="n">Model</span> <span class="nf">addInput</span><span class="o">(</span><span class="n">Model</span> <span class="n">accumulator</span><span class="o">,</span> <span class="n">Event</span> <span class="n&q [...]
- <span class="k">return</span> <span class="n">accumulator</span><span class="o">.</span><span class="na">update</span><span class="o">(</span><span class="n">input</span><span class="o">);</span> <span class="c1">// this is encouraged to mutate, for efficiency</span>
- <span class="o">}</span>
-
- <span class="nd">@Override</span>
- <span class="kd">public</span> <span class="kd">abstract</span> <span class="n">Model</span> <span class="nf">mergeAccumulators</span><span class="o">(</span><span class="n">Iterable</span><span class="o">&lt;</span><span class="n">Model</span><span class="o">&gt;</span> <span c [...]
- <span class="c1">// ?? can you write this ??</span>
- <span class="o">}</span>
-
- <span class="nd">@Override</span>
- <span class="kd">public</span> <span class="kd">abstract</span> <span class="n">Model</span> <span class="nf">extractOutput</span><span class="o">(</span><span class="n">Model</span> <span class="n">accumulator</span><span class="o">)</span> <span class="o">{</span>
- <span class="k">return</span> <span class="n">accumulator</span><span class="o">;</span> <span class="o">}</span>
-<span class="o">}</span>
-</code></pre>
-</div>
-
-<div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="c"># State and timers are not yet supported in Beam's Python SDK.</span>
-<span class="c"># Watch this space!</span>
-</code></pre>
-</div>
-
-<p>Now you have a way to compute the model of a particular user for a window as
-<code class="highlighter-rouge">Combine.perKey(new ModelFromEventsFn())</code>. How would you apply this model to
-the same stream of events from which it is calculated? A standard way to do
-take the result of a <code class="highlighter-rouge">Combine</code> transform and use it while processing the
-elements of a <code class="highlighter-rouge">PCollection</code> is to read it as a side input to a <code class="highlighter-rouge">ParDo</code>
-transform. So you could side input the model and check the stream of events
-against it, outputting the prediction, like so:</p>
-
-<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">UserId</span><span class="o">,</span> <span class="n">Event</span><span class="o& [...]
-
-<span class="kd">final</span> <span class="n">PCollectionView</span><span class="o">&lt;</span><span class="n">Map</span><span class="o">&lt;</span><span class="n">UserId</span><span class="o">,</span> <span class="n">Model</span><span class="o">&gt;&gt;</span> <span class=& [...]
- <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Combine</span><span class="o">.</span><span class="na">perKey</span><span class="o">(</span><span class="k">new</span> <span class="n">ModelFromEventsFn</span><span class="o">()))&l [...]
- <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">View</span><span class="o">.</span><span class="na">asMap</span><span class="o">());</span>
-
-<span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">UserId</span><span class="o">,</span> <span class="n">Prediction</span><span class="o">&gt;&gt;</span> <span class="n">predictions</span> <span cl [...]
- <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span>< [...]
-
- <span class="nd">@ProcessElement</span>
- <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span><span class="n">ProcessContext</span> <span class="n">ctx</span><span class="o">)</span> <span class="o">{</span>
- <span class="n">UserId</span> <span class="n">userId</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="na">element</span><span class="o">().</span><span class="na">getKey</span><span class="o">();</span>
- <span class="n">Event</span> <span class="n">event</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="na">element</span><span class="o">().</span><span class="na">getValue</span><span class="o">();</span>
-
- <span class="n">Model</span> <span class="n">model</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="na">sideinput</span><span class="o">(</span><span class="n">userModels</span><span class="o">).</span><span class="na">get&l [...]
-
- <span class="c1">// Perhaps some logic around when to output a new prediction</span>
- <span class="err">…</span> <span class="n">c</span><span class="o">.</span><span class="na">output</span><span class="o">(</span><span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">userId</span><spa [...]
- <span class="o">}</span>
- <span class="o">}));</span>
-</code></pre>
-</div>
-
-<div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="c"># State and timers are not yet supported in Beam's Python SDK.</span>
-<span class="c"># Watch this space!</span>
-</code></pre>
-</div>
-
-<p>In this pipeline, there is just one model emitted by the <code class="highlighter-rouge">Combine.perKey(...)</code>
-per user, per window, which is then prepared for side input by the <code class="highlighter-rouge">View.asMap()</code>
-transform. The processing of the <code class="highlighter-rouge">ParDo</code> over events will block until that side
-input is ready, buffering events, and will then check each event against the
-model. This is a high latency, high completeness solution: The model takes into
-account all user behavior in the window, but there can be no output until the
-window is complete.</p>
-
-<p>Suppose you want to get some results earlier, or don’t even have any
-natural windowing, but just want continuous analysis with the “model so far”,
-even though your model may not be as complete. How can you control the updates
-to the model against which you are checking your events? Triggers are the
-generic Beam feature for managing completeness versus latency tradeoffs. So here
-is the same pipeline with an added trigger that outputs a new model one second
-after input arrives:</p>
-
-<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">UserId</span><span class="o">,</span> <span class="n">Event</span><span class="o& [...]
-
-<span class="n">PCollectionView</span><span class="o">&lt;</span><span class="n">Map</span><span class="o">&lt;</span><span class="n">UserId</span><span class="o">,</span> <span class="n">Model</span><span class="o">&gt;&gt;</span> <span class="n">userModels</span> <span cla [...]
-
- <span class="c1">// A tradeoff between latency and cost</span>
- <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Window</span><span class="o">.</span><span class="na">triggering</span><span class="o">(</span>
- <span class="n">AfterProcessingTime</span><span class="o">.</span><span class="na">pastFirstElementInPane</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardSeconds</span><span class="o">(</span><span class="mi">1</span><spa [...]
-
- <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Combine</span><span class="o">.</span><span class="na">perKey</span><span class="o">(</span><span class="k">new</span> <span class="n">ModelFromEventsFn</span><span class="o">()))&l [...]
- <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">View</span><span class="o">.</span><span class="na">asMap</span><span class="o">());</span>
-</code></pre>
-</div>
-
-<div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="c"># State and timers are not yet supported in Beam's Python SDK.</span>
-<span class="c"># Watch this space!</span>
-</code></pre>
-</div>
-
-<p>This is often a pretty nice tradeoff between latency and cost: If a huge flood
-of events comes in a second, then you will only emit one new model, so you
-won’t be flooded with model outputs that you cannot even use before they are
-obsolete. In practice, the new model may not be present on the side input
-channel until many more seconds have passed, due to caches and processing
-delays preparing the side input. Many events (maybe an entire batch of
-activity) will have passed through the <code class="highlighter-rouge">ParDo</code> and had their predictions
-calculated according to the prior model. If the runner gave a tight enough
-bound on cache expirations and you used a more aggressive trigger, you might be
-able to improve latency at additional cost.</p>
-
-<p>But there is another cost to consider: you are outputting many uninteresting
-outputs from the <code class="highlighter-rouge">ParDo</code> that will be processed downstream. If the
-“interestingness” of the output is only well-defined relative to the prior
-output, then you cannot use a <code class="highlighter-rouge">Filter</code> transform to reduce data volume downstream.</p>
-
-<p>Stateful processing lets you address both the latency problem of side inputs
-and the cost problem of excessive uninteresting output. Here is the code, using
-only features I have already introduced:</p>
-
-<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">UserId</span><span class="o">,</span> <span class="n"> [...]
-
- <span class="nd">@StateId</span><span class="o">(</span><span class="s">"model"</span><span class="o">)</span>
- <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o">&lt;</span><span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Model</span><span class="o">&gt;&gt;</span> <span class="n">modelSpec</span> &l [...]
- <span class="n">StateSpecs</span><span class="o">.</span><span class="na">value</span><span class="o">(</span><span class="n">Model</span><span class="o">.</span><span class="na">coder</span><span class="o">());</span>
-
- <span class="nd">@StateId</span><span class="o">(</span><span class="s">"previousPrediction"</span><span class="o">)</span>
- <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o">&lt;</span><span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Prediction</span><span class="o">&gt;&gt;</span> <span class="n">previousPredictionSp [...]
- <span class="n">StateSpecs</span><span class="o">.</span><span class="na">value</span><span class="o">(</span><span class="n">Prediction</span><span class="o">.</span><span class="na">coder</span><span class="o">());</span>
-
- <span class="nd">@ProcessElement</span>
- <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span>
- <span class="n">ProcessContext</span> <span class="n">c</span><span class="o">,</span>
- <span class="nd">@StateId</span><span class="o">(</span><span class="s">"previousPrediction"</span><span class="o">)</span> <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Prediction</span><span class="o">&gt;</span> <span class="n">previousPredict [...]
- <span class="nd">@StateId</span><span class="o">(</span><span class="s">"model"</span><span class="o">)</span> <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Model</span><span class="o">&gt;</span> <span class="n">modelState</span><span c [...]
- <span class="n">UserId</span> <span class="n">userId</span> <span class="o">=</span> <span class="n">c</span><span class="o">.</span><span class="na">element</span><span class="o">().</span><span class="na">getKey</span><span class="o">();</span>
- <span class="n">Event</span> <span class="n">event</span> <span class="o">=</span> <span class="n">c</span><span class="o">.</span><span class="na">element</span><span class="o">().</span><span class="na">getValue</span><span class="o">()</span>
-
- <span class="n">Model</span> <span class="n">model</span> <span class="o">=</span> <span class="n">modelState</span><span class="o">.</span><span class="na">read</span><span class="o">();</span>
- <span class="n">Prediction</span> <span class="n">previousPrediction</span> <span class="o">=</span> <span class="n">previousPredictionState</span><span class="o">.</span><span class="na">read</span><span class="o">();</span>
- <span class="n">Prediction</span> <span class="n">newPrediction</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="na">prediction</span><span class="o">(</span><span class="n">event</span><span class="o">);</span>
- <span class="n">model</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">event</span><span class="o">);</span>
- <span class="n">modelState</span><span class="o">.</span><span class="na">write</span><span class="o">(</span><span class="n">model</span><span class="o">);</span>
- <span class="k">if</span> <span class="o">(</span><span class="n">previousPrediction</span> <span class="o">==</span> <span class="kc">null</span>
- <span class="o">||</span> <span class="n">shouldOutputNewPrediction</span><span class="o">(</span><span class="n">previousPrediction</span><span class="o">,</span> <span class="n">newPrediction</span><span class="o">))</span> <span class="o">{</span>
- <span class="n">c</span><span class="o">.</span><span class="na">output</span><span class="o">(</span><span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">userId</span><span class="o">,</span> <span cl [...]
- <span class="n">previousPredictionState</span><span class="o">.</span><span class="na">write</span><span class="o">(</span><span class="n">newPrediction</span><span class="o">);</span>
- <span class="o">}</span>
- <span class="o">}</span>
-<span class="o">};</span>
-</code></pre>
-</div>
-
-<div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="c"># State and timers are not yet supported in Beam's Python SDK.</span>
-<span class="c"># Watch this space!</span>
-</code></pre>
-</div>
-
-<p>Let’s walk through it,</p>
-
-<ul>
- <li>You have two state cells declared, <code class="highlighter-rouge">@StateId("model")</code> to hold the current
-state of the model for a user and <code class="highlighter-rouge">@StateId("previousPrediction")</code> to hold
-the prediction output previously.</li>
- <li>Access to the two state cells by annotation in the <code class="highlighter-rouge">@ProcessElement</code> method
-is as before.</li>
- <li>You read the current model via <code class="highlighter-rouge">modelState.read()</code>. Because state is also
-per-key-and-window, this is a model just for the UserId of the Event
-currently being processed.</li>
- <li>You derive a new prediction <code class="highlighter-rouge">model.prediction(event)</code> and compare it against
-the last one you output, accessed via <code class="highlighter-rouge">previousPredicationState.read()</code>.</li>
- <li>You then update the model <code class="highlighter-rouge">model.update()</code> and write it via
-<code class="highlighter-rouge">modelState.write(...)</code>. It is perfectly fine to mutate the value you pulled
-out of state as long as you also remember to write the mutated value, in the
-same way you are encouraged to mutate <code class="highlighter-rouge">CombineFn</code> accumulators.</li>
- <li>If the prediction has changed a significant amount since the last time you
-output, you emit it via <code class="highlighter-rouge">context.output(...)</code> and save the prediction using
-<code class="highlighter-rouge">previousPredictionState.write(...)</code>. Here the decision is relative to the
-prior prediction output, not the last one computed - realistically you might
-have some complex conditions here.</li>
-</ul>
-
-<p>Most of the above is just talking through Java! But before you go out and
-convert all of your pipelines to use stateful processing, I want to go over
-some considerations as to whether it is a good fit for your use case.</p>
-
-<h2 id="performance-considerations">Performance considerations</h2>
-
-<p>To decide whether to use per-key-and-window state, you need to consider how it
-executes. You can dig into how a particular runner manages state, but there are
-some general things to keep in mind:</p>
-
-<ul>
- <li>Partitioning per-key-and-window: perhaps the most important thing to
-consider is that the runner may have to shuffle your data to colocate all
-the data for a particular key+window. If the data is already shuffled
-correctly, the runner may take advantage of this.</li>
- <li>Synchronization overhead: the API is designed so the runner takes care of
-concurrency control, but this means that the runner cannot parallelize
-processing of elements for a particular key+window even when it would otherwise
-be advantageous.</li>
- <li>Storage and fault tolerance of state: since state is per-key-and-window, the
-more keys and windows you expect to process simultaneously, the more storage
-you will incur. Because state benefits from all the fault tolerance /
-consistency properties of your other data in Beam, it also adds to the cost of
-committing the results of processing.</li>
- <li>Expiration of state: also since state is per-window, the runner can reclaim
-the resources when a window expires (when the watermark exceeds its allowed
-lateness) but this could mean that the runner is tracking an additional timer
-per key and window to cause reclamation code to execute.</li>
-</ul>
-
-<h2 id="go-use-it">Go use it!</h2>
-
-<p>If you are new to Beam, I hope you are now interested in seeing if Beam with
-stateful processing addresses your use case. If you are already using Beam, I
-hope this new addition to the model unlocks new use cases for you. Do check
-the <a href="/documentation/runners/capability-matrix/">capability
-matrix</a> to
-see the level of support for this new model feature on your favorite
-backend(s).</p>
-
-<p>And please do join the community at
-<a href="/get-started/support">user@beam.apache.org</a>. We’d love to
-hear from you.</p>
-</description>
- <pubDate>Mon, 13 Feb 2017 00:00:01 -0800</pubDate>
- <link>https://beam.apache.org/blog/2017/02/13/stateful-processing.html</link>
- <guid isPermaLink="true">https://beam.apache.org/blog/2017/02/13/stateful-processing.html</guid>
-
-
- <category>blog</category>
-
- </item>
-
</channel>
</rss>
diff --git a/content/images/blog/Facebook-AD.png b/content/images/blog/Facebook-AD.png
new file mode 100644
index 0000000..5493545
Binary files /dev/null and b/content/images/blog/Facebook-AD.png differ
diff --git a/content/index.html b/content/index.html
index 33e68cf..dbdaf3d 100644
--- a/content/index.html
+++ b/content/index.html
@@ -162,6 +162,11 @@ limitations under the License.
</div>
<div class="hero__blog__cards">
+ <a class="hero__blog__cards__card" href="/blog/2018/08/21/beam-summit-europe.html">
+ <div class="hero__blog__cards__card__title">Beam Summit Europe 2018</div>
+ <div class="hero__blog__cards__card__date">Aug 21, 2018</div>
+ </a>
+
<a class="hero__blog__cards__card" href="/blog/2018/08/20/review-input-streaming-connectors.html">
<div class="hero__blog__cards__card__title">A review of input streaming connectors</div>
<div class="hero__blog__cards__card__date">Aug 20, 2018</div>
@@ -172,11 +177,6 @@ limitations under the License.
<div class="hero__blog__cards__card__date">Aug 10, 2018</div>
</a>
- <a class="hero__blog__cards__card" href="/blog/2018/06/26/beam-2.5.0.html">
- <div class="hero__blog__cards__card__title">Apache Beam 2.5.0</div>
- <div class="hero__blog__cards__card__date">Jun 26, 2018</div>
- </a>
-
</div>
</div>
</div>