You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by da...@apache.org on 2017/02/15 22:54:30 UTC
[1/3] beam-site git commit: Clarify state further in capability matrix
Repository: beam-site
Updated Branches:
refs/heads/asf-site f48e97f67 -> b6675d294
Clarify state further in capability matrix
Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/466edb35
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/466edb35
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/466edb35
Branch: refs/heads/asf-site
Commit: 466edb3592323b64cc7b580eb06e684af9b0ba2f
Parents: f48e97f
Author: Kenneth Knowles <kl...@google.com>
Authored: Tue Feb 14 19:14:32 2017 -0800
Committer: Davor Bonaci <da...@google.com>
Committed: Wed Feb 15 14:53:50 2017 -0800
----------------------------------------------------------------------
src/_data/capability-matrix.yml | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/beam-site/blob/466edb35/src/_data/capability-matrix.yml
----------------------------------------------------------------------
diff --git a/src/_data/capability-matrix.yml b/src/_data/capability-matrix.yml
index f2e58ac..89f89a2 100644
--- a/src/_data/capability-matrix.yml
+++ b/src/_data/capability-matrix.yml
@@ -196,7 +196,7 @@ categories:
l2: Not implemented in runner.
l3:
- - name: Keyed State
+ - name: Stateful Processing
values:
- class: model
l1: 'Yes'
@@ -205,19 +205,19 @@ categories:
- class: dataflow
l1: 'Partially'
l2: non-merging windows
- l3: Keyed state is fully supported for non-merging windows.
+ l3: State is supported for non-merging windows. SetState and MapState are not yet supported.
- class: flink
l1: 'Partially'
l2: streaming, non-merging windows
- l3: Keyed state is supported in streaming mode for non-merging windows.
+ l3: State is supported in streaming mode for non-merging windows. SetState and MapState are not yet supported.
- class: spark
l1: 'No'
l2: not implemented
- l3: Spark supports keyed state with mapWithState() so support shuold be straight forward.
+ l3: Spark supports per-key state with <tt>mapWithState()</tt> so support should be straightforward.
- class: apex
l1: 'No'
l2: not implemented
- l3: Apex supports keyed state, so adding support for this should be easy.
+ l3: Apex supports per-key state, so adding support for this should be easy.
- description: Where in event time?
anchor: where
[2/3] beam-site git commit: Regenerate website
Posted by da...@apache.org.
Regenerate website
Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/575e4598
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/575e4598
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/575e4598
Branch: refs/heads/asf-site
Commit: 575e45987aa00dca16ee562441dd071362c873b1
Parents: 466edb3
Author: Davor Bonaci <da...@google.com>
Authored: Wed Feb 15 14:54:18 2017 -0800
Committer: Davor Bonaci <da...@google.com>
Committed: Wed Feb 15 14:54:18 2017 -0800
----------------------------------------------------------------------
content/blog/2017/02/13/stateful-processing.html | 19 +++++++++++++------
.../runners/capability-matrix/index.html | 12 ++++++------
content/feed.xml | 19 +++++++++++++------
3 files changed, 32 insertions(+), 18 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/beam-site/blob/575e4598/content/blog/2017/02/13/stateful-processing.html
----------------------------------------------------------------------
diff --git a/content/blog/2017/02/13/stateful-processing.html b/content/blog/2017/02/13/stateful-processing.html
index 833b2fd..a936f4b 100644
--- a/content/blog/2017/02/13/stateful-processing.html
+++ b/content/blog/2017/02/13/stateful-processing.html
@@ -328,7 +328,7 @@ unique and consistent. Before diving into the code for how to do this in a Beam
SDK, I\u2019ll go over this example from the level of the model. In pictures, you
want to write a transform that maps input to output like this:</p>
-<p><img class="center-block" src="/images/blog/stateful-processing/assign-indices.png" alt="Assigning arbitrary but unique indices to each element" width="100" /></p>
+<p><img class="center-block" src="/images/blog/stateful-processing/assign-indices.png" alt="Assigning arbitrary but unique indices to each element" width="180" /></p>
<p>The order of the elements A, B, C, D, E is arbitrary, hence their assigned
indices are arbitrary, but downstream transforms just need to be OK with this.
@@ -400,10 +400,17 @@ key+window pairs, like this:</p>
keys and windows are independent dimensions)</p>
<p>You can provide the opportunity for parallelism by making sure that table has
-enough columns, either via many keys in few windows - for example, a globally
-windowed stateful computation keyed by user ID - or via many windows over few
-keys - for example, a fixed windowed stateful computation over a global key.
-Caveat: all Beam runners today parallelize only over the key.</p>
+enough columns. You might have many keys and many windows, or you might have
+many of just one or the other:</p>
+
+<ul>
+ <li>Many keys in few windows, for example a globally windowed stateful computation
+keyed by user ID.</li>
+ <li>Many windows over few keys, for example a fixed windowed stateful computation
+over a global key.</li>
+</ul>
+
+<p>Caveat: all Beam runners today parallelize only over the key.</p>
<p>Most often your mental model of state can be focused on only a single column of
the table, a single key+window pair. Cross-column interactions do not occur
@@ -610,7 +617,7 @@ outputs from the <code class="highlighter-rouge">ParDo</code> that will be proce
output, then you cannot use a <code class="highlighter-rouge">Filter</code> transform to reduce data volume downstream.</p>
<p>Stateful processing lets you address both the latency problem of side inputs
-and the cost problem of excessive uninterseting output. Here is the code, using
+and the cost problem of excessive uninteresting output. Here is the code, using
only features I have already introduced:</p>
<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="k">new</span> <span class="n">DoFn</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">UserId</span><span class="o">,</span> <span class="n">Event</span><span class="o">>,</span> <span class="n">KV</span><span class="o"><</span><span class="n">UserId</span><span class="o">,</span> <span class="n">Prediction</span><span class="o">>>()</span> <span class="o">{</span>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/575e4598/content/documentation/runners/capability-matrix/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/runners/capability-matrix/index.html b/content/documentation/runners/capability-matrix/index.html
index 60f62b2..88da8eb 100644
--- a/content/documentation/runners/capability-matrix/index.html
+++ b/content/documentation/runners/capability-matrix/index.html
@@ -441,7 +441,7 @@
</tr>
<tr class="cap-summary">
- <th class="cap-summary color-capability format-capability" style="color:#ec3">Keyed State</th>
+ <th class="cap-summary color-capability format-capability" style="color:#ec3">Stateful Processing</th>
@@ -1353,7 +1353,7 @@
</tr>
<tr class="cap">
- <th class="cap color-capability format-capability" style="color:#ec3">Keyed State</th>
+ <th class="cap color-capability format-capability" style="color:#ec3">Stateful Processing</th>
@@ -1362,22 +1362,22 @@
- <td width="25%" class="cap" style="background-color:#fe5;border-color:#ca1"><center><b>Partially: non-merging windows</b></center><br />Keyed state is fully supported for non-merging windows.
+ <td width="25%" class="cap" style="background-color:#fe5;border-color:#ca1"><center><b>Partially: non-merging windows</b></center><br />State is supported for non-merging windows. SetState and MapState are not yet supported.
</td>
- <td width="25%" class="cap" style="background-color:#fe5;border-color:#ca1"><center><b>Partially: streaming, non-merging windows</b></center><br />Keyed state is supported in streaming mode for non-merging windows.
+ <td width="25%" class="cap" style="background-color:#fe5;border-color:#ca1"><center><b>Partially: streaming, non-merging windows</b></center><br />State is supported in streaming mode for non-merging windows. SetState and MapState are not yet supported.
</td>
- <td width="25%" class="cap" style="background-color:#ddd;border-color:#ca1"><center><b>No: not implemented</b></center><br />Spark supports keyed state with mapWithState() so support shuold be straight forward.
+ <td width="25%" class="cap" style="background-color:#ddd;border-color:#ca1"><center><b>No: not implemented</b></center><br />Spark supports per-key state with <tt>mapWithState()</tt> so support should be straightforward.
</td>
- <td width="25%" class="cap" style="background-color:#ddd;border-color:#ca1"><center><b>No: not implemented</b></center><br />Apex supports keyed state, so adding support for this should be easy.
+ <td width="25%" class="cap" style="background-color:#ddd;border-color:#ca1"><center><b>No: not implemented</b></center><br />Apex supports per-key state, so adding support for this should be easy.
</td>
</tr>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/575e4598/content/feed.xml
----------------------------------------------------------------------
diff --git a/content/feed.xml b/content/feed.xml
index 726cdd0..5c0cd90 100644
--- a/content/feed.xml
+++ b/content/feed.xml
@@ -182,7 +182,7 @@ unique and consistent. Before diving into the code for how to do this in a Beam
SDK, I\u2019ll go over this example from the level of the model. In pictures, you
want to write a transform that maps input to output like this:</p>
-<p><img class="center-block" src="/images/blog/stateful-processing/assign-indices.png" alt="Assigning arbitrary but unique indices to each element" width="100" /></p>
+<p><img class="center-block" src="/images/blog/stateful-processing/assign-indices.png" alt="Assigning arbitrary but unique indices to each element" width="180" /></p>
<p>The order of the elements A, B, C, D, E is arbitrary, hence their assigned
indices are arbitrary, but downstream transforms just need to be OK with this.
@@ -254,10 +254,17 @@ key+window pairs, like this:</p>
keys and windows are independent dimensions)</p>
<p>You can provide the opportunity for parallelism by making sure that table has
-enough columns, either via many keys in few windows - for example, a globally
-windowed stateful computation keyed by user ID - or via many windows over few
-keys - for example, a fixed windowed stateful computation over a global key.
-Caveat: all Beam runners today parallelize only over the key.</p>
+enough columns. You might have many keys and many windows, or you might have
+many of just one or the other:</p>
+
+<ul>
+ <li>Many keys in few windows, for example a globally windowed stateful computation
+keyed by user ID.</li>
+ <li>Many windows over few keys, for example a fixed windowed stateful computation
+over a global key.</li>
+</ul>
+
+<p>Caveat: all Beam runners today parallelize only over the key.</p>
<p>Most often your mental model of state can be focused on only a single column of
the table, a single key+window pair. Cross-column interactions do not occur
@@ -464,7 +471,7 @@ outputs from the <code class="highlighter-rouge">ParDo</code&
output, then you cannot use a <code class="highlighter-rouge">Filter</code> transform to reduce data volume downstream.</p>
<p>Stateful processing lets you address both the latency problem of side inputs
-and the cost problem of excessive uninterseting output. Here is the code, using
+and the cost problem of excessive uninteresting output. Here is the code, using
only features I have already introduced:</p>
<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">UserId</span><span class="o">,</span> <span class="n">Event</span><span class="o">&gt;,</span> <span class="n">KV</span><span class="o">&lt;</span><span class="n">UserId</span><span class="o">,</span> <span class="n">Prediction</span><span class="o">&gt;&gt;()</span> <span class="o">{</span>
[3/3] beam-site git commit: This closes #154
Posted by da...@apache.org.
This closes #154
Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/b6675d29
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/b6675d29
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/b6675d29
Branch: refs/heads/asf-site
Commit: b6675d2945b8f8f3fe5ab2baaa228c4f05e70067
Parents: f48e97f 575e459
Author: Davor Bonaci <da...@google.com>
Authored: Wed Feb 15 14:54:18 2017 -0800
Committer: Davor Bonaci <da...@google.com>
Committed: Wed Feb 15 14:54:18 2017 -0800
----------------------------------------------------------------------
content/blog/2017/02/13/stateful-processing.html | 19 +++++++++++++------
.../runners/capability-matrix/index.html | 12 ++++++------
content/feed.xml | 19 +++++++++++++------
src/_data/capability-matrix.yml | 10 +++++-----
4 files changed, 37 insertions(+), 23 deletions(-)
----------------------------------------------------------------------