You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by al...@apache.org on 2017/02/13 20:12:04 UTC
[1/3] beam-site git commit: Update ParDo documentation for Python
Repository: beam-site
Updated Branches:
refs/heads/asf-site a1e2a39f6 -> 4282f6bc5
Update ParDo documentation for Python
Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/8bc63920
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/8bc63920
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/8bc63920
Branch: refs/heads/asf-site
Commit: 8bc639206de6054a15da9601d56a7bf82c4f076f
Parents: a1e2a39
Author: Hadar Hod <ha...@google.com>
Authored: Thu Feb 9 10:37:07 2017 -0800
Committer: Ahmet Altay <al...@google.com>
Committed: Mon Feb 13 12:11:01 2017 -0800
----------------------------------------------------------------------
src/documentation/programming-guide.md | 9 ++++-----
1 file changed, 4 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/beam-site/blob/8bc63920/src/documentation/programming-guide.md
----------------------------------------------------------------------
diff --git a/src/documentation/programming-guide.md b/src/documentation/programming-guide.md
index 587c9f9..641ad2d 100644
--- a/src/documentation/programming-guide.md
+++ b/src/documentation/programming-guide.md
@@ -338,7 +338,7 @@ static class ComputeWordLengthFn extends DoFn<String, Integer> { ... }
Inside your `DoFn` subclass, you'll write a method annotated with `@ProcessElement` where you provide the actual processing logic. You don't need to manually extract the elements from the input collection; the Beam SDKs handle that for you. Your `@ProcessElement` method should accept an object of type `ProcessContext`. The `ProcessContext` object gives you access to an input element and a method for emitting an output element:
{:.language-py}
-Inside your `DoFn` subclass, you'll write a method `process` where you provide the actual processing logic. You don't need to manually extract the elements from the input collection; the Beam SDKs handle that for you. Your `process` method should accept an object of type `context`. The `context` object gives you access to an input element and output is emitted by using `yield` or `return` statement inside `process` method.
+Inside your `DoFn` subclass, you'll write a method `process` where you provide the actual processing logic. You don't need to manually extract the elements from the input collection; the Beam SDKs handle that for you. Your `process` method should accept an object of type `element`. This is the input element and output is emitted by using `yield` or `return` statement inside `process` method.
```java
static class ComputeWordLengthFn extends DoFn<String, Integer> {
@@ -429,11 +429,11 @@ words = ...
#### <a name="transforms-gbk"></a>Using GroupByKey
-`GroupByKey` is a Beam transform for processing collections of key/value pairs. It's a parallel reduction operation, analagous to the Shuffle phase of a Map/Shuffle/Reduce-style algorithm. The input to `GroupByKey` is a collection of key/value pairs that represents a *multimap*, where the collection contains multiple pairs that have the same key, but different values. Given such a collection, you use `GroupByKey` to collect all of the values associated with each unique key.
+`GroupByKey` is a Beam transform for processing collections of key/value pairs. It's a parallel reduction operation, analogous to the Shuffle phase of a Map/Shuffle/Reduce-style algorithm. The input to `GroupByKey` is a collection of key/value pairs that represents a *multimap*, where the collection contains multiple pairs that have the same key, but different values. Given such a collection, you use `GroupByKey` to collect all of the values associated with each unique key.
`GroupByKey` is a good way to aggregate data that has something in common. For example, if you have a collection that stores records of customer orders, you might want to group together all the orders from the same postal code (wherein the "key" of the key/value pair is the postal code field, and the "value" is the remainder of the record).
-Let's examine the mechanics of `GroupByKey` with a simple xample case, where our data set consists of words from a text file and the line number on which they appear. We want to group together all the line numbers (values) that share the same word (key), letting us see all the places in the text where a particular word appears.
+Let's examine the mechanics of `GroupByKey` with a simple example case, where our data set consists of words from a text file and the line number on which they appear. We want to group together all the line numbers (values) that share the same word (key), letting us see all the places in the text where a particular word appears.
Our input is a `PCollection` of key/value pairs where each word is a key, and the value is a line number in the file where the word appears. Here's a list of the key/value pairs in the input collection:
@@ -790,12 +790,11 @@ Side inputs are useful if your `ParDo` needs to inject additional data when proc
%}
# We can also pass side inputs to a ParDo transform, which will get passed to its process method.
-# The only change is that the first arguments are self and a context, rather than the PCollection element itself.
+# The first two arguments for the process method would be self and element.
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets_test.py tag:model_pardo_side_input_dofn
%}
...
-
```
##### Side inputs and windowing:
[2/3] beam-site git commit: Regenerate website
Posted by al...@apache.org.
Regenerate website
Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/aa9b7fea
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/aa9b7fea
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/aa9b7fea
Branch: refs/heads/asf-site
Commit: aa9b7fea05f5e1177ff40f7b3977cdfb9ec0dd19
Parents: 8bc6392
Author: Ahmet Altay <al...@google.com>
Authored: Mon Feb 13 12:11:35 2017 -0800
Committer: Ahmet Altay <al...@google.com>
Committed: Mon Feb 13 12:11:35 2017 -0800
----------------------------------------------------------------------
content/documentation/programming-guide/index.html | 9 ++++-----
content/get-started/wordcount-example/index.html | 4 ++--
2 files changed, 6 insertions(+), 7 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/beam-site/blob/aa9b7fea/content/documentation/programming-guide/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/programming-guide/index.html b/content/documentation/programming-guide/index.html
index f02fd40..0aa0575 100644
--- a/content/documentation/programming-guide/index.html
+++ b/content/documentation/programming-guide/index.html
@@ -515,7 +515,7 @@
<p class="language-java">Inside your <code class="highlighter-rouge">DoFn</code> subclass, you\u2019ll write a method annotated with <code class="highlighter-rouge">@ProcessElement</code> where you provide the actual processing logic. You don\u2019t need to manually extract the elements from the input collection; the Beam SDKs handle that for you. Your <code class="highlighter-rouge">@ProcessElement</code> method should accept an object of type <code class="highlighter-rouge">ProcessContext</code>. The <code class="highlighter-rouge">ProcessContext</code> object gives you access to an input element and a method for emitting an output element:</p>
-<p class="language-py">Inside your <code class="highlighter-rouge">DoFn</code> subclass, you\u2019ll write a method <code class="highlighter-rouge">process</code> where you provide the actual processing logic. You don\u2019t need to manually extract the elements from the input collection; the Beam SDKs handle that for you. Your <code class="highlighter-rouge">process</code> method should accept an object of type <code class="highlighter-rouge">context</code>. The <code class="highlighter-rouge">context</code> object gives you access to an input element and output is emitted by using <code class="highlighter-rouge">yield</code> or <code class="highlighter-rouge">return</code> statement inside <code class="highlighter-rouge">process</code> method.</p>
+<p class="language-py">Inside your <code class="highlighter-rouge">DoFn</code> subclass, you\u2019ll write a method <code class="highlighter-rouge">process</code> where you provide the actual processing logic. You don\u2019t need to manually extract the elements from the input collection; the Beam SDKs handle that for you. Your <code class="highlighter-rouge">process</code> method should accept an object of type <code class="highlighter-rouge">element</code>. This is the input element and output is emitted by using <code class="highlighter-rouge">yield</code> or <code class="highlighter-rouge">return</code> statement inside <code class="highlighter-rouge">process</code> method.</p>
<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="kd">static</span> <span class="kd">class</span> <span class="nc">ComputeWordLengthFn</span> <span class="kd">extends</span> <span class="n">DoFn</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">></span> <span class="o">{</span>
<span class="nd">@ProcessElement</span>
@@ -610,11 +610,11 @@
<h4 id="a-nametransforms-gbkausing-groupbykey"><a name="transforms-gbk"></a>Using GroupByKey</h4>
-<p><code class="highlighter-rouge">GroupByKey</code> is a Beam transform for processing collections of key/value pairs. It\u2019s a parallel reduction operation, analagous to the Shuffle phase of a Map/Shuffle/Reduce-style algorithm. The input to <code class="highlighter-rouge">GroupByKey</code> is a collection of key/value pairs that represents a <em>multimap</em>, where the collection contains multiple pairs that have the same key, but different values. Given such a collection, you use <code class="highlighter-rouge">GroupByKey</code> to collect all of the values associated with each unique key.</p>
+<p><code class="highlighter-rouge">GroupByKey</code> is a Beam transform for processing collections of key/value pairs. It\u2019s a parallel reduction operation, analogous to the Shuffle phase of a Map/Shuffle/Reduce-style algorithm. The input to <code class="highlighter-rouge">GroupByKey</code> is a collection of key/value pairs that represents a <em>multimap</em>, where the collection contains multiple pairs that have the same key, but different values. Given such a collection, you use <code class="highlighter-rouge">GroupByKey</code> to collect all of the values associated with each unique key.</p>
<p><code class="highlighter-rouge">GroupByKey</code> is a good way to aggregate data that has something in common. For example, if you have a collection that stores records of customer orders, you might want to group together all the orders from the same postal code (wherein the \u201ckey\u201d of the key/value pair is the postal code field, and the \u201cvalue\u201d is the remainder of the record).</p>
-<p>Let\u2019s examine the mechanics of <code class="highlighter-rouge">GroupByKey</code> with a simple xample case, where our data set consists of words from a text file and the line number on which they appear. We want to group together all the line numbers (values) that share the same word (key), letting us see all the places in the text where a particular word appears.</p>
+<p>Let\u2019s examine the mechanics of <code class="highlighter-rouge">GroupByKey</code> with a simple example case, where our data set consists of words from a text file and the line number on which they appear. We want to group together all the line numbers (values) that share the same word (key), letting us see all the places in the text where a particular word appears.</p>
<p>Our input is a <code class="highlighter-rouge">PCollection</code> of key/value pairs where each word is a key, and the value is a line number in the file where the word appears. Here\u2019s a list of the key/value pairs in the input collection:</p>
@@ -1046,7 +1046,7 @@ tree, [2]
<span class="c"># We can also pass side inputs to a ParDo transform, which will get passed to its process method.</span>
-<span class="c"># The only change is that the first arguments are self and a context, rather than the PCollection element itself.</span>
+<span class="c"># The first two arguments for the process method would be self and element.</span>
<span class="k">class</span> <span class="nc">FilterUsingLength</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">element</span><span class="p">,</span> <span class="n">lower_bound</span><span class="p">,</span> <span class="n">upper_bound</span><span class="o">=</span><span class="nb">float</span><span class="p">(</span><span class="s">'inf'</span><span class="p">)):</span>
@@ -1056,7 +1056,6 @@ tree, [2]
<span class="n">small_words</span> <span class="o">=</span> <span class="n">words</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">FilterUsingLength</span><span class="p">(),</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span>
<span class="o">...</span>
-
</code></pre>
</div>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/aa9b7fea/content/get-started/wordcount-example/index.html
----------------------------------------------------------------------
diff --git a/content/get-started/wordcount-example/index.html b/content/get-started/wordcount-example/index.html
index 7fc2f71..0295c31 100644
--- a/content/get-started/wordcount-example/index.html
+++ b/content/get-started/wordcount-example/index.html
@@ -456,8 +456,8 @@ Figure 1: The pipeline data flow.</p>
<span class="k">def</span> <span class="nf">expand</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">pcoll</span><span class="p">):</span>
<span class="k">return</span> <span class="p">(</span><span class="n">pcoll</span>
<span class="c"># Convert lines of text into individual words.</span>
- <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">FlatMap</span><span class="p">(</span>
- <span class="s">'ExtractWords'</span><span class="p">,</span> <span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">re</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="s">r'[A-Za-z</span><span class="se">\'</span><span class="s">]+'</span><span class="p">,</span> <span class="n">x</span><span class="p">))</span>
+ <span class="o">|</span> <span class="s">'ExtractWords'</span> <span class="o">>></span> <span class="n">beam</span><span class="o">.</span><span class="n">FlatMap</span><span class="p">(</span>
+ <span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">re</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="s">r'[A-Za-z</span><span class="se">\'</span><span class="s">]+'</span><span class="p">,</span> <span class="n">x</span><span class="p">))</span>
<span class="c"># Count the number of times each word occurs.</span>
<span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">combiners</span><span class="o">.</span><span class="n">Count</span><span class="o">.</span><span class="n">PerElement</span><span class="p">())</span>
[3/3] beam-site git commit: This closes #150
Posted by al...@apache.org.
This closes #150
Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/4282f6bc
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/4282f6bc
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/4282f6bc
Branch: refs/heads/asf-site
Commit: 4282f6bc54a0ccc6bb7a4bac65547e429d78538e
Parents: a1e2a39 aa9b7fe
Author: Ahmet Altay <al...@google.com>
Authored: Mon Feb 13 12:11:36 2017 -0800
Committer: Ahmet Altay <al...@google.com>
Committed: Mon Feb 13 12:11:36 2017 -0800
----------------------------------------------------------------------
content/documentation/programming-guide/index.html | 9 ++++-----
content/get-started/wordcount-example/index.html | 4 ++--
src/documentation/programming-guide.md | 9 ++++-----
3 files changed, 10 insertions(+), 12 deletions(-)
----------------------------------------------------------------------