You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by al...@apache.org on 2017/04/29 00:39:48 UTC
[1/3] beam-site git commit: [BEAM-1695] Improve Python-SDK's
programming guide
Repository: beam-site
Updated Branches:
refs/heads/asf-site bb4221bd8 -> aac389466
[BEAM-1695] Improve Python-SDK's programming guide
Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/faa12ffd
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/faa12ffd
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/faa12ffd
Branch: refs/heads/asf-site
Commit: faa12ffd7b841e01b5a7386c84a856cf4b30c9e9
Parents: bb4221b
Author: Tibor Kiss <ti...@gmail.com>
Authored: Thu Apr 27 14:30:11 2017 +0200
Committer: Ahmet Altay <al...@altay-macbookpro2.roam.corp.google.com>
Committed: Fri Apr 28 17:36:18 2017 -0700
----------------------------------------------------------------------
src/documentation/programming-guide.md | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/beam-site/blob/faa12ffd/src/documentation/programming-guide.md
----------------------------------------------------------------------
diff --git a/src/documentation/programming-guide.md b/src/documentation/programming-guide.md
index 93074fb..8e8b917 100644
--- a/src/documentation/programming-guide.md
+++ b/src/documentation/programming-guide.md
@@ -577,7 +577,7 @@ public class AverageFn extends CombineFn<Integer, AverageFn.Accum, Double> {
```py
pc = ...
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets_test.py tag:combine_custom_average
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets_test.py tag:combine_custom_average_define
%}```
If you are combining a `PCollection` of key-value pairs, [per-key combining](#transforms-combine-per-key) is often enough. If you need the combining strategy to change based on the key (for example, MIN for some users and MAX for other users), you can define a `KeyedCombineFn` to access the key within the combining strategy.
@@ -598,7 +598,7 @@ PCollection<Integer> sum = pc.apply(
# sum combines the elements in the input PCollection.
# The resulting PCollection, called result, contains one value: the sum of all the elements in the input PCollection.
pc = ...
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets_test.py tag:combine_custom_average
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets_test.py tag:combine_custom_average_execute
%}```
##### Global windowing:
@@ -727,6 +727,7 @@ PCollection<Student> fortiethPercentile = studentsByPercentile.get(4);
```py
# Provide an int value with the desired number of result partitions, and a partitioning function (partition_fn in this example).
# Returns a tuple of PCollection objects containing each of the resulting partitions as individual PCollection objects.
+students = ...
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py tag:model_multiple_pcollections_partition
%}
@@ -811,7 +812,7 @@ Side inputs are useful if your `ParDo` needs to inject additional data when proc
# Optional, positional, and keyword arguments are all supported. Deferred arguments are unwrapped into their actual values.
# For example, using pvalue.AsIter(pcoll) at pipeline construction time results in an iterable of the actual elements of pcoll being passed into each process invocation.
# In this example, side inputs are passed to a FlatMap transform as extra arguments and consumed by filter_using_length.
-
+words = ...
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets_test.py tag:model_pardo_side_input
%}
[3/3] beam-site git commit: This closes #225
Posted by al...@apache.org.
This closes #225
Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/aac38946
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/aac38946
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/aac38946
Branch: refs/heads/asf-site
Commit: aac389466fe998e9fb2a65bf538d3f263e2a2561
Parents: bb4221b 1b93162
Author: Ahmet Altay <al...@altay-macbookpro2.roam.corp.google.com>
Authored: Fri Apr 28 17:39:13 2017 -0700
Committer: Ahmet Altay <al...@altay-macbookpro2.roam.corp.google.com>
Committed: Fri Apr 28 17:39:13 2017 -0700
----------------------------------------------------------------------
content/documentation/programming-guide/index.html | 17 ++++++++++++++++-
src/documentation/programming-guide.md | 7 ++++---
2 files changed, 20 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
[2/3] beam-site git commit: Regenerate website
Posted by al...@apache.org.
Regenerate website
Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/1b931625
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/1b931625
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/1b931625
Branch: refs/heads/asf-site
Commit: 1b931625fc692dd81ead8ab4ce8e315d96d5dea6
Parents: faa12ff
Author: Ahmet Altay <al...@altay-macbookpro2.roam.corp.google.com>
Authored: Fri Apr 28 17:39:13 2017 -0700
Committer: Ahmet Altay <al...@altay-macbookpro2.roam.corp.google.com>
Committed: Fri Apr 28 17:39:13 2017 -0700
----------------------------------------------------------------------
content/documentation/programming-guide/index.html | 17 ++++++++++++++++-
1 file changed, 16 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/beam-site/blob/1b931625/content/documentation/programming-guide/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/programming-guide/index.html b/content/documentation/programming-guide/index.html
index 439d48b..b57f89c 100644
--- a/content/documentation/programming-guide/index.html
+++ b/content/documentation/programming-guide/index.html
@@ -774,6 +774,19 @@ guest, [[], [order4]]
</div>
<div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="n">pc</span> <span class="o">=</span> <span class="o">...</span>
+<span class="k">class</span> <span class="nc">AverageFn</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">CombineFn</span><span class="p">):</span>
+ <span class="k">def</span> <span class="nf">create_accumulator</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">return</span> <span class="p">(</span><span class="mf">0.0</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">add_input</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="p">(</span><span class="nb">sum</span><span class="p">,</span> <span class="n">count</span><span class="p">),</span> <span class="nb">input</span><span class="p">):</span>
+ <span class="k">return</span> <span class="nb">sum</span> <span class="o">+</span> <span class="nb">input</span><span class="p">,</span> <span class="n">count</span> <span class="o">+</span> <span class="mi">1</span>
+
+ <span class="k">def</span> <span class="nf">merge_accumulators</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">accumulators</span><span class="p">):</span>
+ <span class="n">sums</span><span class="p">,</span> <span class="n">counts</span> <span class="o">=</span> <span class="nb">zip</span><span class="p">(</span><span class="o">*</span><span class="n">accumulators</span><span class="p">)</span>
+ <span class="k">return</span> <span class="nb">sum</span><span class="p">(</span><span class="n">sums</span><span class="p">),</span> <span class="nb">sum</span><span class="p">(</span><span class="n">counts</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">extract_output</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="p">(</span><span class="nb">sum</span><span class="p">,</span> <span class="n">count</span><span class="p">)):</span>
+ <span class="k">return</span> <span class="nb">sum</span> <span class="o">/</span> <span class="n">count</span> <span class="k">if</span> <span class="n">count</span> <span class="k">else</span> <span class="nb">float</span><span class="p">(</span><span class="s">'NaN'</span><span class="p">)</span>
</code></pre>
</div>
@@ -794,6 +807,7 @@ guest, [[], [order4]]
<div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="c"># sum combines the elements in the input PCollection.</span>
<span class="c"># The resulting PCollection, called result, contains one value: the sum of all the elements in the input PCollection.</span>
<span class="n">pc</span> <span class="o">=</span> <span class="o">...</span>
+<span class="n">average</span> <span class="o">=</span> <span class="n">pc</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">CombineGlobally</span><span class="p">(</span><span class="n">AverageFn</span><span class="p">())</span>
</code></pre>
</div>
@@ -927,6 +941,7 @@ guest, [[], [order4]]
<div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="c"># Provide an int value with the desired number of result partitions, and a partitioning function (partition_fn in this example).</span>
<span class="c"># Returns a tuple of PCollection objects containing each of the resulting partitions as individual PCollection objects.</span>
+<span class="n">students</span> <span class="o">=</span> <span class="o">...</span>
<span class="k">def</span> <span class="nf">partition_fn</span><span class="p">(</span><span class="n">student</span><span class="p">,</span> <span class="n">num_partitions</span><span class="p">):</span>
<span class="k">return</span> <span class="nb">int</span><span class="p">(</span><span class="n">get_percentile</span><span class="p">(</span><span class="n">student</span><span class="p">)</span> <span class="o">*</span> <span class="n">num_partitions</span> <span class="o">/</span> <span class="mi">100</span><span class="p">)</span>
@@ -1019,7 +1034,7 @@ guest, [[], [order4]]
<span class="c"># Optional, positional, and keyword arguments are all supported. Deferred arguments are unwrapped into their actual values.</span>
<span class="c"># For example, using pvalue.AsIter(pcoll) at pipeline construction time results in an iterable of the actual elements of pcoll being passed into each process invocation.</span>
<span class="c"># In this example, side inputs are passed to a FlatMap transform as extra arguments and consumed by filter_using_length.</span>
-
+<span class="n">words</span> <span class="o">=</span> <span class="o">...</span>
<span class="c"># Callable takes additional arguments.</span>
<span class="k">def</span> <span class="nf">filter_using_length</span><span class="p">(</span><span class="n">word</span><span class="p">,</span> <span class="n">lower_bound</span><span class="p">,</span> <span class="n">upper_bound</span><span class="o">=</span><span class="nb">float</span><span class="p">(</span><span class="s">'inf'</span><span class="p">)):</span>
<span class="k">if</span> <span class="n">lower_bound</span> <span class="o"><=</span> <span class="nb">len</span><span class="p">(</span><span class="n">word</span><span class="p">)</span> <span class="o"><=</span> <span class="n">upper_bound</span><span class="p">:</span>