You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by al...@apache.org on 2017/05/03 00:59:45 UTC
[1/3] beam-site git commit: [BEAM-2142] Add composite transforms
Python snippets to programming guide
Repository: beam-site
Updated Branches:
refs/heads/asf-site a0e95bae3 -> 496f24014
[BEAM-2142] Add composite transforms Python snippets to programming guide
Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/62895054
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/62895054
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/62895054
Branch: refs/heads/asf-site
Commit: 62895054e5915cdde9192fc6a866a58790e93c4f
Parents: a0e95ba
Author: melissa <me...@google.com>
Authored: Tue May 2 15:49:41 2017 -0700
Committer: Ahmet Altay <al...@altay-macbookpro2.roam.corp.google.com>
Committed: Tue May 2 17:55:55 2017 -0700
----------------------------------------------------------------------
src/documentation/programming-guide.md | 30 ++++++++++++++++++++++-------
1 file changed, 23 insertions(+), 7 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/beam-site/blob/62895054/src/documentation/programming-guide.md
----------------------------------------------------------------------
diff --git a/src/documentation/programming-guide.md b/src/documentation/programming-guide.md
index 8e8b917..e3cd2d9 100644
--- a/src/documentation/programming-guide.md
+++ b/src/documentation/programming-guide.md
@@ -983,8 +983,8 @@ Your composite transform's parameters and return value must match the initial in
```
```py
- Python code snippet coming soon (BEAM-1926)
-```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py tag:pipeline_monitoring_composite
+%}```
### Creating a composite transform
@@ -1003,8 +1003,8 @@ The following code sample shows how to declare a `PTransform` that accepts a `PC
```
```py
- Python code snippet coming soon (BEAM-1926)
-```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets_test.py tag:model_composite_transform
+%}```
#### Overriding the expand method
@@ -1024,8 +1024,8 @@ The following code sample shows how to override `expand` for the `ComputeWordLen
```
```py
- Python code snippet coming soon (BEAM-1926)
-```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets_test.py tag:model_composite_transform
+%}```
As long as you override the `expand` method in your `PTransform` subclass to accept the appropriate input `PCollection`(s) and return the corresponding output `PCollection`(s), you can include as many transforms as you want. These transforms can include core transforms, composite transforms, or the transforms included in the Beam SDK libraries.
@@ -1620,7 +1620,7 @@ The following example shows a billing scenario, and uses both early and late fir
```
```py
- The Beam SDK for Python does not support triggers.
+ # The Beam SDK for Python does not support triggers.
```
##### <a name="default-trigger"></a>**Default Trigger**
@@ -1655,6 +1655,10 @@ You set the trigger(s) for a `PCollection` by invoking the method `.triggering()
.discardingFiredPanes());
```
+```py
+ # The Beam SDK for Python does not support triggers.
+```
+
This code sample sets a time-based trigger for a `PCollection`, which emits results one minute after the first element in that window has been processed. The last line in the code sample, `.discardingFiredPanes()`, is the window's **accumulation mode**.
#### Window Accumulation Modes
@@ -1707,6 +1711,10 @@ You set the allowed lateness by using `.withAllowedLateness()` when you set your
.withAllowedLateness(Duration.standardMinutes(30));
```
+```py
+ # The Beam SDK for Python does not support triggers.
+```
+
This allowed lateness propagates to all `PCollection`s derived as a result of applying transforms to the original `PCollection`. If you want to change the allowed lateness later in your pipeline, you can apply `Window.withAllowedLateness()` again, explicitly.
@@ -1748,6 +1756,10 @@ You can express this pattern using `AfterWatermark.pastEndOfWindow`. For example
.withAllowedLateness(Duration.standardDays(2)));
```
+```py
+ # The Beam SDK for Python does not support triggers.
+```
+
#### Other Composite Triggers
You can also build other sorts of composite triggers. The following example code shows a simple composite trigger that fires whenever the pane has at least 100 elements, or after a minute.
@@ -1758,3 +1770,7 @@ You can also build other sorts of composite triggers. The following example code
AfterProcessingTime.pastFirstElementInPane().plusDelayOf(Duration.standardMinutes(1))))
```
+```py
+ # The Beam SDK for Python does not support triggers.
+```
+
[2/3] beam-site git commit: Regenerate website
Posted by al...@apache.org.
Regenerate website
Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/5a7282fe
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/5a7282fe
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/5a7282fe
Branch: refs/heads/asf-site
Commit: 5a7282fe42e9a396673aacfac8f466284dffa1ea
Parents: 6289505
Author: Ahmet Altay <al...@altay-macbookpro2.roam.corp.google.com>
Authored: Tue May 2 17:59:11 2017 -0700
Committer: Ahmet Altay <al...@altay-macbookpro2.roam.corp.google.com>
Committed: Tue May 2 17:59:11 2017 -0700
----------------------------------------------------------------------
content/documentation/io/built-in/index.html | 2 +-
.../documentation/programming-guide/index.html | 46 +++++++++++++++++---
.../sdks/python-type-safety/index.html | 10 ++---
3 files changed, 45 insertions(+), 13 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/beam-site/blob/5a7282fe/content/documentation/io/built-in/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/io/built-in/index.html b/content/documentation/io/built-in/index.html
index 639a877..cf262d3 100644
--- a/content/documentation/io/built-in/index.html
+++ b/content/documentation/io/built-in/index.html
@@ -170,7 +170,7 @@
<td>Java</td>
<td>
<p><a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java">AvroIO</a></p>
- <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/hdfs">Apache Hadoop HDFS</a></p>
+ <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/hadoop-file-system">Apache Hadoop File System</a></p>
<p><a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java">TextIO</a></p>
<p><a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/">XML</a></p>
</td>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/5a7282fe/content/documentation/programming-guide/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/programming-guide/index.html b/content/documentation/programming-guide/index.html
index b57f89c..5ad4bea 100644
--- a/content/documentation/programming-guide/index.html
+++ b/content/documentation/programming-guide/index.html
@@ -1200,18 +1200,18 @@ guest, [[], [order4]]
<span class="k">yield</span> <span class="n">element</span>
<span class="k">else</span><span class="p">:</span>
<span class="c"># Emit this word's long length to the 'above_cutoff_lengths' output.</span>
- <span class="k">yield</span> <span class="n">pvalue</span><span class="o">.</span><span class="n">OutputValue</span><span class="p">(</span>
+ <span class="k">yield</span> <span class="n">pvalue</span><span class="o">.</span><span class="n">TaggedOutput</span><span class="p">(</span>
<span class="s">'above_cutoff_lengths'</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">element</span><span class="p">))</span>
<span class="k">if</span> <span class="n">element</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="n">marker</span><span class="p">):</span>
<span class="c"># Emit this word to a different output with the 'marked strings' tag.</span>
- <span class="k">yield</span> <span class="n">pvalue</span><span class="o">.</span><span class="n">OutputValue</span><span class="p">(</span><span class="s">'marked strings'</span><span class="p">,</span> <span class="n">element</span><span class="p">)</span>
+ <span class="k">yield</span> <span class="n">pvalue</span><span class="o">.</span><span class="n">TaggedOutput</span><span class="p">(</span><span class="s">'marked strings'</span><span class="p">,</span> <span class="n">element</span><span class="p">)</span>
<span class="c"># Producing multiple outputs is also available in Map and FlatMap.</span>
<span class="c"># Here is an example that uses FlatMap and shows that the tags do not need to be specified ahead of time.</span>
<span class="k">def</span> <span class="nf">even_odd</span><span class="p">(</span><span class="n">x</span><span class="p">):</span>
- <span class="k">yield</span> <span class="n">pvalue</span><span class="o">.</span><span class="n">OutputValue</span><span class="p">(</span><span class="s">'odd'</span> <span class="k">if</span> <span class="n">x</span> <span class="o">%</span> <span class="mi">2</span> <span class="k">else</span> <span class="s">'even'</span><span class="p">,</span> <span class="n">x</span><span class="p">)</span>
+ <span class="k">yield</span> <span class="n">pvalue</span><span class="o">.</span><span class="n">TaggedOutput</span><span class="p">(</span><span class="s">'odd'</span> <span class="k">if</span> <span class="n">x</span> <span class="o">%</span> <span class="mi">2</span> <span class="k">else</span> <span class="s">'even'</span><span class="p">,</span> <span class="n">x</span><span class="p">)</span>
<span class="k">if</span> <span class="n">x</span> <span class="o">%</span> <span class="mi">10</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">yield</span> <span class="n">x</span>
@@ -1267,7 +1267,17 @@ guest, [[], [order4]]
</code></pre>
</div>
-<div class="language-py highlighter-rouge"><pre class="highlight"><code> <span class="n">Python</span> <span class="n">code</span> <span class="n">snippet</span> <span class="n">coming</span> <span class="n">soon</span> <span class="p">(</span><span class="n">BEAM</span><span class="o">-</span><span class="mi">1926</span><span class="p">)</span>
+<div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="c"># The CountWords Composite Transform inside the WordCount pipeline.</span>
+<span class="k">class</span> <span class="nc">CountWords</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">PTransform</span><span class="p">):</span>
+
+ <span class="k">def</span> <span class="nf">expand</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">pcoll</span><span class="p">):</span>
+ <span class="k">return</span> <span class="p">(</span><span class="n">pcoll</span>
+ <span class="c"># Convert lines of text into individual words.</span>
+ <span class="o">|</span> <span class="s">'ExtractWords'</span> <span class="o">>></span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">ExtractWordsFn</span><span class="p">())</span>
+ <span class="c"># Count the number of times each word occurs.</span>
+ <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">combiners</span><span class="o">.</span><span class="n">Count</span><span class="o">.</span><span class="n">PerElement</span><span class="p">()</span>
+ <span class="c"># Format each word and count into a printable string.</span>
+ <span class="o">|</span> <span class="s">'FormatCounts'</span> <span class="o">>></span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">FormatCountsFn</span><span class="p">()))</span>
</code></pre>
</div>
@@ -1286,7 +1296,10 @@ guest, [[], [order4]]
</code></pre>
</div>
-<div class="language-py highlighter-rouge"><pre class="highlight"><code> <span class="n">Python</span> <span class="n">code</span> <span class="n">snippet</span> <span class="n">coming</span> <span class="n">soon</span> <span class="p">(</span><span class="n">BEAM</span><span class="o">-</span><span class="mi">1926</span><span class="p">)</span>
+<div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="k">class</span> <span class="nc">ComputeWordLengths</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">PTransform</span><span class="p">):</span>
+ <span class="k">def</span> <span class="nf">expand</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">pcoll</span><span class="p">):</span>
+ <span class="c"># transform logic goes here</span>
+ <span class="k">return</span> <span class="n">pcoll</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="nb">len</span><span class="p">(</span><span class="n">x</span><span class="p">))</span>
</code></pre>
</div>
@@ -1307,7 +1320,10 @@ guest, [[], [order4]]
</code></pre>
</div>
-<div class="language-py highlighter-rouge"><pre class="highlight"><code> <span class="n">Python</span> <span class="n">code</span> <span class="n">snippet</span> <span class="n">coming</span> <span class="n">soon</span> <span class="p">(</span><span class="n">BEAM</span><span class="o">-</span><span class="mi">1926</span><span class="p">)</span>
+<div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="k">class</span> <span class="nc">ComputeWordLengths</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">PTransform</span><span class="p">):</span>
+ <span class="k">def</span> <span class="nf">expand</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">pcoll</span><span class="p">):</span>
+ <span class="c"># transform logic goes here</span>
+ <span class="k">return</span> <span class="n">pcoll</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="nb">len</span><span class="p">(</span><span class="n">x</span><span class="p">))</span>
</code></pre>
</div>
@@ -1920,7 +1936,7 @@ Subsequent transforms, however, are applied to the result of the <code class="hi
</code></pre>
</div>
-<div class="language-py highlighter-rouge"><pre class="highlight"><code> <span class="n">The</span> <span class="n">Beam</span> <span class="n">SDK</span> <span class="k">for</span> <span class="n">Python</span> <span class="n">does</span> <span class="ow">not</span> <span class="n">support</span> <span class="n">triggers</span><span class="o">.</span>
+<div class="language-py highlighter-rouge"><pre class="highlight"><code> <span class="c"># The Beam SDK for Python does not support triggers.</span>
</code></pre>
</div>
@@ -1956,6 +1972,10 @@ Subsequent transforms, however, are applied to the result of the <code class="hi
</code></pre>
</div>
+<div class="language-py highlighter-rouge"><pre class="highlight"><code> <span class="c"># The Beam SDK for Python does not support triggers.</span>
+</code></pre>
+</div>
+
<p>This code sample sets a time-based trigger for a <code class="highlighter-rouge">PCollection</code>, which emits results one minute after the first element in that window has been processed. The last line in the code sample, <code class="highlighter-rouge">.discardingFiredPanes()</code>, is the window’s <strong>accumulation mode</strong>.</p>
<h4 id="window-accumulation-modes">Window Accumulation Modes</h4>
@@ -2009,6 +2029,10 @@ Subsequent transforms, however, are applied to the result of the <code class="hi
</code></pre>
</div>
+<div class="language-py highlighter-rouge"><pre class="highlight"><code> <span class="c"># The Beam SDK for Python does not support triggers.</span>
+</code></pre>
+</div>
+
<p>This allowed lateness propagates to all <code class="highlighter-rouge">PCollection</code>s derived as a result of applying transforms to the original <code class="highlighter-rouge">PCollection</code>. If you want to change the allowed lateness later in your pipeline, you can apply <code class="highlighter-rouge">Window.withAllowedLateness()</code> again, explicitly.</p>
<h3 id="a-namecomposite-triggersacomposite-triggers"><a name="composite-triggers"></a>Composite Triggers</h3>
@@ -2055,6 +2079,10 @@ Subsequent transforms, however, are applied to the result of the <code class="hi
</code></pre>
</div>
+<div class="language-py highlighter-rouge"><pre class="highlight"><code> <span class="c"># The Beam SDK for Python does not support triggers.</span>
+</code></pre>
+</div>
+
<h4 id="other-composite-triggers">Other Composite Triggers</h4>
<p>You can also build other sorts of composite triggers. The following example code shows a simple composite trigger that fires whenever the pane has at least 100 elements, or after a minute.</p>
@@ -2065,6 +2093,10 @@ Subsequent transforms, however, are applied to the result of the <code class="hi
</code></pre>
</div>
+<div class="language-py highlighter-rouge"><pre class="highlight"><code> <span class="c"># The Beam SDK for Python does not support triggers.</span>
+</code></pre>
+</div>
+
</div>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/5a7282fe/content/documentation/sdks/python-type-safety/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/sdks/python-type-safety/index.html b/content/documentation/sdks/python-type-safety/index.html
index 24900b8..cd364cf 100644
--- a/content/documentation/sdks/python-type-safety/index.html
+++ b/content/documentation/sdks/python-type-safety/index.html
@@ -167,7 +167,9 @@
<p>Consider the following example, in which <code class="highlighter-rouge">numbers</code> is a <code class="highlighter-rouge">PCollection</code> of <code class="highlighter-rouge">str</code> values:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>numbers = p | beam.Create(['1', '2', '3'])
+<div class="highlighter-rouge"><pre class="highlight"><code>p = TestPipeline(options=PipelineOptions(pipeline_type_check=True))
+
+numbers = p | beam.Create(['1', '2', '3'])
</code></pre>
</div>
@@ -189,8 +191,7 @@
<p>To specify type hints inline, use the methods <code class="highlighter-rouge">with_input_types</code> and <code class="highlighter-rouge">with_output_types</code>. The following example code declares an input type hint inline:</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>p.options.view_as(TypeOptions).pipeline_type_check = True
-evens = numbers | beam.Filter(lambda x: x % 2 == 0).with_input_types(int)
+<div class="highlighter-rouge"><pre class="highlight"><code>evens = numbers | beam.Filter(lambda x: x % 2 == 0).with_input_types(int)
</code></pre>
</div>
@@ -284,8 +285,7 @@ p.run()
<p>However, if you enable runtime type checking, the code passes at pipeline construction and fails at runtime. To enable runtime type checking, set the pipeline option <code class="highlighter-rouge">runtime_type_check</code> to <code class="highlighter-rouge">True</code>.</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>p.options.view_as(TypeOptions).runtime_type_check = True
-p | beam.Create(['a']) | beam.Map(lambda x: 3).with_output_types(str)
+<div class="highlighter-rouge"><pre class="highlight"><code>p | beam.Create(['a']) | beam.Map(lambda x: 3).with_output_types(str)
p.run()
</code></pre>
</div>
[3/3] beam-site git commit: This closes #227
Posted by al...@apache.org.
This closes #227
Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/496f2401
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/496f2401
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/496f2401
Branch: refs/heads/asf-site
Commit: 496f24014f544ed60f211eaf47722fd0e3be02de
Parents: a0e95ba 5a7282f
Author: Ahmet Altay <al...@altay-macbookpro2.roam.corp.google.com>
Authored: Tue May 2 17:59:12 2017 -0700
Committer: Ahmet Altay <al...@altay-macbookpro2.roam.corp.google.com>
Committed: Tue May 2 17:59:12 2017 -0700
----------------------------------------------------------------------
content/documentation/io/built-in/index.html | 2 +-
.../documentation/programming-guide/index.html | 46 +++++++++++++++++---
.../sdks/python-type-safety/index.html | 10 ++---
src/documentation/programming-guide.md | 30 ++++++++++---
4 files changed, 68 insertions(+), 20 deletions(-)
----------------------------------------------------------------------