You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2020/02/06 01:19:03 UTC
[beam] branch asf-site updated: Publishing website 2020/02/06
01:18:56 at commit a8af0e1
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 6ad0c23 Publishing website 2020/02/06 01:18:56 at commit a8af0e1
6ad0c23 is described below
commit 6ad0c23c530bae840823f1266cf8c15163fa4ea3
Author: jenkins <bu...@apache.org>
AuthorDate: Thu Feb 6 01:18:56 2020 +0000
Publishing website 2020/02/06 01:18:56 at commit a8af0e1
---
.../documentation/programming-guide/index.html | 24 ++++++++++++++--------
1 file changed, 15 insertions(+), 9 deletions(-)
diff --git a/website/generated-content/documentation/programming-guide/index.html b/website/generated-content/documentation/programming-guide/index.html
index 178854b..7d1dfae 100644
--- a/website/generated-content/documentation/programming-guide/index.html
+++ b/website/generated-content/documentation/programming-guide/index.html
@@ -3263,10 +3263,6 @@ elements.</p>
<h4 id="managing-late-data">7.4.1. Managing late data</h4>
-<blockquote>
- <p><strong>Note:</strong> Managing late data is not supported in the Beam SDK for Python.</p>
-</blockquote>
-
<p>You can allow late data by invoking the <code class="highlighter-rouge">.withAllowedLateness</code> operation when
you set your <code class="highlighter-rouge">PCollection</code>’s windowing strategy. The following code example
demonstrates a windowing strategy that will allow late data up to two days after
@@ -3278,6 +3274,14 @@ the end of a window.</p>
<span class="o">.</span><span class="na">withAllowedLateness</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardDays</span><span class="o">(</span><span class="mi">2</span><span class="o">)));</span>
</code></pre></div></div>
+<div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="n">pc</span> <span class="o">=</span> <span class="p">[</span><span class="n">Initial</span> <span class="n">PCollection</span><span class="p">]</span>
+ <span class="n">pc</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">WindowInto</span><span class="p">(</span>
+ <span class="n">FixedWindows</span><span class="p">(</span><span class="mi">60</span><span class="p">),</span>
+ <span class="n">trigger</span><span class="o">=</span><span class="n">trigger_fn</span><span class="p">,</span>
+ <span class="n">accumulation_mode</span><span class="o">=</span><span class="n">accumulation_mode</span><span class="p">,</span>
+ <span class="n">timestamp_combiner</span><span class="o">=</span><span class="n">timestamp_combiner</span><span class="p">,</span>
+ <span class="n">allowed_lateness</span><span class="o">=</span><span class="n">Duration</span><span class="p">(</span><span class="n">seconds</span><span class="o">=</span><span class="mi">2</span><span class="o">*</span><span class="mi">24</span><span class="o">*</span><span class="mi">60</span><span class="o">*</span><span class="mi">60</span><span class="p">))</span> <span class="c"># 2 days</span>
+</code></pre></div></div>
<p>When you set <code class="highlighter-rouge">.withAllowedLateness</code> on a <code class="highlighter-rouge">PCollection</code>, that allowed lateness
propagates forward to any subsequent <code class="highlighter-rouge">PCollection</code> derived from the first
<code class="highlighter-rouge">PCollection</code> you applied allowed lateness to. If you want to change the allowed
@@ -3555,10 +3559,6 @@ on each firing:</p>
<h4 id="handling-late-data">8.4.2. Handling late data</h4>
-<blockquote>
- <p>The Beam SDK for Python does not currently support allowed lateness.</p>
-</blockquote>
-
<p>If you want your pipeline to process data that arrives after the watermark
passes the end of the window, you can apply an <em>allowed lateness</em> when you set
your windowing configuration. This gives your trigger the opportunity to react
@@ -3574,7 +3574,13 @@ windowing function:</p>
<span class="o">.</span><span class="na">plusDelayOf</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="mi">1</span><span class="o">)))</span>
<span class="o">.</span><span class="na">withAllowedLateness</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="mi">30</span><span class="o">));</span>
</code></pre></div></div>
-<div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="c"># The Beam SDK for Python does not currently support allowed lateness.</span>
+<div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="n">pc</span> <span class="o">=</span> <span class="p">[</span><span class="n">Initial</span> <span class="n">PCollection</span><span class="p">]</span>
+ <span class="n">pc</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">WindowInto</span><span class="p">(</span>
+ <span class="n">FixedWindows</span><span class="p">(</span><span class="mi">60</span><span class="p">),</span>
+ <span class="n">trigger</span><span class="o">=</span><span class="n">AfterProcessingTime</span><span class="p">(</span><span class="mi">60</span><span class="p">),</span>
+ <span class="n">allowed_lateness</span><span class="o">=</span><span class="mi">1800</span><span class="p">)</span> <span class="c"># 30 minutes</span>
+ <span class="o">|</span> <span class="o">...</span>
+
</code></pre></div></div>
<p>This allowed lateness propagates to all <code class="highlighter-rouge">PCollection</code>s derived as a result of