You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2022/05/03 04:17:09 UTC
[beam] branch asf-site updated: Publishing website 2022/05/03 04:17:00 at commit 0daef62
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new a470b0a395b Publishing website 2022/05/03 04:17:00 at commit 0daef62
a470b0a395b is described below
commit a470b0a395b3ee0f038a6742e0911b73b081c4be
Author: jenkins <bu...@apache.org>
AuthorDate: Tue May 3 04:17:01 2022 +0000
Publishing website 2022/05/03 04:17:00 at commit 0daef62
---
website/generated-content/documentation/index.xml | 47 +++++++++++++++++-
.../documentation/programming-guide/index.html | 56 +++++++++++++++++++++-
website/generated-content/sitemap.xml | 2 +-
3 files changed, 101 insertions(+), 4 deletions(-)
diff --git a/website/generated-content/documentation/index.xml b/website/generated-content/documentation/index.xml
index 336a66df3f5..f535f993226 100644
--- a/website/generated-content/documentation/index.xml
+++ b/website/generated-content/documentation/index.xml
@@ -11627,7 +11627,52 @@ watermark estimator implementation. You can also provide your own watermark esti
<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
<img src="/images/copy-icon.svg"/>
</a>
-<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="nx">This</span> <span class="nx">is</span> <span class="nx">not</span> <span class="nx">supported</span> <span class="nx">yet</span><span class="p">,</span> <span class="nx">see</span> <span class="nx">BEAM</span><span class="o">-</span><span class="mf">11105.</span></code></pre></div>
+<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="c1">// WatermarkState is a custom type.`
+</span><span class="c1">//
+</span><span class="c1">// It is optional to write your own state type when making a custom estimator.
+</span><span class="c1"></span><span class="kd">type</span> <span class="nx">WatermarkState</span> <span class="kd">struct</span> <span class="p">{</span>
+<span class="nx">Watermark</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Time</span>
+<span class="p">}</span>
+<span class="c1">// CustomWatermarkEstimator is a custom watermark estimator.
+</span><span class="c1">// You may use any type here, including some of Beam&#39;s built in watermark estimator types,
+</span><span class="c1">// e.g. sdf.WallTimeWatermarkEstimator, sdf.TimestampObservingWatermarkEstimator, and sdf.ManualWatermarkEstimator
+</span><span class="c1"></span><span class="kd">type</span> <span class="nx">CustomWatermarkEstimator</span> <span class="kd">struct</span> <span class="p">{</span>
+<span class="nx">state</span> <span class="nx">WatermarkState</span>
+<span class="p">}</span>
+<span class="c1">// CurrentWatermark returns the current watermark and is invoked on DoFn splits and self-checkpoints.
+</span><span class="c1">// Watermark estimators must implement CurrentWatermark() time.Time
+</span><span class="c1"></span><span class="kd">func</span> <span class="p">(</span><span class="nx">e</span> <span class="o">*</span><span class="nx">CustomWatermarkEstimator</span><span class="p">)</span> <span class="nf">CurrentWatermark</span><span class="p">()</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Time</span> <span class="p">{</span>
+<span class="k">return</span> <span class="nx">e</span><span class="p">.</span><span class="nx">state</span><span class="p">.</span><span class="nx">Watermark</span>
+<span class="p">}</span>
+<span class="c1">// ObserveTimestamp is called on the output timestamps of all
+</span><span class="c1">// emitted elements to update the watermark. It is optional
+</span><span class="c1"></span><span class="kd">func</span> <span class="p">(</span><span class="nx">e</span> <span class="o">*</span><span class="nx">CustomWatermarkEstimator</span><span class="p">)</span> <span class="nf">ObserveTimestamp</span><span class="p">(</span><span class="nx">ts</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Time</span><span class="p">)</span> &l [...]
+<span class="nx">e</span><span class="p">.</span><span class="nx">state</span><span class="p">.</span><span class="nx">Watermark</span> <span class="p">=</span> <span class="nx">ts</span>
+<span class="p">}</span>
+<span class="c1">// InitialWatermarkEstimatorState defines an initial state used to initialize the watermark
+</span><span class="c1">// estimator. It is optional. If this is not defined, WatermarkEstimatorState may not be
+</span><span class="c1">// defined and CreateWatermarkEstimator must not take in parameters.
+</span><span class="c1"></span><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">weDoFn</span><span class="p">)</span> <span class="nf">InitialWatermarkEstimatorState</span><span class="p">(</span><span class="nx">et</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">EventTime</span><span class="p">,</span> [...]
+<span class="c1">// Return some watermark state
+</span><span class="c1"></span> <span class="k">return</span> <span class="nx">WatermarkState</span><span class="p">{</span><span class="nx">Watermark</span><span class="p">:</span> <span class="nx">time</span><span class="p">.</span><span class="nf">Now</span><span class="p">()}</span>
+<span class="p">}</span>
+<span class="c1">// CreateWatermarkEstimator creates the watermark estimator used by this Splittable DoFn.
+</span><span class="c1">// Must take in a state parameter if InitialWatermarkEstimatorState is defined, otherwise takes no parameters.
+</span><span class="c1"></span><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">weDoFn</span><span class="p">)</span> <span class="nf">CreateWatermarkEstimator</span><span class="p">(</span><span class="nx">initialState</span> <span class="nx">WatermarkState</span><span class="p">)</span> <span class="o">*</span><span class="nx">Custo [...]
+<span class="k">return</span> <span class="o">&amp;</span><span class="nx">CustomWatermarkEstimator</span><span class="p">{</span><span class="nx">state</span><span class="p">:</span> <span class="nx">initialState</span><span class="p">}</span>
+<span class="p">}</span>
+<span class="c1">// WatermarkEstimatorState returns the state used to resume future watermark estimation
+</span><span class="c1">// after a checkpoint/split. It is required if InitialWatermarkEstimatorState is defined,
+</span><span class="c1">// otherwise it must not be defined.
+</span><span class="c1"></span><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">weDoFn</span><span class="p">)</span> <span class="nf">WatermarkEstimatorState</span><span class="p">(</span><span class="nx">e</span> <span class="o">*</span><span class="nx">CustomWatermarkEstimator</span><span class="p">)</span> <span class="nx">Waterma [...]
+<span class="k">return</span> <span class="nx">e</span><span class="p">.</span><span class="nx">state</span>
+<span class="p">}</span>
+<span class="c1">// ProcessElement is the method to execute for each element.
+</span><span class="c1">// It can optionally take in a watermark estimator.
+</span><span class="c1"></span><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">weDoFn</span><span class="p">)</span> <span class="nf">ProcessElement</span><span class="p">(</span><span class="nx">e</span> <span class="o">*</span><span class="nx">CustomWatermarkEstimator</span><span class="p">,</span> <span class="nx">element</span [...]
+<span class="c1">// ...
+</span><span class="c1"></span> <span class="nx">e</span><span class="p">.</span><span class="nx">state</span><span class="p">.</span><span class="nx">Watermark</span> <span class="p">=</span> <span class="nx">time</span><span class="p">.</span><span class="nf">Now</span><span class="p">()</span>
+<span class="p">}</span></code></pre></div>
</div>
</div>
<h3 id="truncating-during-drain">12.6. Truncating during drain</h3>
diff --git a/website/generated-content/documentation/programming-guide/index.html b/website/generated-content/documentation/programming-guide/index.html
index 9ca51a73d33..578f66a485d 100644
--- a/website/generated-content/documentation/programming-guide/index.html
+++ b/website/generated-content/documentation/programming-guide/index.html
@@ -4116,7 +4116,59 @@ watermark estimator implementation. You can also provide your own watermark esti
<span class=n>watermark_estimator</span><span class=o>=</span><span class=n>beam</span><span class=o>.</span><span class=n>DoFn</span><span class=o>.</span><span class=n>WatermarkEstimatorParam</span><span class=p>(</span>
<span class=n>MyWatermarkEstimatorProvider</span><span class=p>())):</span>
<span class=c1># The current watermark can be inspected.</span>
- <span class=n>watermark_estimator</span><span class=o>.</span><span class=n>current_watermark</span><span class=p>()</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-go data-lang=go><span class=nx>This</span> <span class=nx>is</span [...]
+ <span class=n>watermark_estimator</span><span class=o>.</span><span class=n>current_watermark</span><span class=p>()</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-go data-lang=go><span class=c1>// WatermarkState is a custom type.`
+</span><span class=c1>//
+</span><span class=c1>// It is optional to write your own state type when making a custom estimator.
+</span><span class=c1></span><span class=kd>type</span> <span class=nx>WatermarkState</span> <span class=kd>struct</span> <span class=p>{</span>
+ <span class=nx>Watermark</span> <span class=nx>time</span><span class=p>.</span><span class=nx>Time</span>
+<span class=p>}</span>
+
+<span class=c1>// CustomWatermarkEstimator is a custom watermark estimator.
+</span><span class=c1>// You may use any type here, including some of Beam's built in watermark estimator types,
+</span><span class=c1>// e.g. sdf.WallTimeWatermarkEstimator, sdf.TimestampObservingWatermarkEstimator, and sdf.ManualWatermarkEstimator
+</span><span class=c1></span><span class=kd>type</span> <span class=nx>CustomWatermarkEstimator</span> <span class=kd>struct</span> <span class=p>{</span>
+ <span class=nx>state</span> <span class=nx>WatermarkState</span>
+<span class=p>}</span>
+
+<span class=c1>// CurrentWatermark returns the current watermark and is invoked on DoFn splits and self-checkpoints.
+</span><span class=c1>// Watermark estimators must implement CurrentWatermark() time.Time
+</span><span class=c1></span><span class=kd>func</span> <span class=p>(</span><span class=nx>e</span> <span class=o>*</span><span class=nx>CustomWatermarkEstimator</span><span class=p>)</span> <span class=nf>CurrentWatermark</span><span class=p>()</span> <span class=nx>time</span><span class=p>.</span><span class=nx>Time</span> <span class=p>{</span>
+ <span class=k>return</span> <span class=nx>e</span><span class=p>.</span><span class=nx>state</span><span class=p>.</span><span class=nx>Watermark</span>
+<span class=p>}</span>
+
+<span class=c1>// ObserveTimestamp is called on the output timestamps of all
+</span><span class=c1>// emitted elements to update the watermark. It is optional
+</span><span class=c1></span><span class=kd>func</span> <span class=p>(</span><span class=nx>e</span> <span class=o>*</span><span class=nx>CustomWatermarkEstimator</span><span class=p>)</span> <span class=nf>ObserveTimestamp</span><span class=p>(</span><span class=nx>ts</span> <span class=nx>time</span><span class=p>.</span><span class=nx>Time</span><span class=p>)</span> <span class=p>{</span>
+ <span class=nx>e</span><span class=p>.</span><span class=nx>state</span><span class=p>.</span><span class=nx>Watermark</span> <span class=p>=</span> <span class=nx>ts</span>
+<span class=p>}</span>
+
+<span class=c1>// InitialWatermarkEstimatorState defines an initial state used to initialize the watermark
+</span><span class=c1>// estimator. It is optional. If this is not defined, WatermarkEstimatorState may not be
+</span><span class=c1>// defined and CreateWatermarkEstimator must not take in parameters.
+</span><span class=c1></span><span class=kd>func</span> <span class=p>(</span><span class=nx>fn</span> <span class=o>*</span><span class=nx>weDoFn</span><span class=p>)</span> <span class=nf>InitialWatermarkEstimatorState</span><span class=p>(</span><span class=nx>et</span> <span class=nx>beam</span><span class=p>.</span><span class=nx>EventTime</span><span class=p>,</span> <span class=nx>rest</span> <span class=nx>offsetrange</span><span class=p>.</span><span class=nx>Restriction</span> [...]
+ <span class=c1>// Return some watermark state
+</span><span class=c1></span> <span class=k>return</span> <span class=nx>WatermarkState</span><span class=p>{</span><span class=nx>Watermark</span><span class=p>:</span> <span class=nx>time</span><span class=p>.</span><span class=nf>Now</span><span class=p>()}</span>
+<span class=p>}</span>
+
+<span class=c1>// CreateWatermarkEstimator creates the watermark estimator used by this Splittable DoFn.
+</span><span class=c1>// Must take in a state parameter if InitialWatermarkEstimatorState is defined, otherwise takes no parameters.
+</span><span class=c1></span><span class=kd>func</span> <span class=p>(</span><span class=nx>fn</span> <span class=o>*</span><span class=nx>weDoFn</span><span class=p>)</span> <span class=nf>CreateWatermarkEstimator</span><span class=p>(</span><span class=nx>initialState</span> <span class=nx>WatermarkState</span><span class=p>)</span> <span class=o>*</span><span class=nx>CustomWatermarkEstimator</span> <span class=p>{</span>
+ <span class=k>return</span> <span class=o>&</span><span class=nx>CustomWatermarkEstimator</span><span class=p>{</span><span class=nx>state</span><span class=p>:</span> <span class=nx>initialState</span><span class=p>}</span>
+<span class=p>}</span>
+
+<span class=c1>// WatermarkEstimatorState returns the state used to resume future watermark estimation
+</span><span class=c1>// after a checkpoint/split. It is required if InitialWatermarkEstimatorState is defined,
+</span><span class=c1>// otherwise it must not be defined.
+</span><span class=c1></span><span class=kd>func</span> <span class=p>(</span><span class=nx>fn</span> <span class=o>*</span><span class=nx>weDoFn</span><span class=p>)</span> <span class=nf>WatermarkEstimatorState</span><span class=p>(</span><span class=nx>e</span> <span class=o>*</span><span class=nx>CustomWatermarkEstimator</span><span class=p>)</span> <span class=nx>WatermarkState</span> <span class=p>{</span>
+ <span class=k>return</span> <span class=nx>e</span><span class=p>.</span><span class=nx>state</span>
+<span class=p>}</span>
+
+<span class=c1>// ProcessElement is the method to execute for each element.
+</span><span class=c1>// It can optionally take in a watermark estimator.
+</span><span class=c1></span><span class=kd>func</span> <span class=p>(</span><span class=nx>fn</span> <span class=o>*</span><span class=nx>weDoFn</span><span class=p>)</span> <span class=nf>ProcessElement</span><span class=p>(</span><span class=nx>e</span> <span class=o>*</span><span class=nx>CustomWatermarkEstimator</span><span class=p>,</span> <span class=nx>element</span> <span class=kt>string</span><span class=p>)</span> <span class=p>{</span>
+ <span class=c1>// ...
+</span><span class=c1></span> <span class=nx>e</span><span class=p>.</span><span class=nx>state</span><span class=p>.</span><span class=nx>Watermark</span> <span class=p>=</span> <span class=nx>time</span><span class=p>.</span><span class=nf>Now</span><span class=p>()</span>
+<span class=p>}</span></code></pre></div></div></div><h3 id=truncating-during-drain>12.6. Truncating during drain</h3><p>Runners which support draining pipelines need the ability to drain SDFs; otherwise, the
pipeline may never stop. By default, bounded restrictions process the remainder of the restriction while
unbounded restrictions finish processing at the next SDF-initiated checkpoint or runner-initiated split.
You are able to override this default behavior by defining the appropriate method on the restriction
@@ -4340,7 +4392,7 @@ functions as shortcuts for single, unnamed inputs/outputs or define a map for na
<span class=nx>outT</span> <span class=o>:=</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>UnnamedOutput</span><span class=p>(</span><span class=nx>typex</span><span class=p>.</span><span class=nf>New</span><span class=p>(</span><span class=nx>reflectx</span><span class=p>.</span><span class=nx>String</span><span class=p>))</span>
<span class=nx>res</span> <span class=o>:=</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>CrossLanguage</span><span class=p>(</span><span class=nx>s</span><span class=p>,</span> <span class=nx>urn</span><span class=p>,</span> <span class=nx>payload</span><span class=p>,</span> <span class=nx>expansionAddr</span><span class=p>,</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>UnnamedInput</span><span class=p>(</span><span class=nx>inputPCol</spa [...]
</code></pre></div></li><li><p>After the job has been submitted to the Beam runner, shutdown the expansion service by
-terminating the expansion service process.</p></li></ol><h3 id=x-lang-transform-runner-support>13.3. Runner Support</h3><p>Currently, portable runners such as Flink, Spark, and the direct runner can be used with multi-language pipelines.</p><p>Dataflow supports multi-language pipelines through the Dataflow Runner v2 backend architecture.</p><h3 id=x-lang-transform-tips-troubleshooting>13.4 Tips and Troubleshooting</h3><p>For additional tips and troubleshooting information, see <a href=ht [...]
+terminating the expansion service process.</p></li></ol><h3 id=x-lang-transform-runner-support>13.3. Runner Support</h3><p>Currently, portable runners such as Flink, Spark, and the direct runner can be used with multi-language pipelines.</p><p>Dataflow supports multi-language pipelines through the Dataflow Runner v2 backend architecture.</p><h3 id=x-lang-transform-tips-troubleshooting>13.4 Tips and Troubleshooting</h3><p>For additional tips and troubleshooting information, see <a href=ht [...]
<a href=http://www.apache.org>The Apache Software Foundation</a>
| <a href=/privacy_policy>Privacy Policy</a>
| <a href=/feed.xml>RSS Feed</a><br><br>Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation. All other products or name brands are trademarks of their respective holders, including The Apache Software Foundation.</div></div></div></div></footer></body></html>
\ No newline at end of file
diff --git a/website/generated-content/sitemap.xml b/website/generated-content/sitemap.xml
index 6eb3a9963ae..e2bc4573dab 100644
--- a/website/generated-content/sitemap.xml
+++ b/website/generated-content/sitemap.xml
@@ -1 +1 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/categories/blog/</loc><lastmod>2022-04-29T15:54:00-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2022-04-29T15:54:00-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2022-04-29T15:54:00-07:00</lastmod></url><url><loc>/blog/beam-sql-with-notebooks/</loc><lastmod>2022-04-29T15:54:00-07:00</lastmod></url><url> [...]
\ No newline at end of file
+<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/categories/blog/</loc><lastmod>2022-04-29T15:54:00-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2022-04-29T15:54:00-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2022-04-29T15:54:00-07:00</lastmod></url><url><loc>/blog/beam-sql-with-notebooks/</loc><lastmod>2022-04-29T15:54:00-07:00</lastmod></url><url> [...]
\ No newline at end of file