You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2022/04/18 16:17:37 UTC
[beam] branch asf-site updated: Publishing website 2022/04/18 16:17:30 at commit 3ce67b2
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 687f6568426 Publishing website 2022/04/18 16:17:30 at commit 3ce67b2
687f6568426 is described below
commit 687f656842617cc01fc117d2b5daa6801fae868e
Author: jenkins <bu...@apache.org>
AuthorDate: Mon Apr 18 16:17:31 2022 +0000
Publishing website 2022/04/18 16:17:30 at commit 3ce67b2
---
.../documentation/patterns/side-inputs/index.html | 132 +++++++++++----------
1 file changed, 67 insertions(+), 65 deletions(-)
diff --git a/website/generated-content/documentation/patterns/side-inputs/index.html b/website/generated-content/documentation/patterns/side-inputs/index.html
index 0f05681b161..9859f10bfdc 100644
--- a/website/generated-content/documentation/patterns/side-inputs/index.html
+++ b/website/generated-content/documentation/patterns/side-inputs/index.html
@@ -18,72 +18,74 @@
function addPlaceholder(){$('input:text').attr('placeholder',"What are you looking for?");}
function endSearch(){var search=document.querySelector(".searchBar");search.classList.add("disappear");var icons=document.querySelector("#iconsBar");icons.classList.remove("disappear");}
function blockScroll(){$("body").toggleClass("fixedPosition");}
-function openMenu(){addPlaceholder();blockScroll();}</script><div class="clearfix container-main-content"><div class="section-nav closed" data-offset-top=90 data-offset-bottom=500><span class="section-nav-back glyphicon glyphicon-menu-left"></span><nav><ul class=section-nav-list data-section-nav><li><span class=section-nav-list-main-title>Documentation</span></li><li><a href=/documentation>Using the Documentation</a></li><li class=section-nav-item--collapsible><span class=section-nav-lis [...]
- <span class=c1>// This pipeline uses View.asSingleton for a placeholder external service.
-</span><span class=c1></span> <span class=c1>// Run in debug mode to see the output.
-</span><span class=c1></span> <span class=n>Pipeline</span> <span class=n>p</span> <span class=o>=</span> <span class=n>Pipeline</span><span class=o>.</span><span class=na>create</span><span class=o>();</span>
-
- <span class=c1>// Create a side input that updates each second.
-</span><span class=c1></span> <span class=n>PCollectionView</span><span class=o><</span><span class=n>Map</span><span class=o><</span><span class=n>String</span><span class=o>,</span> <span class=n>String</span><span class=o>>></span> <span class=n>map</span> <span class=o>=</span>
- <span class=n>p</span><span class=o>.</span><span class=na>apply</span><span class=o>(</span><span class=n>GenerateSequence</span><span class=o>.</span><span class=na>from</span><span class=o>(</span><span class=n>0</span><span class=o>).</span><span class=na>withRate</span><span class=o>(</span><span class=n>1</span><span class=o>,</span> <span class=n>Duration</span><span class=o>.</span><span class=na>standardSeconds</span><span class=o>(</span><span class=n>5L</span><span class [...]
- <span class=o>.</span><span class=na>apply</span><span class=o>(</span>
- <span class=n>Window</span><span class=o>.<</span><span class=n>Long</span><span class=o>></span><span class=n>into</span><span class=o>(</span><span class=k>new</span> <span class=n>GlobalWindows</span><span class=o>())</span>
- <span class=o>.</span><span class=na>triggering</span><span class=o>(</span><span class=n>Repeatedly</span><span class=o>.</span><span class=na>forever</span><span class=o>(</span><span class=n>AfterProcessingTime</span><span class=o>.</span><span class=na>pastFirstElementInPane</span><span class=o>()))</span>
- <span class=o>.</span><span class=na>discardingFiredPanes</span><span class=o>())</span>
- <span class=o>.</span><span class=na>apply</span><span class=o>(</span>
- <span class=n>ParDo</span><span class=o>.</span><span class=na>of</span><span class=o>(</span>
- <span class=k>new</span> <span class=n>DoFn</span><span class=o><</span><span class=n>Long</span><span class=o>,</span> <span class=n>Map</span><span class=o><</span><span class=n>String</span><span class=o>,</span> <span class=n>String</span><span class=o>>>()</span> <span class=o>{</span>
-
- <span class=nd>@ProcessElement</span>
- <span class=kd>public</span> <span class=kt>void</span> <span class=nf>process</span><span class=o>(</span>
- <span class=nd>@Element</span> <span class=n>Long</span> <span class=n>input</span><span class=o>,</span> <span class=n>OutputReceiver</span><span class=o><</span><span class=n>Map</span><span class=o><</span><span class=n>String</span><span class=o>,</span> <span class=n>String</span><span class=o>>></span> <span class=n>o</span><span class=o>)</span> <span class=o>{</span>
- <span class=c1>// Replace map with test data from the placeholder external service.
-</span><span class=c1></span> <span class=c1>// Add external reads here.
-</span><span class=c1></span> <span class=n>o</span><span class=o>.</span><span class=na>output</span><span class=o>(</span><span class=n>PlaceholderExternalService</span><span class=o>.</span><span class=na>readTestData</span><span class=o>());</span>
- <span class=o>}</span>
- <span class=o>}))</span>
- <span class=o>.</span><span class=na>apply</span><span class=o>(</span><span class=n>View</span><span class=o>.</span><span class=na>asSingleton</span><span class=o>());</span>
-
- <span class=c1>// Consume side input. GenerateSequence generates test data.
-</span><span class=c1></span> <span class=c1>// Use a real source (like PubSubIO or KafkaIO) in production.
-</span><span class=c1></span> <span class=n>p</span><span class=o>.</span><span class=na>apply</span><span class=o>(</span><span class=n>GenerateSequence</span><span class=o>.</span><span class=na>from</span><span class=o>(</span><span class=n>0</span><span class=o>).</span><span class=na>withRate</span><span class=o>(</span><span class=n>1</span><span class=o>,</span> <span class=n>Duration</span><span class=o>.</span><span class=na>standardSeconds</span><span class=o>(</span><span cla [...]
- <span class=o>.</span><span class=na>apply</span><span class=o>(</span><span class=n>Window</span><span class=o>.</span><span class=na>into</span><span class=o>(</span><span class=n>FixedWindows</span><span class=o>.</span><span class=na>of</span><span class=o>(</span><span class=n>Duration</span><span class=o>.</span><span class=na>standardSeconds</span><span class=o>(</span><span class=n>1</span><span class=o>))))</span>
- <span class=o>.</span><span class=na>apply</span><span class=o>(</span><span class=n>Sum</span><span class=o>.</span><span class=na>longsGlobally</span><span class=o>().</span><span class=na>withoutDefaults</span><span class=o>())</span>
- <span class=o>.</span><span class=na>apply</span><span class=o>(</span>
- <span class=n>ParDo</span><span class=o>.</span><span class=na>of</span><span class=o>(</span>
- <span class=k>new</span> <span class=n>DoFn</span><span class=o><</span><span class=n>Long</span><span class=o>,</span> <span class=n>KV</span><span class=o><</span><span class=n>Long</span><span class=o>,</span> <span class=n>Long</span><span class=o>>>()</span> <span class=o>{</span>
-
- <span class=nd>@ProcessElement</span>
- <span class=kd>public</span> <span class=kt>void</span> <span class=nf>process</span><span class=o>(</span><span class=n>ProcessContext</span> <span class=n>c</span><span class=o>)</span> <span class=o>{</span>
- <span class=n>Map</span><span class=o><</span><span class=n>String</span><span class=o>,</span> <span class=n>String</span><span class=o>></span> <span class=n>keyMap</span> <span class=o>=</span> <span class=n>c</span><span class=o>.</span><span class=na>sideInput</span><span class=o>(</span><span class=n>map</span><span class=o>);</span>
- <span class=n>c</span><span class=o>.</span><span class=na>outputWithTimestamp</span><span class=o>(</span><span class=n>KV</span><span class=o>.</span><span class=na>of</span><span class=o>(</span><span class=n>1L</span><span class=o>,</span> <span class=n>c</span><span class=o>.</span><span class=na>element</span><span class=o>()),</span> <span class=n>Instant</span><span class=o>.</span><span class=na>now</span><span class=o>());</span>
-
- <span class=n>LOG</span><span class=o>.</span><span class=na>debug</span><span class=o>(</span>
- <span class=s>"Value is {}, key A is {}, and key B is {}."</span><span class=o>,</span>
- <span class=n>c</span><span class=o>.</span><span class=na>element</span><span class=o>(),</span>
- <span class=n>keyMap</span><span class=o>.</span><span class=na>get</span><span class=o>(</span><span class=s>"Key_A"</span><span class=o>),</span>
- <span class=n>keyMap</span><span class=o>.</span><span class=na>get</span><span class=o>(</span><span class=s>"Key_B"</span><span class=o>));</span>
- <span class=o>}</span>
- <span class=o>})</span>
- <span class=o>.</span><span class=na>withSideInputs</span><span class=o>(</span><span class=n>map</span><span class=o>));</span>
-<span class=o>}</span>
-
-<span class=cm>/** Placeholder class that represents an external service generating test data. */</span>
-<span class=kd>public</span> <span class=kd>static</span> <span class=kd>class</span> <span class=nc>PlaceholderExternalService</span> <span class=o>{</span>
-
- <span class=kd>public</span> <span class=kd>static</span> <span class=n>Map</span><span class=o><</span><span class=n>String</span><span class=o>,</span> <span class=n>String</span><span class=o>></span> <span class=nf>readTestData</span><span class=o>()</span> <span class=o>{</span>
-
- <span class=n>Map</span><span class=o><</span><span class=n>String</span><span class=o>,</span> <span class=n>String</span><span class=o>></span> <span class=n>map</span> <span class=o>=</span> <span class=k>new</span> <span class=n>HashMap</span><span class=o><>();</span>
- <span class=n>Instant</span> <span class=n>now</span> <span class=o>=</span> <span class=n>Instant</span><span class=o>.</span><span class=na>now</span><span class=o>();</span>
-
- <span class=n>DateTimeFormatter</span> <span class=n>dtf</span> <span class=o>=</span> <span class=n>DateTimeFormat</span><span class=o>.</span><span class=na>forPattern</span><span class=o>(</span><span class=s>"HH:MM:SS"</span><span class=o>);</span>
-
- <span class=n>map</span><span class=o>.</span><span class=na>put</span><span class=o>(</span><span class=s>"Key_A"</span><span class=o>,</span> <span class=n>now</span><span class=o>.</span><span class=na>minus</span><span class=o>(</span><span class=n>Duration</span><span class=o>.</span><span class=na>standardSeconds</span><span class=o>(</span><span class=n>30</span><span class=o>)).</span><span class=na>toString</span><span class=o>(</span><span class=n>dtf</span><span cl [...]
- <span class=n>map</span><span class=o>.</span><span class=na>put</span><span class=o>(</span><span class=s>"Key_B"</span><span class=o>,</span> <span class=n>now</span><span class=o>.</span><span class=na>minus</span><span class=o>(</span><span class=n>Duration</span><span class=o>.</span><span class=na>standardSeconds</span><span class=o>(</span><span class=n>30</span><span class=o>)).</span><span class=na>toString</span><span class=o>());</span>
-
- <span class=k>return</span> <span class=n>map</span><span class=o>;</span>
+function openMenu(){addPlaceholder();blockScroll();}</script><div class="clearfix container-main-content"><div class="section-nav closed" data-offset-top=90 data-offset-bottom=500><span class="section-nav-back glyphicon glyphicon-menu-left"></span><nav><ul class=section-nav-list data-section-nav><li><span class=section-nav-list-main-title>Documentation</span></li><li><a href=/documentation>Using the Documentation</a></li><li class=section-nav-item--collapsible><span class=section-nav-lis [...]
+ <span class=c1>// This pipeline uses View.asSingleton for a placeholder external service.
+</span><span class=c1></span> <span class=c1>// Run in debug mode to see the output.
+</span><span class=c1></span> <span class=n>Pipeline</span> <span class=n>p</span> <span class=o>=</span> <span class=n>Pipeline</span><span class=o>.</span><span class=na>create</span><span class=o>();</span>
+
+ <span class=c1>// Create a side input that updates each second.
+</span><span class=c1></span> <span class=n>PCollectionView</span><span class=o><</span><span class=n>Map</span><span class=o><</span><span class=n>String</span><span class=o>,</span> <span class=n>String</span><span class=o>>></span> <span class=n>map</span> <span class=o>=</span>
+ <span class=n>p</span><span class=o>.</span><span class=na>apply</span><span class=o>(</span><span class=n>GenerateSequence</span><span class=o>.</span><span class=na>from</span><span class=o>(</span><span class=n>0</span><span class=o>).</span><span class=na>withRate</span><span class=o>(</span><span class=n>1</span><span class=o>,</span> <span class=n>Duration</span><span class=o>.</span><span class=na>standardSeconds</span><span class=o>(</span><span class=n>5L</span><span cla [...]
+ <span class=o>.</span><span class=na>apply</span><span class=o>(</span><span class=n>Window</span><span class=o>.</span><span class=na>into</span><span class=o>(</span><span class=n>FixedWindows</span><span class=o>.</span><span class=na>of</span><span class=o>(</span><span class=n>Duration</span><span class=o>.</span><span class=na>standardSeconds</span><span class=o>(</span><span class=n>5</span><span class=o>))))</span>
+ <span class=o>.</span><span class=na>apply</span><span class=o>(</span><span class=n>Sum</span><span class=o>.</span><span class=na>longsGlobally</span><span class=o>().</span><span class=na>withoutDefaults</span><span class=o>())</span>
+ <span class=o>.</span><span class=na>apply</span><span class=o>(</span>
+ <span class=n>ParDo</span><span class=o>.</span><span class=na>of</span><span class=o>(</span>
+ <span class=k>new</span> <span class=n>DoFn</span><span class=o><</span><span class=n>Long</span><span class=o>,</span> <span class=n>Map</span><span class=o><</span><span class=n>String</span><span class=o>,</span> <span class=n>String</span><span class=o>>>()</span> <span class=o>{</span>
+
+ <span class=nd>@ProcessElement</span>
+ <span class=kd>public</span> <span class=kt>void</span> <span class=nf>process</span><span class=o>(</span>
+ <span class=nd>@Element</span> <span class=n>Long</span> <span class=n>input</span><span class=o>,</span>
+ <span class=nd>@Timestamp</span> <span class=n>Instant</span> <span class=n>timestamp</span><span class=o>,</span>
+ <span class=n>OutputReceiver</span><span class=o><</span><span class=n>Map</span><span class=o><</span><span class=n>String</span><span class=o>,</span> <span class=n>String</span><span class=o>>></span> <span class=n>o</span><span class=o>)</span> <span class=o>{</span>
+ <span class=c1>// Replace map with test data from the placeholder external service.
+</span><span class=c1></span> <span class=c1>// Add external reads here.
+</span><span class=c1></span> <span class=n>o</span><span class=o>.</span><span class=na>output</span><span class=o>(</span><span class=n>PlaceholderExternalService</span><span class=o>.</span><span class=na>readTestData</span><span class=o>(</span><span class=n>timestamp</span><span class=o>));</span>
+ <span class=o>}</span>
+ <span class=o>}))</span>
+ <span class=o>.</span><span class=na>apply</span><span class=o>(</span>
+ <span class=n>Window</span><span class=o>.<</span><span class=n>Map</span><span class=o><</span><span class=n>String</span><span class=o>,</span> <span class=n>String</span><span class=o>>></span><span class=n>into</span><span class=o>(</span><span class=k>new</span> <span class=n>GlobalWindows</span><span class=o>())</span>
+ <span class=o>.</span><span class=na>triggering</span><span class=o>(</span><span class=n>Repeatedly</span><span class=o>.</span><span class=na>forever</span><span class=o>(</span><span class=n>AfterProcessingTime</span><span class=o>.</span><span class=na>pastFirstElementInPane</span><span class=o>()))</span>
+ <span class=o>.</span><span class=na>discardingFiredPanes</span><span class=o>())</span>
+ <span class=o>.</span><span class=na>apply</span><span class=o>(</span><span class=n>View</span><span class=o>.</span><span class=na>asSingleton</span><span class=o>());</span>
+
+ <span class=c1>// Consume side input. GenerateSequence generates test data.
+</span><span class=c1></span> <span class=c1>// Use a real source (like PubSubIO or KafkaIO) in production.
+</span><span class=c1></span> <span class=n>p</span><span class=o>.</span><span class=na>apply</span><span class=o>(</span><span class=n>GenerateSequence</span><span class=o>.</span><span class=na>from</span><span class=o>(</span><span class=n>0</span><span class=o>).</span><span class=na>withRate</span><span class=o>(</span><span class=n>1</span><span class=o>,</span> <span class=n>Duration</span><span class=o>.</span><span class=na>standardSeconds</span><span class=o>(</span><span c [...]
+ <span class=o>.</span><span class=na>apply</span><span class=o>(</span><span class=n>Window</span><span class=o>.</span><span class=na>into</span><span class=o>(</span><span class=n>FixedWindows</span><span class=o>.</span><span class=na>of</span><span class=o>(</span><span class=n>Duration</span><span class=o>.</span><span class=na>standardSeconds</span><span class=o>(</span><span class=n>1</span><span class=o>))))</span>
+ <span class=o>.</span><span class=na>apply</span><span class=o>(</span><span class=n>Sum</span><span class=o>.</span><span class=na>longsGlobally</span><span class=o>().</span><span class=na>withoutDefaults</span><span class=o>())</span>
+ <span class=o>.</span><span class=na>apply</span><span class=o>(</span>
+ <span class=n>ParDo</span><span class=o>.</span><span class=na>of</span><span class=o>(</span>
+ <span class=k>new</span> <span class=n>DoFn</span><span class=o><</span><span class=n>Long</span><span class=o>,</span> <span class=n>KV</span><span class=o><</span><span class=n>Long</span><span class=o>,</span> <span class=n>Long</span><span class=o>>>()</span> <span class=o>{</span>
+
+ <span class=nd>@ProcessElement</span>
+ <span class=kd>public</span> <span class=kt>void</span> <span class=nf>process</span><span class=o>(</span><span class=n>ProcessContext</span> <span class=n>c</span><span class=o>,</span> <span class=nd>@Timestamp</span> <span class=n>Instant</span> <span class=n>timestamp</span><span class=o>)</span> <span class=o>{</span>
+ <span class=n>Map</span><span class=o><</span><span class=n>String</span><span class=o>,</span> <span class=n>String</span><span class=o>></span> <span class=n>keyMap</span> <span class=o>=</span> <span class=n>c</span><span class=o>.</span><span class=na>sideInput</span><span class=o>(</span><span class=n>map</span><span class=o>);</span>
+ <span class=n>c</span><span class=o>.</span><span class=na>outputWithTimestamp</span><span class=o>(</span><span class=n>KV</span><span class=o>.</span><span class=na>of</span><span class=o>(</span><span class=n>1L</span><span class=o>,</span> <span class=n>c</span><span class=o>.</span><span class=na>element</span><span class=o>()),</span> <span class=n>Instant</span><span class=o>.</span><span class=na>now</span><span class=o>());</span>
+
+ <span class=n>LOG</span><span class=o>.</span><span class=na>info</span><span class=o>(</span>
+ <span class=s>"Value is {} with timestamp {}, using key A from side input with time {}."</span><span class=o>,</span>
+ <span class=n>c</span><span class=o>.</span><span class=na>element</span><span class=o>(),</span>
+ <span class=n>timestamp</span><span class=o>.</span><span class=na>toString</span><span class=o>(</span><span class=n>DateTimeFormat</span><span class=o>.</span><span class=na>forPattern</span><span class=o>(</span><span class=s>"HH:mm:ss"</span><span class=o>)),</span>
+ <span class=n>keyMap</span><span class=o>.</span><span class=na>get</span><span class=o>(</span><span class=s>"Key_A"</span><span class=o>));</span>
+ <span class=o>}</span>
+ <span class=o>})</span>
+ <span class=o>.</span><span class=na>withSideInputs</span><span class=o>(</span><span class=n>map</span><span class=o>));</span>
+
+ <span class=n>p</span><span class=o>.</span><span class=na>run</span><span class=o>();</span>
<span class=o>}</span>
-<span class=o>}</span></code></pre></div></div></div><div class="language-py snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-py data-lang=py><span class=n>No</span> <span class=n>sample</span> <span class=n>present</span><span class=o>.</span></code></pre></div></div></div><h2 id=slowly-updating [...]
+
+ <span class=cm>/** Placeholder class that represents an external service generating test data. */</span>
+ <span class=kd>public</span> <span class=kd>static</span> <span class=kd>class</span> <span class=nc>PlaceholderExternalService</span> <span class=o>{</span>
+
+ <span class=kd>public</span> <span class=kd>static</span> <span class=n>Map</span><span class=o><</span><span class=n>String</span><span class=o>,</span> <span class=n>String</span><span class=o>></span> <span class=nf>readTestData</span><span class=o>(</span><span class=n>Instant</span> <span class=n>timestamp</span><span class=o>)</span> <span class=o>{</span>
+
+ <span class=n>Map</span><span class=o><</span><span class=n>String</span><span class=o>,</span> <span class=n>String</span><span class=o>></span> <span class=n>map</span> <span class=o>=</span> <span class=k>new</span> <span class=n>HashMap</span><span class=o><>();</span>
+
+ <span class=n>map</span><span class=o>.</span><span class=na>put</span><span class=o>(</span><span class=s>"Key_A"</span><span class=o>,</span> <span class=n>timestamp</span><span class=o>.</span><span class=na>toString</span><span class=o>(</span><span class=n>DateTimeFormat</span><span class=o>.</span><span class=na>forPattern</span><span class=o>(</span><span class=s>"HH:mm:ss"</span><span class=o>)));</span>
+
+ <span class=k>return</span> <span class=n>map</span><span class=o>;</span>
+ <span class=o>}</span>
+ <span class=o>}</span></code></pre></div></div></div><div class="language-py snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-py data-lang=py><span class=n>No</span> <span class=n>sample</span> <span class=n>present</span><span class=o>.</span></code></pre></div></div></div><h2 id=slowly-updati [...]
When you apply the side input to your main input, each main input
window is automatically matched to a single side input window.
This guarantees consistency on the duration of the single window,