You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2018/11/09 17:30:45 UTC
[beam] branch asf-site updated: Publishing website 2018/11/09
17:30:40 at commit f911be4
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new e969cc2 Publishing website 2018/11/09 17:30:40 at commit f911be4
e969cc2 is described below
commit e969cc2034711412bf1b0859810105788f30c98c
Author: jenkins <bu...@apache.org>
AuthorDate: Fri Nov 9 17:30:41 2018 +0000
Publishing website 2018/11/09 17:30:40 at commit f911be4
---
.../documentation/sdks/java/euphoria/index.html | 156 ++++++++-------------
1 file changed, 57 insertions(+), 99 deletions(-)
diff --git a/website/generated-content/documentation/sdks/java/euphoria/index.html b/website/generated-content/documentation/sdks/java/euphoria/index.html
index 737ec98..d5db0a5 100644
--- a/website/generated-content/documentation/sdks/java/euphoria/index.html
+++ b/website/generated-content/documentation/sdks/java/euphoria/index.html
@@ -243,13 +243,11 @@
<li><a href="#wordcount-example">WordCount Example</a></li>
<li><a href="#euphoria-guide">Euphoria Guide</a>
<ul>
- <li><a href="#datasets">Datasets</a></li>
<li><a href="#inputs-and-outputs">Inputs and Outputs</a></li>
<li><a href="#adding-operators">Adding Operators</a></li>
<li><a href="#coders-and-types">Coders and Types</a></li>
<li><a href="#metrics-and-accumulators">Metrics and Accumulators</a></li>
<li><a href="#windowing">Windowing</a></li>
- <li><a href="#integration-of-euphoria-into-existing-pipelines">Integration of Euphoria into existing pipelines</a></li>
</ul>
</li>
<li><a href="#how-to-get-euphoria">How to get Euphoria</a></li>
@@ -311,7 +309,7 @@ For each of the assigned windows the extracted value is accumulated using a user
-->
<h2 id="what-is-euphoria">What is Euphoria</h2>
-<p>Easy to use Java 8 API build on top of the Beam’s Java SDK. API provides a <a href="#operator-reference">high-level abstraction</a> of data transformations, with focus on the Java 8 language features (e.g. lambdas and streams). It is fully inter-operable with existing Beam SDK and convertible back and forth. It allows fast prototyping through use of (optional) <a href="https://github.com/EsotericSoftware/kryo">Kryo</a> based coders, lambdas and high level operators and can be <a href= [...]
+<p>Easy to use Java 8 API build on top of the Beam’s Java SDK. API provides a <a href="#operator-reference">high-level abstraction</a> of data transformations, with focus on the Java 8 language features (e.g. lambdas and streams). It is fully inter-operable with existing Beam SDK and convertible back and forth. It allows fast prototyping through use of (optional) <a href="https://github.com/EsotericSoftware/kryo">Kryo</a> based coders, lambdas and high level operators and can be seamless [...]
<p><a href="https://github.com/seznam/euphoria">Euphoria API</a> project has been started in 2014, with a clear goal of providing the main building block for <a href="https://www.seznam.cz/">Seznam.cz’s</a> data infrastructure.
In 2015, <a href="http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf">DataFlow whitepaper</a> inspired original authors to go one step further and also provide the unified API for both stream and batch processing.
@@ -334,12 +332,8 @@ the API as a high level DSL over Beam Java SDK and share our effort with the com
<span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Create</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">textLineByLine</span><span class="o">))</span>
<span class="o">.</span><span class="na">setTypeDescriptor</span><span class="o">(</span><span class="n">TypeDescriptor</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">String</span><span class="o">.</span><span class="na">class</span><span class="o">));</span>
-<span class="c1">// Transform PCollection to euphoria's Dataset.</span>
-<span class="n">Dataset</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">lines</span> <span class="o">=</span> <span class="n">Dataset</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">input</span><span class="o">);</span>
-
-<span class="c1">// FlatMap processes one input element at a time and allows user code to emit</span>
<span class="c1">// zero, one, or more output elements. From input lines we will get data set of words.</span>
-<span class="n">Dataset</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">words</span> <span class="o">=</span>
+<span class="n">PCollection</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">words</span> <span class="o">=</span>
<span class="n">FlatMap</span><span class="o">.</span><span class="na">named</span><span class="o">(</span><span class="s">"TOKENIZER"</span><span class="o">)</span>
<span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">lines</span><span class="o">)</span>
<span class="o">.</span><span class="na">using</span><span class="o">(</span>
@@ -352,26 +346,23 @@ the API as a high level DSL over Beam Java SDK and share our effort with the com
<span class="c1">// Now we can count input words - the operator ensures that all values for the same</span>
<span class="c1">// key (word in this case) end up being processed together. Then it counts number of appearances</span>
-<span class="c1">// of the same key in 'words' dataset and emits it to output.</span>
-<span class="n">Dataset</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">>></span> <span class="n">counted</span> <span class="o">=</span>
+<span class="c1">// of the same key in 'words' PCollection and emits it to output.</span>
+<span class="n">PCollection</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">>></span> <span class="n">counted</span> <span class="o">=</span>
<span class="n">CountByKey</span><span class="o">.</span><span class="na">named</span><span class="o">(</span><span class="s">"COUNT"</span><span class="o">)</span>
<span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">words</span><span class="o">)</span>
<span class="o">.</span><span class="na">keyBy</span><span class="o">(</span><span class="n">w</span> <span class="o">-></span> <span class="n">w</span><span class="o">)</span>
<span class="o">.</span><span class="na">output</span><span class="o">();</span>
<span class="c1">// Format output.</span>
-<span class="n">Dataset</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">output</span> <span class="o">=</span>
+<span class="n">PCollection</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">output</span> <span class="o">=</span>
<span class="n">MapElements</span><span class="o">.</span><span class="na">named</span><span class="o">(</span><span class="s">"FORMAT"</span><span class="o">)</span>
<span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">counted</span><span class="o">)</span>
<span class="o">.</span><span class="na">using</span><span class="o">(</span><span class="n">p</span> <span class="o">-></span> <span class="n">p</span><span class="o">.</span><span class="na">getKey</span><span class="o">()</span> <span class="o">+</span> <span class="s">": "</span> <span class="o">+</span> <span class="n">p</span><span class="o">.</span><span class="na">getValue</span><span class="o">())</span>
<span class="o">.</span><span class="na">output</span><span class="o">();</span>
-<span class="c1">// Transform Dataset back to PCollection. It can be done anytime.</span>
-<span class="n">PCollection</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">outputCollection</span> <span class="o">=</span> <span class="n">output</span><span class="o">.</span><span class="na">getPCollection</span><span class="o">();</span>
-
<span class="c1">// Now we can again use Beam transformation. In this case we save words and their count</span>
<span class="c1">// into the text file.</span>
-<span class="n">outputCollection</span>
+<span class="n">output</span>
<span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">TextIO</span><span class="o">.</span><span class="na">write</span><span class="o">()</span>
<span class="o">.</span><span class="na">to</span><span class="o">(</span><span class="s">"counted_words"</span><span class="o">));</span>
@@ -383,38 +374,23 @@ the API as a high level DSL over Beam Java SDK and share our effort with the com
<p>Euphoria API is composed from a set of operators, which allows you to construct <code class="highlighter-rouge">Pipeline</code> according to your application needs.</p>
-<h3 id="datasets">Datasets</h3>
-<p>Euphoria uses the concept of ‘Datasets’ to describe data pipeline between <code class="highlighter-rouge">Operators</code>. This concept is similar to Beam’s <code class="highlighter-rouge">PCollection</code> and can be converted back and forth through:</p>
-<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">PCollection</span><span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="n">someCollection</span> <span class="o">=</span> <span class="o">...</span>
-
-<span class="c1">// PCollection -> Dataset</span>
-<span class="n">Dataset</span><span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="n">dataset</span> <span class="o">=</span> <span class="n">Dataset</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">someCollection</span><span class="o">);</span>
-
-<span class="c1">//And now back: Dataset -> PCollection</span>
-<span class="n">PCollection</span><span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="n">collection</span> <span class="o">=</span> <span class="n">dataset</span><span class="o">.</span><span class="na">getPCollection</span><span class="o">();</span>
-</code></pre>
-</div>
-
<h3 id="inputs-and-outputs">Inputs and Outputs</h3>
-<p>Input data can be supplied through Beams IO into <code class="highlighter-rouge">PCollection</code>, the same way as in Beam, and wrapped by <code class="highlighter-rouge">Dataset.of(PCollection<T> pCollection)</code> into <code class="highlighter-rouge">Dataset</code> later on.</p>
+<p>Input data can be supplied through Beams IO into <code class="highlighter-rouge">PCollection</code>, the same way as in Beam.</p>
<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">PCollection</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">input</span> <span class="o">=</span>
<span class="n">pipeline</span>
<span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Create</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">"mouse"</span><span class="o">,</span> <span class="s">"rat"</span><span class="o">,</span> <span class="s">"elephant"</span><span class="o">,</span> <span class="s">"cat"</span><span class="o">,</span> <span class="s">"X"</span><span class="o">,</span> <span class="s">"duck"</span><span cla [...]
<span class="o">.</span><span class="na">setTypeDescriptor</span><span class="o">(</span><span class="n">TypeDescriptor</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">String</span><span class="o">.</span><span class="na">class</span><span class="o">));</span>
-
-<span class="n">Dataset</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">dataset</span> <span class="o">=</span> <span class="n">Dataset</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">input</span><span class="o">);</span>
</code></pre>
</div>
-<p>Outputs can be treated the same way as inputs, last <code class="highlighter-rouge">Dataset</code> is converted to <code class="highlighter-rouge">PCollection</code> and dumped into appropriate IO.</p>
<h3 id="adding-operators">Adding Operators</h3>
-<p>Real power of Euphoria API is in its <a href="#operator-reference">operators suite</a>. Once we get our hands on <code class="highlighter-rouge">Dataset</code> we are able to create and connect operators. Each Operator consumes one or more input and produces one output
-<code class="highlighter-rouge">Dataset</code>. Lets take a look at simple <code class="highlighter-rouge">MapElements</code> example.</p>
+<p>Real power of Euphoria API is in its <a href="#operator-reference">operators suite</a>. Each Operator consumes one or more input and produces one output
+<code class="highlighter-rouge">PCollection</code>. Lets take a look at simple <code class="highlighter-rouge">MapElements</code> example.</p>
-<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">Dataset</span><span class="o"><</span><span class="n">Integer</span><span class="o">></span> <span class="n">input</span> <span class="o">=</span> <span class="o">...</span>
+<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">PCollection</span><span class="o"><</span><span class="n">Integer</span><span class="o">></span> <span class="n">input</span> <span class="o">=</span> <span class="o">...</span>
-<span class="n">Dataset</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">mappedElements</span> <span class="o">=</span>
+<span class="n">PCollection</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">mappedElements</span> <span class="o">=</span>
<span class="n">MapElements</span>
<span class="o">.</span><span class="na">named</span><span class="o">(</span><span class="s">"Int2Str"</span><span class="o">)</span>
<span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">input</span><span class="o">)</span>
@@ -422,7 +398,7 @@ the API as a high level DSL over Beam Java SDK and share our effort with the com
<span class="o">.</span><span class="na">output</span><span class="o">();</span>
</code></pre>
</div>
-<p>The operator consumes <code class="highlighter-rouge">input</code>, it applies given lambda expression (<code class="highlighter-rouge">String::valueOf</code>) on each element of <code class="highlighter-rouge">input</code> and returns mapped <code class="highlighter-rouge">Dataset</code>. Developer is guided through series of steps when creating operator so the declaration of an operator is straightforward. To start building operator just wrote its name and ‘.’ (dot). Your IDE will g [...]
+<p>The operator consumes <code class="highlighter-rouge">input</code>, it applies given lambda expression (<code class="highlighter-rouge">String::valueOf</code>) on each element of <code class="highlighter-rouge">input</code> and returns mapped <code class="highlighter-rouge">PCollection</code>. Developer is guided through series of steps when creating operator so the declaration of an operator is straightforward. To start building operator just wrote its name and ‘.’ (dot). Your IDE wi [...]
<p>First step to build any operator is to give it a name through <code class="highlighter-rouge">named()</code> method. The name is propagated through system and can latter be used when debugging.</p>
@@ -470,7 +446,7 @@ the API as a high level DSL over Beam Java SDK and share our effort with the com
</code></pre>
</div>
<p>Beam resolves coders using types of elements. Type information is not available at runtime when element type is described by lambda implementation. It is due to type erasure and dynamic nature of lambda expressions. So there is an optional way of supplying <code class="highlighter-rouge">TypeDescriptor</code> every time new type is introduced during Operator construction.</p>
-<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">Dataset</span><span class="o"><</span><span class="n">Integer</span><span class="o">></span> <span class="n">input</span> <span class="o">=</span> <span class="o">...</span>
+<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">PCollection</span><span class="o"><</span><span class="n">Integer</span><span class="o">></span> <span class="n">input</span> <span class="o">=</span> <span class="o">...</span>
<span class="n">MapElements</span>
<span class="o">.</span><span class="na">named</span><span class="o">(</span><span class="s">"Int2Str"</span><span class="o">)</span>
@@ -479,14 +455,14 @@ the API as a high level DSL over Beam Java SDK and share our effort with the com
<span class="o">.</span><span class="na">output</span><span class="o">();</span>
</code></pre>
</div>
-<p>Euphoria operator’s will use <code class="highlighter-rouge">TypeDescriptor<Object></code>, when <code class="highlighter-rouge">TypeDescriptors</code> is not supplied by user. So <code class="highlighter-rouge">KryoCoderProvider</code> may return <code class="highlighter-rouge">KryoCOder<Object></code> for every element with unknown type, if allowed by <code class="highlighter-rouge">KryoOptions</code>. Supplying <code class="highlighter-rouge">TypeDescriptors</code> beco [...]
+<p>Euphoria operator’s will use <code class="highlighter-rouge">TypeDescriptor<Object></code>, when <code class="highlighter-rouge">TypeDescriptors</code> is not supplied by user. So <code class="highlighter-rouge">KryoCoderProvider</code> may return <code class="highlighter-rouge">KryoCoder<Object></code> for every element with unknown type, if allowed by <code class="highlighter-rouge">KryoOptions</code>. Supplying <code class="highlighter-rouge">TypeDescriptors</code> beco [...]
<h3 id="metrics-and-accumulators">Metrics and Accumulators</h3>
<p>Statistics about job’s internals are very helpful during development of distributed jobs. Euphoria calls them accumulators. They are accessible through environment <code class="highlighter-rouge">Context</code>, which can be obtained from <code class="highlighter-rouge">Collector</code>, whenever working with it. It is usually present when zero-to-many output elements are expected from operator. For example in case of <code class="highlighter-rouge">FlatMap</code>.</p>
<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">Pipeline</span> <span class="n">pipeline</span> <span class="o">=</span> <span class="o">...</span>
-<span class="n">Dataset</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">dataset</span> <span class="o">=</span> <span class="o">..</span>
+<span class="n">PCollection</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">dataset</span> <span class="o">=</span> <span class="o">..</span>
-<span class="n">Dataset</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">mapped</span> <span class="o">=</span>
+<span class="n">PCollection</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">mapped</span> <span class="o">=</span>
<span class="n">FlatMap</span>
<span class="o">.</span><span class="na">named</span><span class="o">(</span><span class="s">"FlatMap1"</span><span class="o">)</span>
<span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">dataset</span><span class="o">)</span>
@@ -500,9 +476,9 @@ the API as a high level DSL over Beam Java SDK and share our effort with the com
</div>
<p><code class="highlighter-rouge">MapElements</code> also allows for <code class="highlighter-rouge">Context</code> to be accessed by supplying implementations of <code class="highlighter-rouge">UnaryFunctionEnv</code> (add second context argument) instead of <code class="highlighter-rouge">UnaryFunctor</code>.</p>
<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">Pipeline</span> <span class="n">pipeline</span> <span class="o">=</span> <span class="o">...</span>
-<span class="n">Dataset</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">dataset</span> <span class="o">=</span> <span class="o">...</span>
+<span class="n">PCollection</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">dataset</span> <span class="o">=</span> <span class="o">...</span>
-<span class="n">Dataset</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">mapped</span> <span class="o">=</span>
+<span class="n">PCollection</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">mapped</span> <span class="o">=</span>
<span class="n">MapElements</span>
<span class="o">.</span><span class="na">named</span><span class="o">(</span><span class="s">"MapThem"</span><span class="o">)</span>
<span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">dataset</span><span class="o">)</span>
@@ -510,7 +486,6 @@ the API as a high level DSL over Beam Java SDK and share our effort with the com
<span class="o">(</span><span class="n">input</span><span class="o">,</span> <span class="n">context</span><span class="o">)</span> <span class="o">-></span> <span class="o">{</span>
<span class="c1">// use simple counter</span>
<span class="n">context</span><span class="o">.</span><span class="na">getCounter</span><span class="o">(</span><span class="s">"my-counter"</span><span class="o">).</span><span class="na">increment</span><span class="o">();</span>
-
<span class="k">return</span> <span class="n">input</span><span class="o">.</span><span class="na">toLowerCase</span><span class="o">();</span>
<span class="o">})</span>
<span class="o">.</span><span class="na">output</span><span class="o">();</span>
@@ -520,39 +495,22 @@ the API as a high level DSL over Beam Java SDK and share our effort with the com
<h3 id="windowing">Windowing</h3>
<p>Euphoria follows the same <a href="/documentation/programming-guide/#windowing">windowing principles</a> as Beam Java SDK. Every shuffle operator (operator which needs to shuffle data over the network) allows you to set it. The same parameters as in Beam are required. <code class="highlighter-rouge">WindowFn</code>, <code class="highlighter-rouge">Trigger</code>, <code class="highlighter-rouge">WindowingStrategy</code> and other. Users are guided to either set all mandatory and severa [...]
-<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">Dtaset</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Long</span><span class="o">>></span> <span class="n">countedElements</span> <span class="o">=</span>
-<span class="n">CountByKey</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">input</span><span class="o">)</span>
- <span class="o">.</span><span class="na">keyBy</span><span class="o">(</span><span class="n">e</span> <span class="o">-></span> <span class="n">e</span><span class="o">)</span>
- <span class="o">.</span><span class="na">windowBy</span><span class="o">(</span><span class="n">FixedWindows</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardSeconds</span><span class="o">(</span><span class="mi">1</span><span class="o">)))</span>
- <span class="o">.</span><span class="na">triggeredBy</span><span class="o">(</span><span class="n">DefaultTrigger</span><span class="o">.</span><span class="na">of</span><span class="o">())</span>
- <span class="o">.</span><span class="na">discardingFiredPanes</span><span class="o">()</span>
- <span class="o">.</span><span class="na">withAllowedLateness</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardSeconds</span><span class="o">(</span><span class="mi">5</span><span class="o">))</span>
- <span class="o">.</span><span class="na">withOnTimeBehavior</span><span class="o">(</span><span class="n">OnTimeBehavior</span><span class="o">.</span><span class="na">FIRE_IF_NON_EMPTY</span><span class="o">)</span>
- <span class="o">.</span><span class="na">withTimestampCombiner</span><span class="o">(</span><span class="n">TimestampCombiner</span><span class="o">.</span><span class="na">EARLIEST</span><span class="o">)</span>
- <span class="o">.</span><span class="na">output</span><span class="o">();</span>
-</code></pre>
-</div>
-
-<h3 id="integration-of-euphoria-into-existing-pipelines">Integration of Euphoria into existing pipelines</h3>
-<p><code class="highlighter-rouge">Euphoria</code> allows to define composite <code class="highlighter-rouge">PTransform</code> so Euphoria can be seamlessly integrated to already existing Beam <code class="highlighter-rouge">Pipelines</code>. User only need to provide implementation of function which takes input <code class="highlighter-rouge">Dataset</code> and outputs another <code class="highlighter-rouge">Datatset</code>. The input dataset is nothing else than mirror of a input <co [...]
-<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="c1">//suppose inputs PCollection contains: [ "a", "b", "c", "A", "a", "C", "x"]</span>
-<span class="n">PCollection</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">>></span> <span class="n">lettersWithCounts</span> <span class="o">=</span>
- <span class="n">inputs</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">"count-uppercase-letters-in-Euphoria"</span><span class="o">,</span>
- <span class="n">Euphoria</span><span class="o">.</span><span class="na">of</span><span class="o">(</span>
- <span class="o">(</span><span class="n">Dataset</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">input</span><span class="o">)</span> <span class="o">-></span> <span class="o">{</span>
- <span class="n">Dataset</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">upperCase</span> <span class="o">=</span>
- <span class="n">MapElements</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">input</span><span class="o">)</span>
- <span class="o">.</span><span class="na">using</span><span class="o">((</span><span class="n">UnaryFunction</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">>)</span> <span class="nl">String:</span><span class="o">:</span><span class="n">toUpperCase</span><span class="o">)</span>
- <span class="o">.</span><span class="na">output</span><span class="o">();</span>
-
- <span class="k">return</span> <span class="n">CountByKey</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">upperCase</span><span class="o">).</span><span class="na">keyBy</span><span class="o">(</span><span class="n">e</span> <span class="o">-></span> <span class="n">e</span><span class="o">).</span><span class="na">output</span><span class="o">();</span>
- <span class="o">}));</span>
-<span class="c1">//now the 'lettersWithCounts' will conntain [ KV("A", 3L), KV("B", 1L), KV("C", 2L), KV("X", 1L) ]</span>
+<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">PCollection</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Long</span><span class="o">>></span> <span class="n">countedElements</span> <span class="o">=</span>
+ <span class="n">CountByKey</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">input</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">keyBy</span><span class="o">(</span><span class="n">e</span> <span class="o">-></span> <span class="n">e</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">windowBy</span><span class="o">(</span><span class="n">FixedWindows</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardSeconds</span><span class="o">(</span><span class="mi">1</span><span class="o">)))</span>
+ <span class="o">.</span><span class="na">triggeredBy</span><span class="o">(</span><span class="n">DefaultTrigger</span><span class="o">.</span><span class="na">of</span><span class="o">())</span>
+ <span class="o">.</span><span class="na">discardingFiredPanes</span><span class="o">()</span>
+ <span class="o">.</span><span class="na">withAllowedLateness</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardSeconds</span><span class="o">(</span><span class="mi">5</span><span class="o">))</span>
+ <span class="o">.</span><span class="na">withOnTimeBehavior</span><span class="o">(</span><span class="n">OnTimeBehavior</span><span class="o">.</span><span class="na">FIRE_IF_NON_EMPTY</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">withTimestampCombiner</span><span class="o">(</span><span class="n">TimestampCombiner</span><span class="o">.</span><span class="na">EARLIEST</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">output</span><span class="o">();</span>
</code></pre>
</div>
<h2 id="how-to-get-euphoria">How to get Euphoria</h2>
<p>Euphoria is located in <code class="highlighter-rouge">dsl-euphoria</code> branch, <code class="highlighter-rouge">beam-sdks-java-extensions-euphoria</code> module of The Apache Beam project. To build <code class="highlighter-rouge">euphoria</code> subproject call:</p>
+
<div class="highlighter-rouge"><pre class="highlight"><code>./gradlew beam-sdks-java-extensions-euphoria:build
</code></pre>
</div>
@@ -563,7 +521,7 @@ the API as a high level DSL over Beam Java SDK and share our effort with the com
<h3 id="countbykey"><code class="highlighter-rouge">CountByKey</code></h3>
<p>Counting elements with the same key. Requires input dataset to be mapped by given key extractor (<code class="highlighter-rouge">UnaryFunction</code>) to keys which are then counted. Output is emitted as <code class="highlighter-rouge">KV<K, Long></code> (<code class="highlighter-rouge">K</code> is key type) where each <code class="highlighter-rouge">KV</code> contains key and number of element in input dataset for the key.</p>
<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="c1">// suppose input: [1, 2, 4, 1, 1, 3]</span>
-<span class="n">Dataset</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Long</span><span class="o">>></span> <span class="n">output</span> <span class="o">=</span>
+<span class="n">PCollection</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Long</span><span class="o">>></span> <span class="n">output</span> <span class="o">=</span>
<span class="n">CountByKey</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">input</span><span class="o">)</span>
<span class="o">.</span><span class="na">keyBy</span><span class="o">(</span><span class="n">e</span> <span class="o">-></span> <span class="n">e</span><span class="o">)</span>
<span class="o">.</span><span class="na">output</span><span class="o">();</span>
@@ -594,7 +552,7 @@ the API as a high level DSL over Beam Java SDK and share our effort with the com
<p>Represents inner join of two (left and right) datasets on given key producing a new dataset. Key is extracted from both datasets by separate extractors so elements in left and right can have different types denoted as <code class="highlighter-rouge">LeftT</code> and <code class="highlighter-rouge">RightT</code>. The join itself is performed by user-supplied <code class="highlighter-rouge">BinaryFunctor</code> which consumes elements from both dataset sharing the same key. And outputs [...]
<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="c1">// suppose that left contains: [1, 2, 3, 0, 4, 3, 1]</span>
<span class="c1">// suppose that right contains: ["mouse", "rat", "elephant", "cat", "X", "duck"]</span>
-<span class="n">Dataset</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">String</span><span class="o">>></span> <span class="n">joined</span> <span class="o">=</span>
+<span class="n">PCollection</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">String</span><span class="o">>></span> <span class="n">joined</span> <span class="o">=</span>
<span class="n">Join</span><span class="o">.</span><span class="na">named</span><span class="o">(</span><span class="s">"join-length-to-words"</span><span class="o">)</span>
<span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">left</span><span class="o">,</span> <span class="n">right</span><span class="o">)</span>
<span class="o">.</span><span class="na">by</span><span class="o">(</span><span class="n">le</span> <span class="o">-></span> <span class="n">le</span><span class="o">,</span> <span class="nl">String:</span><span class="o">:</span><span class="n">length</span><span class="o">)</span> <span class="c1">// key extractors</span>
@@ -609,26 +567,26 @@ the API as a high level DSL over Beam Java SDK and share our effort with the com
<p>Represents left join of two (left and right) datasets on given key producing single new dataset. Key is extracted from both datasets by separate extractors so elements in left and right can have different types denoted as <code class="highlighter-rouge">LeftT</code> and <code class="highlighter-rouge">RightT</code>. The join itself is performed by user-supplied <code class="highlighter-rouge">BinaryFunctor</code> which consumes one element from both dataset, where right is present opt [...]
<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="c1">// suppose that left contains: [1, 2, 3, 0, 4, 3, 1]</span>
<span class="c1">// suppose that right contains: ["mouse", "rat", "elephant", "cat", "X", "duck"]</span>
- <span class="n">Dataset</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">String</span><span class="o">>></span> <span class="n">joined</span> <span class="o">=</span>
- <span class="n">LeftJoin</span><span class="o">.</span><span class="na">named</span><span class="o">(</span><span class="s">"left-join-length-to-words"</span><span class="o">)</span>
- <span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">left</span><span class="o">,</span> <span class="n">right</span><span class="o">)</span>
- <span class="o">.</span><span class="na">by</span><span class="o">(</span><span class="n">le</span> <span class="o">-></span> <span class="n">le</span><span class="o">,</span> <span class="nl">String:</span><span class="o">:</span><span class="n">length</span><span class="o">)</span> <span class="c1">// key extractors</span>
- <span class="o">.</span><span class="na">using</span><span class="o">(</span>
- <span class="o">(</span><span class="n">Integer</span> <span class="n">l</span><span class="o">,</span> <span class="n">Optional</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">r</span><span class="o">,</span> <span class="n">Collector</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">c</span><span class="o">)</span> <span class="o">-></span>
- <span class="n">c</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span><span class="n">l</span> <span class="o">+</span> <span class="s">"+"</span> <span class="o">+</span> <span class="n">r</span><span class="o">.</span><span class="na">orElse</span><span class="o">(</span><span class="kc">null</span><span class="o">)))</span>
- <span class="o">.</span><span class="na">output</span><span class="o">();</span>
+<span class="n">PCollection</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">String</span><span class="o">>></span> <span class="n">joined</span> <span class="o">=</span>
+ <span class="n">LeftJoin</span><span class="o">.</span><span class="na">named</span><span class="o">(</span><span class="s">"left-join-length-to-words"</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">left</span><span class="o">,</span> <span class="n">right</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">by</span><span class="o">(</span><span class="n">le</span> <span class="o">-></span> <span class="n">le</span><span class="o">,</span> <span class="nl">String:</span><span class="o">:</span><span class="n">length</span><span class="o">)</span> <span class="c1">// key extractors</span>
+ <span class="o">.</span><span class="na">using</span><span class="o">(</span>
+ <span class="o">(</span><span class="n">Integer</span> <span class="n">l</span><span class="o">,</span> <span class="n">Optional</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">r</span><span class="o">,</span> <span class="n">Collector</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">c</span><span class="o">)</span> <span class="o">-></span>
+ <span class="n">c</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span><span class="n">l</span> <span class="o">+</span> <span class="s">"+"</span> <span class="o">+</span> <span class="n">r</span><span class="o">.</span><span class="na">orElse</span><span class="o">(</span><span class="kc">null</span><span class="o">)))</span>
+ <span class="o">.</span><span class="na">output</span><span class="o">();</span>
<span class="c1">// joined will contain: [KV(1, "1+X"), KV(2, "2+null"), KV(3, "3+cat"),</span>
<span class="c1">// KV(3, "3+rat"), KV(0, "0+null"), KV(4, "4+duck"), KV(3, "3+cat"),</span>
<span class="c1">// KV(3, "3+rat"), KV(1, "1+X")]</span>
</code></pre>
</div>
-<p>Euphoria support performance optimization called ‘BroadcastHashJoin’ for the <code class="highlighter-rouge">LeftJoin</code>. User can indicate through previous operator’s output hint <code class="highlighter-rouge">.output(SizeHint.FITS_IN_MEMORY)</code> that output <code class="highlighter-rouge">Dataset</code> of that operator fits in executors memory. And when the <code class="highlighter-rouge">Dataset</code> is used as right input, Euphoria will automatically translated <code cl [...]
+<p>Euphoria support performance optimization called ‘BroadcastHashJoin’ for the <code class="highlighter-rouge">LeftJoin</code>. User can indicate through previous operator’s output hint <code class="highlighter-rouge">.output(SizeHint.FITS_IN_MEMORY)</code> that output <code class="highlighter-rouge">PCollection</code> of that operator fits in executors memory. And when the <code class="highlighter-rouge">PCollection</code> is used as right input, Euphoria will automatically translated [...]
<h3 id="rightjoin"><code class="highlighter-rouge">RightJoin</code></h3>
<p>Represents right join of two (left and right) datasets on given key producing single new dataset. Key is extracted from both datasets by separate extractors so elements in left and right can have different types denoted as <code class="highlighter-rouge">LeftT</code> and <code class="highlighter-rouge">RightT</code>. The join itself is performed by user-supplied <code class="highlighter-rouge">BinaryFunctor</code> which consumes one element from both dataset, where left is present opt [...]
<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="c1">// suppose that left contains: [1, 2, 3, 0, 4, 3, 1]</span>
<span class="c1">// suppose that right contains: ["mouse", "rat", "elephant", "cat", "X", "duck"]</span>
-<span class="n">Dataset</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">String</span><span class="o">>></span> <span class="n">joined</span> <span class="o">=</span>
+<span class="n">PCollection</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">String</span><span class="o">>></span> <span class="n">joined</span> <span class="o">=</span>
<span class="n">RightJoin</span><span class="o">.</span><span class="na">named</span><span class="o">(</span><span class="s">"right-join-length-to-words"</span><span class="o">)</span>
<span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">left</span><span class="o">,</span> <span class="n">right</span><span class="o">)</span>
<span class="o">.</span><span class="na">by</span><span class="o">(</span><span class="n">le</span> <span class="o">-></span> <span class="n">le</span><span class="o">,</span> <span class="nl">String:</span><span class="o">:</span><span class="n">length</span><span class="o">)</span> <span class="c1">// key extractors</span>
@@ -641,13 +599,13 @@ the API as a high level DSL over Beam Java SDK and share our effort with the com
<span class="c1">// KV(8, "null+elephant"), KV(5, "null+mouse")]</span>
</code></pre>
</div>
-<p>Euphoria support performance optimization called ‘Broadcast Hash Join’ for the <code class="highlighter-rouge">RightJoin</code>. User can indicate through previous operator’s output hint <code class="highlighter-rouge">.output(SizeHint.FITS_IN_MEMORY)</code> that output <code class="highlighter-rouge">Dataset</code> of that operator fits in executors memory. And when the <code class="highlighter-rouge">Dataset</code> is used as left input, Euphoria will automatically translated <code [...]
+<p>Euphoria support performance optimization called ‘Broadcast Hash Join’ for the <code class="highlighter-rouge">RightJoin</code>. User can indicate through previous operator’s output hint <code class="highlighter-rouge">.output(SizeHint.FITS_IN_MEMORY)</code> that output <code class="highlighter-rouge">PCollection</code> of that operator fits in executors memory. And when the <code class="highlighter-rouge">PCollection</code> is used as left input, Euphoria will automatically translate [...]
<h3 id="fulljoin"><code class="highlighter-rouge">FullJoin</code></h3>
<p>Represents full outer join of two (left and right) datasets on given key producing single new dataset. Key is extracted from both datasets by separate extractors so elements in left and right can have different types denoted as <code class="highlighter-rouge">LeftT</code> and <code class="highlighter-rouge">RightT</code>. The join itself is performed by user-supplied <code class="highlighter-rouge">BinaryFunctor</code> which consumes one element from both dataset, where both are prese [...]
<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="c1">// suppose that left contains: [1, 2, 3, 0, 4, 3, 1]</span>
<span class="c1">// suppose that right contains: ["mouse", "rat", "elephant", "cat", "X", "duck"]</span>
-<span class="n">Dataset</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">String</span><span class="o">>></span> <span class="n">joined</span> <span class="o">=</span>
+<span class="n">PCollection</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">String</span><span class="o">>></span> <span class="n">joined</span> <span class="o">=</span>
<span class="n">FullJoin</span><span class="o">.</span><span class="na">named</span><span class="o">(</span><span class="s">"join-length-to-words"</span><span class="o">)</span>
<span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">left</span><span class="o">,</span> <span class="n">right</span><span class="o">)</span>
<span class="o">.</span><span class="na">by</span><span class="o">(</span><span class="n">le</span> <span class="o">-></span> <span class="n">le</span><span class="o">,</span> <span class="nl">String:</span><span class="o">:</span><span class="n">length</span><span class="o">)</span> <span class="c1">// key extractors</span>
@@ -664,7 +622,7 @@ the API as a high level DSL over Beam Java SDK and share our effort with the com
<h3 id="mapelements"><code class="highlighter-rouge">MapElements</code></h3>
<p>Transforms one input element of input type <code class="highlighter-rouge">InputT</code> to one output element of another (potentially the same) <code class="highlighter-rouge">OutputT</code> type. Transformation is done through user specified <code class="highlighter-rouge">UnaryFunction</code>.</p>
<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="c1">// suppose inputs contains: [ 0, 1, 2, 3, 4, 5]</span>
-<span class="n">Dataset</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">strings</span> <span class="o">=</span>
+<span class="n">PCollection</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">strings</span> <span class="o">=</span>
<span class="n">MapElements</span><span class="o">.</span><span class="na">named</span><span class="o">(</span><span class="s">"int2str"</span><span class="o">)</span>
<span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">input</span><span class="o">)</span>
<span class="o">.</span><span class="na">using</span><span class="o">(</span><span class="n">i</span> <span class="o">-></span> <span class="s">"#"</span> <span class="o">+</span> <span class="n">i</span><span class="o">)</span>
@@ -676,7 +634,7 @@ the API as a high level DSL over Beam Java SDK and share our effort with the com
<h3 id="flatmap"><code class="highlighter-rouge">FlatMap</code></h3>
<p>Transforms one input element of input type <code class="highlighter-rouge">InputT</code> to zero or more output elements of another (potentially the same) <code class="highlighter-rouge">OutputT</code> type. Transformation is done through user specified <code class="highlighter-rouge">UnaryFunctor</code>, where <code class="highlighter-rouge">Collector<OutputT></code> is utilized to emit output elements. Notice similarity with <code class="highlighter-rouge">MapElements</code> w [...]
<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="c1">// suppose words contain: ["Brown", "fox", ".", ""]</span>
-<span class="n">Dataset</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">letters</span> <span class="o">=</span>
+<span class="n">PCollection</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">letters</span> <span class="o">=</span>
<span class="n">FlatMap</span><span class="o">.</span><span class="na">named</span><span class="o">(</span><span class="s">"str2char"</span><span class="o">)</span>
<span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">words</span><span class="o">)</span>
<span class="o">.</span><span class="na">using</span><span class="o">(</span>
@@ -692,7 +650,7 @@ the API as a high level DSL over Beam Java SDK and share our effort with the com
</div>
<p><code class="highlighter-rouge">FlatMap</code> may be used to determine time-stamp of elements. It is done by supplying implementation of <code class="highlighter-rouge">ExtractEventTime</code> time extractor when building it. There is specialized <code class="highlighter-rouge">AssignEventTime</code> operator to assign time-stamp to elements. Consider using it, you code may be more readable.</p>
<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="c1">// suppose events contain events of SomeEventObject, its 'getEventTimeInMillis()' methods returns time-stamp</span>
-<span class="n">Dataset</span><span class="o"><</span><span class="n">SomeEventObject</span><span class="o">></span> <span class="n">timeStampedEvents</span> <span class="o">=</span>
+<span class="n">PCollection</span><span class="o"><</span><span class="n">SomeEventObject</span><span class="o">></span> <span class="n">timeStampedEvents</span> <span class="o">=</span>
<span class="n">FlatMap</span><span class="o">.</span><span class="na">named</span><span class="o">(</span><span class="s">"extract-event-time"</span><span class="o">)</span>
<span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">events</span><span class="o">)</span>
<span class="o">.</span><span class="na">using</span><span class="o">(</span> <span class="o">(</span><span class="n">SomeEventObject</span> <span class="n">e</span><span class="o">,</span> <span class="n">Collector</span><span class="o"><</span><span class="n">SomeEventObject</span><span class="o">></span> <span class="n">c</span><span class="o">)</span> <span class="o">-></span> <span class="n">c</span><span class="o">.</span><span class="na">collect</span><span class="o"> [...]
@@ -705,7 +663,7 @@ the API as a high level DSL over Beam Java SDK and share our effort with the com
<h3 id="filter"><code class="highlighter-rouge">Filter</code></h3>
<p><code class="highlighter-rouge">Filter</code> throws away all the elements which do not pass given condition. The condition is supplied by the user as implementation of <code class="highlighter-rouge">UnaryPredicate</code>. Input and output elements are of the same type.</p>
<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="c1">// suppose nums contains: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]</span>
-<span class="n">Dataset</span><span class="o"><</span><span class="n">Integer</span><span class="o">></span> <span class="n">divisibleBythree</span> <span class="o">=</span>
+<span class="n">PCollection</span><span class="o"><</span><span class="n">Integer</span><span class="o">></span> <span class="n">divisibleBythree</span> <span class="o">=</span>
<span class="n">Filter</span><span class="o">.</span><span class="na">named</span><span class="o">(</span><span class="s">"divisibleByThree"</span><span class="o">).</span><span class="na">of</span><span class="o">(</span><span class="n">nums</span><span class="o">).</span><span class="na">by</span><span class="o">(</span><span class="n">e</span> <span class="o">-></span> <span class="n">e</span> <span class="o">%</span> <span class="mi">3</span> <span class="o">==</span> <span clas [...]
<span class="c1">//divisibleBythree will contain: [ 0, 3, 6, 9]</span>
</code></pre>
@@ -718,7 +676,7 @@ the API as a high level DSL over Beam Java SDK and share our effort with the com
<p>Following example shows basic usage of <code class="highlighter-rouge">ReduceByKey</code> operator including value extraction.</p>
<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="c1">//suppose animals contains : [ "mouse", "rat", "elephant", "cat", "X", "duck"]</span>
-<span class="n">Dataset</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Long</span><span class="o">>></span> <span class="n">countOfAnimalNamesByLength</span> <span class="o">=</span>
+<span class="n">PCollection</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Long</span><span class="o">>></span> <span class="n">countOfAnimalNamesByLength</span> <span class="o">=</span>
<span class="n">ReduceByKey</span><span class="o">.</span><span class="na">named</span><span class="o">(</span><span class="s">"to-letters-couts"</span><span class="o">)</span>
<span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">animals</span><span class="o">)</span>
<span class="o">.</span><span class="na">keyBy</span><span class="o">(</span><span class="nl">String:</span><span class="o">:</span><span class="n">length</span><span class="o">)</span> <span class="c1">// length of animal name will be used as groupping key</span>
@@ -732,7 +690,7 @@ the API as a high level DSL over Beam Java SDK and share our effort with the com
<p>Now suppose that we want to track our <code class="highlighter-rouge">ReduceByKey</code> internals using counter.</p>
<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="c1">//suppose animals contains : [ "mouse", "rat", "elephant", "cat", "X", "duck"]</span>
-<span class="n">Dataset</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Long</span><span class="o">>></span> <span class="n">countOfAnimalNamesByLenght</span> <span class="o">=</span>
+<span class="n">PCollection</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Long</span><span class="o">>></span> <span class="n">countOfAnimalNamesByLenght</span> <span class="o">=</span>
<span class="n">ReduceByKey</span><span class="o">.</span><span class="na">named</span><span class="o">(</span><span class="s">"to-letters-couts"</span><span class="o">)</span>
<span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">animals</span><span class="o">)</span>
<span class="o">.</span><span class="na">keyBy</span><span class="o">(</span><span class="nl">String:</span><span class="o">:</span><span class="n">length</span><span class="o">)</span> <span class="c1">// length of animal name will be used as grouping key</span>
@@ -750,7 +708,7 @@ the API as a high level DSL over Beam Java SDK and share our effort with the com
<p>Again the same example with optimized combinable output.</p>
<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="c1">//suppose animals contains : [ "mouse", "rat", "elephant", "cat", "X", "duck"]</span>
-<span class="n">Dataset</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Long</span><span class="o">>></span> <span class="n">countOfAnimalNamesByLenght</span> <span class="o">=</span>
+<span class="n">PCollection</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Long</span><span class="o">>></span> <span class="n">countOfAnimalNamesByLenght</span> <span class="o">=</span>
<span class="n">ReduceByKey</span><span class="o">.</span><span class="na">named</span><span class="o">(</span><span class="s">"to-letters-couts"</span><span class="o">)</span>
<span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">animals</span><span class="o">)</span>
<span class="o">.</span><span class="na">keyBy</span><span class="o">(</span><span class="nl">String:</span><span class="o">:</span><span class="n">length</span><span class="o">)</span> <span class="c1">// length of animal name will e used as grouping key</span>
@@ -765,7 +723,7 @@ the API as a high level DSL over Beam Java SDK and share our effort with the com
<p>Euphoria aims to make code easy to write and read. Therefore some support to write combinable reduce functions in form of <code class="highlighter-rouge">Fold</code> or folding function is already there. It allows user to supply only the reduction logic (<code class="highlighter-rouge">BinaryFunction</code>) and creates <code class="highlighter-rouge">CombinableReduceFunction</code> out of it. Supplied <code class="highlighter-rouge">BinaryFunction</code> still have to be associative.</p>
<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="c1">//suppose animals contains : [ "mouse", "rat", "elephant", "cat", "X", "duck"]</span>
-<span class="n">Dataset</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Long</span><span class="o">>></span> <span class="n">countOfAnimalNamesByLenght</span> <span class="o">=</span>
+<span class="n">PCollection</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Long</span><span class="o">>></span> <span class="n">countOfAnimalNamesByLenght</span> <span class="o">=</span>
<span class="n">ReduceByKey</span><span class="o">.</span><span class="na">named</span><span class="o">(</span><span class="s">"to-letters-couts"</span><span class="o">)</span>
<span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">animals</span><span class="o">)</span>
<span class="o">.</span><span class="na">keyBy</span><span class="o">(</span><span class="nl">String:</span><span class="o">:</span><span class="n">length</span><span class="o">)</span> <span class="c1">// length of animal name will be used as grouping key</span>
@@ -781,9 +739,9 @@ the API as a high level DSL over Beam Java SDK and share our effort with the com
<p>Reduces all elements in a <a href="#windowing">window</a>. The operator corresponds to <code class="highlighter-rouge">ReduceByKey</code> with the same key for all elements, so the actual key is defined only by window.</p>
<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="c1">//suppose input contains [ 1, 2, 3, 4, 5, 6, 7, 8 ]</span>
<span class="c1">//lets assign time-stamp to each input element</span>
-<span class="n">Dataset</span><span class="o"><</span><span class="n">Integer</span><span class="o">></span> <span class="n">withEventTime</span> <span class="o">=</span> <span class="n">AssignEventTime</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">input</span><span class="o">).</span><span class="na">using</span><span class="o">(</span><span class="n">i</span> <span class="o">-></span> <span class="mi">1000L</span> <span class=" [...]
+<span class="n">PCollection</span><span class="o"><</span><span class="n">Integer</span><span class="o">></span> <span class="n">withEventTime</span> <span class="o">=</span> <span class="n">AssignEventTime</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">input</span><span class="o">).</span><span class="na">using</span><span class="o">(</span><span class="n">i</span> <span class="o">-></span> <span class="mi">1000L</span> <span cla [...]
-<span class="n">Dataset</span><span class="o"><</span><span class="n">Integer</span><span class="o">></span> <span class="n">output</span> <span class="o">=</span>
+<span class="n">PCollection</span><span class="o"><</span><span class="n">Integer</span><span class="o">></span> <span class="n">output</span> <span class="o">=</span>
<span class="n">ReduceWindow</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">withEventTime</span><span class="o">)</span>
<span class="o">.</span><span class="na">combineBy</span><span class="o">(</span><span class="n">Fold</span><span class="o">.</span><span class="na">of</span><span class="o">((</span><span class="n">i1</span><span class="o">,</span> <span class="n">i2</span><span class="o">)</span> <span class="o">-></span> <span class="n">i1</span> <span class="o">+</span> <span class="n">i2</span><span class="o">))</span>
<span class="o">.</span><span class="na">windowBy</span><span class="o">(</span><span class="n">FixedWindows</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">millis</span><span class="o">(</span><span class="mi">5000</span><span class="o">)))</span>
@@ -797,7 +755,7 @@ the API as a high level DSL over Beam Java SDK and share our effort with the com
<h3 id="sumbykey"><code class="highlighter-rouge">SumByKey</code></h3>
<p>Summing elements with same key. Requires input dataset to be mapped by given key extractor (<code class="highlighter-rouge">UnaryFunction</code>) to keys. By value extractor, also <code class="highlighter-rouge">UnaryFunction</code> which outputs to <code class="highlighter-rouge">Long</code>, to values. Those values are then grouped by key and summed. Output is emitted as <code class="highlighter-rouge">KV<K, Long></code> (<code class="highlighter-rouge">K</code> is key type) w [...]
<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="c1">//suppose input contains: [ 1, 2, 3, 4, 5, 6, 7, 8, 9 ]</span>
-<span class="n">Dataset</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Long</span><span class="o">>></span> <span class="n">output</span> <span class="o">=</span>
+<span class="n">PCollection</span><span class="o"><</span><span class="n">KV</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Long</span><span class="o">>></span> <span class="n">output</span> <span class="o">=</span>
<span class="n">SumByKey</span><span class="o">.</span><span class="na">named</span><span class="o">(</span><span class="s">"sum-odd-and-even"</span><span class="o">)</span>
<span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">input</span><span class="o">)</span>
<span class="o">.</span><span class="na">keyBy</span><span class="o">(</span><span class="n">e</span> <span class="o">-></span> <span class="n">e</span> <span class="o">%</span> <span class="mi">2</span><span class="o">)</span>
@@ -811,7 +769,7 @@ the API as a high level DSL over Beam Java SDK and share our effort with the com
<p>Merge of at least two datasets of the same type without any guarantee about elements ordering.</p>
<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="c1">//suppose cats contains: [ "cheetah", "cat", "lynx", "jaguar" ]</span>
<span class="c1">//suppose rodents conains: [ "squirrel", "mouse", "rat", "lemming", "beaver" ]</span>
-<span class="n">Dataset</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">animals</span> <span class="o">=</span>
+<span class="n">PCollection</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">animals</span> <span class="o">=</span>
<span class="n">Union</span><span class="o">.</span><span class="na">named</span><span class="o">(</span><span class="s">"to-animals"</span><span class="o">)</span>
<span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">cats</span><span class="o">,</span> <span class="n">rodents</span><span class="o">)</span>
<span class="o">.</span><span class="na">output</span><span class="o">();</span>
@@ -822,7 +780,7 @@ the API as a high level DSL over Beam Java SDK and share our effort with the com
<h3 id="topperkey"><code class="highlighter-rouge">TopPerKey</code></h3>
<p>Emits one top-rated element per key. Key of type <code class="highlighter-rouge">K</code> is extracted by given <code class="highlighter-rouge">UnaryFunction</code>. Another <code class="highlighter-rouge">UnaryFunction</code> extractor allows for conversion input elements to values of type <code class="highlighter-rouge">V</code>. Selection of top element is based on <em>score</em>, which is obtained from each element by user supplied <code class="highlighter-rouge">UnaryFunction</co [...]
<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="c1">// suppose 'animals contain: [ "mouse", "elk", "rat", "mule", "elephant", "dinosaur", "cat", "duck", "caterpillar" ]</span>
-<span class="n">Dataset</span><span class="o"><</span><span class="n">Triple</span><span class="o"><</span><span class="n">Character</span><span class="o">,</span> <span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">>></span> <span class="n">longestNamesByLetter</span> <span class="o">=</span>
+<span class="n">PCollection</span><span class="o"><</span><span class="n">Triple</span><span class="o"><</span><span class="n">Character</span><span class="o">,</span> <span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">>></span> <span class="n">longestNamesByLetter</span> <span class="o">=</span>
<span class="n">TopPerKey</span><span class="o">.</span><span class="na">named</span><span class="o">(</span><span class="s">"longest-animal-names"</span><span class="o">)</span>
<span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">animals</span><span class="o">)</span>
<span class="o">.</span><span class="na">keyBy</span><span class="o">(</span><span class="n">name</span> <span class="o">-></span> <span class="n">name</span><span class="o">.</span><span class="na">charAt</span><span class="o">(</span><span class="mi">0</span><span class="o">))</span> <span class="c1">// first character is the key</span>
@@ -837,7 +795,7 @@ the API as a high level DSL over Beam Java SDK and share our effort with the com
<h3 id="assigneventtime"><code class="highlighter-rouge">AssignEventTime</code></h3>
<p>Euphoria needs to know how to extract time-stamp from elements when <a href="#windowing">windowing</a> is applied. <code class="highlighter-rouge">AssignEventTime</code> tells Euphoria how to do that through given implementation of <code class="highlighter-rouge">ExtractEventTime</code> function.</p>
<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="c1">// suppose events contain events of SomeEventObject, its 'getEventTimeInMillis()' methods returns time-stamp</span>
-<span class="n">Dataset</span><span class="o"><</span><span class="n">SomeEventObject</span><span class="o">></span> <span class="n">timeStampedEvents</span> <span class="o">=</span>
+<span class="n">PCollection</span><span class="o"><</span><span class="n">SomeEventObject</span><span class="o">></span> <span class="n">timeStampedEvents</span> <span class="o">=</span>
<span class="n">AssignEventTime</span><span class="o">.</span><span class="na">named</span><span class="o">(</span><span class="s">"extract-event-tyme"</span><span class="o">)</span>
<span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">events</span><span class="o">)</span>
<span class="o">.</span><span class="na">using</span><span class="o">(</span><span class="nl">SomeEventObject:</span><span class="o">:</span><span class="n">getEventTimeInMillis</span><span class="o">)</span>