You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2022/04/08 22:17:46 UTC
[beam] branch asf-site updated: Publishing website 2022/04/08 22:17:40 at commit 2d27f44
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 3a202e7e536 Publishing website 2022/04/08 22:17:40 at commit 2d27f44
3a202e7e536 is described below
commit 3a202e7e5364a61f3c1b9cf7b9f05acdc838de76
Author: jenkins <bu...@apache.org>
AuthorDate: Fri Apr 8 22:17:41 2022 +0000
Publishing website 2022/04/08 22:17:40 at commit 2d27f44
---
.../get-started/from-spark/index.html | 26 +++++++++++++---------
website/generated-content/get-started/index.xml | 26 +++++++++++++---------
website/generated-content/sitemap.xml | 2 +-
3 files changed, 33 insertions(+), 21 deletions(-)
diff --git a/website/generated-content/get-started/from-spark/index.html b/website/generated-content/get-started/from-spark/index.html
index 14a5fadc834..659e1951f3a 100644
--- a/website/generated-content/get-started/from-spark/index.html
+++ b/website/generated-content/get-started/from-spark/index.html
@@ -19,8 +19,8 @@ function addPlaceholder(){$('input:text').attr('placeholder',"What are you looki
function endSearch(){var search=document.querySelector(".searchBar");search.classList.add("disappear");var icons=document.querySelector("#iconsBar");icons.classList.remove("disappear");}
function blockScroll(){$("body").toggleClass("fixedPosition");}
function openMenu(){addPlaceholder();blockScroll();}</script><div class="clearfix container-main-content"><div class="section-nav closed" data-offset-top=90 data-offset-bottom=500><span class="section-nav-back glyphicon glyphicon-menu-left"></span><nav><ul class=section-nav-list data-section-nav><li><span class=section-nav-list-main-title>Get started</span></li><li><a href=/get-started/beam-overview/>Beam Overview</a></li><li><a href=/get-started/tour-of-beam/>Tour of Beam</a></li><li><s [...]
-learning <em>Apache Beam</em> is familiar.
-The Beam and Spark APIs are similar, so you already know the basic concepts.</p><p>Spark stores data <em>Spark DataFrames</em> for structured data,
+using Beam should be easy.
+The basic concepts are the same, and the APIs are similar as well.</p><p>Spark stores data <em>Spark DataFrames</em> for structured data,
and in <em>Resilient Distributed Datasets</em> (RDD) for unstructured data.
We are using RDDs for this guide.</p><p>A Spark RDD represents a collection of elements,
while in Beam it’s called a <em>Parallel Collection</em> (PCollection).
@@ -46,7 +46,8 @@ methods like <code>data.map(...)</code>, but they’re doing the same thing.
<span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>)</span>
<span class=p>)</span></code></pre></div></div></div><blockquote><p>ℹ️ Note that we called <code>print</code> inside a <code>Map</code> transform.
That’s because we can only access the elements of a PCollection
-from within a PTransform.</p></blockquote><p>Another thing to note is that Beam pipelines are constructed lazily.
+from within a PTransform.
+To inspect the data locally, you can use the <a href=https://cloud.google.com/dataflow/docs/guides/interactive-pipeline-development#creating_your_pipeline>InteractiveRunner</a></p></blockquote><p>Another thing to note is that Beam pipelines are constructed lazily.
This means that when you pipe <code>|</code> data you’re only declaring the
transformations and the order you want them to happen,
but the actual computation doesn’t happen.
@@ -74,10 +75,11 @@ we can’t guarantee that the results we’ve calculated are available a
<span class=n>sc</span> <span class=o>=</span> <span class=n>pyspark</span><span class=o>.</span><span class=n>SparkContext</span><span class=p>()</span>
<span class=n>values</span> <span class=o>=</span> <span class=n>sc</span><span class=o>.</span><span class=n>parallelize</span><span class=p>([</span><span class=mi>1</span><span class=p>,</span> <span class=mi>2</span><span class=p>,</span> <span class=mi>3</span><span class=p>,</span> <span class=mi>4</span><span class=p>])</span>
-<span class=n>total</span> <span class=o>=</span> <span class=n>values</span><span class=o>.</span><span class=n>reduce</span><span class=p>(</span><span class=k>lambda</span> <span class=n>x</span><span class=p>,</span> <span class=n>y</span><span class=p>:</span> <span class=n>x</span> <span class=o>+</span> <span class=n>y</span><span class=p>)</span>
+<span class=n>min_value</span> <span class=o>=</span> <span class=n>values</span><span class=o>.</span><span class=n>reduce</span><span class=p>(</span><span class=nb>min</span><span class=p>)</span>
+<span class=n>max_value</span> <span class=o>=</span> <span class=n>values</span><span class=o>.</span><span class=n>reduce</span><span class=p>(</span><span class=nb>max</span><span class=p>)</span>
-<span class=c1># We can simply use `total` since it's already a Python `int` value from `reduce`.</span>
-<span class=n>scaled_values</span> <span class=o>=</span> <span class=n>values</span><span class=o>.</span><span class=n>map</span><span class=p>(</span><span class=k>lambda</span> <span class=n>x</span><span class=p>:</span> <span class=n>x</span> <span class=o>/</span> <span class=n>total</span><span class=p>)</span>
+<span class=c1># We can simply use `min_value` and `max_value` since it's already a Python `int` value from `reduce`.</span>
+<span class=n>scaled_values</span> <span class=o>=</span> <span class=n>values</span><span class=o>.</span><span class=n>map</span><span class=p>(</span><span class=k>lambda</span> <span class=n>x</span><span class=p>:</span> <span class=p>(</span><span class=n>x</span> <span class=o>-</span> <span class=n>min_value</span><span class=p>)</span> <span class=o>/</span> <span class=p>(</span><span class=n>max_value</span> <span class=o>-</span> <span class=n>min_value</span><span class=p>))</span>
<span class=c1># But to access `scaled_values`, we need to call `collect`.</span>
<span class=k>print</span><span class=p>(</span><span class=n>scaled_values</span><span class=o>.</span><span class=n>collect</span><span class=p>())</span></code></pre></div></div></div><p>In Beam the results from all transforms result in a PCollection.
@@ -93,15 +95,19 @@ and access them as an <a href=https://docs.python.org/3/glossary.html#term-itera
<span class=k>with</span> <span class=n>beam</span><span class=o>.</span><span class=n>Pipeline</span><span class=p>()</span> <span class=k>as</span> <span class=n>pipeline</span><span class=p>:</span>
<span class=n>values</span> <span class=o>=</span> <span class=n>pipeline</span> <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Create</span><span class=p>([</span><span class=mi>1</span><span class=p>,</span> <span class=mi>2</span><span class=p>,</span> <span class=mi>3</span><span class=p>,</span> <span class=mi>4</span><span class=p>])</span>
- <span class=n>total</span> <span class=o>=</span> <span class=n>values</span> <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>CombineGlobally</span><span class=p>(</span><span class=nb>sum</span><span class=p>)</span>
+ <span class=n>min_value</span> <span class=o>=</span> <span class=n>values</span> <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>CombineGlobally</span><span class=p>(</span><span class=nb>min</span><span class=p>)</span>
+ <span class=n>max_value</span> <span class=o>=</span> <span class=n>values</span> <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>CombineGlobally</span><span class=p>(</span><span class=nb>max</span><span class=p>)</span>
<span class=c1># To access `total`, we need to pass it as a side input.</span>
<span class=n>scaled_values</span> <span class=o>=</span> <span class=n>values</span> <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span>
- <span class=k>lambda</span> <span class=n>x</span><span class=p>,</span> <span class=n>total</span><span class=p>:</span> <span class=n>x</span> <span class=o>/</span> <span class=n>total</span><span class=p>,</span>
- <span class=n>total</span><span class=o>=</span><span class=n>beam</span><span class=o>.</span><span class=n>pvalue</span><span class=o>.</span><span class=n>AsSingleton</span><span class=p>(</span><span class=n>total</span><span class=p>))</span>
+ <span class=k>lambda</span> <span class=n>x</span><span class=p>,</span> <span class=n>min_value</span><span class=p>,</span> <span class=n>max_value</span><span class=p>:</span> <span class=n>x</span> <span class=o>/</span> <span class=k>lambda</span> <span class=n>x</span><span class=p>:</span> <span class=p>(</span><span class=n>x</span> <span class=o>-</span> <span class=n>min_value</span><span class=p>)</span> <span class=o>/</span> <span class=p>(</span><span class=n>max_va [...]
+ <span class=n>min_value</span> <span class=o>=</span><span class=n>beam</span><span class=o>.</span><span class=n>pvalue</span><span class=o>.</span><span class=n>AsSingleton</span><span class=p>(</span><span class=n>min_value</span><span class=p>),</span>
+ <span class=n>max_value</span> <span class=o>=</span><span class=n>beam</span><span class=o>.</span><span class=n>pvalue</span><span class=o>.</span><span class=n>AsSingleton</span><span class=p>(</span><span class=n>max_value</span><span class=p>))</span>
<span class=n>scaled_values</span> <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>print</span><span class=p>)</span></code></pre></div></div></div><blockquote><p>ℹ️ In Beam we need to pass a side input explicitly, but we get the
-benefit that a reduction or aggregation does <em>not</em> have to fit into memory.</p></blockquote><h2 id=next-steps>Next Steps</h2><ul><li>Take a look at all the available transforms in the <a href=/documentation/transforms/python/overview>Python transform gallery</a>.</li><li>Learn how to read from and write to files in the <a href=/documentation/programming-guide/#pipeline-io><em>Pipeline I/O</em> section of the <em>Programming guide</em></a></li><li>Walk through additional WordCount [...]
+benefit that a reduction or aggregation does <em>not</em> have to fit into memory.
+Lazily computing side inputs also allows us to compute <code>values</code> only once,
+rather than for each distinct reduction (or requiring explicit caching of the RDD).</p></blockquote><h2 id=next-steps>Next Steps</h2><ul><li>Take a look at all the available transforms in the <a href=/documentation/transforms/python/overview>Python transform gallery</a>.</li><li>Learn how to read from and write to files in the <a href=/documentation/programming-guide/#pipeline-io><em>Pipeline I/O</em> section of the <em>Programming guide</em></a></li><li>Walk through additional WordCount [...]
<a href=http://www.apache.org>The Apache Software Foundation</a>
| <a href=/privacy_policy>Privacy Policy</a>
| <a href=/feed.xml>RSS Feed</a><br><br>Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation. All other products or name brands are trademarks of their respective holders, including The Apache Software Foundation.</div></div></div></div></footer></body></html>
\ No newline at end of file
diff --git a/website/generated-content/get-started/index.xml b/website/generated-content/get-started/index.xml
index 0092b70f547..85b9db78a56 100644
--- a/website/generated-content/get-started/index.xml
+++ b/website/generated-content/get-started/index.xml
@@ -4300,8 +4300,8 @@ limitations under the License.
localStorage.setItem("language", "language-py")
</script>
<p>If you already know <a href="http://spark.apache.org/"><em>Apache Spark</em></a>,
-learning <em>Apache Beam</em> is familiar.
-The Beam and Spark APIs are similar, so you already know the basic concepts.</p>
+using Beam should be easy.
+The basic concepts are the same, and the APIs are similar as well.</p>
<p>Spark stores data <em>Spark DataFrames</em> for structured data,
and in <em>Resilient Distributed Datasets</em> (RDD) for unstructured data.
We are using RDDs for this guide.</p>
@@ -4351,7 +4351,8 @@ methods like <code>data.map(...)</code>, but they&rsquo;re doing the s
<blockquote>
<p>ℹ️ Note that we called <code>print</code> inside a <code>Map</code> transform.
That&rsquo;s because we can only access the elements of a PCollection
-from within a PTransform.</p>
+from within a PTransform.
+To inspect the data locally, you can use the <a href="https://cloud.google.com/dataflow/docs/guides/interactive-pipeline-development#creating_your_pipeline">InteractiveRunner</a></p>
</blockquote>
<p>Another thing to note is that Beam pipelines are constructed lazily.
This means that when you pipe <code>|</code> data you&rsquo;re only declaring the
@@ -4549,9 +4550,10 @@ we can&rsquo;t guarantee that the results we&rsquo;ve calculated are ava
<div class="highlight"><pre class="chroma"><code class="language-py" data-lang="py"><span class="kn">import</span> <span class="nn">pyspark</span>
<span class="n">sc</span> <span class="o">=</span> <span class="n">pyspark</span><span class="o">.</span><span class="n">SparkContext</span><span class="p">()</span>
<span class="n">values</span> <span class="o">=</span> <span class="n">sc</span><span class="o">.</span><span class="n">parallelize</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">])</span>
-<span class="n">total</span> <span class="o">=</span> <span class="n">values</span><span class="o">.</span><span class="n">reduce</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">:</span> <span class="n">x</span> <span class="o">+</span> <span class="n">y</span><span class="p">)</span>
-<span class="c1"># We can simply use `total` since it&#39;s already a Python `int` value from `reduce`.</span>
-<span class="n">scaled_values</span> <span class="o">=</span> <span class="n">values</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span> <span class="o">/</span> <span class="n">total</span><span class="p">)</span>
+<span class="n">min_value</span> <span class="o">=</span> <span class="n">values</span><span class="o">.</span><span class="n">reduce</span><span class="p">(</span><span class="nb">min</span><span class="p">)</span>
+<span class="n">max_value</span> <span class="o">=</span> <span class="n">values</span><span class="o">.</span><span class="n">reduce</span><span class="p">(</span><span class="nb">max</span><span class="p">)</span>
+<span class="c1"># We can simply use `min_value` and `max_value` since it&#39;s already a Python `int` value from `reduce`.</span>
+<span class="n">scaled_values</span> <span class="o">=</span> <span class="n">values</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="p">(</span><span class="n">x</span> <span class="o">-</span> <span class="n">min_value</span><span class="p">)</span> <span class="o">/</span> &l [...]
<span class="c1"># But to access `scaled_values`, we need to call `collect`.</span>
<span class="k">print</span><span class="p">(</span><span class="n">scaled_values</span><span class="o">.</span><span class="n">collect</span><span class="p">())</span></code></pre></div>
</div>
@@ -4575,17 +4577,21 @@ and access them as an <a href="https://docs.python.org/3/glossary.html#term-i
<div class="highlight"><pre class="chroma"><code class="language-py" data-lang="py"><span class="kn">import</span> <span class="nn">apache_beam</span> <span class="kn">as</span> <span class="nn">beam</span>
<span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> <span class="k">as</span> <span class="n">pipeline</span><span class="p">:</span>
<span class="n">values</span> <span class="o">=</span> <span class="n">pipeline</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span c [...]
-<span class="n">total</span> <span class="o">=</span> <span class="n">values</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">CombineGlobally</span><span class="p">(</span><span class="nb">sum</span><span class="p">)</span>
+<span class="n">min_value</span> <span class="o">=</span> <span class="n">values</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">CombineGlobally</span><span class="p">(</span><span class="nb">min</span><span class="p">)</span>
+<span class="n">max_value</span> <span class="o">=</span> <span class="n">values</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">CombineGlobally</span><span class="p">(</span><span class="nb">max</span><span class="p">)</span>
<span class="c1"># To access `total`, we need to pass it as a side input.</span>
<span class="n">scaled_values</span> <span class="o">=</span> <span class="n">values</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span>
-<span class="k">lambda</span> <span class="n">x</span><span class="p">,</span> <span class="n">total</span><span class="p">:</span> <span class="n">x</span> <span class="o">/</span> <span class="n">total</span><span class="p">,</span>
-<span class="n">total</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">pvalue</span><span class="o">.</span><span class="n">AsSingleton</span><span class="p">(</span><span class="n">total</span><span class="p">))</span>
+<span class="k">lambda</span> <span class="n">x</span><span class="p">,</span> <span class="n">min_value</span><span class="p">,</span> <span class="n">max_value</span><span class="p">:</span> <span class="n">x</span> <span class="o">/</span> <span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="p">(</span><span class="n">x</span> <span class="o">-</span> <sp [...]
+<span class="n">min_value</span> <span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">pvalue</span><span class="o">.</span><span class="n">AsSingleton</span><span class="p">(</span><span class="n">min_value</span><span class="p">),</span>
+<span class="n">max_value</span> <span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">pvalue</span><span class="o">.</span><span class="n">AsSingleton</span><span class="p">(</span><span class="n">max_value</span><span class="p">))</span>
<span class="n">scaled_values</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">print</span><span class="p">)</span></code></pre></div>
</div>
</div>
<blockquote>
<p>ℹ️ In Beam we need to pass a side input explicitly, but we get the
-benefit that a reduction or aggregation does <em>not</em> have to fit into memory.</p>
+benefit that a reduction or aggregation does <em>not</em> have to fit into memory.
+Lazily computing side inputs also allows us to compute <code>values</code> only once,
+rather than for each distinct reduction (or requiring explicit caching of the RDD).</p>
</blockquote>
<h2 id="next-steps">Next Steps</h2>
<ul>
diff --git a/website/generated-content/sitemap.xml b/website/generated-content/sitemap.xml
index a01d82b2f92..567e77a0641 100644
--- a/website/generated-content/sitemap.xml
+++ b/website/generated-content/sitemap.xml
@@ -1 +1 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/beam-2.37.0/</loc><lastmod>2022-03-04T10:14:02-08:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2022-03-28T08:41:34-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2022-03-28T08:41:34-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2022-03-28T08:41:34-07:00</lastmod></url><url><loc>/blog/u [...]
\ No newline at end of file
+<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/beam-2.37.0/</loc><lastmod>2022-03-04T10:14:02-08:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2022-03-28T08:41:34-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2022-03-28T08:41:34-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2022-03-28T08:41:34-07:00</lastmod></url><url><loc>/blog/u [...]
\ No newline at end of file