You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/09/29 15:54:35 UTC
[GitHub] [beam] davidcavazos opened a new pull request #12963: [BEAM-10983] Add getting started from Spark page
davidcavazos opened a new pull request #12963:
URL: https://github.com/apache/beam/pull/12963
Add a new page to get started for users that already know PySpark.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace | Typescript
--- | --- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #12963: [BEAM-10983] Add getting started from Spark page
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #12963:
URL: https://github.com/apache/beam/pull/12963#issuecomment-738513065
@davidcavazos (if you are back to office) - what is the next step on this PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rosetn commented on pull request #12963: [BEAM-10983] Add getting started from Spark page
Posted by GitBox <gi...@apache.org>.
rosetn commented on pull request #12963:
URL: https://github.com/apache/beam/pull/12963#issuecomment-702992353
This looks great so far! I'll do a review after seeing how you incorporate Kyle's feedback.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidcavazos commented on a change in pull request #12963: [BEAM-10983] Add getting started from Spark page
Posted by GitBox <gi...@apache.org>.
davidcavazos commented on a change in pull request #12963:
URL: https://github.com/apache/beam/pull/12963#discussion_r542520456
##########
File path: website/www/site/content/en/get-started/from-spark.md
##########
@@ -0,0 +1,268 @@
+---
+title: "Getting started from Apache Spark"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Getting started from Apache Spark
+
+{{< localstorage language language-py >}}
+
+If you already know [_Apache Spark_](http://spark.apache.org/),
+learning _Apache Beam_ is familiar.
+The Beam and Spark APIs are similar, so you already know the basic concepts.
+
+Spark stores data _Spark DataFrames_ for structured data,
+and in _Resilient Distributed Datasets_ (RDD) for unstructured data.
+We are using RDDs for this guide.
+
+A Spark RDD represents a collection of elements,
+while in Beam it's called a _Parallel Collection_ (PCollection).
+A PCollection in Beam does _not_ have any ordering guarantees.
+
+Likewise, a transform in Beam is called a _Parallel Transform_ (PTransform).
+
+Here are some examples of common operations and their equivalent between PySpark and Beam.
+
+## Overview
+
+Here's a simple example of a PySpark pipeline that takes the numbers from one to four,
+multiplies them by two, adds all the values together, and prints the result.
+
+{{< highlight py >}}
+import pyspark
+
+sc = pyspark.SparkContext()
+result = (
+ sc.parallelize([1, 2, 3, 4])
+ .map(lambda x: x * 2)
+ .reduce(lambda x, y: x + y)
+)
+print(result)
+{{< /highlight >}}
+
+In Beam you _pipe_ your data through the pipeline using the
+_pipe operator_ `|` like `data | beam.Map(...)` instead of chaining
+methods like `data.map(...)`, but they're doing the same thing.
+
+Here's what an equivalent pipeline looks like in Beam.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | beam.Create([1, 2, 3, 4])
+ | beam.Map(lambda x: x * 2)
+ | beam.CombineGlobally(sum)
+ | beam.Map(print)
+ )
+{{< /highlight >}}
+
+> ℹ️ Note that we called `print` inside a `Map` transform.
+> That's because we can only access the elements of a PCollection
+> from within a PTransform.
+
+Another thing to note is that Beam pipelines are constructed _lazily_.
+This means that when you pipe `|` data you're only _decalring_ the
Review comment:
Thanks, fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on pull request #12963: [BEAM-10983] Add getting started from Spark page
Posted by GitBox <gi...@apache.org>.
iemejia commented on pull request #12963:
URL: https://github.com/apache/beam/pull/12963#issuecomment-728329415
Is there something important still missing to get this one merged? Maybe we can merge and ask/do minor fixes after?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rosetn commented on pull request #12963: [BEAM-10983] Add getting started from Spark page
Posted by GitBox <gi...@apache.org>.
rosetn commented on pull request #12963:
URL: https://github.com/apache/beam/pull/12963#issuecomment-730648630
I think we're fairly close to merging. David has been OOO since my review, so that might be the cause of delay.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #12963: [BEAM-10983] Add getting started from Spark page
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #12963:
URL: https://github.com/apache/beam/pull/12963#issuecomment-700939028
@ibzib or @iemejia would be able to review this better.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidcavazos commented on a change in pull request #12963: [BEAM-10983] Add getting started from Spark page
Posted by GitBox <gi...@apache.org>.
davidcavazos commented on a change in pull request #12963:
URL: https://github.com/apache/beam/pull/12963#discussion_r508090271
##########
File path: website/www/site/content/en/get-started/from-spark.md
##########
@@ -0,0 +1,245 @@
+---
+title: "Getting started from Apache Spark"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Getting started from Apache Spark
+
+{{< localstorage language language-py >}}
+
+If you already know [_Apache Spark_](http://spark.apache.org/),
+learning _Apache Beam_ is easy.
+Beam and Spark are mostly equivalent, so you already know the basic concepts.
+
+A collection of elements in Spark is called a _Resilient Distributed Dataset_ (RDD),
+while in Beam it's called a _Parallel Collection_ (PCollection).
+A PCollection in Beam does _not_ have any ordering guarantees.
+
+Likewise, a transform in Beam is called a _Parallel Transform_ (PTransform).
+
+Here are some examples of common operations and their equivalent between PySpark and Beam.
+
+## Overview
+
+Here's a simple example of a PySpark pipeline that takes the numbers from one to four,
+multiplies them by two, adds all the values together, and prints the result.
+
+{{< highlight py >}}
+import pyspark
+
+with pyspark.SparkContext() as sc:
+ result = (
+ sc.parallelize([1, 2, 3, 4])
+ .map(lambda x: x * 2)
+ .reduce(lambda x, y: x + y)
+ )
+ print(result)
+{{< /highlight >}}
+
+In Beam you _pipe_ your data through the pipeline using the
+_pipe operator_ `|` like `data | beam.Map(...)` instead of chaining
+methods like `data.map(...)`, but they're doing the same thing.
+
+Here's how an equivalent pipeline looks like in Beam.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | beam.Create([1, 2, 3, 4])
+ | beam.Map(lambda x: x * 2)
+ | beam.CombineGlobally(sum)
+ | beam.Map(print)
+ )
+{{< /highlight >}}
+
+> ℹ️ Note that we called `print` inside a `Map` transform.
+> That's because we can only access the elements of a PCollection
+> from within a PTransform.
+
+A label can optionally be added to a transform using the
+_right shift operator_ `>>` like `data | 'My description' >> beam.Map(...)`.
+This serves both as comments and makes your pipeline easier to debug.
+
+This is how the pipeline looks after adding labels.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | 'Create numbers' >> beam.Create([1, 2, 3, 4])
+ | 'Multiply by two' >> beam.Map(lambda x: x * 2)
+ | 'Sum everything' >> beam.CombineGlobally(sum)
+ | beam.Map(print)
+ )
+{{< /highlight >}}
+
+## Setup
+
+Here's a comparison on how to get started both in PySpark and Beam.
+
+{{< table >}}
+<table>
+<tr>
+ <th></th>
+ <th>PySpark</th>
+ <th>Beam</th>
+</tr>
+<tr>
+ <td><b>Install</b></td>
+ <td><code>$ pip install pyspark</code></td>
+ <td><code>$ pip install apache-beam</code></td>
+</tr>
+<tr>
+ <td><b>Imports</b></td>
+ <td><code>import pyspark</code></td>
+ <td><code>import apache_beam as beam</code></td>
+</tr>
+<tr>
+ <td><b>Creating a<br>local pipeline</b></td>
+ <td>
+ <code>with pyspark.SparkContext() as sc:</code><br>
+ <code> # Your pipeline code here.</code>
+ </td>
+ <td>
+ <code>with beam.Pipeline() as pipeline:</code><br>
+ <code> # Your pipeline code here.</code>
+ </td>
+</tr>
+<tr>
+ <td><b>Creating values</b></td>
+ <td><code>values = sc.parallelize([1, 2, 3, 4])</code></td>
+ <td><code>values = pipeline | beam.Create([1, 2, 3, 4])</code></td>
+</tr>
+<tr>
+ <td><b>Creating<br>key-value pairs</b></td>
+ <td>
+ <code>pairs = sc.parallelize([</code><br>
+ <code> ('key1', 'value1'),</code><br>
+ <code> ('key2', 'value2'),</code><br>
+ <code> ('key3', 'value3'),</code><br>
+ <code>])</code>
+ </td>
+ <td>
+ <code>pairs = pipeline | beam.Create([</code><br>
+ <code> ('key1', 'value1'),</code><br>
+ <code> ('key2', 'value2'),</code><br>
+ <code> ('key3', 'value3'),</code><br>
+ <code>])</code>
+ </td>
+</tr>
+<tr>
+ <td><b>Running a<br>local pipeline</b></td>
+ <td><code>$ spark-submit spark_pipeline.py</code></td>
+ <td><code>$ python beam_pipeline.py</code></td>
+</tr>
+</table>
+{{< /table >}}
+
+## Transforms
+
+Here are the equivalents of some common transforms in both PySpark and Beam.
+
+{{< table >}}
+| | PySpark | Beam |
+|-------------------|---------------------------------------|---------------------------------------------------------|
+| **Map** | `values.map(lambda x: x * 2)` | `values | beam.Map(lambda x: x * 2)` |
+| **Filter** | `values.filter(lambda x: x % 2 == 0)` | `values | beam.Filter(lambda x: x % 2 == 0)` |
+| **FlatMap** | `values.flatMap(lambda x: range(x))` | `values | beam.FlatMap(lambda x: range(x))` |
+| **Group by key** | `pairs.groupByKey()` | `pairs | beam.GroupByKey()` |
+| **Reduce** | `values.reduce(lambda x, y: x+y)` | `values | beam.CombineGlobally(sum)` |
+| **Reduce by key** | `pairs.reduceByKey(lambda x, y: x+y)` | `pairs | beam.CombinePerKey(sum)` |
+| **Distinct** | `values.distinct()` | `values | beam.Distinct()` |
+| **Count** | `values.count()` | `values | beam.combiners.Count.Globally()` |
+| **Count by key** | `pairs.countByKey()` | `pairs | beam.combiners.Count.PerKey()` |
+| **Take smallest** | `values.takeOrdered(3)` | `values | beam.combiners.Top.Smallest(3)` |
+| **Take largest** | `values.takeOrdered(3, lambda x: -x)` | `values | beam.combiners.Top.Largest(3)` |
+| **Random sample** | `values.takeSample(False, 3)` | `values | beam.combiners.Sample.FixedSizeGlobally(3)` |
+| **Union** | `values.union(otherValues)` | `(values, otherValues) | beam.Flatten()` |
+| **Co-group** | `pairs.cogroup(otherPairs)` | `{'Xs': pairs, 'Ys': otherPairs} | beam.CoGroupByKey()` |
+{{< /table >}}
+
+> ℹ️ To learn more about the transforms available in Beam, check the
+> [Python transform gallery](/documentation/transforms/python/overview).
+
+## Using calculated values
+
+Since we are working in potentially distributed environments,
+we can't guarantee that the results we've calculated are available at any given machine.
+
+In PySpark, we can get a result from a collection of elements (RDD) by using
+`data.collect()`, or other aggregations such as `reduce()`, `count()` and more.
+
+Here's an example to scale numbers into a range between zero and one.
+
+{{< highlight py >}}
+import pyspark
+
+with pyspark.SparkContext() as sc:
+ values = sc.parallelize([1, 2, 3, 4])
+ total = values.reduce(lambda x, y: x + y)
+
+ # We can simply use `total` since it's already a Python value from `reduce`.
Review comment:
Done
##########
File path: website/www/site/content/en/get-started/from-spark.md
##########
@@ -0,0 +1,245 @@
+---
+title: "Getting started from Apache Spark"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Getting started from Apache Spark
+
+{{< localstorage language language-py >}}
+
+If you already know [_Apache Spark_](http://spark.apache.org/),
+learning _Apache Beam_ is easy.
+Beam and Spark are mostly equivalent, so you already know the basic concepts.
+
+A collection of elements in Spark is called a _Resilient Distributed Dataset_ (RDD),
+while in Beam it's called a _Parallel Collection_ (PCollection).
+A PCollection in Beam does _not_ have any ordering guarantees.
+
+Likewise, a transform in Beam is called a _Parallel Transform_ (PTransform).
+
+Here are some examples of common operations and their equivalent between PySpark and Beam.
+
+## Overview
+
+Here's a simple example of a PySpark pipeline that takes the numbers from one to four,
+multiplies them by two, adds all the values together, and prints the result.
+
+{{< highlight py >}}
+import pyspark
+
+with pyspark.SparkContext() as sc:
+ result = (
+ sc.parallelize([1, 2, 3, 4])
+ .map(lambda x: x * 2)
+ .reduce(lambda x, y: x + y)
+ )
+ print(result)
+{{< /highlight >}}
+
+In Beam you _pipe_ your data through the pipeline using the
+_pipe operator_ `|` like `data | beam.Map(...)` instead of chaining
+methods like `data.map(...)`, but they're doing the same thing.
+
+Here's how an equivalent pipeline looks like in Beam.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | beam.Create([1, 2, 3, 4])
+ | beam.Map(lambda x: x * 2)
+ | beam.CombineGlobally(sum)
+ | beam.Map(print)
+ )
+{{< /highlight >}}
+
+> ℹ️ Note that we called `print` inside a `Map` transform.
+> That's because we can only access the elements of a PCollection
+> from within a PTransform.
+
+A label can optionally be added to a transform using the
+_right shift operator_ `>>` like `data | 'My description' >> beam.Map(...)`.
+This serves both as comments and makes your pipeline easier to debug.
+
+This is how the pipeline looks after adding labels.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | 'Create numbers' >> beam.Create([1, 2, 3, 4])
+ | 'Multiply by two' >> beam.Map(lambda x: x * 2)
+ | 'Sum everything' >> beam.CombineGlobally(sum)
+ | beam.Map(print)
+ )
+{{< /highlight >}}
+
+## Setup
+
+Here's a comparison on how to get started both in PySpark and Beam.
+
+{{< table >}}
+<table>
+<tr>
+ <th></th>
+ <th>PySpark</th>
+ <th>Beam</th>
+</tr>
+<tr>
+ <td><b>Install</b></td>
+ <td><code>$ pip install pyspark</code></td>
+ <td><code>$ pip install apache-beam</code></td>
+</tr>
+<tr>
+ <td><b>Imports</b></td>
+ <td><code>import pyspark</code></td>
+ <td><code>import apache_beam as beam</code></td>
+</tr>
+<tr>
+ <td><b>Creating a<br>local pipeline</b></td>
+ <td>
+ <code>with pyspark.SparkContext() as sc:</code><br>
+ <code> # Your pipeline code here.</code>
+ </td>
+ <td>
+ <code>with beam.Pipeline() as pipeline:</code><br>
+ <code> # Your pipeline code here.</code>
+ </td>
+</tr>
+<tr>
+ <td><b>Creating values</b></td>
+ <td><code>values = sc.parallelize([1, 2, 3, 4])</code></td>
+ <td><code>values = pipeline | beam.Create([1, 2, 3, 4])</code></td>
+</tr>
+<tr>
+ <td><b>Creating<br>key-value pairs</b></td>
+ <td>
+ <code>pairs = sc.parallelize([</code><br>
+ <code> ('key1', 'value1'),</code><br>
+ <code> ('key2', 'value2'),</code><br>
+ <code> ('key3', 'value3'),</code><br>
+ <code>])</code>
+ </td>
+ <td>
+ <code>pairs = pipeline | beam.Create([</code><br>
+ <code> ('key1', 'value1'),</code><br>
+ <code> ('key2', 'value2'),</code><br>
+ <code> ('key3', 'value3'),</code><br>
+ <code>])</code>
+ </td>
+</tr>
+<tr>
+ <td><b>Running a<br>local pipeline</b></td>
+ <td><code>$ spark-submit spark_pipeline.py</code></td>
+ <td><code>$ python beam_pipeline.py</code></td>
+</tr>
+</table>
+{{< /table >}}
+
+## Transforms
+
+Here are the equivalents of some common transforms in both PySpark and Beam.
+
+{{< table >}}
+| | PySpark | Beam |
+|-------------------|---------------------------------------|---------------------------------------------------------|
+| **Map** | `values.map(lambda x: x * 2)` | `values | beam.Map(lambda x: x * 2)` |
+| **Filter** | `values.filter(lambda x: x % 2 == 0)` | `values | beam.Filter(lambda x: x % 2 == 0)` |
+| **FlatMap** | `values.flatMap(lambda x: range(x))` | `values | beam.FlatMap(lambda x: range(x))` |
+| **Group by key** | `pairs.groupByKey()` | `pairs | beam.GroupByKey()` |
+| **Reduce** | `values.reduce(lambda x, y: x+y)` | `values | beam.CombineGlobally(sum)` |
+| **Reduce by key** | `pairs.reduceByKey(lambda x, y: x+y)` | `pairs | beam.CombinePerKey(sum)` |
+| **Distinct** | `values.distinct()` | `values | beam.Distinct()` |
+| **Count** | `values.count()` | `values | beam.combiners.Count.Globally()` |
+| **Count by key** | `pairs.countByKey()` | `pairs | beam.combiners.Count.PerKey()` |
+| **Take smallest** | `values.takeOrdered(3)` | `values | beam.combiners.Top.Smallest(3)` |
+| **Take largest** | `values.takeOrdered(3, lambda x: -x)` | `values | beam.combiners.Top.Largest(3)` |
+| **Random sample** | `values.takeSample(False, 3)` | `values | beam.combiners.Sample.FixedSizeGlobally(3)` |
+| **Union** | `values.union(otherValues)` | `(values, otherValues) | beam.Flatten()` |
+| **Co-group** | `pairs.cogroup(otherPairs)` | `{'Xs': pairs, 'Ys': otherPairs} | beam.CoGroupByKey()` |
+{{< /table >}}
+
+> ℹ️ To learn more about the transforms available in Beam, check the
+> [Python transform gallery](/documentation/transforms/python/overview).
+
+## Using calculated values
+
+Since we are working in potentially distributed environments,
+we can't guarantee that the results we've calculated are available at any given machine.
+
+In PySpark, we can get a result from a collection of elements (RDD) by using
+`data.collect()`, or other aggregations such as `reduce()`, `count()` and more.
+
+Here's an example to scale numbers into a range between zero and one.
+
+{{< highlight py >}}
+import pyspark
+
+with pyspark.SparkContext() as sc:
+ values = sc.parallelize([1, 2, 3, 4])
+ total = values.reduce(lambda x, y: x + y)
+
+ # We can simply use `total` since it's already a Python value from `reduce`.
+ scaled_values = values.map(lambda x: x / total)
+
+ # But to access `scaled_values`, we need to call `collect`.
+ print(scaled_values.collect())
+{{< /highlight >}}
+
+In Beam the results from _all_ transforms result in a PCollection.
+We use _side inputs_ to feed a PCollection into a transform and access its values.
Review comment:
Done
##########
File path: website/www/site/content/en/get-started/from-spark.md
##########
@@ -0,0 +1,245 @@
+---
+title: "Getting started from Apache Spark"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Getting started from Apache Spark
+
+{{< localstorage language language-py >}}
+
+If you already know [_Apache Spark_](http://spark.apache.org/),
+learning _Apache Beam_ is easy.
+Beam and Spark are mostly equivalent, so you already know the basic concepts.
+
+A collection of elements in Spark is called a _Resilient Distributed Dataset_ (RDD),
+while in Beam it's called a _Parallel Collection_ (PCollection).
+A PCollection in Beam does _not_ have any ordering guarantees.
+
+Likewise, a transform in Beam is called a _Parallel Transform_ (PTransform).
+
+Here are some examples of common operations and their equivalent between PySpark and Beam.
+
+## Overview
+
+Here's a simple example of a PySpark pipeline that takes the numbers from one to four,
+multiplies them by two, adds all the values together, and prints the result.
+
+{{< highlight py >}}
+import pyspark
+
+with pyspark.SparkContext() as sc:
+ result = (
+ sc.parallelize([1, 2, 3, 4])
+ .map(lambda x: x * 2)
+ .reduce(lambda x, y: x + y)
+ )
+ print(result)
+{{< /highlight >}}
+
+In Beam you _pipe_ your data through the pipeline using the
+_pipe operator_ `|` like `data | beam.Map(...)` instead of chaining
+methods like `data.map(...)`, but they're doing the same thing.
+
+Here's how an equivalent pipeline looks like in Beam.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | beam.Create([1, 2, 3, 4])
+ | beam.Map(lambda x: x * 2)
+ | beam.CombineGlobally(sum)
+ | beam.Map(print)
+ )
+{{< /highlight >}}
+
+> ℹ️ Note that we called `print` inside a `Map` transform.
+> That's because we can only access the elements of a PCollection
+> from within a PTransform.
+
+A label can optionally be added to a transform using the
+_right shift operator_ `>>` like `data | 'My description' >> beam.Map(...)`.
+This serves both as comments and makes your pipeline easier to debug.
+
+This is how the pipeline looks after adding labels.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | 'Create numbers' >> beam.Create([1, 2, 3, 4])
+ | 'Multiply by two' >> beam.Map(lambda x: x * 2)
+ | 'Sum everything' >> beam.CombineGlobally(sum)
+ | beam.Map(print)
+ )
+{{< /highlight >}}
+
+## Setup
+
+Here's a comparison on how to get started both in PySpark and Beam.
+
+{{< table >}}
+<table>
+<tr>
+ <th></th>
+ <th>PySpark</th>
+ <th>Beam</th>
+</tr>
+<tr>
+ <td><b>Install</b></td>
+ <td><code>$ pip install pyspark</code></td>
+ <td><code>$ pip install apache-beam</code></td>
+</tr>
+<tr>
+ <td><b>Imports</b></td>
+ <td><code>import pyspark</code></td>
+ <td><code>import apache_beam as beam</code></td>
+</tr>
+<tr>
+ <td><b>Creating a<br>local pipeline</b></td>
+ <td>
+ <code>with pyspark.SparkContext() as sc:</code><br>
+ <code> # Your pipeline code here.</code>
+ </td>
+ <td>
+ <code>with beam.Pipeline() as pipeline:</code><br>
+ <code> # Your pipeline code here.</code>
+ </td>
+</tr>
+<tr>
+ <td><b>Creating values</b></td>
+ <td><code>values = sc.parallelize([1, 2, 3, 4])</code></td>
+ <td><code>values = pipeline | beam.Create([1, 2, 3, 4])</code></td>
+</tr>
+<tr>
+ <td><b>Creating<br>key-value pairs</b></td>
+ <td>
+ <code>pairs = sc.parallelize([</code><br>
+ <code> ('key1', 'value1'),</code><br>
+ <code> ('key2', 'value2'),</code><br>
+ <code> ('key3', 'value3'),</code><br>
+ <code>])</code>
+ </td>
+ <td>
+ <code>pairs = pipeline | beam.Create([</code><br>
+ <code> ('key1', 'value1'),</code><br>
+ <code> ('key2', 'value2'),</code><br>
+ <code> ('key3', 'value3'),</code><br>
+ <code>])</code>
+ </td>
+</tr>
+<tr>
+ <td><b>Running a<br>local pipeline</b></td>
+ <td><code>$ spark-submit spark_pipeline.py</code></td>
+ <td><code>$ python beam_pipeline.py</code></td>
+</tr>
+</table>
+{{< /table >}}
+
+## Transforms
+
+Here are the equivalents of some common transforms in both PySpark and Beam.
+
+{{< table >}}
+| | PySpark | Beam |
+|-------------------|---------------------------------------|---------------------------------------------------------|
+| **Map** | `values.map(lambda x: x * 2)` | `values | beam.Map(lambda x: x * 2)` |
+| **Filter** | `values.filter(lambda x: x % 2 == 0)` | `values | beam.Filter(lambda x: x % 2 == 0)` |
+| **FlatMap** | `values.flatMap(lambda x: range(x))` | `values | beam.FlatMap(lambda x: range(x))` |
+| **Group by key** | `pairs.groupByKey()` | `pairs | beam.GroupByKey()` |
+| **Reduce** | `values.reduce(lambda x, y: x+y)` | `values | beam.CombineGlobally(sum)` |
+| **Reduce by key** | `pairs.reduceByKey(lambda x, y: x+y)` | `pairs | beam.CombinePerKey(sum)` |
+| **Distinct** | `values.distinct()` | `values | beam.Distinct()` |
+| **Count** | `values.count()` | `values | beam.combiners.Count.Globally()` |
+| **Count by key** | `pairs.countByKey()` | `pairs | beam.combiners.Count.PerKey()` |
+| **Take smallest** | `values.takeOrdered(3)` | `values | beam.combiners.Top.Smallest(3)` |
+| **Take largest** | `values.takeOrdered(3, lambda x: -x)` | `values | beam.combiners.Top.Largest(3)` |
+| **Random sample** | `values.takeSample(False, 3)` | `values | beam.combiners.Sample.FixedSizeGlobally(3)` |
+| **Union** | `values.union(otherValues)` | `(values, otherValues) | beam.Flatten()` |
+| **Co-group** | `pairs.cogroup(otherPairs)` | `{'Xs': pairs, 'Ys': otherPairs} | beam.CoGroupByKey()` |
+{{< /table >}}
+
+> ℹ️ To learn more about the transforms available in Beam, check the
+> [Python transform gallery](/documentation/transforms/python/overview).
+
+## Using calculated values
+
+Since we are working in potentially distributed environments,
+we can't guarantee that the results we've calculated are available at any given machine.
+
+In PySpark, we can get a result from a collection of elements (RDD) by using
+`data.collect()`, or other aggregations such as `reduce()`, `count()` and more.
+
+Here's an example to scale numbers into a range between zero and one.
+
+{{< highlight py >}}
+import pyspark
+
+with pyspark.SparkContext() as sc:
+ values = sc.parallelize([1, 2, 3, 4])
+ total = values.reduce(lambda x, y: x + y)
+
+ # We can simply use `total` since it's already a Python value from `reduce`.
+ scaled_values = values.map(lambda x: x / total)
+
+ # But to access `scaled_values`, we need to call `collect`.
+ print(scaled_values.collect())
+{{< /highlight >}}
+
+In Beam the results from _all_ transforms result in a PCollection.
+We use _side inputs_ to feed a PCollection into a transform and access its values.
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rosetn commented on a change in pull request #12963: [BEAM-10983] Add getting started from Spark page
Posted by GitBox <gi...@apache.org>.
rosetn commented on a change in pull request #12963:
URL: https://github.com/apache/beam/pull/12963#discussion_r516785479
##########
File path: website/www/site/layouts/partials/section-menu/en/get-started.html
##########
@@ -22,12 +22,13 @@
<li><a href="/get-started/quickstart-go/">Quickstart - Go</a></li>
</ul>
</li>
+<li><a href="/get-started/from-spark/">From Apache Spark</a></li>
Review comment:
The actual Overview page has a Getting Started section you can include this new doc in.
##########
File path: website/www/site/content/en/get-started/from-spark.md
##########
@@ -0,0 +1,261 @@
+---
+title: "Getting started from Apache Spark"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Getting started from Apache Spark
+
+{{< localstorage language language-py >}}
+
+If you already know [_Apache Spark_](http://spark.apache.org/),
+learning _Apache Beam_ is easy.
Review comment:
WDYT about replacing "is easy" with "is familiar" or removing this sentence, since you explain the connection in the next sentence?
##########
File path: website/www/site/content/en/get-started/from-spark.md
##########
@@ -0,0 +1,261 @@
+---
+title: "Getting started from Apache Spark"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Getting started from Apache Spark
+
+{{< localstorage language language-py >}}
+
+If you already know [_Apache Spark_](http://spark.apache.org/),
+learning _Apache Beam_ is easy.
+The Beam and Spark APIs are similar, so you already know the basic concepts.
+
+Spark stores data _Spark DataFrames_ for structured data,
+and in _Resilient Distributed Datasets_ (RDD) for unstructured data.
+We are using RDDs for this guide.
+
+A _Spark RDD_ represents a collection of elements,
+while in Beam it's called a _Parallel Collection_ (PCollection).
+A PCollection in Beam does _not_ have any ordering guarantees.
+
+Likewise, a transform in Beam is called a _Parallel Transform_ (PTransform).
+
+Here are some examples of common operations and their equivalent between PySpark and Beam.
+
+## Overview
+
+Here's a simple example of a PySpark pipeline that takes the numbers from one to four,
+multiplies them by two, adds all the values together, and prints the result.
+
+{{< highlight py >}}
+import pyspark
+
+sc = pyspark.SparkContext()
+result = (
+ sc.parallelize([1, 2, 3, 4])
+ .map(lambda x: x * 2)
+ .reduce(lambda x, y: x + y)
+)
+print(result)
+{{< /highlight >}}
+
+In Beam you _pipe_ your data through the pipeline using the
+_pipe operator_ `|` like `data | beam.Map(...)` instead of chaining
+methods like `data.map(...)`, but they're doing the same thing.
+
+Here's how an equivalent pipeline looks like in Beam.
Review comment:
Replace "how" with "what"
##########
File path: website/www/site/content/en/get-started/from-spark.md
##########
@@ -0,0 +1,261 @@
+---
+title: "Getting started from Apache Spark"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Getting started from Apache Spark
Review comment:
I'd consider only italicizing the new Spark terms. The font emphasis can create accessibility issues if it's too frequent.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidcavazos commented on pull request #12963: [BEAM-10983] Add getting started from Spark page
Posted by GitBox <gi...@apache.org>.
davidcavazos commented on pull request #12963:
URL: https://github.com/apache/beam/pull/12963#issuecomment-744559372
Run Website_Stage_GCS PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bradmiro commented on pull request #12963: [BEAM-10983] Add getting started from Spark page
Posted by GitBox <gi...@apache.org>.
bradmiro commented on pull request #12963:
URL: https://github.com/apache/beam/pull/12963#issuecomment-712964924
Generally, Spark RDDs are used for unstructured data, whereas Spark Dataframes are used for structured data as it can be supplied a schema. The transformations in this case tend to be more efficient. Does Beam have any sort of similar split, or would you want to use PCollections in both cases?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on pull request #12963: [BEAM-10983] Add getting started from Spark page
Posted by GitBox <gi...@apache.org>.
iemejia commented on pull request #12963:
URL: https://github.com/apache/beam/pull/12963#issuecomment-713565407
Really nice idea, an idea for a future part 2 or extra section is to do an introduction based on Spark's Dataframe/Dataset (and SQL) vs Beam's Schema and Schema-baded transforms (and SQL)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] iemejia commented on a change in pull request #12963: [BEAM-10983] Add getting started from Spark page
Posted by GitBox <gi...@apache.org>.
iemejia commented on a change in pull request #12963:
URL: https://github.com/apache/beam/pull/12963#discussion_r509276556
##########
File path: website/www/site/content/en/get-started/from-spark.md
##########
@@ -0,0 +1,261 @@
+---
+title: "Getting started from Apache Spark"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Getting started from Apache Spark
+
+{{< localstorage language language-py >}}
+
+If you already know [_Apache Spark_](http://spark.apache.org/),
+learning _Apache Beam_ is easy.
+The Beam and Spark APIs are similar, so you already know the basic concepts.
+
+Spark stores data _Spark DataFrames_ for structured data,
+and in _Resilient Distributed Datasets_ (RDD) for unstructured data.
+We are using RDDs for this guide.
+
+A _Spark RDD_ represents a collection of elements,
+while in Beam it's called a _Parallel Collection_ (PCollection).
+A PCollection in Beam does _not_ have any ordering guarantees.
+
+Likewise, a transform in Beam is called a _Parallel Transform_ (PTransform).
+
+Here are some examples of common operations and their equivalent between PySpark and Beam.
+
+## Overview
+
+Here's a simple example of a PySpark pipeline that takes the numbers from one to four,
+multiplies them by two, adds all the values together, and prints the result.
+
+{{< highlight py >}}
+import pyspark
+
+sc = pyspark.SparkContext()
+result = (
+ sc.parallelize([1, 2, 3, 4])
+ .map(lambda x: x * 2)
+ .reduce(lambda x, y: x + y)
+)
+print(result)
+{{< /highlight >}}
+
+In Beam you _pipe_ your data through the pipeline using the
+_pipe operator_ `|` like `data | beam.Map(...)` instead of chaining
+methods like `data.map(...)`, but they're doing the same thing.
+
+Here's how an equivalent pipeline looks like in Beam.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | beam.Create([1, 2, 3, 4])
+ | beam.Map(lambda x: x * 2)
+ | beam.CombineGlobally(sum)
+ | beam.Map(print)
+ )
+{{< /highlight >}}
+
+> ℹ️ Note that we called `print` inside a `Map` transform.
+> That's because we can only access the elements of a PCollection
+> from within a PTransform.
+
+Another thing to note is that Beam pipelines are constructed _lazily_.
+This means that when you _pipe_ `|` data you're only _declaring_ the
+transformations and the order you want them to happen,
+but the actual computation doesn't happen.
+The pipeline is run _after_ the `with beam.Pipeline() as pipeline` context has
+closed.
+The pipeline is then sent to your runner of choice and it processes the data.
+
+> ℹ️ When the `with beam.Pipeline() as pipeline` context closes,
+> it implicitly calls `pipeline.run()` which triggers the computation to happen.
+
+A label can optionally be added to a transform using the
+_right shift operator_ `>>` like `data | 'My description' >> beam.Map(...)`.
+This serves both as comments and makes your pipeline easier to debug.
+
+This is how the pipeline looks after adding labels.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | 'Create numbers' >> beam.Create([1, 2, 3, 4])
+ | 'Multiply by two' >> beam.Map(lambda x: x * 2)
+ | 'Sum everything' >> beam.CombineGlobally(sum)
+ | 'Print results' >> beam.Map(print)
+ )
+{{< /highlight >}}
+
+## Setup
+
+Here's a comparison on how to get started both in PySpark and Beam.
+
+{{< table >}}
+<table>
+<tr>
+ <th></th>
+ <th>PySpark</th>
+ <th>Beam</th>
+</tr>
+<tr>
+ <td><b>Install</b></td>
+ <td><code>$ pip install pyspark</code></td>
+ <td><code>$ pip install apache-beam</code></td>
+</tr>
+<tr>
+ <td><b>Imports</b></td>
+ <td><code>import pyspark</code></td>
+ <td><code>import apache_beam as beam</code></td>
+</tr>
+<tr>
+ <td><b>Creating a<br>local pipeline</b></td>
+ <td>
+ <code>sc = pyspark.SparkContext() as sc:</code><br>
+ <code># Your pipeline code here.</code>
+ </td>
+ <td>
+ <code>with beam.Pipeline() as pipeline:</code><br>
+ <code> # Your pipeline code here.</code>
+ </td>
+</tr>
+<tr>
+ <td><b>Creating values</b></td>
+ <td><code>values = sc.parallelize([1, 2, 3, 4])</code></td>
+ <td><code>values = pipeline | beam.Create([1, 2, 3, 4])</code></td>
+</tr>
+<tr>
+ <td><b>Creating<br>key-value pairs</b></td>
+ <td>
+ <code>pairs = sc.parallelize([</code><br>
+ <code> ('key1', 'value1'),</code><br>
+ <code> ('key2', 'value2'),</code><br>
+ <code> ('key3', 'value3'),</code><br>
+ <code>])</code>
+ </td>
+ <td>
+ <code>pairs = pipeline | beam.Create([</code><br>
+ <code> ('key1', 'value1'),</code><br>
+ <code> ('key2', 'value2'),</code><br>
+ <code> ('key3', 'value3'),</code><br>
+ <code>])</code>
+ </td>
+</tr>
Review comment:
I am not sure if worth but since we mention lazy computation we probably may mention what triggers 'producing results', in Spark is done by Spark's `actions` e.g. collect(), etc and in Beam by outputting the data (you can relate this to the print mention). Notice that I saw the warning of the `with` section but we can double mention this here and/or mention `p.run()`).
Note this is just an extra suggestion but it is not mandatory at all, it is already clear, this is 'extra details'.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidcavazos commented on pull request #12963: [BEAM-10983] Add getting started from Spark page
Posted by GitBox <gi...@apache.org>.
davidcavazos commented on pull request #12963:
URL: https://github.com/apache/beam/pull/12963#issuecomment-743231702
Sorry for the delay, I was working on a higher priority project and was also OOO for some days.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rosetn commented on pull request #12963: [BEAM-10983] Add getting started from Spark page
Posted by GitBox <gi...@apache.org>.
rosetn commented on pull request #12963:
URL: https://github.com/apache/beam/pull/12963#issuecomment-721247468
This is great! FYI @rezarokni as we've discussed something similar!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidcavazos commented on pull request #12963: [BEAM-10983] Add getting started from Spark page
Posted by GitBox <gi...@apache.org>.
davidcavazos commented on pull request #12963:
URL: https://github.com/apache/beam/pull/12963#issuecomment-744555294
Thanks, fixed those typos. I also removed most of the italics except for terms definitions and other important notes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidcavazos commented on a change in pull request #12963: [BEAM-10983] Add getting started from Spark page
Posted by GitBox <gi...@apache.org>.
davidcavazos commented on a change in pull request #12963:
URL: https://github.com/apache/beam/pull/12963#discussion_r542520658
##########
File path: website/www/site/content/en/get-started/from-spark.md
##########
@@ -0,0 +1,268 @@
+---
+title: "Getting started from Apache Spark"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Getting started from Apache Spark
+
+{{< localstorage language language-py >}}
+
+If you already know [_Apache Spark_](http://spark.apache.org/),
+learning _Apache Beam_ is familiar.
+The Beam and Spark APIs are similar, so you already know the basic concepts.
+
+Spark stores data _Spark DataFrames_ for structured data,
+and in _Resilient Distributed Datasets_ (RDD) for unstructured data.
+We are using RDDs for this guide.
+
+A Spark RDD represents a collection of elements,
+while in Beam it's called a _Parallel Collection_ (PCollection).
+A PCollection in Beam does _not_ have any ordering guarantees.
+
+Likewise, a transform in Beam is called a _Parallel Transform_ (PTransform).
+
+Here are some examples of common operations and their equivalent between PySpark and Beam.
+
+## Overview
+
+Here's a simple example of a PySpark pipeline that takes the numbers from one to four,
+multiplies them by two, adds all the values together, and prints the result.
+
+{{< highlight py >}}
+import pyspark
+
+sc = pyspark.SparkContext()
+result = (
+ sc.parallelize([1, 2, 3, 4])
+ .map(lambda x: x * 2)
+ .reduce(lambda x, y: x + y)
+)
+print(result)
+{{< /highlight >}}
+
+In Beam you _pipe_ your data through the pipeline using the
+_pipe operator_ `|` like `data | beam.Map(...)` instead of chaining
+methods like `data.map(...)`, but they're doing the same thing.
+
+Here's what an equivalent pipeline looks like in Beam.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | beam.Create([1, 2, 3, 4])
+ | beam.Map(lambda x: x * 2)
+ | beam.CombineGlobally(sum)
+ | beam.Map(print)
+ )
+{{< /highlight >}}
+
+> ℹ️ Note that we called `print` inside a `Map` transform.
+> That's because we can only access the elements of a PCollection
+> from within a PTransform.
+
+Another thing to note is that Beam pipelines are constructed _lazily_.
+This means that when you pipe `|` data you're only _decalring_ the
+transformations and the order you want them to happen,
+but the actual computation doesn't happen.
+The pipeline is run _after_ the `with beam.Pipeline() as pipeline` context has
+closed.
+
+> ℹ️ When the `with beam.Pipeline() as pipeline` context closes,
+> it implicitly calls `pipeline.run()` which triggers the computation to happen.
+
+The pipeline is then sent to your
+[runner of choice](https://beam.apache.org/documentation/runners/capability-matrix/)
+and it processes the data.
+
+> ℹ️ The pipeline can run locally with the _DirectRunner_,
+> or in a distributed runner such as Flink, Spark, or Dataflow.
+> The Spark runner is not related to PySpark.
+
+A label can optionally be added to a transform using the
+_right shift operator_ `>>` like `data | 'My description' >> beam.Map(...)`.
+This serves both as comments and makes your pipeline easier to debug.
+
+This is how the pipeline looks after adding labels.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | 'Create numbers' >> beam.Create([1, 2, 3, 4])
+ | 'Multiply by two' >> beam.Map(lambda x: x * 2)
+ | 'Sum everything' >> beam.CombineGlobally(sum)
+ | 'Print results' >> beam.Map(print)
+ )
+{{< /highlight >}}
+
+## Setup
+
+Here's a comparison on how to get started both in PySpark and Beam.
+
+{{< table >}}
+<table>
+<tr>
+ <th></th>
+ <th>PySpark</th>
+ <th>Beam</th>
+</tr>
+<tr>
+ <td><b>Install</b></td>
+ <td><code>$ pip install pyspark</code></td>
+ <td><code>$ pip install apache-beam</code></td>
+</tr>
+<tr>
+ <td><b>Imports</b></td>
+ <td><code>import pyspark</code></td>
+ <td><code>import apache_beam as beam</code></td>
+</tr>
+<tr>
+ <td><b>Creating a<br>local pipeline</b></td>
+ <td>
+ <code>sc = pyspark.SparkContext() as sc:</code><br>
+ <code># Your pipeline code here.</code>
+ </td>
+ <td>
+ <code>with beam.Pipeline() as pipeline:</code><br>
+ <code> # Your pipeline code here.</code>
+ </td>
+</tr>
+<tr>
+ <td><b>Creating values</b></td>
+ <td><code>values = sc.parallelize([1, 2, 3, 4])</code></td>
+ <td><code>values = pipeline | beam.Create([1, 2, 3, 4])</code></td>
+</tr>
+<tr>
+ <td><b>Creating<br>key-value pairs</b></td>
+ <td>
+ <code>pairs = sc.parallelize([</code><br>
+ <code> ('key1', 'value1'),</code><br>
+ <code> ('key2', 'value2'),</code><br>
+ <code> ('key3', 'value3'),</code><br>
+ <code>])</code>
+ </td>
+ <td>
+ <code>pairs = pipeline | beam.Create([</code><br>
+ <code> ('key1', 'value1'),</code><br>
+ <code> ('key2', 'value2'),</code><br>
+ <code> ('key3', 'value3'),</code><br>
+ <code>])</code>
+ </td>
+</tr>
+<tr>
+ <td><b>Running a<br>local pipeline</b></td>
+ <td><code>$ spark-submit spark_pipeline.py</code></td>
+ <td><code>$ python beam_pipeline.py</code></td>
+</tr>
+</table>
+{{< /table >}}
+
+## Transforms
+
+Here are the equivalents of some common transforms in both PySpark and Beam.
+
+{{< table >}}
+| | PySpark | Beam |
+|----------------------------------------------------------------------------------|---------------------------------------|---------------------------------------------------------|
+| [**Map**](/documentation/transforms/python/elementwise/map/) | `values.map(lambda x: x * 2)` | `values | beam.Map(lambda x: x * 2)` |
+| [**Filter**](/documentation/transforms/python/elementwise/filter/) | `values.filter(lambda x: x % 2 == 0)` | `values | beam.Filter(lambda x: x % 2 == 0)` |
+| [**FlatMap**](/documentation/transforms/python/elementwise/flatmap/) | `values.flatMap(lambda x: range(x))` | `values | beam.FlatMap(lambda x: range(x))` |
+| [**Group by key**](/documentation/transforms/python/aggregation/groupbykey/) | `pairs.groupByKey()` | `pairs | beam.GroupByKey()` |
+| [**Reduce**](/documentation/transforms/python/aggregation/combineglobally/) | `values.reduce(lambda x, y: x+y)` | `values | beam.CombineGlobally(sum)` |
+| [**Reduce by key**](/documentation/transforms/python/aggregation/combineperkey/) | `pairs.reduceByKey(lambda x, y: x+y)` | `pairs | beam.CombinePerKey(sum)` |
+| [**Distinct**](/documentation/transforms/python/aggregation/distinct/) | `values.distinct()` | `values | beam.Distinct()` |
+| [**Count**](/documentation/transforms/python/aggregation/count/) | `values.count()` | `values | beam.combiners.Count.Globally()` |
+| [**Count by key**](/documentation/transforms/python/aggregation/count/) | `pairs.countByKey()` | `pairs | beam.combiners.Count.PerKey()` |
+| [**Take smallest**](/documentation/transforms/python/aggregation/top/) | `values.takeOrdered(3)` | `values | beam.combiners.Top.Smallest(3)` |
+| [**Take largest**](/documentation/transforms/python/aggregation/top/) | `values.takeOrdered(3, lambda x: -x)` | `values | beam.combiners.Top.Largest(3)` |
+| [**Random sample**](/documentation/transforms/python/aggregation/sample/) | `values.takeSample(False, 3)` | `values | beam.combiners.Sample.FixedSizeGlobally(3)` |
+| [**Union**](/documentation/transforms/python/other/flatten/) | `values.union(otherValues)` | `(values, otherValues) | beam.Flatten()` |
+| [**Co-group**](/documentation/transforms/python/aggregation/cogroupbykey/) | `pairs.cogroup(otherPairs)` | `{'Xs': pairs, 'Ys': otherPairs} | beam.CoGroupByKey()` |
+{{< /table >}}
+
+> ℹ️ To learn more about the transforms available in Beam, check the
+> [Python transform gallery](/documentation/transforms/python/overview).
+
+## Using calculated values
+
+Since we are working in potentially distributed environments,
+we can't guarantee that the results we've calculated are available at any given machine.
+
+In PySpark, we can get a result from a collection of elements (RDD) by using
+`data.collect()`, or other aggregations such as `reduce()`, `count()` and more.
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidcavazos removed a comment on pull request #12963: [BEAM-10983] Add getting started from Spark page
Posted by GitBox <gi...@apache.org>.
davidcavazos removed a comment on pull request #12963:
URL: https://github.com/apache/beam/pull/12963#issuecomment-744559192
Run Website_Stage_GCS
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidcavazos edited a comment on pull request #12963: [BEAM-10983] Add getting started from Spark page
Posted by GitBox <gi...@apache.org>.
davidcavazos edited a comment on pull request #12963:
URL: https://github.com/apache/beam/pull/12963#issuecomment-712953191
> You may also want to make some mention of the Spark `DataFrame` [API](https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrame), at least in the introduction. This is technically newer and in many cases superior to the RDD API, and where a lot of users would be coming from.
@bradmiro What kind of mention do you think would be useful? I'm not familiar with Spark nor Spark Dataframes. I was looking at the API and it looks like it has about the same functionality, but some method names are different, and it supports SQL. Right now BeamSQL is only available in Java so we won't mention it here for the time being.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rosetn commented on a change in pull request #12963: [BEAM-10983] Add getting started from Spark page
Posted by GitBox <gi...@apache.org>.
rosetn commented on a change in pull request #12963:
URL: https://github.com/apache/beam/pull/12963#discussion_r541421022
##########
File path: website/www/site/content/en/get-started/from-spark.md
##########
@@ -0,0 +1,268 @@
+---
+title: "Getting started from Apache Spark"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Getting started from Apache Spark
+
+{{< localstorage language language-py >}}
+
+If you already know [_Apache Spark_](http://spark.apache.org/),
+learning _Apache Beam_ is familiar.
+The Beam and Spark APIs are similar, so you already know the basic concepts.
+
+Spark stores data _Spark DataFrames_ for structured data,
+and in _Resilient Distributed Datasets_ (RDD) for unstructured data.
+We are using RDDs for this guide.
+
+A Spark RDD represents a collection of elements,
+while in Beam it's called a _Parallel Collection_ (PCollection).
+A PCollection in Beam does _not_ have any ordering guarantees.
+
+Likewise, a transform in Beam is called a _Parallel Transform_ (PTransform).
+
+Here are some examples of common operations and their equivalent between PySpark and Beam.
+
+## Overview
+
+Here's a simple example of a PySpark pipeline that takes the numbers from one to four,
+multiplies them by two, adds all the values together, and prints the result.
+
+{{< highlight py >}}
+import pyspark
+
+sc = pyspark.SparkContext()
+result = (
+ sc.parallelize([1, 2, 3, 4])
+ .map(lambda x: x * 2)
+ .reduce(lambda x, y: x + y)
+)
+print(result)
+{{< /highlight >}}
+
+In Beam you _pipe_ your data through the pipeline using the
+_pipe operator_ `|` like `data | beam.Map(...)` instead of chaining
+methods like `data.map(...)`, but they're doing the same thing.
+
+Here's what an equivalent pipeline looks like in Beam.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | beam.Create([1, 2, 3, 4])
+ | beam.Map(lambda x: x * 2)
+ | beam.CombineGlobally(sum)
+ | beam.Map(print)
+ )
+{{< /highlight >}}
+
+> ℹ️ Note that we called `print` inside a `Map` transform.
+> That's because we can only access the elements of a PCollection
+> from within a PTransform.
+
+Another thing to note is that Beam pipelines are constructed _lazily_.
+This means that when you pipe `|` data you're only _decalring_ the
Review comment:
typo--declaring
##########
File path: website/www/site/content/en/get-started/from-spark.md
##########
@@ -0,0 +1,268 @@
+---
+title: "Getting started from Apache Spark"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Getting started from Apache Spark
+
+{{< localstorage language language-py >}}
+
+If you already know [_Apache Spark_](http://spark.apache.org/),
+learning _Apache Beam_ is familiar.
+The Beam and Spark APIs are similar, so you already know the basic concepts.
+
+Spark stores data _Spark DataFrames_ for structured data,
+and in _Resilient Distributed Datasets_ (RDD) for unstructured data.
+We are using RDDs for this guide.
+
+A Spark RDD represents a collection of elements,
+while in Beam it's called a _Parallel Collection_ (PCollection).
+A PCollection in Beam does _not_ have any ordering guarantees.
+
+Likewise, a transform in Beam is called a _Parallel Transform_ (PTransform).
+
+Here are some examples of common operations and their equivalent between PySpark and Beam.
+
+## Overview
+
+Here's a simple example of a PySpark pipeline that takes the numbers from one to four,
+multiplies them by two, adds all the values together, and prints the result.
+
+{{< highlight py >}}
+import pyspark
+
+sc = pyspark.SparkContext()
+result = (
+ sc.parallelize([1, 2, 3, 4])
+ .map(lambda x: x * 2)
+ .reduce(lambda x, y: x + y)
+)
+print(result)
+{{< /highlight >}}
+
+In Beam you _pipe_ your data through the pipeline using the
+_pipe operator_ `|` like `data | beam.Map(...)` instead of chaining
+methods like `data.map(...)`, but they're doing the same thing.
+
+Here's what an equivalent pipeline looks like in Beam.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | beam.Create([1, 2, 3, 4])
+ | beam.Map(lambda x: x * 2)
+ | beam.CombineGlobally(sum)
+ | beam.Map(print)
+ )
+{{< /highlight >}}
+
+> ℹ️ Note that we called `print` inside a `Map` transform.
+> That's because we can only access the elements of a PCollection
+> from within a PTransform.
+
+Another thing to note is that Beam pipelines are constructed _lazily_.
+This means that when you pipe `|` data you're only _decalring_ the
+transformations and the order you want them to happen,
+but the actual computation doesn't happen.
+The pipeline is run _after_ the `with beam.Pipeline() as pipeline` context has
+closed.
+
+> ℹ️ When the `with beam.Pipeline() as pipeline` context closes,
+> it implicitly calls `pipeline.run()` which triggers the computation to happen.
+
+The pipeline is then sent to your
+[runner of choice](https://beam.apache.org/documentation/runners/capability-matrix/)
+and it processes the data.
+
+> ℹ️ The pipeline can run locally with the _DirectRunner_,
+> or in a distributed runner such as Flink, Spark, or Dataflow.
+> The Spark runner is not related to PySpark.
+
+A label can optionally be added to a transform using the
+_right shift operator_ `>>` like `data | 'My description' >> beam.Map(...)`.
+This serves both as comments and makes your pipeline easier to debug.
+
+This is how the pipeline looks after adding labels.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | 'Create numbers' >> beam.Create([1, 2, 3, 4])
+ | 'Multiply by two' >> beam.Map(lambda x: x * 2)
+ | 'Sum everything' >> beam.CombineGlobally(sum)
+ | 'Print results' >> beam.Map(print)
+ )
+{{< /highlight >}}
+
+## Setup
+
+Here's a comparison on how to get started both in PySpark and Beam.
+
+{{< table >}}
+<table>
+<tr>
+ <th></th>
+ <th>PySpark</th>
+ <th>Beam</th>
+</tr>
+<tr>
+ <td><b>Install</b></td>
+ <td><code>$ pip install pyspark</code></td>
+ <td><code>$ pip install apache-beam</code></td>
+</tr>
+<tr>
+ <td><b>Imports</b></td>
+ <td><code>import pyspark</code></td>
+ <td><code>import apache_beam as beam</code></td>
+</tr>
+<tr>
+ <td><b>Creating a<br>local pipeline</b></td>
+ <td>
+ <code>sc = pyspark.SparkContext() as sc:</code><br>
+ <code># Your pipeline code here.</code>
+ </td>
+ <td>
+ <code>with beam.Pipeline() as pipeline:</code><br>
+ <code> # Your pipeline code here.</code>
+ </td>
+</tr>
+<tr>
+ <td><b>Creating values</b></td>
+ <td><code>values = sc.parallelize([1, 2, 3, 4])</code></td>
+ <td><code>values = pipeline | beam.Create([1, 2, 3, 4])</code></td>
+</tr>
+<tr>
+ <td><b>Creating<br>key-value pairs</b></td>
+ <td>
+ <code>pairs = sc.parallelize([</code><br>
+ <code> ('key1', 'value1'),</code><br>
+ <code> ('key2', 'value2'),</code><br>
+ <code> ('key3', 'value3'),</code><br>
+ <code>])</code>
+ </td>
+ <td>
+ <code>pairs = pipeline | beam.Create([</code><br>
+ <code> ('key1', 'value1'),</code><br>
+ <code> ('key2', 'value2'),</code><br>
+ <code> ('key3', 'value3'),</code><br>
+ <code>])</code>
+ </td>
+</tr>
+<tr>
+ <td><b>Running a<br>local pipeline</b></td>
+ <td><code>$ spark-submit spark_pipeline.py</code></td>
+ <td><code>$ python beam_pipeline.py</code></td>
+</tr>
+</table>
+{{< /table >}}
+
+## Transforms
+
+Here are the equivalents of some common transforms in both PySpark and Beam.
+
+{{< table >}}
+| | PySpark | Beam |
+|----------------------------------------------------------------------------------|---------------------------------------|---------------------------------------------------------|
+| [**Map**](/documentation/transforms/python/elementwise/map/) | `values.map(lambda x: x * 2)` | `values | beam.Map(lambda x: x * 2)` |
+| [**Filter**](/documentation/transforms/python/elementwise/filter/) | `values.filter(lambda x: x % 2 == 0)` | `values | beam.Filter(lambda x: x % 2 == 0)` |
+| [**FlatMap**](/documentation/transforms/python/elementwise/flatmap/) | `values.flatMap(lambda x: range(x))` | `values | beam.FlatMap(lambda x: range(x))` |
+| [**Group by key**](/documentation/transforms/python/aggregation/groupbykey/) | `pairs.groupByKey()` | `pairs | beam.GroupByKey()` |
+| [**Reduce**](/documentation/transforms/python/aggregation/combineglobally/) | `values.reduce(lambda x, y: x+y)` | `values | beam.CombineGlobally(sum)` |
+| [**Reduce by key**](/documentation/transforms/python/aggregation/combineperkey/) | `pairs.reduceByKey(lambda x, y: x+y)` | `pairs | beam.CombinePerKey(sum)` |
+| [**Distinct**](/documentation/transforms/python/aggregation/distinct/) | `values.distinct()` | `values | beam.Distinct()` |
+| [**Count**](/documentation/transforms/python/aggregation/count/) | `values.count()` | `values | beam.combiners.Count.Globally()` |
+| [**Count by key**](/documentation/transforms/python/aggregation/count/) | `pairs.countByKey()` | `pairs | beam.combiners.Count.PerKey()` |
+| [**Take smallest**](/documentation/transforms/python/aggregation/top/) | `values.takeOrdered(3)` | `values | beam.combiners.Top.Smallest(3)` |
+| [**Take largest**](/documentation/transforms/python/aggregation/top/) | `values.takeOrdered(3, lambda x: -x)` | `values | beam.combiners.Top.Largest(3)` |
+| [**Random sample**](/documentation/transforms/python/aggregation/sample/) | `values.takeSample(False, 3)` | `values | beam.combiners.Sample.FixedSizeGlobally(3)` |
+| [**Union**](/documentation/transforms/python/other/flatten/) | `values.union(otherValues)` | `(values, otherValues) | beam.Flatten()` |
+| [**Co-group**](/documentation/transforms/python/aggregation/cogroupbykey/) | `pairs.cogroup(otherPairs)` | `{'Xs': pairs, 'Ys': otherPairs} | beam.CoGroupByKey()` |
+{{< /table >}}
+
+> ℹ️ To learn more about the transforms available in Beam, check the
+> [Python transform gallery](/documentation/transforms/python/overview).
+
+## Using calculated values
+
+Since we are working in potentially distributed environments,
+we can't guarantee that the results we've calculated are available at any given machine.
+
+In PySpark, we can get a result from a collection of elements (RDD) by using
+`data.collect()`, or other aggregations such as `reduce()`, `count()` and more.
Review comment:
comm after `count()`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidcavazos commented on pull request #12963: [BEAM-10983] Add getting started from Spark page
Posted by GitBox <gi...@apache.org>.
davidcavazos commented on pull request #12963:
URL: https://github.com/apache/beam/pull/12963#issuecomment-713004034
I just added a clarification at the introduction.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rosetn commented on pull request #12963: [BEAM-10983] Add getting started from Spark page
Posted by GitBox <gi...@apache.org>.
rosetn commented on pull request #12963:
URL: https://github.com/apache/beam/pull/12963#issuecomment-730628410
Run Website_Stage_GCS PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidcavazos commented on pull request #12963: [BEAM-10983] Add getting started from Spark page
Posted by GitBox <gi...@apache.org>.
davidcavazos commented on pull request #12963:
URL: https://github.com/apache/beam/pull/12963#issuecomment-756259242
@aaltay I think all comments have been addressed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on a change in pull request #12963: [BEAM-10983] Add getting started from Spark page
Posted by GitBox <gi...@apache.org>.
ibzib commented on a change in pull request #12963:
URL: https://github.com/apache/beam/pull/12963#discussion_r517704555
##########
File path: website/www/site/content/en/get-started/from-spark.md
##########
@@ -0,0 +1,261 @@
+---
+title: "Getting started from Apache Spark"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Getting started from Apache Spark
+
+{{< localstorage language language-py >}}
+
+If you already know [_Apache Spark_](http://spark.apache.org/),
+learning _Apache Beam_ is easy.
+The Beam and Spark APIs are similar, so you already know the basic concepts.
+
+Spark stores data _Spark DataFrames_ for structured data,
+and in _Resilient Distributed Datasets_ (RDD) for unstructured data.
+We are using RDDs for this guide.
+
+A _Spark RDD_ represents a collection of elements,
+while in Beam it's called a _Parallel Collection_ (PCollection).
+A PCollection in Beam does _not_ have any ordering guarantees.
+
+Likewise, a transform in Beam is called a _Parallel Transform_ (PTransform).
+
+Here are some examples of common operations and their equivalent between PySpark and Beam.
+
+## Overview
+
+Here's a simple example of a PySpark pipeline that takes the numbers from one to four,
+multiplies them by two, adds all the values together, and prints the result.
+
+{{< highlight py >}}
+import pyspark
+
+sc = pyspark.SparkContext()
+result = (
+ sc.parallelize([1, 2, 3, 4])
+ .map(lambda x: x * 2)
+ .reduce(lambda x, y: x + y)
+)
+print(result)
+{{< /highlight >}}
+
+In Beam you _pipe_ your data through the pipeline using the
+_pipe operator_ `|` like `data | beam.Map(...)` instead of chaining
+methods like `data.map(...)`, but they're doing the same thing.
+
+Here's how an equivalent pipeline looks like in Beam.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | beam.Create([1, 2, 3, 4])
+ | beam.Map(lambda x: x * 2)
+ | beam.CombineGlobally(sum)
+ | beam.Map(print)
+ )
+{{< /highlight >}}
+
+> ℹ️ Note that we called `print` inside a `Map` transform.
+> That's because we can only access the elements of a PCollection
+> from within a PTransform.
+
+Another thing to note is that Beam pipelines are constructed _lazily_.
+This means that when you _pipe_ `|` data you're only _declaring_ the
+transformations and the order you want them to happen,
+but the actual computation doesn't happen.
+The pipeline is run _after_ the `with beam.Pipeline() as pipeline` context has
+closed.
+The pipeline is then sent to your runner of choice and it processes the data.
Review comment:
I realized we should probably clarify somewhere that "Beam Python on the Spark runner" and PySpark are completed unrelated, since that seems to be a common misconception.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidcavazos commented on a change in pull request #12963: [BEAM-10983] Add getting started from Spark page
Posted by GitBox <gi...@apache.org>.
davidcavazos commented on a change in pull request #12963:
URL: https://github.com/apache/beam/pull/12963#discussion_r508069958
##########
File path: website/www/site/content/en/get-started/from-spark.md
##########
@@ -0,0 +1,245 @@
+---
+title: "Getting started from Apache Spark"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Getting started from Apache Spark
+
+{{< localstorage language language-py >}}
+
+If you already know [_Apache Spark_](http://spark.apache.org/),
+learning _Apache Beam_ is easy.
+Beam and Spark are mostly equivalent, so you already know the basic concepts.
Review comment:
Thanks, done
##########
File path: website/www/site/content/en/get-started/from-spark.md
##########
@@ -0,0 +1,245 @@
+---
+title: "Getting started from Apache Spark"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Getting started from Apache Spark
+
+{{< localstorage language language-py >}}
+
+If you already know [_Apache Spark_](http://spark.apache.org/),
+learning _Apache Beam_ is easy.
+Beam and Spark are mostly equivalent, so you already know the basic concepts.
+
+A collection of elements in Spark is called a _Resilient Distributed Dataset_ (RDD),
+while in Beam it's called a _Parallel Collection_ (PCollection).
+A PCollection in Beam does _not_ have any ordering guarantees.
+
+Likewise, a transform in Beam is called a _Parallel Transform_ (PTransform).
+
+Here are some examples of common operations and their equivalent between PySpark and Beam.
+
+## Overview
+
+Here's a simple example of a PySpark pipeline that takes the numbers from one to four,
+multiplies them by two, adds all the values together, and prints the result.
+
+{{< highlight py >}}
+import pyspark
+
+with pyspark.SparkContext() as sc:
+ result = (
+ sc.parallelize([1, 2, 3, 4])
+ .map(lambda x: x * 2)
+ .reduce(lambda x, y: x + y)
+ )
+ print(result)
+{{< /highlight >}}
+
+In Beam you _pipe_ your data through the pipeline using the
Review comment:
I added a note just below the example.
##########
File path: website/www/site/content/en/get-started/from-spark.md
##########
@@ -0,0 +1,245 @@
+---
+title: "Getting started from Apache Spark"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Getting started from Apache Spark
+
+{{< localstorage language language-py >}}
+
+If you already know [_Apache Spark_](http://spark.apache.org/),
+learning _Apache Beam_ is easy.
+Beam and Spark are mostly equivalent, so you already know the basic concepts.
+
+A collection of elements in Spark is called a _Resilient Distributed Dataset_ (RDD),
+while in Beam it's called a _Parallel Collection_ (PCollection).
+A PCollection in Beam does _not_ have any ordering guarantees.
+
+Likewise, a transform in Beam is called a _Parallel Transform_ (PTransform).
+
+Here are some examples of common operations and their equivalent between PySpark and Beam.
+
+## Overview
+
+Here's a simple example of a PySpark pipeline that takes the numbers from one to four,
+multiplies them by two, adds all the values together, and prints the result.
+
+{{< highlight py >}}
+import pyspark
+
+with pyspark.SparkContext() as sc:
+ result = (
+ sc.parallelize([1, 2, 3, 4])
+ .map(lambda x: x * 2)
+ .reduce(lambda x, y: x + y)
+ )
+ print(result)
+{{< /highlight >}}
+
+In Beam you _pipe_ your data through the pipeline using the
+_pipe operator_ `|` like `data | beam.Map(...)` instead of chaining
+methods like `data.map(...)`, but they're doing the same thing.
+
+Here's how an equivalent pipeline looks like in Beam.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | beam.Create([1, 2, 3, 4])
+ | beam.Map(lambda x: x * 2)
+ | beam.CombineGlobally(sum)
+ | beam.Map(print)
+ )
+{{< /highlight >}}
+
+> ℹ️ Note that we called `print` inside a `Map` transform.
+> That's because we can only access the elements of a PCollection
+> from within a PTransform.
+
+A label can optionally be added to a transform using the
+_right shift operator_ `>>` like `data | 'My description' >> beam.Map(...)`.
+This serves both as comments and makes your pipeline easier to debug.
+
+This is how the pipeline looks after adding labels.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | 'Create numbers' >> beam.Create([1, 2, 3, 4])
+ | 'Multiply by two' >> beam.Map(lambda x: x * 2)
+ | 'Sum everything' >> beam.CombineGlobally(sum)
+ | beam.Map(print)
+ )
+{{< /highlight >}}
+
+## Setup
+
+Here's a comparison on how to get started both in PySpark and Beam.
+
+{{< table >}}
+<table>
+<tr>
+ <th></th>
+ <th>PySpark</th>
+ <th>Beam</th>
+</tr>
+<tr>
+ <td><b>Install</b></td>
+ <td><code>$ pip install pyspark</code></td>
+ <td><code>$ pip install apache-beam</code></td>
+</tr>
+<tr>
+ <td><b>Imports</b></td>
+ <td><code>import pyspark</code></td>
+ <td><code>import apache_beam as beam</code></td>
+</tr>
+<tr>
+ <td><b>Creating a<br>local pipeline</b></td>
+ <td>
+ <code>with pyspark.SparkContext() as sc:</code><br>
+ <code> # Your pipeline code here.</code>
+ </td>
+ <td>
+ <code>with beam.Pipeline() as pipeline:</code><br>
+ <code> # Your pipeline code here.</code>
+ </td>
+</tr>
+<tr>
+ <td><b>Creating values</b></td>
+ <td><code>values = sc.parallelize([1, 2, 3, 4])</code></td>
+ <td><code>values = pipeline | beam.Create([1, 2, 3, 4])</code></td>
+</tr>
+<tr>
+ <td><b>Creating<br>key-value pairs</b></td>
+ <td>
+ <code>pairs = sc.parallelize([</code><br>
+ <code> ('key1', 'value1'),</code><br>
+ <code> ('key2', 'value2'),</code><br>
+ <code> ('key3', 'value3'),</code><br>
+ <code>])</code>
+ </td>
+ <td>
+ <code>pairs = pipeline | beam.Create([</code><br>
+ <code> ('key1', 'value1'),</code><br>
+ <code> ('key2', 'value2'),</code><br>
+ <code> ('key3', 'value3'),</code><br>
+ <code>])</code>
+ </td>
+</tr>
+<tr>
+ <td><b>Running a<br>local pipeline</b></td>
+ <td><code>$ spark-submit spark_pipeline.py</code></td>
+ <td><code>$ python beam_pipeline.py</code></td>
+</tr>
+</table>
+{{< /table >}}
+
+## Transforms
+
+Here are the equivalents of some common transforms in both PySpark and Beam.
+
+{{< table >}}
Review comment:
Sure, the table gets a little uglier, and definitely over 80 characters long, but I can do that.
PS: I tried using reference links, but the website doesn't like them to be inside a table (even HTML instead of Markdown), so they have to be inlined if we want them here.
##########
File path: website/www/site/content/en/get-started/from-spark.md
##########
@@ -0,0 +1,245 @@
+---
+title: "Getting started from Apache Spark"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Getting started from Apache Spark
+
+{{< localstorage language language-py >}}
+
+If you already know [_Apache Spark_](http://spark.apache.org/),
+learning _Apache Beam_ is easy.
+Beam and Spark are mostly equivalent, so you already know the basic concepts.
+
+A collection of elements in Spark is called a _Resilient Distributed Dataset_ (RDD),
+while in Beam it's called a _Parallel Collection_ (PCollection).
+A PCollection in Beam does _not_ have any ordering guarantees.
+
+Likewise, a transform in Beam is called a _Parallel Transform_ (PTransform).
+
+Here are some examples of common operations and their equivalent between PySpark and Beam.
+
+## Overview
+
+Here's a simple example of a PySpark pipeline that takes the numbers from one to four,
+multiplies them by two, adds all the values together, and prints the result.
+
+{{< highlight py >}}
+import pyspark
+
+with pyspark.SparkContext() as sc:
+ result = (
+ sc.parallelize([1, 2, 3, 4])
+ .map(lambda x: x * 2)
+ .reduce(lambda x, y: x + y)
+ )
+ print(result)
+{{< /highlight >}}
+
+In Beam you _pipe_ your data through the pipeline using the
+_pipe operator_ `|` like `data | beam.Map(...)` instead of chaining
+methods like `data.map(...)`, but they're doing the same thing.
+
+Here's how an equivalent pipeline looks like in Beam.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | beam.Create([1, 2, 3, 4])
+ | beam.Map(lambda x: x * 2)
+ | beam.CombineGlobally(sum)
+ | beam.Map(print)
+ )
+{{< /highlight >}}
+
+> ℹ️ Note that we called `print` inside a `Map` transform.
+> That's because we can only access the elements of a PCollection
+> from within a PTransform.
+
+A label can optionally be added to a transform using the
+_right shift operator_ `>>` like `data | 'My description' >> beam.Map(...)`.
+This serves both as comments and makes your pipeline easier to debug.
+
+This is how the pipeline looks after adding labels.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | 'Create numbers' >> beam.Create([1, 2, 3, 4])
+ | 'Multiply by two' >> beam.Map(lambda x: x * 2)
+ | 'Sum everything' >> beam.CombineGlobally(sum)
+ | beam.Map(print)
+ )
+{{< /highlight >}}
+
+## Setup
+
+Here's a comparison on how to get started both in PySpark and Beam.
+
+{{< table >}}
+<table>
+<tr>
+ <th></th>
+ <th>PySpark</th>
+ <th>Beam</th>
+</tr>
+<tr>
+ <td><b>Install</b></td>
+ <td><code>$ pip install pyspark</code></td>
+ <td><code>$ pip install apache-beam</code></td>
+</tr>
+<tr>
+ <td><b>Imports</b></td>
+ <td><code>import pyspark</code></td>
+ <td><code>import apache_beam as beam</code></td>
+</tr>
+<tr>
+ <td><b>Creating a<br>local pipeline</b></td>
+ <td>
+ <code>with pyspark.SparkContext() as sc:</code><br>
+ <code> # Your pipeline code here.</code>
+ </td>
+ <td>
+ <code>with beam.Pipeline() as pipeline:</code><br>
+ <code> # Your pipeline code here.</code>
+ </td>
+</tr>
+<tr>
+ <td><b>Creating values</b></td>
+ <td><code>values = sc.parallelize([1, 2, 3, 4])</code></td>
+ <td><code>values = pipeline | beam.Create([1, 2, 3, 4])</code></td>
+</tr>
+<tr>
+ <td><b>Creating<br>key-value pairs</b></td>
+ <td>
+ <code>pairs = sc.parallelize([</code><br>
+ <code> ('key1', 'value1'),</code><br>
+ <code> ('key2', 'value2'),</code><br>
+ <code> ('key3', 'value3'),</code><br>
+ <code>])</code>
+ </td>
+ <td>
+ <code>pairs = pipeline | beam.Create([</code><br>
+ <code> ('key1', 'value1'),</code><br>
Review comment:
I don't have an @apache.org email so I can't open the discussion.
I thought tuples were supported better than lists. I've been using tuples for all the samples and I've never had an issue. I also think type-wise, tuples are "allowed" to have elements of any/different types, while lists should only be of one type. On a type checker like `mypy` or any other typed language, it would be a compile error to have a list of two different types, unless you use `Any` which should be discouraged.
I think I would prefer to keep these as tuples, @aaltay what do you think?
##########
File path: website/www/site/content/en/get-started/from-spark.md
##########
@@ -0,0 +1,245 @@
+---
+title: "Getting started from Apache Spark"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Getting started from Apache Spark
+
+{{< localstorage language language-py >}}
+
+If you already know [_Apache Spark_](http://spark.apache.org/),
+learning _Apache Beam_ is easy.
+Beam and Spark are mostly equivalent, so you already know the basic concepts.
+
+A collection of elements in Spark is called a _Resilient Distributed Dataset_ (RDD),
+while in Beam it's called a _Parallel Collection_ (PCollection).
+A PCollection in Beam does _not_ have any ordering guarantees.
+
+Likewise, a transform in Beam is called a _Parallel Transform_ (PTransform).
+
+Here are some examples of common operations and their equivalent between PySpark and Beam.
+
+## Overview
+
+Here's a simple example of a PySpark pipeline that takes the numbers from one to four,
+multiplies them by two, adds all the values together, and prints the result.
+
+{{< highlight py >}}
+import pyspark
+
+with pyspark.SparkContext() as sc:
+ result = (
+ sc.parallelize([1, 2, 3, 4])
+ .map(lambda x: x * 2)
+ .reduce(lambda x, y: x + y)
+ )
+ print(result)
+{{< /highlight >}}
+
+In Beam you _pipe_ your data through the pipeline using the
+_pipe operator_ `|` like `data | beam.Map(...)` instead of chaining
+methods like `data.map(...)`, but they're doing the same thing.
+
+Here's how an equivalent pipeline looks like in Beam.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | beam.Create([1, 2, 3, 4])
+ | beam.Map(lambda x: x * 2)
+ | beam.CombineGlobally(sum)
+ | beam.Map(print)
+ )
+{{< /highlight >}}
+
+> ℹ️ Note that we called `print` inside a `Map` transform.
+> That's because we can only access the elements of a PCollection
+> from within a PTransform.
+
+A label can optionally be added to a transform using the
+_right shift operator_ `>>` like `data | 'My description' >> beam.Map(...)`.
+This serves both as comments and makes your pipeline easier to debug.
+
+This is how the pipeline looks after adding labels.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | 'Create numbers' >> beam.Create([1, 2, 3, 4])
+ | 'Multiply by two' >> beam.Map(lambda x: x * 2)
+ | 'Sum everything' >> beam.CombineGlobally(sum)
+ | beam.Map(print)
Review comment:
Added
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib merged pull request #12963: [BEAM-10983] Add getting started from Spark page
Posted by GitBox <gi...@apache.org>.
ibzib merged pull request #12963:
URL: https://github.com/apache/beam/pull/12963
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidcavazos commented on a change in pull request #12963: [BEAM-10983] Add getting started from Spark page
Posted by GitBox <gi...@apache.org>.
davidcavazos commented on a change in pull request #12963:
URL: https://github.com/apache/beam/pull/12963#discussion_r529828968
##########
File path: website/www/site/content/en/get-started/from-spark.md
##########
@@ -0,0 +1,261 @@
+---
+title: "Getting started from Apache Spark"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Getting started from Apache Spark
+
+{{< localstorage language language-py >}}
+
+If you already know [_Apache Spark_](http://spark.apache.org/),
+learning _Apache Beam_ is easy.
Review comment:
Sounds good, done
##########
File path: website/www/site/layouts/partials/section-menu/en/get-started.html
##########
@@ -22,12 +22,13 @@
<li><a href="/get-started/quickstart-go/">Quickstart - Go</a></li>
</ul>
</li>
+<li><a href="/get-started/from-spark/">From Apache Spark</a></li>
Review comment:
Thanks, I just added a link there too
##########
File path: website/www/site/content/en/get-started/from-spark.md
##########
@@ -0,0 +1,261 @@
+---
+title: "Getting started from Apache Spark"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Getting started from Apache Spark
+
+{{< localstorage language language-py >}}
+
+If you already know [_Apache Spark_](http://spark.apache.org/),
+learning _Apache Beam_ is easy.
+The Beam and Spark APIs are similar, so you already know the basic concepts.
+
+Spark stores data _Spark DataFrames_ for structured data,
+and in _Resilient Distributed Datasets_ (RDD) for unstructured data.
+We are using RDDs for this guide.
+
+A _Spark RDD_ represents a collection of elements,
+while in Beam it's called a _Parallel Collection_ (PCollection).
+A PCollection in Beam does _not_ have any ordering guarantees.
+
+Likewise, a transform in Beam is called a _Parallel Transform_ (PTransform).
+
+Here are some examples of common operations and their equivalent between PySpark and Beam.
+
+## Overview
+
+Here's a simple example of a PySpark pipeline that takes the numbers from one to four,
+multiplies them by two, adds all the values together, and prints the result.
+
+{{< highlight py >}}
+import pyspark
+
+sc = pyspark.SparkContext()
+result = (
+ sc.parallelize([1, 2, 3, 4])
+ .map(lambda x: x * 2)
+ .reduce(lambda x, y: x + y)
+)
+print(result)
+{{< /highlight >}}
+
+In Beam you _pipe_ your data through the pipeline using the
+_pipe operator_ `|` like `data | beam.Map(...)` instead of chaining
+methods like `data.map(...)`, but they're doing the same thing.
+
+Here's how an equivalent pipeline looks like in Beam.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | beam.Create([1, 2, 3, 4])
+ | beam.Map(lambda x: x * 2)
+ | beam.CombineGlobally(sum)
+ | beam.Map(print)
+ )
+{{< /highlight >}}
+
+> ℹ️ Note that we called `print` inside a `Map` transform.
+> That's because we can only access the elements of a PCollection
+> from within a PTransform.
+
+Another thing to note is that Beam pipelines are constructed _lazily_.
+This means that when you _pipe_ `|` data you're only _declaring_ the
+transformations and the order you want them to happen,
+but the actual computation doesn't happen.
+The pipeline is run _after_ the `with beam.Pipeline() as pipeline` context has
+closed.
+The pipeline is then sent to your runner of choice and it processes the data.
+
+> ℹ️ When the `with beam.Pipeline() as pipeline` context closes,
+> it implicitly calls `pipeline.run()` which triggers the computation to happen.
+
+A label can optionally be added to a transform using the
+_right shift operator_ `>>` like `data | 'My description' >> beam.Map(...)`.
+This serves both as comments and makes your pipeline easier to debug.
+
+This is how the pipeline looks after adding labels.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | 'Create numbers' >> beam.Create([1, 2, 3, 4])
+ | 'Multiply by two' >> beam.Map(lambda x: x * 2)
+ | 'Sum everything' >> beam.CombineGlobally(sum)
+ | 'Print results' >> beam.Map(print)
+ )
+{{< /highlight >}}
+
+## Setup
+
+Here's a comparison on how to get started both in PySpark and Beam.
+
+{{< table >}}
+<table>
+<tr>
+ <th></th>
+ <th>PySpark</th>
+ <th>Beam</th>
+</tr>
+<tr>
+ <td><b>Install</b></td>
+ <td><code>$ pip install pyspark</code></td>
+ <td><code>$ pip install apache-beam</code></td>
+</tr>
+<tr>
+ <td><b>Imports</b></td>
+ <td><code>import pyspark</code></td>
+ <td><code>import apache_beam as beam</code></td>
+</tr>
+<tr>
+ <td><b>Creating a<br>local pipeline</b></td>
+ <td>
+ <code>sc = pyspark.SparkContext() as sc:</code><br>
+ <code># Your pipeline code here.</code>
+ </td>
+ <td>
+ <code>with beam.Pipeline() as pipeline:</code><br>
+ <code> # Your pipeline code here.</code>
+ </td>
+</tr>
+<tr>
+ <td><b>Creating values</b></td>
+ <td><code>values = sc.parallelize([1, 2, 3, 4])</code></td>
+ <td><code>values = pipeline | beam.Create([1, 2, 3, 4])</code></td>
+</tr>
+<tr>
+ <td><b>Creating<br>key-value pairs</b></td>
+ <td>
+ <code>pairs = sc.parallelize([</code><br>
+ <code> ('key1', 'value1'),</code><br>
+ <code> ('key2', 'value2'),</code><br>
+ <code> ('key3', 'value3'),</code><br>
+ <code>])</code>
+ </td>
+ <td>
+ <code>pairs = pipeline | beam.Create([</code><br>
+ <code> ('key1', 'value1'),</code><br>
+ <code> ('key2', 'value2'),</code><br>
+ <code> ('key3', 'value3'),</code><br>
+ <code>])</code>
+ </td>
+</tr>
Review comment:
If it's already clear, I think I'll leave it out in favor of simplicity. Sometimes less is more :)
##########
File path: website/www/site/content/en/get-started/from-spark.md
##########
@@ -0,0 +1,261 @@
+---
+title: "Getting started from Apache Spark"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Getting started from Apache Spark
+
+{{< localstorage language language-py >}}
+
+If you already know [_Apache Spark_](http://spark.apache.org/),
+learning _Apache Beam_ is easy.
+The Beam and Spark APIs are similar, so you already know the basic concepts.
+
+Spark stores data _Spark DataFrames_ for structured data,
+and in _Resilient Distributed Datasets_ (RDD) for unstructured data.
+We are using RDDs for this guide.
+
+A _Spark RDD_ represents a collection of elements,
+while in Beam it's called a _Parallel Collection_ (PCollection).
+A PCollection in Beam does _not_ have any ordering guarantees.
+
+Likewise, a transform in Beam is called a _Parallel Transform_ (PTransform).
+
+Here are some examples of common operations and their equivalent between PySpark and Beam.
+
+## Overview
+
+Here's a simple example of a PySpark pipeline that takes the numbers from one to four,
+multiplies them by two, adds all the values together, and prints the result.
+
+{{< highlight py >}}
+import pyspark
+
+sc = pyspark.SparkContext()
+result = (
+ sc.parallelize([1, 2, 3, 4])
+ .map(lambda x: x * 2)
+ .reduce(lambda x, y: x + y)
+)
+print(result)
+{{< /highlight >}}
+
+In Beam you _pipe_ your data through the pipeline using the
+_pipe operator_ `|` like `data | beam.Map(...)` instead of chaining
+methods like `data.map(...)`, but they're doing the same thing.
+
+Here's how an equivalent pipeline looks like in Beam.
Review comment:
Done
##########
File path: website/www/site/content/en/get-started/from-spark.md
##########
@@ -0,0 +1,261 @@
+---
+title: "Getting started from Apache Spark"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Getting started from Apache Spark
Review comment:
Got it, thanks
##########
File path: website/www/site/content/en/get-started/from-spark.md
##########
@@ -0,0 +1,261 @@
+---
+title: "Getting started from Apache Spark"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Getting started from Apache Spark
+
+{{< localstorage language language-py >}}
+
+If you already know [_Apache Spark_](http://spark.apache.org/),
+learning _Apache Beam_ is easy.
+The Beam and Spark APIs are similar, so you already know the basic concepts.
+
+Spark stores data _Spark DataFrames_ for structured data,
+and in _Resilient Distributed Datasets_ (RDD) for unstructured data.
+We are using RDDs for this guide.
+
+A _Spark RDD_ represents a collection of elements,
+while in Beam it's called a _Parallel Collection_ (PCollection).
+A PCollection in Beam does _not_ have any ordering guarantees.
+
+Likewise, a transform in Beam is called a _Parallel Transform_ (PTransform).
+
+Here are some examples of common operations and their equivalent between PySpark and Beam.
+
+## Overview
+
+Here's a simple example of a PySpark pipeline that takes the numbers from one to four,
+multiplies them by two, adds all the values together, and prints the result.
+
+{{< highlight py >}}
+import pyspark
+
+sc = pyspark.SparkContext()
+result = (
+ sc.parallelize([1, 2, 3, 4])
+ .map(lambda x: x * 2)
+ .reduce(lambda x, y: x + y)
+)
+print(result)
+{{< /highlight >}}
+
+In Beam you _pipe_ your data through the pipeline using the
+_pipe operator_ `|` like `data | beam.Map(...)` instead of chaining
+methods like `data.map(...)`, but they're doing the same thing.
+
+Here's how an equivalent pipeline looks like in Beam.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | beam.Create([1, 2, 3, 4])
+ | beam.Map(lambda x: x * 2)
+ | beam.CombineGlobally(sum)
+ | beam.Map(print)
+ )
+{{< /highlight >}}
+
+> ℹ️ Note that we called `print` inside a `Map` transform.
+> That's because we can only access the elements of a PCollection
+> from within a PTransform.
+
+Another thing to note is that Beam pipelines are constructed _lazily_.
+This means that when you _pipe_ `|` data you're only _declaring_ the
+transformations and the order you want them to happen,
+but the actual computation doesn't happen.
+The pipeline is run _after_ the `with beam.Pipeline() as pipeline` context has
+closed.
+The pipeline is then sent to your runner of choice and it processes the data.
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bradmiro commented on pull request #12963: [BEAM-10983] Add getting started from Spark page
Posted by GitBox <gi...@apache.org>.
bradmiro commented on pull request #12963:
URL: https://github.com/apache/beam/pull/12963#issuecomment-700873996
This looks great! A couple notes: I haven't commonly seen **with** blocks used in examples for the sparkContext. Though it should technically work, I might remove it in favor of just `sc = spark.SparkContext()`.
You may also want to make some mention of the Spark `DataFrame` [API](https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrame), at least in the introduction. This is technically newer and in many cases superior to the RDD API, and where a lot of users would be coming from.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #12963: [BEAM-10983] Add getting started from Spark page
Posted by GitBox <gi...@apache.org>.
ibzib commented on pull request #12963:
URL: https://github.com/apache/beam/pull/12963#issuecomment-712982490
> What kind of mention do you think would be useful?
I'm fine with just a clarification up front that we are using RDDs in this tutorial.
> Does Beam have any sort of similar split, or would you want to use PCollections in both cases?
Beam lets you assign schemas to PCollections. But they probably need their own how-to guide first; they don't seem to be well-documented yet.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidcavazos commented on pull request #12963: [BEAM-10983] Add getting started from Spark page
Posted by GitBox <gi...@apache.org>.
davidcavazos commented on pull request #12963:
URL: https://github.com/apache/beam/pull/12963#issuecomment-744559192
Run Website_Stage_GCS
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidcavazos commented on pull request #12963: [BEAM-10983] Add getting started from Spark page
Posted by GitBox <gi...@apache.org>.
davidcavazos commented on pull request #12963:
URL: https://github.com/apache/beam/pull/12963#issuecomment-712953191
> You may also want to make some mention of the Spark `DataFrame` [API](https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrame), at least in the introduction. This is technically newer and in many cases superior to the RDD API, and where a lot of users would be coming from.
What kind of mention do you think would be useful? I'm not familiar with Spark nor Spark Dataframes. I was looking at the API and it looks like it has about the same functionality, but some method names are different, and it supports SQL. Right now BeamSQL is only available in Java so we won't mention it here for the time being.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on a change in pull request #12963: [BEAM-10983] Add getting started from Spark page
Posted by GitBox <gi...@apache.org>.
ibzib commented on a change in pull request #12963:
URL: https://github.com/apache/beam/pull/12963#discussion_r497018651
##########
File path: website/www/site/content/en/get-started/from-spark.md
##########
@@ -0,0 +1,245 @@
+---
+title: "Getting started from Apache Spark"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Getting started from Apache Spark
+
+{{< localstorage language language-py >}}
+
+If you already know [_Apache Spark_](http://spark.apache.org/),
+learning _Apache Beam_ is easy.
+Beam and Spark are mostly equivalent, so you already know the basic concepts.
Review comment:
The Beam and Spark *APIs* are mostly equivalent. The backend stuff, not so much.
I might reword this as, "The Beam and Spark APIs are similar."
##########
File path: website/www/site/content/en/get-started/from-spark.md
##########
@@ -0,0 +1,245 @@
+---
+title: "Getting started from Apache Spark"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Getting started from Apache Spark
+
+{{< localstorage language language-py >}}
+
+If you already know [_Apache Spark_](http://spark.apache.org/),
+learning _Apache Beam_ is easy.
+Beam and Spark are mostly equivalent, so you already know the basic concepts.
+
+A collection of elements in Spark is called a _Resilient Distributed Dataset_ (RDD),
+while in Beam it's called a _Parallel Collection_ (PCollection).
+A PCollection in Beam does _not_ have any ordering guarantees.
+
+Likewise, a transform in Beam is called a _Parallel Transform_ (PTransform).
+
+Here are some examples of common operations and their equivalent between PySpark and Beam.
+
+## Overview
+
+Here's a simple example of a PySpark pipeline that takes the numbers from one to four,
+multiplies them by two, adds all the values together, and prints the result.
+
+{{< highlight py >}}
+import pyspark
+
+with pyspark.SparkContext() as sc:
+ result = (
+ sc.parallelize([1, 2, 3, 4])
+ .map(lambda x: x * 2)
+ .reduce(lambda x, y: x + y)
+ )
+ print(result)
+{{< /highlight >}}
+
+In Beam you _pipe_ your data through the pipeline using the
Review comment:
It might be worth mentioning a couple other important details here:
1. Beam pipelines are constructed lazily, in other words, no computation happens until `pipeline.run()` is called.
2. `with beam.Pipeline() as pipeline` implicitly calls `pipeline.run()`.
##########
File path: website/www/site/content/en/get-started/from-spark.md
##########
@@ -0,0 +1,245 @@
+---
+title: "Getting started from Apache Spark"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Getting started from Apache Spark
+
+{{< localstorage language language-py >}}
+
+If you already know [_Apache Spark_](http://spark.apache.org/),
+learning _Apache Beam_ is easy.
+Beam and Spark are mostly equivalent, so you already know the basic concepts.
+
+A collection of elements in Spark is called a _Resilient Distributed Dataset_ (RDD),
+while in Beam it's called a _Parallel Collection_ (PCollection).
+A PCollection in Beam does _not_ have any ordering guarantees.
+
+Likewise, a transform in Beam is called a _Parallel Transform_ (PTransform).
+
+Here are some examples of common operations and their equivalent between PySpark and Beam.
+
+## Overview
+
+Here's a simple example of a PySpark pipeline that takes the numbers from one to four,
+multiplies them by two, adds all the values together, and prints the result.
+
+{{< highlight py >}}
+import pyspark
+
+with pyspark.SparkContext() as sc:
+ result = (
+ sc.parallelize([1, 2, 3, 4])
+ .map(lambda x: x * 2)
+ .reduce(lambda x, y: x + y)
+ )
+ print(result)
+{{< /highlight >}}
+
+In Beam you _pipe_ your data through the pipeline using the
+_pipe operator_ `|` like `data | beam.Map(...)` instead of chaining
+methods like `data.map(...)`, but they're doing the same thing.
+
+Here's how an equivalent pipeline looks like in Beam.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | beam.Create([1, 2, 3, 4])
+ | beam.Map(lambda x: x * 2)
+ | beam.CombineGlobally(sum)
+ | beam.Map(print)
+ )
+{{< /highlight >}}
+
+> ℹ️ Note that we called `print` inside a `Map` transform.
+> That's because we can only access the elements of a PCollection
+> from within a PTransform.
+
+A label can optionally be added to a transform using the
+_right shift operator_ `>>` like `data | 'My description' >> beam.Map(...)`.
+This serves both as comments and makes your pipeline easier to debug.
+
+This is how the pipeline looks after adding labels.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | 'Create numbers' >> beam.Create([1, 2, 3, 4])
+ | 'Multiply by two' >> beam.Map(lambda x: x * 2)
+ | 'Sum everything' >> beam.CombineGlobally(sum)
+ | beam.Map(print)
+ )
+{{< /highlight >}}
+
+## Setup
+
+Here's a comparison on how to get started both in PySpark and Beam.
+
+{{< table >}}
+<table>
+<tr>
+ <th></th>
+ <th>PySpark</th>
+ <th>Beam</th>
+</tr>
+<tr>
+ <td><b>Install</b></td>
+ <td><code>$ pip install pyspark</code></td>
+ <td><code>$ pip install apache-beam</code></td>
+</tr>
+<tr>
+ <td><b>Imports</b></td>
+ <td><code>import pyspark</code></td>
+ <td><code>import apache_beam as beam</code></td>
+</tr>
+<tr>
+ <td><b>Creating a<br>local pipeline</b></td>
+ <td>
+ <code>with pyspark.SparkContext() as sc:</code><br>
+ <code> # Your pipeline code here.</code>
+ </td>
+ <td>
+ <code>with beam.Pipeline() as pipeline:</code><br>
+ <code> # Your pipeline code here.</code>
+ </td>
+</tr>
+<tr>
+ <td><b>Creating values</b></td>
+ <td><code>values = sc.parallelize([1, 2, 3, 4])</code></td>
+ <td><code>values = pipeline | beam.Create([1, 2, 3, 4])</code></td>
+</tr>
+<tr>
+ <td><b>Creating<br>key-value pairs</b></td>
+ <td>
+ <code>pairs = sc.parallelize([</code><br>
+ <code> ('key1', 'value1'),</code><br>
+ <code> ('key2', 'value2'),</code><br>
+ <code> ('key3', 'value3'),</code><br>
+ <code>])</code>
+ </td>
+ <td>
+ <code>pairs = pipeline | beam.Create([</code><br>
+ <code> ('key1', 'value1'),</code><br>
+ <code> ('key2', 'value2'),</code><br>
+ <code> ('key3', 'value3'),</code><br>
+ <code>])</code>
+ </td>
+</tr>
+<tr>
+ <td><b>Running a<br>local pipeline</b></td>
+ <td><code>$ spark-submit spark_pipeline.py</code></td>
+ <td><code>$ python beam_pipeline.py</code></td>
+</tr>
+</table>
+{{< /table >}}
+
+## Transforms
+
+Here are the equivalents of some common transforms in both PySpark and Beam.
+
+{{< table >}}
+| | PySpark | Beam |
+|-------------------|---------------------------------------|---------------------------------------------------------|
+| **Map** | `values.map(lambda x: x * 2)` | `values | beam.Map(lambda x: x * 2)` |
+| **Filter** | `values.filter(lambda x: x % 2 == 0)` | `values | beam.Filter(lambda x: x % 2 == 0)` |
+| **FlatMap** | `values.flatMap(lambda x: range(x))` | `values | beam.FlatMap(lambda x: range(x))` |
+| **Group by key** | `pairs.groupByKey()` | `pairs | beam.GroupByKey()` |
+| **Reduce** | `values.reduce(lambda x, y: x+y)` | `values | beam.CombineGlobally(sum)` |
+| **Reduce by key** | `pairs.reduceByKey(lambda x, y: x+y)` | `pairs | beam.CombinePerKey(sum)` |
+| **Distinct** | `values.distinct()` | `values | beam.Distinct()` |
+| **Count** | `values.count()` | `values | beam.combiners.Count.Globally()` |
+| **Count by key** | `pairs.countByKey()` | `pairs | beam.combiners.Count.PerKey()` |
+| **Take smallest** | `values.takeOrdered(3)` | `values | beam.combiners.Top.Smallest(3)` |
+| **Take largest** | `values.takeOrdered(3, lambda x: -x)` | `values | beam.combiners.Top.Largest(3)` |
+| **Random sample** | `values.takeSample(False, 3)` | `values | beam.combiners.Sample.FixedSizeGlobally(3)` |
+| **Union** | `values.union(otherValues)` | `(values, otherValues) | beam.Flatten()` |
+| **Co-group** | `pairs.cogroup(otherPairs)` | `{'Xs': pairs, 'Ys': otherPairs} | beam.CoGroupByKey()` |
+{{< /table >}}
+
+> ℹ️ To learn more about the transforms available in Beam, check the
+> [Python transform gallery](/documentation/transforms/python/overview).
+
+## Using calculated values
+
+Since we are working in potentially distributed environments,
+we can't guarantee that the results we've calculated are available at any given machine.
+
+In PySpark, we can get a result from a collection of elements (RDD) by using
+`data.collect()`, or other aggregations such as `reduce()`, `count()` and more.
+
+Here's an example to scale numbers into a range between zero and one.
+
+{{< highlight py >}}
+import pyspark
+
+with pyspark.SparkContext() as sc:
+ values = sc.parallelize([1, 2, 3, 4])
+ total = values.reduce(lambda x, y: x + y)
+
+ # We can simply use `total` since it's already a Python value from `reduce`.
+ scaled_values = values.map(lambda x: x / total)
+
+ # But to access `scaled_values`, we need to call `collect`.
+ print(scaled_values.collect())
+{{< /highlight >}}
+
+In Beam the results from _all_ transforms result in a PCollection.
+We use _side inputs_ to feed a PCollection into a transform and access its values.
Review comment:
Link to `/documentation/programming-guide/#side-inputs`.
##########
File path: website/www/site/content/en/get-started/from-spark.md
##########
@@ -0,0 +1,245 @@
+---
+title: "Getting started from Apache Spark"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Getting started from Apache Spark
+
+{{< localstorage language language-py >}}
+
+If you already know [_Apache Spark_](http://spark.apache.org/),
+learning _Apache Beam_ is easy.
+Beam and Spark are mostly equivalent, so you already know the basic concepts.
+
+A collection of elements in Spark is called a _Resilient Distributed Dataset_ (RDD),
+while in Beam it's called a _Parallel Collection_ (PCollection).
+A PCollection in Beam does _not_ have any ordering guarantees.
+
+Likewise, a transform in Beam is called a _Parallel Transform_ (PTransform).
+
+Here are some examples of common operations and their equivalent between PySpark and Beam.
+
+## Overview
+
+Here's a simple example of a PySpark pipeline that takes the numbers from one to four,
+multiplies them by two, adds all the values together, and prints the result.
+
+{{< highlight py >}}
+import pyspark
+
+with pyspark.SparkContext() as sc:
+ result = (
+ sc.parallelize([1, 2, 3, 4])
+ .map(lambda x: x * 2)
+ .reduce(lambda x, y: x + y)
+ )
+ print(result)
+{{< /highlight >}}
+
+In Beam you _pipe_ your data through the pipeline using the
+_pipe operator_ `|` like `data | beam.Map(...)` instead of chaining
+methods like `data.map(...)`, but they're doing the same thing.
+
+Here's how an equivalent pipeline looks like in Beam.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | beam.Create([1, 2, 3, 4])
+ | beam.Map(lambda x: x * 2)
+ | beam.CombineGlobally(sum)
+ | beam.Map(print)
+ )
+{{< /highlight >}}
+
+> ℹ️ Note that we called `print` inside a `Map` transform.
+> That's because we can only access the elements of a PCollection
+> from within a PTransform.
+
+A label can optionally be added to a transform using the
+_right shift operator_ `>>` like `data | 'My description' >> beam.Map(...)`.
+This serves both as comments and makes your pipeline easier to debug.
+
+This is how the pipeline looks after adding labels.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | 'Create numbers' >> beam.Create([1, 2, 3, 4])
+ | 'Multiply by two' >> beam.Map(lambda x: x * 2)
+ | 'Sum everything' >> beam.CombineGlobally(sum)
+ | beam.Map(print)
+ )
+{{< /highlight >}}
+
+## Setup
+
+Here's a comparison on how to get started both in PySpark and Beam.
+
+{{< table >}}
+<table>
+<tr>
+ <th></th>
+ <th>PySpark</th>
+ <th>Beam</th>
+</tr>
+<tr>
+ <td><b>Install</b></td>
+ <td><code>$ pip install pyspark</code></td>
+ <td><code>$ pip install apache-beam</code></td>
+</tr>
+<tr>
+ <td><b>Imports</b></td>
+ <td><code>import pyspark</code></td>
+ <td><code>import apache_beam as beam</code></td>
+</tr>
+<tr>
+ <td><b>Creating a<br>local pipeline</b></td>
+ <td>
+ <code>with pyspark.SparkContext() as sc:</code><br>
+ <code> # Your pipeline code here.</code>
+ </td>
+ <td>
+ <code>with beam.Pipeline() as pipeline:</code><br>
+ <code> # Your pipeline code here.</code>
+ </td>
+</tr>
+<tr>
+ <td><b>Creating values</b></td>
+ <td><code>values = sc.parallelize([1, 2, 3, 4])</code></td>
+ <td><code>values = pipeline | beam.Create([1, 2, 3, 4])</code></td>
+</tr>
+<tr>
+ <td><b>Creating<br>key-value pairs</b></td>
+ <td>
+ <code>pairs = sc.parallelize([</code><br>
+ <code> ('key1', 'value1'),</code><br>
+ <code> ('key2', 'value2'),</code><br>
+ <code> ('key3', 'value3'),</code><br>
+ <code>])</code>
+ </td>
+ <td>
+ <code>pairs = pipeline | beam.Create([</code><br>
+ <code> ('key1', 'value1'),</code><br>
+ <code> ('key2', 'value2'),</code><br>
+ <code> ('key3', 'value3'),</code><br>
+ <code>])</code>
+ </td>
+</tr>
+<tr>
+ <td><b>Running a<br>local pipeline</b></td>
+ <td><code>$ spark-submit spark_pipeline.py</code></td>
+ <td><code>$ python beam_pipeline.py</code></td>
+</tr>
+</table>
+{{< /table >}}
+
+## Transforms
+
+Here are the equivalents of some common transforms in both PySpark and Beam.
+
+{{< table >}}
+| | PySpark | Beam |
+|-------------------|---------------------------------------|---------------------------------------------------------|
+| **Map** | `values.map(lambda x: x * 2)` | `values | beam.Map(lambda x: x * 2)` |
+| **Filter** | `values.filter(lambda x: x % 2 == 0)` | `values | beam.Filter(lambda x: x % 2 == 0)` |
+| **FlatMap** | `values.flatMap(lambda x: range(x))` | `values | beam.FlatMap(lambda x: range(x))` |
+| **Group by key** | `pairs.groupByKey()` | `pairs | beam.GroupByKey()` |
+| **Reduce** | `values.reduce(lambda x, y: x+y)` | `values | beam.CombineGlobally(sum)` |
+| **Reduce by key** | `pairs.reduceByKey(lambda x, y: x+y)` | `pairs | beam.CombinePerKey(sum)` |
+| **Distinct** | `values.distinct()` | `values | beam.Distinct()` |
+| **Count** | `values.count()` | `values | beam.combiners.Count.Globally()` |
+| **Count by key** | `pairs.countByKey()` | `pairs | beam.combiners.Count.PerKey()` |
+| **Take smallest** | `values.takeOrdered(3)` | `values | beam.combiners.Top.Smallest(3)` |
+| **Take largest** | `values.takeOrdered(3, lambda x: -x)` | `values | beam.combiners.Top.Largest(3)` |
+| **Random sample** | `values.takeSample(False, 3)` | `values | beam.combiners.Sample.FixedSizeGlobally(3)` |
+| **Union** | `values.union(otherValues)` | `(values, otherValues) | beam.Flatten()` |
+| **Co-group** | `pairs.cogroup(otherPairs)` | `{'Xs': pairs, 'Ys': otherPairs} | beam.CoGroupByKey()` |
+{{< /table >}}
+
+> ℹ️ To learn more about the transforms available in Beam, check the
+> [Python transform gallery](/documentation/transforms/python/overview).
+
+## Using calculated values
+
+Since we are working in potentially distributed environments,
+we can't guarantee that the results we've calculated are available at any given machine.
+
+In PySpark, we can get a result from a collection of elements (RDD) by using
+`data.collect()`, or other aggregations such as `reduce()`, `count()` and more.
+
+Here's an example to scale numbers into a range between zero and one.
+
+{{< highlight py >}}
+import pyspark
+
+with pyspark.SparkContext() as sc:
+ values = sc.parallelize([1, 2, 3, 4])
+ total = values.reduce(lambda x, y: x + y)
+
+ # We can simply use `total` since it's already a Python value from `reduce`.
Review comment:
Nit: clarify what you mean here by "Python value" (in this case it's an `int` value).
##########
File path: website/www/site/content/en/get-started/from-spark.md
##########
@@ -0,0 +1,245 @@
+---
+title: "Getting started from Apache Spark"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Getting started from Apache Spark
+
+{{< localstorage language language-py >}}
+
+If you already know [_Apache Spark_](http://spark.apache.org/),
+learning _Apache Beam_ is easy.
+Beam and Spark are mostly equivalent, so you already know the basic concepts.
+
+A collection of elements in Spark is called a _Resilient Distributed Dataset_ (RDD),
+while in Beam it's called a _Parallel Collection_ (PCollection).
+A PCollection in Beam does _not_ have any ordering guarantees.
+
+Likewise, a transform in Beam is called a _Parallel Transform_ (PTransform).
+
+Here are some examples of common operations and their equivalent between PySpark and Beam.
+
+## Overview
+
+Here's a simple example of a PySpark pipeline that takes the numbers from one to four,
+multiplies them by two, adds all the values together, and prints the result.
+
+{{< highlight py >}}
+import pyspark
+
+with pyspark.SparkContext() as sc:
+ result = (
+ sc.parallelize([1, 2, 3, 4])
+ .map(lambda x: x * 2)
+ .reduce(lambda x, y: x + y)
+ )
+ print(result)
+{{< /highlight >}}
+
+In Beam you _pipe_ your data through the pipeline using the
+_pipe operator_ `|` like `data | beam.Map(...)` instead of chaining
+methods like `data.map(...)`, but they're doing the same thing.
+
+Here's how an equivalent pipeline looks like in Beam.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | beam.Create([1, 2, 3, 4])
+ | beam.Map(lambda x: x * 2)
+ | beam.CombineGlobally(sum)
+ | beam.Map(print)
+ )
+{{< /highlight >}}
+
+> ℹ️ Note that we called `print` inside a `Map` transform.
+> That's because we can only access the elements of a PCollection
+> from within a PTransform.
+
+A label can optionally be added to a transform using the
+_right shift operator_ `>>` like `data | 'My description' >> beam.Map(...)`.
+This serves both as comments and makes your pipeline easier to debug.
+
+This is how the pipeline looks after adding labels.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | 'Create numbers' >> beam.Create([1, 2, 3, 4])
+ | 'Multiply by two' >> beam.Map(lambda x: x * 2)
+ | 'Sum everything' >> beam.CombineGlobally(sum)
+ | beam.Map(print)
Review comment:
Nit: should we add a label here too for consistency?
##########
File path: website/www/site/content/en/get-started/from-spark.md
##########
@@ -0,0 +1,245 @@
+---
+title: "Getting started from Apache Spark"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Getting started from Apache Spark
+
+{{< localstorage language language-py >}}
+
+If you already know [_Apache Spark_](http://spark.apache.org/),
+learning _Apache Beam_ is easy.
+Beam and Spark are mostly equivalent, so you already know the basic concepts.
+
+A collection of elements in Spark is called a _Resilient Distributed Dataset_ (RDD),
+while in Beam it's called a _Parallel Collection_ (PCollection).
+A PCollection in Beam does _not_ have any ordering guarantees.
+
+Likewise, a transform in Beam is called a _Parallel Transform_ (PTransform).
+
+Here are some examples of common operations and their equivalent between PySpark and Beam.
+
+## Overview
+
+Here's a simple example of a PySpark pipeline that takes the numbers from one to four,
+multiplies them by two, adds all the values together, and prints the result.
+
+{{< highlight py >}}
+import pyspark
+
+with pyspark.SparkContext() as sc:
+ result = (
+ sc.parallelize([1, 2, 3, 4])
+ .map(lambda x: x * 2)
+ .reduce(lambda x, y: x + y)
+ )
+ print(result)
+{{< /highlight >}}
+
+In Beam you _pipe_ your data through the pipeline using the
+_pipe operator_ `|` like `data | beam.Map(...)` instead of chaining
+methods like `data.map(...)`, but they're doing the same thing.
+
+Here's how an equivalent pipeline looks like in Beam.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | beam.Create([1, 2, 3, 4])
+ | beam.Map(lambda x: x * 2)
+ | beam.CombineGlobally(sum)
+ | beam.Map(print)
+ )
+{{< /highlight >}}
+
+> ℹ️ Note that we called `print` inside a `Map` transform.
+> That's because we can only access the elements of a PCollection
+> from within a PTransform.
+
+A label can optionally be added to a transform using the
+_right shift operator_ `>>` like `data | 'My description' >> beam.Map(...)`.
+This serves both as comments and makes your pipeline easier to debug.
+
+This is how the pipeline looks after adding labels.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | 'Create numbers' >> beam.Create([1, 2, 3, 4])
+ | 'Multiply by two' >> beam.Map(lambda x: x * 2)
+ | 'Sum everything' >> beam.CombineGlobally(sum)
+ | beam.Map(print)
+ )
+{{< /highlight >}}
+
+## Setup
+
+Here's a comparison on how to get started both in PySpark and Beam.
+
+{{< table >}}
+<table>
+<tr>
+ <th></th>
+ <th>PySpark</th>
+ <th>Beam</th>
+</tr>
+<tr>
+ <td><b>Install</b></td>
+ <td><code>$ pip install pyspark</code></td>
+ <td><code>$ pip install apache-beam</code></td>
+</tr>
+<tr>
+ <td><b>Imports</b></td>
+ <td><code>import pyspark</code></td>
+ <td><code>import apache_beam as beam</code></td>
+</tr>
+<tr>
+ <td><b>Creating a<br>local pipeline</b></td>
+ <td>
+ <code>with pyspark.SparkContext() as sc:</code><br>
+ <code> # Your pipeline code here.</code>
+ </td>
+ <td>
+ <code>with beam.Pipeline() as pipeline:</code><br>
+ <code> # Your pipeline code here.</code>
+ </td>
+</tr>
+<tr>
+ <td><b>Creating values</b></td>
+ <td><code>values = sc.parallelize([1, 2, 3, 4])</code></td>
+ <td><code>values = pipeline | beam.Create([1, 2, 3, 4])</code></td>
+</tr>
+<tr>
+ <td><b>Creating<br>key-value pairs</b></td>
+ <td>
+ <code>pairs = sc.parallelize([</code><br>
+ <code> ('key1', 'value1'),</code><br>
+ <code> ('key2', 'value2'),</code><br>
+ <code> ('key3', 'value3'),</code><br>
+ <code>])</code>
+ </td>
+ <td>
+ <code>pairs = pipeline | beam.Create([</code><br>
+ <code> ('key1', 'value1'),</code><br>
Review comment:
Probably better to use lists instead of tuples in Beam. I'm not sure tuples are adequately tested/supported: https://the-asf.slack.com/archives/CBDNLQZM1/p1598454585014000
##########
File path: website/www/site/content/en/get-started/from-spark.md
##########
@@ -0,0 +1,245 @@
+---
+title: "Getting started from Apache Spark"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Getting started from Apache Spark
+
+{{< localstorage language language-py >}}
+
+If you already know [_Apache Spark_](http://spark.apache.org/),
+learning _Apache Beam_ is easy.
+Beam and Spark are mostly equivalent, so you already know the basic concepts.
+
+A collection of elements in Spark is called a _Resilient Distributed Dataset_ (RDD),
+while in Beam it's called a _Parallel Collection_ (PCollection).
+A PCollection in Beam does _not_ have any ordering guarantees.
+
+Likewise, a transform in Beam is called a _Parallel Transform_ (PTransform).
+
+Here are some examples of common operations and their equivalent between PySpark and Beam.
+
+## Overview
+
+Here's a simple example of a PySpark pipeline that takes the numbers from one to four,
+multiplies them by two, adds all the values together, and prints the result.
+
+{{< highlight py >}}
+import pyspark
+
+with pyspark.SparkContext() as sc:
+ result = (
+ sc.parallelize([1, 2, 3, 4])
+ .map(lambda x: x * 2)
+ .reduce(lambda x, y: x + y)
+ )
+ print(result)
+{{< /highlight >}}
+
+In Beam you _pipe_ your data through the pipeline using the
+_pipe operator_ `|` like `data | beam.Map(...)` instead of chaining
+methods like `data.map(...)`, but they're doing the same thing.
+
+Here's how an equivalent pipeline looks like in Beam.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | beam.Create([1, 2, 3, 4])
+ | beam.Map(lambda x: x * 2)
+ | beam.CombineGlobally(sum)
+ | beam.Map(print)
+ )
+{{< /highlight >}}
+
+> ℹ️ Note that we called `print` inside a `Map` transform.
+> That's because we can only access the elements of a PCollection
+> from within a PTransform.
+
+A label can optionally be added to a transform using the
+_right shift operator_ `>>` like `data | 'My description' >> beam.Map(...)`.
+This serves both as comments and makes your pipeline easier to debug.
+
+This is how the pipeline looks after adding labels.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | 'Create numbers' >> beam.Create([1, 2, 3, 4])
+ | 'Multiply by two' >> beam.Map(lambda x: x * 2)
+ | 'Sum everything' >> beam.CombineGlobally(sum)
+ | beam.Map(print)
+ )
+{{< /highlight >}}
+
+## Setup
+
+Here's a comparison on how to get started both in PySpark and Beam.
+
+{{< table >}}
+<table>
+<tr>
+ <th></th>
+ <th>PySpark</th>
+ <th>Beam</th>
+</tr>
+<tr>
+ <td><b>Install</b></td>
+ <td><code>$ pip install pyspark</code></td>
+ <td><code>$ pip install apache-beam</code></td>
+</tr>
+<tr>
+ <td><b>Imports</b></td>
+ <td><code>import pyspark</code></td>
+ <td><code>import apache_beam as beam</code></td>
+</tr>
+<tr>
+ <td><b>Creating a<br>local pipeline</b></td>
+ <td>
+ <code>with pyspark.SparkContext() as sc:</code><br>
+ <code> # Your pipeline code here.</code>
+ </td>
+ <td>
+ <code>with beam.Pipeline() as pipeline:</code><br>
+ <code> # Your pipeline code here.</code>
+ </td>
+</tr>
+<tr>
+ <td><b>Creating values</b></td>
+ <td><code>values = sc.parallelize([1, 2, 3, 4])</code></td>
+ <td><code>values = pipeline | beam.Create([1, 2, 3, 4])</code></td>
+</tr>
+<tr>
+ <td><b>Creating<br>key-value pairs</b></td>
+ <td>
+ <code>pairs = sc.parallelize([</code><br>
+ <code> ('key1', 'value1'),</code><br>
+ <code> ('key2', 'value2'),</code><br>
+ <code> ('key3', 'value3'),</code><br>
+ <code>])</code>
+ </td>
+ <td>
+ <code>pairs = pipeline | beam.Create([</code><br>
+ <code> ('key1', 'value1'),</code><br>
+ <code> ('key2', 'value2'),</code><br>
+ <code> ('key3', 'value3'),</code><br>
+ <code>])</code>
+ </td>
+</tr>
+<tr>
+ <td><b>Running a<br>local pipeline</b></td>
+ <td><code>$ spark-submit spark_pipeline.py</code></td>
+ <td><code>$ python beam_pipeline.py</code></td>
+</tr>
+</table>
+{{< /table >}}
+
+## Transforms
+
+Here are the equivalents of some common transforms in both PySpark and Beam.
+
+{{< table >}}
Review comment:
It'd be really nice if each Beam transform was linked to its respective `documentation/transforms` page.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidcavazos commented on pull request #12963: [BEAM-10983] Add getting started from Spark page
Posted by GitBox <gi...@apache.org>.
davidcavazos commented on pull request #12963:
URL: https://github.com/apache/beam/pull/12963#issuecomment-713056012
@rosetn I think this is ready for your review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on a change in pull request #12963: [BEAM-10983] Add getting started from Spark page
Posted by GitBox <gi...@apache.org>.
ibzib commented on a change in pull request #12963:
URL: https://github.com/apache/beam/pull/12963#discussion_r508095053
##########
File path: website/www/site/content/en/get-started/from-spark.md
##########
@@ -0,0 +1,245 @@
+---
+title: "Getting started from Apache Spark"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Getting started from Apache Spark
+
+{{< localstorage language language-py >}}
+
+If you already know [_Apache Spark_](http://spark.apache.org/),
+learning _Apache Beam_ is easy.
+Beam and Spark are mostly equivalent, so you already know the basic concepts.
+
+A collection of elements in Spark is called a _Resilient Distributed Dataset_ (RDD),
+while in Beam it's called a _Parallel Collection_ (PCollection).
+A PCollection in Beam does _not_ have any ordering guarantees.
+
+Likewise, a transform in Beam is called a _Parallel Transform_ (PTransform).
+
+Here are some examples of common operations and their equivalent between PySpark and Beam.
+
+## Overview
+
+Here's a simple example of a PySpark pipeline that takes the numbers from one to four,
+multiplies them by two, adds all the values together, and prints the result.
+
+{{< highlight py >}}
+import pyspark
+
+with pyspark.SparkContext() as sc:
+ result = (
+ sc.parallelize([1, 2, 3, 4])
+ .map(lambda x: x * 2)
+ .reduce(lambda x, y: x + y)
+ )
+ print(result)
+{{< /highlight >}}
+
+In Beam you _pipe_ your data through the pipeline using the
+_pipe operator_ `|` like `data | beam.Map(...)` instead of chaining
+methods like `data.map(...)`, but they're doing the same thing.
+
+Here's how an equivalent pipeline looks like in Beam.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | beam.Create([1, 2, 3, 4])
+ | beam.Map(lambda x: x * 2)
+ | beam.CombineGlobally(sum)
+ | beam.Map(print)
+ )
+{{< /highlight >}}
+
+> ℹ️ Note that we called `print` inside a `Map` transform.
+> That's because we can only access the elements of a PCollection
+> from within a PTransform.
+
+A label can optionally be added to a transform using the
+_right shift operator_ `>>` like `data | 'My description' >> beam.Map(...)`.
+This serves both as comments and makes your pipeline easier to debug.
+
+This is how the pipeline looks after adding labels.
+
+{{< highlight py >}}
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ result = (
+ pipeline
+ | 'Create numbers' >> beam.Create([1, 2, 3, 4])
+ | 'Multiply by two' >> beam.Map(lambda x: x * 2)
+ | 'Sum everything' >> beam.CombineGlobally(sum)
+ | beam.Map(print)
+ )
+{{< /highlight >}}
+
+## Setup
+
+Here's a comparison on how to get started both in PySpark and Beam.
+
+{{< table >}}
+<table>
+<tr>
+ <th></th>
+ <th>PySpark</th>
+ <th>Beam</th>
+</tr>
+<tr>
+ <td><b>Install</b></td>
+ <td><code>$ pip install pyspark</code></td>
+ <td><code>$ pip install apache-beam</code></td>
+</tr>
+<tr>
+ <td><b>Imports</b></td>
+ <td><code>import pyspark</code></td>
+ <td><code>import apache_beam as beam</code></td>
+</tr>
+<tr>
+ <td><b>Creating a<br>local pipeline</b></td>
+ <td>
+ <code>with pyspark.SparkContext() as sc:</code><br>
+ <code> # Your pipeline code here.</code>
+ </td>
+ <td>
+ <code>with beam.Pipeline() as pipeline:</code><br>
+ <code> # Your pipeline code here.</code>
+ </td>
+</tr>
+<tr>
+ <td><b>Creating values</b></td>
+ <td><code>values = sc.parallelize([1, 2, 3, 4])</code></td>
+ <td><code>values = pipeline | beam.Create([1, 2, 3, 4])</code></td>
+</tr>
+<tr>
+ <td><b>Creating<br>key-value pairs</b></td>
+ <td>
+ <code>pairs = sc.parallelize([</code><br>
+ <code> ('key1', 'value1'),</code><br>
+ <code> ('key2', 'value2'),</code><br>
+ <code> ('key3', 'value3'),</code><br>
+ <code>])</code>
+ </td>
+ <td>
+ <code>pairs = pipeline | beam.Create([</code><br>
+ <code> ('key1', 'value1'),</code><br>
Review comment:
> I don't have an @apache.org email so I can't open the discussion.
Anyone should be able to join the ASF slack. There is an invite link on the Beam website: https://beam.apache.org/community/contact-us/
The case I linked might be an isolated bug. If tuples have worked for you, I'm fine with keeping them in the documentation.
##########
File path: website/www/site/content/en/get-started/from-spark.md
##########
@@ -71,6 +71,17 @@ with beam.Pipeline() as pipeline:
> That's because we can only access the elements of a PCollection
> from within a PTransform.
+Another thing to note is that Beam pipelines are constructed _lazily_.
+This means that when you _pipe_ `|` data you're only _decalring_ the
Review comment:
```suggestion
This means that when you _pipe_ `|` data you're only _declaring_ the
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org