You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/09/21 18:49:37 UTC
[GitHub] [beam] davidcavazos opened a new pull request #12892: [BEAM-10937] Add first introductory notebook
davidcavazos opened a new pull request #12892:
URL: https://github.com/apache/beam/pull/12892
This is the first notebook of a new _"Tour of Beam"_ section we want to create to help improve the onboarding experience for beginners.
The plan is to have the _"Tour of Beam"_ section to include links to all the notebooks we currently have, as well as to offer a new set of introductory notebooks to explain concepts gradually in an interactive environment. The landing page will come up as following PR.
This notebook covers the following:
* Basic concepts: pipeline, `PCollection`, `PTransform`
* Explanation of the `|` and `>>` operators
* Basic transforms: `Map`, `FlatMap`, `Filter`, `Combine`, `GroupByKey`
The _"What's next?"_ section at the end of the notebook contains links to existing documentation, but will be updated once we have more notebooks for new topics.
R: @aaltay
R: @rosetn
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [x] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [x] Update `CHANGES.md` with noteworthy changes.
- [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace | Typescript
--- | --- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #12892: [BEAM-10937] Add first introductory notebook
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #12892:
URL: https://github.com/apache/beam/pull/12892#issuecomment-700370878
Could you update the staged version after your next commit?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidcavazos commented on a change in pull request #12892: [BEAM-10937] Add first introductory notebook
Posted by GitBox <gi...@apache.org>.
davidcavazos commented on a change in pull request #12892:
URL: https://github.com/apache/beam/pull/12892#discussion_r494483180
##########
File path: examples/notebooks/tour-of-beam/getting-started.ipynb
##########
@@ -0,0 +1,1021 @@
+{
+ "license": [
+ "Licensed to the Apache Software Foundation (ASF) under one",
+ "or more contributor license agreements. See the NOTICE file",
+ "distributed with this work for additional information",
+ "regarding copyright ownership. The ASF licenses this file",
+ "to you under the Apache License, Version 2.0 (the",
+ "\"License\"); you may not use this file except in compliance",
+ "with the License. You may obtain a copy of the License at",
+ "",
+ " http://www.apache.org/licenses/LICENSE-2.0",
+ "",
+ "Unless required by applicable law or agreed to in writing,",
+ "software distributed under the License is distributed on an",
+ "\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY",
+ "KIND, either express or implied. See the License for the",
+ "specific language governing permissions and limitations",
+ "under the License."
+ ],
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Getting started -- Tour of Beam",
+ "provenance": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python3",
+ "display_name": "Python 3"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/tour-of-beam/getting-started.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5UC_aGanx6oE",
+ "colab_type": "text"
+ },
+ "source": [
+ "# Getting started -- *Tour of Beam*\n",
+ "\n",
+ "[Apache Beam](https://beam.apache.org/get-started/beam-overview/)\n",
+ "is a library for parallel data processing.\n",
+ "\n",
+ "Beam is commonly used for\n",
+ "[Extract-Transform-Load (ETL)](https://en.wikipedia.org/wiki/Extract,_transform,_load)\n",
+ "jobs, where we _extract_ data from a data source, _transform_ that data, and _load_ it into a data sink like a database.\n",
+ "It does particularly well with large amounts of data since it can use mutliple machines to process everything at the same time.\n",
+ "\n",
+ "Let's begin by installing the `apache-beam` package with `pip`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "R_Yhoc6N_Flg",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "source": [
+ "# Install apache-beam with pip.\n",
+ "!pip install --quiet apache-beam"
+ ],
+ "execution_count": 25,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "WwxLB5EiVKP_",
+ "colab_type": "text"
+ },
+ "source": [
+ "You can express a _data processing pipeline_, and then run it on the\n",
+ "[_runner_ of your choice](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "For now, we use the `DirectRunner` which runs locally for simplicity."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "8rfQeLk2y9lx",
+ "colab_type": "text"
+ },
+ "source": [
+ "# What is a _pipeline_?\n",
+ "\n",
+ "A __pipeline__ is a __sequence of data transformations__.\n",
+ "You can think of it like a production line,\n",
+ "data comes in from one end,\n",
+ "it gets transformed by each step.\n",
+ "The outputs from one step are passed as inputs to the next step.\n",
+ "\n",
+ "In Beam, your data lives in a __`PCollection`__,\n",
+ "which stands for _Parallel Collection_.\n",
+ "A `PCollection` is like a __list of elements__,\n",
+ "but without any order guarantees.\n",
+ "This allows Beam to easily parallelize and distribute\n",
+ "the `PCollection`'s elements."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "81VNY478gep_",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "LaljqKvs4UU5",
+ "colab_type": "text"
+ },
+ "source": [
+ "Once you have your data, the next step is to transform it.\n",
+ "In Beam, you transform data using **`PTransform`**s,\n",
+ "which stands for _Parallel Transform_.\n",
+ "A `PTransform` is like a __function__,\n",
+ "they take some inputs, transform them and create some outputs."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Me_hvd6XgtPC",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "SVk4BsuKVlLj",
+ "colab_type": "text"
+ },
+ "source": [
+ "Now let's dive into creating our first pipeline.\n",
+ "\n",
+ "For this first pipeline, let's just feed it some data from a Python list and print the results.\n",
+ "\n",
+ "Each _step_ in the pipeline is delimited by the _pipe operator_ `|`.\n",
+ "The outputs of each transform are passed to the next transform as inputs.\n",
+ "And we can save the final results into a `PCollection` variable.\n",
+ "\n",
+ "```py\n",
+ "# We pass the elements from step1 through step3 and save the results into `outputs`.\n",
+ "outputs = pipeline | step1 | step2 | step3\n",
+ "```\n",
+ "\n",
+ "Pipelines can quickly grow long, so it's sometimes easier to read if we surround them with parentheses and break them into multiple lines.\n",
+ "\n",
+ "```py\n",
+ "# This is equivalent to the example above.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | step1\n",
+ " | step2\n",
+ " | step3\n",
+ ")\n",
+ "```\n",
+ "\n",
+ "Also, Beam expects each transform, or step, to have a unique _label_, or description.\n",
+ "This makes it a lot easier to debug, and it's in general a good practice to start.\n",
+ "You can use the _right shift operator_ `>>` to add a label to your transforms, like `'My description' >> MyTransform`.\n",
+ "\n",
+ "```py\n",
+ "# Try to give short but descriptive labels.\n",
+ "# These serve both as comments and help debug later on.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | 'First step' >> step1\n",
+ " | 'Second step' >> step2\n",
+ " | 'Third step' >> step3\n",
+ ")\n",
+ "```\n",
+ "\n",
+ "> ℹ️ The syntax might seem a little strange at first, but you'll get used to it.\n",
+ "> Once you're familiar with it, you'll see it's actually easy to read and write.\n",
+ "\n",
+ "We use the `Create` transform to feed the pipeline with an\n",
+ "[`iterable`](https://docs.python.org/3/glossary.html#term-iterable)\n",
+ "of elements, like a `list`.\n",
+ "\n",
+ "> ℹ️ _Note:_ in Beam, you can __NOT__ access the elements from a `PCollection` directly like a Python list.\n",
+ "> This means, we can't simply `print` the output `PCollection` to see the elements.\n",
+ ">\n",
+ "> This is because, depending on the runner,\n",
+ "> the `PCollection` elements might live in multiple worker machines.\n",
+ "\n",
+ "We'll do a little trick to print the elements, but we'll explain it shortly."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "XAke4d5lV5f5",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "outputId": "8e032211-c70a-4a89-9904-58d6fa2c8cb7"
+ },
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "inputs = [0, 1, 2, 3]\n",
+ "\n",
+ "# Create a pipeline.\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " # Feed it some input elements with `Create`.\n",
+ " outputs = (\n",
+ " pipeline\n",
+ " | 'Create initial values' >> beam.Create(inputs)\n",
+ " )\n",
+ "\n",
+ " # `outputs` is a PCollection with our input elements.\n",
+ " # But printing it directly won't show us its contents :(\n",
+ " print(f\"printing won't show us the elements: {outputs}\\n-----\\n\")\n",
+ "\n",
+ " # We can only access the elements through another transform.\n",
+ " # Don't worry if you don't know what's happening here,\n",
+ " # we'll get to it just next :)\n",
+ " outputs | beam.Map(print)"
+ ],
+ "execution_count": 26,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "printing won't show us the elements: PCollection[[26]: Create initial values/Map(decode).None]\n",
+ "-----\n",
+ "\n",
+ "0\n",
+ "1\n",
+ "2\n",
+ "3\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "FY9TP3Tw5eZt",
+ "colab_type": "text"
+ },
+ "source": [
+ "# Transforming data\n",
+ "\n",
+ "Apache Beam is designed with a [functional paradigm](https://en.wikipedia.org/wiki/Functional_programming).\n",
+ "This means that, instead of _loops_, it uses `PTransform`s alongside with _functions_ to process each element in a `PCollection`.\n",
+ "\n",
+ "Let's go through some of the most common and basic data transforms in Beam."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "UMKgi9LD6-xb",
+ "colab_type": "text"
+ },
+ "source": [
+ "## Map -- *one to one*\n",
+ "\n",
+ "Let's say we have some elements and we want to do something with each element.\n",
+ "\n",
+ "We want to `map` a function to each element of the collection.\n",
+ "\n",
+ "`map` takes a _function_ that transforms a single input `a` into a single output `b`.\n",
+ "\n",
+ "> 🐌 -- For example, we want to multiply each element by 2."
Review comment:
It was just a cute emoji, but that makes sense. I'm replacing them with ▶️
##########
File path: examples/notebooks/tour-of-beam/getting-started.ipynb
##########
@@ -0,0 +1,1021 @@
+{
+ "license": [
+ "Licensed to the Apache Software Foundation (ASF) under one",
+ "or more contributor license agreements. See the NOTICE file",
+ "distributed with this work for additional information",
+ "regarding copyright ownership. The ASF licenses this file",
+ "to you under the Apache License, Version 2.0 (the",
+ "\"License\"); you may not use this file except in compliance",
+ "with the License. You may obtain a copy of the License at",
+ "",
+ " http://www.apache.org/licenses/LICENSE-2.0",
+ "",
+ "Unless required by applicable law or agreed to in writing,",
+ "software distributed under the License is distributed on an",
+ "\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY",
+ "KIND, either express or implied. See the License for the",
+ "specific language governing permissions and limitations",
+ "under the License."
+ ],
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Getting started -- Tour of Beam",
+ "provenance": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python3",
+ "display_name": "Python 3"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/tour-of-beam/getting-started.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5UC_aGanx6oE",
+ "colab_type": "text"
+ },
+ "source": [
+ "# Getting started -- *Tour of Beam*\n",
+ "\n",
+ "[Apache Beam](https://beam.apache.org/get-started/beam-overview/)\n",
+ "is a library for parallel data processing.\n",
+ "\n",
+ "Beam is commonly used for\n",
+ "[Extract-Transform-Load (ETL)](https://en.wikipedia.org/wiki/Extract,_transform,_load)\n",
+ "jobs, where we _extract_ data from a data source, _transform_ that data, and _load_ it into a data sink like a database.\n",
+ "It does particularly well with large amounts of data since it can use mutliple machines to process everything at the same time.\n",
+ "\n",
+ "Let's begin by installing the `apache-beam` package with `pip`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "R_Yhoc6N_Flg",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "source": [
+ "# Install apache-beam with pip.\n",
+ "!pip install --quiet apache-beam"
+ ],
+ "execution_count": 25,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "WwxLB5EiVKP_",
+ "colab_type": "text"
+ },
+ "source": [
+ "You can express a _data processing pipeline_, and then run it on the\n",
+ "[_runner_ of your choice](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "For now, we use the `DirectRunner` which runs locally for simplicity."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "8rfQeLk2y9lx",
+ "colab_type": "text"
+ },
+ "source": [
+ "# What is a _pipeline_?\n",
+ "\n",
+ "A __pipeline__ is a __sequence of data transformations__.\n",
+ "You can think of it like a production line,\n",
+ "data comes in from one end,\n",
+ "it gets transformed by each step.\n",
+ "The outputs from one step are passed as inputs to the next step.\n",
+ "\n",
+ "In Beam, your data lives in a __`PCollection`__,\n",
+ "which stands for _Parallel Collection_.\n",
+ "A `PCollection` is like a __list of elements__,\n",
+ "but without any order guarantees.\n",
+ "This allows Beam to easily parallelize and distribute\n",
+ "the `PCollection`'s elements."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "81VNY478gep_",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "LaljqKvs4UU5",
+ "colab_type": "text"
+ },
+ "source": [
+ "Once you have your data, the next step is to transform it.\n",
+ "In Beam, you transform data using **`PTransform`**s,\n",
+ "which stands for _Parallel Transform_.\n",
+ "A `PTransform` is like a __function__,\n",
+ "they take some inputs, transform them and create some outputs."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Me_hvd6XgtPC",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "SVk4BsuKVlLj",
+ "colab_type": "text"
+ },
+ "source": [
+ "Now let's dive into creating our first pipeline.\n",
+ "\n",
+ "For this first pipeline, let's just feed it some data from a Python list and print the results.\n",
+ "\n",
+ "Each _step_ in the pipeline is delimited by the _pipe operator_ `|`.\n",
+ "The outputs of each transform are passed to the next transform as inputs.\n",
+ "And we can save the final results into a `PCollection` variable.\n",
+ "\n",
+ "```py\n",
+ "# We pass the elements from step1 through step3 and save the results into `outputs`.\n",
+ "outputs = pipeline | step1 | step2 | step3\n",
+ "```\n",
+ "\n",
+ "Pipelines can quickly grow long, so it's sometimes easier to read if we surround them with parentheses and break them into multiple lines.\n",
+ "\n",
+ "```py\n",
+ "# This is equivalent to the example above.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | step1\n",
+ " | step2\n",
+ " | step3\n",
+ ")\n",
+ "```\n",
+ "\n",
+ "Also, Beam expects each transform, or step, to have a unique _label_, or description.\n",
+ "This makes it a lot easier to debug, and it's in general a good practice to start.\n",
+ "You can use the _right shift operator_ `>>` to add a label to your transforms, like `'My description' >> MyTransform`.\n",
+ "\n",
+ "```py\n",
+ "# Try to give short but descriptive labels.\n",
+ "# These serve both as comments and help debug later on.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | 'First step' >> step1\n",
+ " | 'Second step' >> step2\n",
+ " | 'Third step' >> step3\n",
+ ")\n",
+ "```\n",
+ "\n",
+ "> ℹ️ The syntax might seem a little strange at first, but you'll get used to it.\n",
+ "> Once you're familiar with it, you'll see it's actually easy to read and write.\n",
+ "\n",
+ "We use the `Create` transform to feed the pipeline with an\n",
+ "[`iterable`](https://docs.python.org/3/glossary.html#term-iterable)\n",
+ "of elements, like a `list`.\n",
+ "\n",
+ "> ℹ️ _Note:_ in Beam, you can __NOT__ access the elements from a `PCollection` directly like a Python list.\n",
+ "> This means, we can't simply `print` the output `PCollection` to see the elements.\n",
+ ">\n",
+ "> This is because, depending on the runner,\n",
+ "> the `PCollection` elements might live in multiple worker machines.\n",
+ "\n",
+ "We'll do a little trick to print the elements, but we'll explain it shortly."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "XAke4d5lV5f5",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "outputId": "8e032211-c70a-4a89-9904-58d6fa2c8cb7"
+ },
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "inputs = [0, 1, 2, 3]\n",
+ "\n",
+ "# Create a pipeline.\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " # Feed it some input elements with `Create`.\n",
+ " outputs = (\n",
+ " pipeline\n",
+ " | 'Create initial values' >> beam.Create(inputs)\n",
+ " )\n",
+ "\n",
+ " # `outputs` is a PCollection with our input elements.\n",
+ " # But printing it directly won't show us its contents :(\n",
+ " print(f\"printing won't show us the elements: {outputs}\\n-----\\n\")\n",
+ "\n",
+ " # We can only access the elements through another transform.\n",
+ " # Don't worry if you don't know what's happening here,\n",
+ " # we'll get to it just next :)\n",
+ " outputs | beam.Map(print)"
+ ],
+ "execution_count": 26,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "printing won't show us the elements: PCollection[[26]: Create initial values/Map(decode).None]\n",
+ "-----\n",
+ "\n",
+ "0\n",
+ "1\n",
+ "2\n",
+ "3\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "FY9TP3Tw5eZt",
+ "colab_type": "text"
+ },
+ "source": [
+ "# Transforming data\n",
+ "\n",
+ "Apache Beam is designed with a [functional paradigm](https://en.wikipedia.org/wiki/Functional_programming).\n",
+ "This means that, instead of _loops_, it uses `PTransform`s alongside with _functions_ to process each element in a `PCollection`.\n",
+ "\n",
+ "Let's go through some of the most common and basic data transforms in Beam."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "UMKgi9LD6-xb",
+ "colab_type": "text"
+ },
+ "source": [
+ "## Map -- *one to one*\n",
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #12892: [BEAM-10937] Add first introductory notebook
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #12892:
URL: https://github.com/apache/beam/pull/12892#issuecomment-696371040
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rosetn commented on a change in pull request #12892: [BEAM-10937] Add first introductory notebook
Posted by GitBox <gi...@apache.org>.
rosetn commented on a change in pull request #12892:
URL: https://github.com/apache/beam/pull/12892#discussion_r495124669
##########
File path: examples/notebooks/tour-of-beam/getting-started.ipynb
##########
@@ -0,0 +1,1020 @@
+{
+ "license": [
+ "Licensed to the Apache Software Foundation (ASF) under one",
+ "or more contributor license agreements. See the NOTICE file",
+ "distributed with this work for additional information",
+ "regarding copyright ownership. The ASF licenses this file",
+ "to you under the Apache License, Version 2.0 (the",
+ "\"License\"); you may not use this file except in compliance",
+ "with the License. You may obtain a copy of the License at",
+ "",
+ " http://www.apache.org/licenses/LICENSE-2.0",
+ "",
+ "Unless required by applicable law or agreed to in writing,",
+ "software distributed under the License is distributed on an",
+ "\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY",
+ "KIND, either express or implied. See the License for the",
+ "specific language governing permissions and limitations",
+ "under the License."
+ ],
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Getting started -- Tour of Beam",
Review comment:
Nit: For each header in the notebook, can you replace these em dashes with spaces around them, " -- ", with a colon? https://developers.google.com/style/dashes#colons-instead-of-dashes-in-lists
##########
File path: examples/notebooks/tour-of-beam/getting-started.ipynb
##########
@@ -0,0 +1,1020 @@
+{
+ "license": [
+ "Licensed to the Apache Software Foundation (ASF) under one",
+ "or more contributor license agreements. See the NOTICE file",
+ "distributed with this work for additional information",
+ "regarding copyright ownership. The ASF licenses this file",
+ "to you under the Apache License, Version 2.0 (the",
+ "\"License\"); you may not use this file except in compliance",
+ "with the License. You may obtain a copy of the License at",
+ "",
+ " http://www.apache.org/licenses/LICENSE-2.0",
+ "",
+ "Unless required by applicable law or agreed to in writing,",
+ "software distributed under the License is distributed on an",
+ "\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY",
+ "KIND, either express or implied. See the License for the",
+ "specific language governing permissions and limitations",
+ "under the License."
+ ],
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Getting started -- Tour of Beam",
+ "provenance": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python3",
+ "display_name": "Python 3"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/tour-of-beam/getting-started.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5UC_aGanx6oE",
+ "colab_type": "text"
+ },
+ "source": [
+ "# Getting started -- *Tour of Beam*\n",
+ "\n",
+ "[Apache Beam](https://beam.apache.org/get-started/beam-overview/)\n",
+ "is a library for parallel data processing.\n",
+ "\n",
+ "Beam is commonly used for\n",
+ "[Extract-Transform-Load (ETL)](https://en.wikipedia.org/wiki/Extract,_transform,_load)\n",
+ "jobs, where we _extract_ data from a data source, _transform_ that data, and _load_ it into a data sink like a database.\n",
+ "It does particularly well with large amounts of data since it can use mutliple machines to process everything at the same time.\n",
+ "\n",
+ "Let's begin by installing the `apache-beam` package with `pip`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "R_Yhoc6N_Flg",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "source": [
+ "# Install apache-beam with pip.\n",
+ "!pip install --quiet apache-beam"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "WwxLB5EiVKP_",
+ "colab_type": "text"
+ },
+ "source": [
+ "You can express a _data processing pipeline_, and then run it on the\n",
+ "[_runner_ of your choice](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "For now, we use the `DirectRunner` which runs locally for simplicity."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "8rfQeLk2y9lx",
+ "colab_type": "text"
+ },
+ "source": [
+ "# What is a _pipeline_?\n",
+ "\n",
+ "A __pipeline__ is a __sequence of data transformations__.\n",
+ "You can think of it like a production line,\n",
+ "data comes in from one end,\n",
+ "it gets transformed by each step.\n",
+ "The outputs from one step are passed as inputs to the next step.\n",
+ "\n",
+ "In Beam, your data lives in a __`PCollection`__,\n",
+ "which stands for _Parallel Collection_.\n",
+ "A `PCollection` is like a __list of elements__,\n",
+ "but without any order guarantees.\n",
+ "This allows Beam to easily parallelize and distribute\n",
+ "the `PCollection`'s elements."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "81VNY478gep_",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "LaljqKvs4UU5",
+ "colab_type": "text"
+ },
+ "source": [
+ "Once you have your data, the next step is to transform it.\n",
+ "In Beam, you transform data using **`PTransform`**s,\n",
+ "which stands for _Parallel Transform_.\n",
+ "A `PTransform` is like a __function__,\n",
+ "they take some inputs, transform them and create some outputs."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Me_hvd6XgtPC",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "SVk4BsuKVlLj",
+ "colab_type": "text"
+ },
+ "source": [
+ "Now let's dive into creating our first pipeline.\n",
+ "\n",
+ "For this first pipeline, let's just feed it some data from a Python list and print the results.\n",
+ "\n",
+ "Each _step_ in the pipeline is delimited by the _pipe operator_ `|`.\n",
+ "The outputs of each transform are passed to the next transform as inputs.\n",
+ "And we can save the final results into a `PCollection` variable.\n",
+ "\n",
+ "```py\n",
+ "# We pass the elements from step1 through step3 and save the results into `outputs`.\n",
+ "outputs = pipeline | step1 | step2 | step3\n",
+ "```\n",
+ "\n",
+ "Pipelines can quickly grow long, so it's sometimes easier to read if we surround them with parentheses and break them into multiple lines.\n",
+ "\n",
+ "```py\n",
+ "# This is equivalent to the example above.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | step1\n",
+ " | step2\n",
+ " | step3\n",
+ ")\n",
+ "```\n",
+ "\n",
+ "Also, Beam expects each transform, or step, to have a unique _label_, or description.\n",
+ "This makes it a lot easier to debug, and it's in general a good practice to start.\n",
+ "You can use the _right shift operator_ `>>` to add a label to your transforms, like `'My description' >> MyTransform`.\n",
+ "\n",
+ "```py\n",
+ "# Try to give short but descriptive labels.\n",
+ "# These serve both as comments and help debug later on.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | 'First step' >> step1\n",
+ " | 'Second step' >> step2\n",
+ " | 'Third step' >> step3\n",
+ ")\n",
+ "```\n",
+ "\n",
+ "> ℹ️ The syntax might seem a little strange at first, but you'll become familiar with it.\n",
+ "\n",
+ "We use the `Create` transform to feed the pipeline with an\n",
+ "[`iterable`](https://docs.python.org/3/glossary.html#term-iterable)\n",
+ "of elements, like a `list`.\n",
+ "\n",
+ "> ℹ️ In Beam, you can __NOT__ access the elements from a `PCollection` directly like a Python list.\n",
Review comment:
Great note. This is a large source of confusion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #12892: [BEAM-10937] Add first introductory notebook
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #12892:
URL: https://github.com/apache/beam/pull/12892#issuecomment-696371040
@rosetn could you make the first review pass?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on a change in pull request #12892: [BEAM-10937] Add first introductory notebook
Posted by GitBox <gi...@apache.org>.
aaltay commented on a change in pull request #12892:
URL: https://github.com/apache/beam/pull/12892#discussion_r495353278
##########
File path: examples/notebooks/tour-of-beam/getting-started.ipynb
##########
@@ -0,0 +1,1020 @@
+{
+ "license": [
+ "Licensed to the Apache Software Foundation (ASF) under one",
+ "or more contributor license agreements. See the NOTICE file",
+ "distributed with this work for additional information",
+ "regarding copyright ownership. The ASF licenses this file",
+ "to you under the Apache License, Version 2.0 (the",
+ "\"License\"); you may not use this file except in compliance",
+ "with the License. You may obtain a copy of the License at",
+ "",
+ " http://www.apache.org/licenses/LICENSE-2.0",
+ "",
+ "Unless required by applicable law or agreed to in writing,",
+ "software distributed under the License is distributed on an",
+ "\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY",
+ "KIND, either express or implied. See the License for the",
+ "specific language governing permissions and limitations",
+ "under the License."
+ ],
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Getting started -- Tour of Beam",
+ "provenance": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python3",
+ "display_name": "Python 3"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/tour-of-beam/getting-started.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5UC_aGanx6oE",
+ "colab_type": "text"
+ },
+ "source": [
+ "# Getting started -- *Tour of Beam*\n",
+ "\n",
+ "[Apache Beam](https://beam.apache.org/get-started/beam-overview/)\n",
+ "is a library for parallel data processing.\n",
+ "\n",
+ "Beam is commonly used for\n",
+ "[Extract-Transform-Load (ETL)](https://en.wikipedia.org/wiki/Extract,_transform,_load)\n",
+ "jobs, where we _extract_ data from a data source, _transform_ that data, and _load_ it into a data sink like a database.\n",
+ "It does particularly well with large amounts of data since it can use mutliple machines to process everything at the same time.\n",
+ "\n",
+ "Let's begin by installing the `apache-beam` package with `pip`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "R_Yhoc6N_Flg",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "source": [
+ "# Install apache-beam with pip.\n",
+ "!pip install --quiet apache-beam"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "WwxLB5EiVKP_",
+ "colab_type": "text"
+ },
+ "source": [
+ "You can express a _data processing pipeline_, and then run it on the\n",
+ "[_runner_ of your choice](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "For now, we use the `DirectRunner` which runs locally for simplicity."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "8rfQeLk2y9lx",
+ "colab_type": "text"
+ },
+ "source": [
+ "# What is a _pipeline_?\n",
+ "\n",
+ "A __pipeline__ is a __sequence of data transformations__.\n",
+ "You can think of it like a production line,\n",
+ "data comes in from one end,\n",
+ "it gets transformed by each step.\n",
+ "The outputs from one step are passed as inputs to the next step.\n",
+ "\n",
+ "In Beam, your data lives in a __`PCollection`__,\n",
+ "which stands for _Parallel Collection_.\n",
+ "A `PCollection` is like a __list of elements__,\n",
+ "but without any order guarantees.\n",
+ "This allows Beam to easily parallelize and distribute\n",
+ "the `PCollection`'s elements."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "81VNY478gep_",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "LaljqKvs4UU5",
+ "colab_type": "text"
+ },
+ "source": [
+ "Once you have your data, the next step is to transform it.\n",
+ "In Beam, you transform data using **`PTransform`**s,\n",
+ "which stands for _Parallel Transform_.\n",
+ "A `PTransform` is like a __function__,\n",
+ "they take some inputs, transform them and create some outputs."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Me_hvd6XgtPC",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "SVk4BsuKVlLj",
+ "colab_type": "text"
+ },
+ "source": [
+ "Now let's dive into creating our first pipeline.\n",
+ "\n",
+ "For this first pipeline, let's just feed it some data from a Python list and print the results.\n",
+ "\n",
+ "Each _step_ in the pipeline is delimited by the _pipe operator_ `|`.\n",
+ "The outputs of each transform are passed to the next transform as inputs.\n",
+ "And we can save the final results into a `PCollection` variable.\n",
+ "\n",
+ "```py\n",
+ "# We pass the elements from step1 through step3 and save the results into `outputs`.\n",
+ "outputs = pipeline | step1 | step2 | step3\n",
+ "```\n",
+ "\n",
+ "Pipelines can quickly grow long, so it's sometimes easier to read if we surround them with parentheses and break them into multiple lines.\n",
+ "\n",
+ "```py\n",
+ "# This is equivalent to the example above.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | step1\n",
+ " | step2\n",
+ " | step3\n",
+ ")\n",
+ "```\n",
+ "\n",
+ "Also, Beam expects each transform, or step, to have a unique _label_, or description.\n",
+ "This makes it a lot easier to debug, and it's in general a good practice to start.\n",
+ "You can use the _right shift operator_ `>>` to add a label to your transforms, like `'My description' >> MyTransform`.\n",
+ "\n",
+ "```py\n",
+ "# Try to give short but descriptive labels.\n",
+ "# These serve both as comments and help debug later on.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | 'First step' >> step1\n",
+ " | 'Second step' >> step2\n",
+ " | 'Third step' >> step3\n",
+ ")\n",
+ "```\n",
+ "\n",
+ "> ℹ️ The syntax might seem a little strange at first, but you'll become familiar with it.\n",
Review comment:
Change "little strange" to something less judgemental? :)
##########
File path: examples/notebooks/tour-of-beam/getting-started.ipynb
##########
@@ -0,0 +1,1020 @@
+{
+ "license": [
+ "Licensed to the Apache Software Foundation (ASF) under one",
+ "or more contributor license agreements. See the NOTICE file",
+ "distributed with this work for additional information",
+ "regarding copyright ownership. The ASF licenses this file",
+ "to you under the Apache License, Version 2.0 (the",
+ "\"License\"); you may not use this file except in compliance",
+ "with the License. You may obtain a copy of the License at",
+ "",
+ " http://www.apache.org/licenses/LICENSE-2.0",
+ "",
+ "Unless required by applicable law or agreed to in writing,",
+ "software distributed under the License is distributed on an",
+ "\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY",
+ "KIND, either express or implied. See the License for the",
+ "specific language governing permissions and limitations",
+ "under the License."
+ ],
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Getting started -- Tour of Beam",
+ "provenance": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python3",
+ "display_name": "Python 3"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/tour-of-beam/getting-started.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5UC_aGanx6oE",
+ "colab_type": "text"
+ },
+ "source": [
+ "# Getting started -- *Tour of Beam*\n",
+ "\n",
+ "[Apache Beam](https://beam.apache.org/get-started/beam-overview/)\n",
+ "is a library for parallel data processing.\n",
+ "\n",
+ "Beam is commonly used for\n",
+ "[Extract-Transform-Load (ETL)](https://en.wikipedia.org/wiki/Extract,_transform,_load)\n",
+ "jobs, where we _extract_ data from a data source, _transform_ that data, and _load_ it into a data sink like a database.\n",
+ "It does particularly well with large amounts of data since it can use mutliple machines to process everything at the same time.\n",
+ "\n",
+ "Let's begin by installing the `apache-beam` package with `pip`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "R_Yhoc6N_Flg",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "source": [
+ "# Install apache-beam with pip.\n",
+ "!pip install --quiet apache-beam"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "WwxLB5EiVKP_",
+ "colab_type": "text"
+ },
+ "source": [
+ "You can express a _data processing pipeline_, and then run it on the\n",
+ "[_runner_ of your choice](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "For now, we use the `DirectRunner` which runs locally for simplicity."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "8rfQeLk2y9lx",
+ "colab_type": "text"
+ },
+ "source": [
+ "# What is a _pipeline_?\n",
+ "\n",
+ "A __pipeline__ is a __sequence of data transformations__.\n",
+ "You can think of it like a production line,\n",
+ "data comes in from one end,\n",
+ "it gets transformed by each step.\n",
+ "The outputs from one step are passed as inputs to the next step.\n",
+ "\n",
+ "In Beam, your data lives in a __`PCollection`__,\n",
+ "which stands for _Parallel Collection_.\n",
+ "A `PCollection` is like a __list of elements__,\n",
+ "but without any order guarantees.\n",
+ "This allows Beam to easily parallelize and distribute\n",
+ "the `PCollection`'s elements."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "81VNY478gep_",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "LaljqKvs4UU5",
+ "colab_type": "text"
+ },
+ "source": [
+ "Once you have your data, the next step is to transform it.\n",
+ "In Beam, you transform data using **`PTransform`**s,\n",
+ "which stands for _Parallel Transform_.\n",
+ "A `PTransform` is like a __function__,\n",
+ "they take some inputs, transform them and create some outputs."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Me_hvd6XgtPC",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "SVk4BsuKVlLj",
+ "colab_type": "text"
+ },
+ "source": [
+ "Now let's dive into creating our first pipeline.\n",
+ "\n",
+ "For this first pipeline, let's just feed it some data from a Python list and print the results.\n",
+ "\n",
+ "Each _step_ in the pipeline is delimited by the _pipe operator_ `|`.\n",
+ "The outputs of each transform are passed to the next transform as inputs.\n",
+ "And we can save the final results into a `PCollection` variable.\n",
+ "\n",
+ "```py\n",
+ "# We pass the elements from step1 through step3 and save the results into `outputs`.\n",
+ "outputs = pipeline | step1 | step2 | step3\n",
+ "```\n",
+ "\n",
+ "Pipelines can quickly grow long, so it's sometimes easier to read if we surround them with parentheses and break them into multiple lines.\n",
+ "\n",
+ "```py\n",
+ "# This is equivalent to the example above.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | step1\n",
+ " | step2\n",
+ " | step3\n",
+ ")\n",
+ "```\n",
+ "\n",
+ "Also, Beam expects each transform, or step, to have a unique _label_, or description.\n",
+ "This makes it a lot easier to debug, and it's in general a good practice to start.\n",
+ "You can use the _right shift operator_ `>>` to add a label to your transforms, like `'My description' >> MyTransform`.\n",
+ "\n",
+ "```py\n",
+ "# Try to give short but descriptive labels.\n",
+ "# These serve both as comments and help debug later on.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | 'First step' >> step1\n",
+ " | 'Second step' >> step2\n",
+ " | 'Third step' >> step3\n",
+ ")\n",
+ "```\n",
+ "\n",
+ "> ℹ️ The syntax might seem a little strange at first, but you'll become familiar with it.\n",
+ "\n",
+ "We use the `Create` transform to feed the pipeline with an\n",
+ "[`iterable`](https://docs.python.org/3/glossary.html#term-iterable)\n",
+ "of elements, like a `list`.\n",
+ "\n",
+ "> ℹ️ In Beam, you can __NOT__ access the elements from a `PCollection` directly like a Python list.\n",
+ "> This means, we can't simply `print` the output `PCollection` to see the elements.\n",
+ ">\n",
+ "> This is because, depending on the runner,\n",
+ "> the `PCollection` elements might live in multiple worker machines.\n",
+ "\n",
+ "We'll do a little trick to print the elements, but we'll explain it shortly."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "XAke4d5lV5f5",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "outputId": "8e032211-c70a-4a89-9904-58d6fa2c8cb7"
+ },
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "inputs = [0, 1, 2, 3]\n",
+ "\n",
+ "# Create a pipeline.\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " # Feed it some input elements with `Create`.\n",
+ " outputs = (\n",
+ " pipeline\n",
+ " | 'Create initial values' >> beam.Create(inputs)\n",
+ " )\n",
+ "\n",
+ " # `outputs` is a PCollection with our input elements.\n",
+ " # But printing it directly won't show us its contents :(\n",
+ " print(f\"printing won't show us the elements: {outputs}\\n-----\\n\")\n",
+ "\n",
+ " # We can only access the elements through another transform.\n",
+ " # Don't worry if you don't know what's happening here,\n",
+ " # we'll get to it just next :)\n",
+ " outputs | beam.Map(print)"
+ ],
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "printing won't show us the elements: PCollection[[26]: Create initial values/Map(decode).None]\n",
Review comment:
This is a bit confusing here because the output is literally right under. Maybe break it into 2 separate execution boxes?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #12892: [BEAM-10937] Add first introductory notebook
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #12892:
URL: https://github.com/apache/beam/pull/12892#issuecomment-696371040
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay merged pull request #12892: [BEAM-10937] Add first introductory notebook
Posted by GitBox <gi...@apache.org>.
aaltay merged pull request #12892:
URL: https://github.com/apache/beam/pull/12892
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on a change in pull request #12892: [BEAM-10937] Add first introductory notebook
Posted by GitBox <gi...@apache.org>.
aaltay commented on a change in pull request #12892:
URL: https://github.com/apache/beam/pull/12892#discussion_r496323074
##########
File path: examples/notebooks/tour-of-beam/getting-started.ipynb
##########
@@ -0,0 +1,1020 @@
+{
+ "license": [
+ "Licensed to the Apache Software Foundation (ASF) under one",
+ "or more contributor license agreements. See the NOTICE file",
+ "distributed with this work for additional information",
+ "regarding copyright ownership. The ASF licenses this file",
+ "to you under the Apache License, Version 2.0 (the",
+ "\"License\"); you may not use this file except in compliance",
+ "with the License. You may obtain a copy of the License at",
+ "",
+ " http://www.apache.org/licenses/LICENSE-2.0",
+ "",
+ "Unless required by applicable law or agreed to in writing,",
+ "software distributed under the License is distributed on an",
+ "\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY",
+ "KIND, either express or implied. See the License for the",
+ "specific language governing permissions and limitations",
+ "under the License."
+ ],
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Getting started -- Tour of Beam",
+ "provenance": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python3",
+ "display_name": "Python 3"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/tour-of-beam/getting-started.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5UC_aGanx6oE",
+ "colab_type": "text"
+ },
+ "source": [
+ "# Getting started -- *Tour of Beam*\n",
+ "\n",
+ "[Apache Beam](https://beam.apache.org/get-started/beam-overview/)\n",
+ "is a library for parallel data processing.\n",
+ "\n",
+ "Beam is commonly used for\n",
+ "[Extract-Transform-Load (ETL)](https://en.wikipedia.org/wiki/Extract,_transform,_load)\n",
+ "jobs, where we _extract_ data from a data source, _transform_ that data, and _load_ it into a data sink like a database.\n",
+ "It does particularly well with large amounts of data since it can use mutliple machines to process everything at the same time.\n",
+ "\n",
+ "Let's begin by installing the `apache-beam` package with `pip`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "R_Yhoc6N_Flg",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "source": [
+ "# Install apache-beam with pip.\n",
+ "!pip install --quiet apache-beam"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "WwxLB5EiVKP_",
+ "colab_type": "text"
+ },
+ "source": [
+ "You can express a _data processing pipeline_, and then run it on the\n",
+ "[_runner_ of your choice](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "For now, we use the `DirectRunner` which runs locally for simplicity."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "8rfQeLk2y9lx",
+ "colab_type": "text"
+ },
+ "source": [
+ "# What is a _pipeline_?\n",
+ "\n",
+ "A __pipeline__ is a __sequence of data transformations__.\n",
+ "You can think of it like a production line,\n",
+ "data comes in from one end,\n",
+ "it gets transformed by each step.\n",
+ "The outputs from one step are passed as inputs to the next step.\n",
+ "\n",
+ "In Beam, your data lives in a __`PCollection`__,\n",
+ "which stands for _Parallel Collection_.\n",
+ "A `PCollection` is like a __list of elements__,\n",
+ "but without any order guarantees.\n",
+ "This allows Beam to easily parallelize and distribute\n",
+ "the `PCollection`'s elements."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "81VNY478gep_",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "LaljqKvs4UU5",
+ "colab_type": "text"
+ },
+ "source": [
+ "Once you have your data, the next step is to transform it.\n",
+ "In Beam, you transform data using **`PTransform`**s,\n",
+ "which stands for _Parallel Transform_.\n",
+ "A `PTransform` is like a __function__,\n",
+ "they take some inputs, transform them and create some outputs."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Me_hvd6XgtPC",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "SVk4BsuKVlLj",
+ "colab_type": "text"
+ },
+ "source": [
+ "Now let's dive into creating our first pipeline.\n",
+ "\n",
+ "For this first pipeline, let's just feed it some data from a Python list and print the results.\n",
+ "\n",
+ "Each _step_ in the pipeline is delimited by the _pipe operator_ `|`.\n",
+ "The outputs of each transform are passed to the next transform as inputs.\n",
+ "And we can save the final results into a `PCollection` variable.\n",
+ "\n",
+ "```py\n",
+ "# We pass the elements from step1 through step3 and save the results into `outputs`.\n",
+ "outputs = pipeline | step1 | step2 | step3\n",
+ "```\n",
+ "\n",
+ "Pipelines can quickly grow long, so it's sometimes easier to read if we surround them with parentheses and break them into multiple lines.\n",
+ "\n",
+ "```py\n",
+ "# This is equivalent to the example above.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | step1\n",
+ " | step2\n",
+ " | step3\n",
+ ")\n",
+ "```\n",
+ "\n",
+ "Also, Beam expects each transform, or step, to have a unique _label_, or description.\n",
+ "This makes it a lot easier to debug, and it's in general a good practice to start.\n",
+ "You can use the _right shift operator_ `>>` to add a label to your transforms, like `'My description' >> MyTransform`.\n",
+ "\n",
+ "```py\n",
+ "# Try to give short but descriptive labels.\n",
+ "# These serve both as comments and help debug later on.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | 'First step' >> step1\n",
+ " | 'Second step' >> step2\n",
+ " | 'Third step' >> step3\n",
+ ")\n",
+ "```\n",
+ "\n",
+ "> ℹ️ The syntax might seem a little strange at first, but you'll become familiar with it.\n",
Review comment:
"a little different" - SGTM.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rosetn commented on pull request #12892: [BEAM-10937] Add first introductory notebook
Posted by GitBox <gi...@apache.org>.
rosetn commented on pull request #12892:
URL: https://github.com/apache/beam/pull/12892#issuecomment-699061670
@aaltay I'm done with my review
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidcavazos commented on pull request #12892: [BEAM-10937] Add first introductory notebook
Posted by GitBox <gi...@apache.org>.
davidcavazos commented on pull request #12892:
URL: https://github.com/apache/beam/pull/12892#issuecomment-699031148
I've updated the link to the staged version to the latest commit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rosetn commented on a change in pull request #12892: [BEAM-10937] Add first introductory notebook
Posted by GitBox <gi...@apache.org>.
rosetn commented on a change in pull request #12892:
URL: https://github.com/apache/beam/pull/12892#discussion_r493754999
##########
File path: examples/notebooks/tour-of-beam/getting-started.ipynb
##########
@@ -0,0 +1,1021 @@
+{
+ "license": [
+ "Licensed to the Apache Software Foundation (ASF) under one",
+ "or more contributor license agreements. See the NOTICE file",
+ "distributed with this work for additional information",
+ "regarding copyright ownership. The ASF licenses this file",
+ "to you under the Apache License, Version 2.0 (the",
+ "\"License\"); you may not use this file except in compliance",
+ "with the License. You may obtain a copy of the License at",
+ "",
+ " http://www.apache.org/licenses/LICENSE-2.0",
+ "",
+ "Unless required by applicable law or agreed to in writing,",
+ "software distributed under the License is distributed on an",
+ "\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY",
+ "KIND, either express or implied. See the License for the",
+ "specific language governing permissions and limitations",
+ "under the License."
+ ],
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Getting started -- Tour of Beam",
+ "provenance": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python3",
+ "display_name": "Python 3"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/tour-of-beam/getting-started.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5UC_aGanx6oE",
+ "colab_type": "text"
+ },
+ "source": [
+ "# Getting started -- *Tour of Beam*\n",
+ "\n",
+ "[Apache Beam](https://beam.apache.org/get-started/beam-overview/)\n",
+ "is a library for parallel data processing.\n",
+ "\n",
+ "Beam is commonly used for\n",
+ "[Extract-Transform-Load (ETL)](https://en.wikipedia.org/wiki/Extract,_transform,_load)\n",
+ "jobs, where we _extract_ data from a data source, _transform_ that data, and _load_ it into a data sink like a database.\n",
+ "It does particularly well with large amounts of data since it can use mutliple machines to process everything at the same time.\n",
+ "\n",
+ "Let's begin by installing the `apache-beam` package with `pip`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "R_Yhoc6N_Flg",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "source": [
+ "# Install apache-beam with pip.\n",
+ "!pip install --quiet apache-beam"
+ ],
+ "execution_count": 25,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "WwxLB5EiVKP_",
+ "colab_type": "text"
+ },
+ "source": [
+ "You can express a _data processing pipeline_, and then run it on the\n",
+ "[_runner_ of your choice](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "For now, we use the `DirectRunner` which runs locally for simplicity."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "8rfQeLk2y9lx",
+ "colab_type": "text"
+ },
+ "source": [
+ "# What is a _pipeline_?\n",
+ "\n",
+ "A __pipeline__ is a __sequence of data transformations__.\n",
+ "You can think of it like a production line,\n",
+ "data comes in from one end,\n",
+ "it gets transformed by each step.\n",
+ "The outputs from one step are passed as inputs to the next step.\n",
+ "\n",
+ "In Beam, your data lives in a __`PCollection`__,\n",
+ "which stands for _Parallel Collection_.\n",
+ "A `PCollection` is like a __list of elements__,\n",
+ "but without any order guarantees.\n",
+ "This allows Beam to easily parallelize and distribute\n",
+ "the `PCollection`'s elements."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "81VNY478gep_",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "LaljqKvs4UU5",
+ "colab_type": "text"
+ },
+ "source": [
+ "Once you have your data, the next step is to transform it.\n",
+ "In Beam, you transform data using **`PTransform`**s,\n",
+ "which stands for _Parallel Transform_.\n",
+ "A `PTransform` is like a __function__,\n",
+ "they take some inputs, transform them and create some outputs."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Me_hvd6XgtPC",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "SVk4BsuKVlLj",
+ "colab_type": "text"
+ },
+ "source": [
+ "Now let's dive into creating our first pipeline.\n",
+ "\n",
+ "For this first pipeline, let's just feed it some data from a Python list and print the results.\n",
+ "\n",
+ "Each _step_ in the pipeline is delimited by the _pipe operator_ `|`.\n",
+ "The outputs of each transform are passed to the next transform as inputs.\n",
+ "And we can save the final results into a `PCollection` variable.\n",
+ "\n",
+ "```py\n",
+ "# We pass the elements from step1 through step3 and save the results into `outputs`.\n",
+ "outputs = pipeline | step1 | step2 | step3\n",
+ "```\n",
+ "\n",
+ "Pipelines can quickly grow long, so it's sometimes easier to read if we surround them with parentheses and break them into multiple lines.\n",
+ "\n",
+ "```py\n",
+ "# This is equivalent to the example above.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | step1\n",
+ " | step2\n",
+ " | step3\n",
+ ")\n",
+ "```\n",
+ "\n",
+ "Also, Beam expects each transform, or step, to have a unique _label_, or description.\n",
+ "This makes it a lot easier to debug, and it's in general a good practice to start.\n",
+ "You can use the _right shift operator_ `>>` to add a label to your transforms, like `'My description' >> MyTransform`.\n",
+ "\n",
+ "```py\n",
+ "# Try to give short but descriptive labels.\n",
+ "# These serve both as comments and help debug later on.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | 'First step' >> step1\n",
+ " | 'Second step' >> step2\n",
+ " | 'Third step' >> step3\n",
+ ")\n",
+ "```\n",
+ "\n",
+ "> ℹ️ The syntax might seem a little strange at first, but you'll get used to it.\n",
+ "> Once you're familiar with it, you'll see it's actually easy to read and write.\n",
Review comment:
Caution against using "easy" to make the content more inclusive: https://developers.google.com/style/word-list#letter-e
Maybe: "The syntax might seem a little strange at first, but you'll become familiar with it."
##########
File path: examples/notebooks/tour-of-beam/getting-started.ipynb
##########
@@ -0,0 +1,1021 @@
+{
+ "license": [
+ "Licensed to the Apache Software Foundation (ASF) under one",
+ "or more contributor license agreements. See the NOTICE file",
+ "distributed with this work for additional information",
+ "regarding copyright ownership. The ASF licenses this file",
+ "to you under the Apache License, Version 2.0 (the",
+ "\"License\"); you may not use this file except in compliance",
+ "with the License. You may obtain a copy of the License at",
+ "",
+ " http://www.apache.org/licenses/LICENSE-2.0",
+ "",
+ "Unless required by applicable law or agreed to in writing,",
+ "software distributed under the License is distributed on an",
+ "\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY",
+ "KIND, either express or implied. See the License for the",
+ "specific language governing permissions and limitations",
+ "under the License."
+ ],
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Getting started -- Tour of Beam",
+ "provenance": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python3",
+ "display_name": "Python 3"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/tour-of-beam/getting-started.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5UC_aGanx6oE",
+ "colab_type": "text"
+ },
+ "source": [
+ "# Getting started -- *Tour of Beam*\n",
+ "\n",
+ "[Apache Beam](https://beam.apache.org/get-started/beam-overview/)\n",
+ "is a library for parallel data processing.\n",
+ "\n",
+ "Beam is commonly used for\n",
+ "[Extract-Transform-Load (ETL)](https://en.wikipedia.org/wiki/Extract,_transform,_load)\n",
+ "jobs, where we _extract_ data from a data source, _transform_ that data, and _load_ it into a data sink like a database.\n",
+ "It does particularly well with large amounts of data since it can use mutliple machines to process everything at the same time.\n",
+ "\n",
+ "Let's begin by installing the `apache-beam` package with `pip`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "R_Yhoc6N_Flg",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "source": [
+ "# Install apache-beam with pip.\n",
+ "!pip install --quiet apache-beam"
+ ],
+ "execution_count": 25,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "WwxLB5EiVKP_",
+ "colab_type": "text"
+ },
+ "source": [
+ "You can express a _data processing pipeline_, and then run it on the\n",
+ "[_runner_ of your choice](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "For now, we use the `DirectRunner` which runs locally for simplicity."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "8rfQeLk2y9lx",
+ "colab_type": "text"
+ },
+ "source": [
+ "# What is a _pipeline_?\n",
+ "\n",
+ "A __pipeline__ is a __sequence of data transformations__.\n",
+ "You can think of it like a production line,\n",
+ "data comes in from one end,\n",
+ "it gets transformed by each step.\n",
+ "The outputs from one step are passed as inputs to the next step.\n",
+ "\n",
+ "In Beam, your data lives in a __`PCollection`__,\n",
+ "which stands for _Parallel Collection_.\n",
+ "A `PCollection` is like a __list of elements__,\n",
+ "but without any order guarantees.\n",
+ "This allows Beam to easily parallelize and distribute\n",
+ "the `PCollection`'s elements."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "81VNY478gep_",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "LaljqKvs4UU5",
+ "colab_type": "text"
+ },
+ "source": [
+ "Once you have your data, the next step is to transform it.\n",
+ "In Beam, you transform data using **`PTransform`**s,\n",
+ "which stands for _Parallel Transform_.\n",
+ "A `PTransform` is like a __function__,\n",
+ "they take some inputs, transform them and create some outputs."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Me_hvd6XgtPC",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "SVk4BsuKVlLj",
+ "colab_type": "text"
+ },
+ "source": [
+ "Now let's dive into creating our first pipeline.\n",
+ "\n",
+ "For this first pipeline, let's just feed it some data from a Python list and print the results.\n",
+ "\n",
+ "Each _step_ in the pipeline is delimited by the _pipe operator_ `|`.\n",
+ "The outputs of each transform are passed to the next transform as inputs.\n",
+ "And we can save the final results into a `PCollection` variable.\n",
+ "\n",
+ "```py\n",
+ "# We pass the elements from step1 through step3 and save the results into `outputs`.\n",
+ "outputs = pipeline | step1 | step2 | step3\n",
+ "```\n",
+ "\n",
+ "Pipelines can quickly grow long, so it's sometimes easier to read if we surround them with parentheses and break them into multiple lines.\n",
+ "\n",
+ "```py\n",
+ "# This is equivalent to the example above.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | step1\n",
+ " | step2\n",
+ " | step3\n",
+ ")\n",
+ "```\n",
+ "\n",
+ "Also, Beam expects each transform, or step, to have a unique _label_, or description.\n",
+ "This makes it a lot easier to debug, and it's in general a good practice to start.\n",
+ "You can use the _right shift operator_ `>>` to add a label to your transforms, like `'My description' >> MyTransform`.\n",
+ "\n",
+ "```py\n",
+ "# Try to give short but descriptive labels.\n",
+ "# These serve both as comments and help debug later on.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | 'First step' >> step1\n",
+ " | 'Second step' >> step2\n",
+ " | 'Third step' >> step3\n",
+ ")\n",
+ "```\n",
+ "\n",
+ "> ℹ️ The syntax might seem a little strange at first, but you'll get used to it.\n",
+ "> Once you're familiar with it, you'll see it's actually easy to read and write.\n",
+ "\n",
+ "We use the `Create` transform to feed the pipeline with an\n",
+ "[`iterable`](https://docs.python.org/3/glossary.html#term-iterable)\n",
+ "of elements, like a `list`.\n",
+ "\n",
+ "> ℹ️ _Note:_ in Beam, you can __NOT__ access the elements from a `PCollection` directly like a Python list.\n",
+ "> This means, we can't simply `print` the output `PCollection` to see the elements.\n",
+ ">\n",
+ "> This is because, depending on the runner,\n",
+ "> the `PCollection` elements might live in multiple worker machines.\n",
+ "\n",
+ "We'll do a little trick to print the elements, but we'll explain it shortly."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "XAke4d5lV5f5",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "outputId": "8e032211-c70a-4a89-9904-58d6fa2c8cb7"
+ },
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "inputs = [0, 1, 2, 3]\n",
+ "\n",
+ "# Create a pipeline.\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " # Feed it some input elements with `Create`.\n",
+ " outputs = (\n",
+ " pipeline\n",
+ " | 'Create initial values' >> beam.Create(inputs)\n",
+ " )\n",
+ "\n",
+ " # `outputs` is a PCollection with our input elements.\n",
+ " # But printing it directly won't show us its contents :(\n",
+ " print(f\"printing won't show us the elements: {outputs}\\n-----\\n\")\n",
+ "\n",
+ " # We can only access the elements through another transform.\n",
+ " # Don't worry if you don't know what's happening here,\n",
+ " # we'll get to it just next :)\n",
+ " outputs | beam.Map(print)"
+ ],
+ "execution_count": 26,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "printing won't show us the elements: PCollection[[26]: Create initial values/Map(decode).None]\n",
+ "-----\n",
+ "\n",
+ "0\n",
+ "1\n",
+ "2\n",
+ "3\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "FY9TP3Tw5eZt",
+ "colab_type": "text"
+ },
+ "source": [
+ "# Transforming data\n",
+ "\n",
+ "Apache Beam is designed with a [functional paradigm](https://en.wikipedia.org/wiki/Functional_programming).\n",
+ "This means that, instead of _loops_, it uses `PTransform`s alongside with _functions_ to process each element in a `PCollection`.\n",
+ "\n",
+ "Let's go through some of the most common and basic data transforms in Beam."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "UMKgi9LD6-xb",
+ "colab_type": "text"
+ },
+ "source": [
+ "## Map -- *one to one*\n",
Review comment:
one-to-one
##########
File path: examples/notebooks/tour-of-beam/getting-started.ipynb
##########
@@ -0,0 +1,1021 @@
+{
+ "license": [
+ "Licensed to the Apache Software Foundation (ASF) under one",
+ "or more contributor license agreements. See the NOTICE file",
+ "distributed with this work for additional information",
+ "regarding copyright ownership. The ASF licenses this file",
+ "to you under the Apache License, Version 2.0 (the",
+ "\"License\"); you may not use this file except in compliance",
+ "with the License. You may obtain a copy of the License at",
+ "",
+ " http://www.apache.org/licenses/LICENSE-2.0",
+ "",
+ "Unless required by applicable law or agreed to in writing,",
+ "software distributed under the License is distributed on an",
+ "\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY",
+ "KIND, either express or implied. See the License for the",
+ "specific language governing permissions and limitations",
+ "under the License."
+ ],
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Getting started -- Tour of Beam",
+ "provenance": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python3",
+ "display_name": "Python 3"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/tour-of-beam/getting-started.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5UC_aGanx6oE",
+ "colab_type": "text"
+ },
+ "source": [
+ "# Getting started -- *Tour of Beam*\n",
+ "\n",
+ "[Apache Beam](https://beam.apache.org/get-started/beam-overview/)\n",
+ "is a library for parallel data processing.\n",
+ "\n",
+ "Beam is commonly used for\n",
+ "[Extract-Transform-Load (ETL)](https://en.wikipedia.org/wiki/Extract,_transform,_load)\n",
+ "jobs, where we _extract_ data from a data source, _transform_ that data, and _load_ it into a data sink like a database.\n",
+ "It does particularly well with large amounts of data since it can use mutliple machines to process everything at the same time.\n",
+ "\n",
+ "Let's begin by installing the `apache-beam` package with `pip`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "R_Yhoc6N_Flg",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "source": [
+ "# Install apache-beam with pip.\n",
+ "!pip install --quiet apache-beam"
+ ],
+ "execution_count": 25,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "WwxLB5EiVKP_",
+ "colab_type": "text"
+ },
+ "source": [
+ "You can express a _data processing pipeline_, and then run it on the\n",
+ "[_runner_ of your choice](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "For now, we use the `DirectRunner` which runs locally for simplicity."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "8rfQeLk2y9lx",
+ "colab_type": "text"
+ },
+ "source": [
+ "# What is a _pipeline_?\n",
+ "\n",
+ "A __pipeline__ is a __sequence of data transformations__.\n",
+ "You can think of it like a production line,\n",
+ "data comes in from one end,\n",
+ "it gets transformed by each step.\n",
+ "The outputs from one step are passed as inputs to the next step.\n",
+ "\n",
+ "In Beam, your data lives in a __`PCollection`__,\n",
+ "which stands for _Parallel Collection_.\n",
+ "A `PCollection` is like a __list of elements__,\n",
+ "but without any order guarantees.\n",
+ "This allows Beam to easily parallelize and distribute\n",
+ "the `PCollection`'s elements."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "81VNY478gep_",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "LaljqKvs4UU5",
+ "colab_type": "text"
+ },
+ "source": [
+ "Once you have your data, the next step is to transform it.\n",
+ "In Beam, you transform data using **`PTransform`**s,\n",
+ "which stands for _Parallel Transform_.\n",
+ "A `PTransform` is like a __function__,\n",
+ "they take some inputs, transform them and create some outputs."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Me_hvd6XgtPC",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "SVk4BsuKVlLj",
+ "colab_type": "text"
+ },
+ "source": [
+ "Now let's dive into creating our first pipeline.\n",
+ "\n",
+ "For this first pipeline, let's just feed it some data from a Python list and print the results.\n",
+ "\n",
+ "Each _step_ in the pipeline is delimited by the _pipe operator_ `|`.\n",
+ "The outputs of each transform are passed to the next transform as inputs.\n",
+ "And we can save the final results into a `PCollection` variable.\n",
+ "\n",
+ "```py\n",
+ "# We pass the elements from step1 through step3 and save the results into `outputs`.\n",
+ "outputs = pipeline | step1 | step2 | step3\n",
+ "```\n",
+ "\n",
+ "Pipelines can quickly grow long, so it's sometimes easier to read if we surround them with parentheses and break them into multiple lines.\n",
+ "\n",
+ "```py\n",
+ "# This is equivalent to the example above.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | step1\n",
+ " | step2\n",
+ " | step3\n",
+ ")\n",
+ "```\n",
+ "\n",
+ "Also, Beam expects each transform, or step, to have a unique _label_, or description.\n",
+ "This makes it a lot easier to debug, and it's in general a good practice to start.\n",
+ "You can use the _right shift operator_ `>>` to add a label to your transforms, like `'My description' >> MyTransform`.\n",
+ "\n",
+ "```py\n",
+ "# Try to give short but descriptive labels.\n",
+ "# These serve both as comments and help debug later on.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | 'First step' >> step1\n",
+ " | 'Second step' >> step2\n",
+ " | 'Third step' >> step3\n",
+ ")\n",
+ "```\n",
+ "\n",
+ "> ℹ️ The syntax might seem a little strange at first, but you'll get used to it.\n",
+ "> Once you're familiar with it, you'll see it's actually easy to read and write.\n",
+ "\n",
+ "We use the `Create` transform to feed the pipeline with an\n",
+ "[`iterable`](https://docs.python.org/3/glossary.html#term-iterable)\n",
+ "of elements, like a `list`.\n",
+ "\n",
+ "> ℹ️ _Note:_ in Beam, you can __NOT__ access the elements from a `PCollection` directly like a Python list.\n",
+ "> This means, we can't simply `print` the output `PCollection` to see the elements.\n",
+ ">\n",
+ "> This is because, depending on the runner,\n",
+ "> the `PCollection` elements might live in multiple worker machines.\n",
+ "\n",
+ "We'll do a little trick to print the elements, but we'll explain it shortly."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "XAke4d5lV5f5",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "outputId": "8e032211-c70a-4a89-9904-58d6fa2c8cb7"
+ },
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "inputs = [0, 1, 2, 3]\n",
+ "\n",
+ "# Create a pipeline.\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " # Feed it some input elements with `Create`.\n",
+ " outputs = (\n",
+ " pipeline\n",
+ " | 'Create initial values' >> beam.Create(inputs)\n",
+ " )\n",
+ "\n",
+ " # `outputs` is a PCollection with our input elements.\n",
+ " # But printing it directly won't show us its contents :(\n",
+ " print(f\"printing won't show us the elements: {outputs}\\n-----\\n\")\n",
+ "\n",
+ " # We can only access the elements through another transform.\n",
+ " # Don't worry if you don't know what's happening here,\n",
+ " # we'll get to it just next :)\n",
+ " outputs | beam.Map(print)"
+ ],
+ "execution_count": 26,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "printing won't show us the elements: PCollection[[26]: Create initial values/Map(decode).None]\n",
+ "-----\n",
+ "\n",
+ "0\n",
+ "1\n",
+ "2\n",
+ "3\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "FY9TP3Tw5eZt",
+ "colab_type": "text"
+ },
+ "source": [
+ "# Transforming data\n",
+ "\n",
+ "Apache Beam is designed with a [functional paradigm](https://en.wikipedia.org/wiki/Functional_programming).\n",
+ "This means that, instead of _loops_, it uses `PTransform`s alongside with _functions_ to process each element in a `PCollection`.\n",
+ "\n",
+ "Let's go through some of the most common and basic data transforms in Beam."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "UMKgi9LD6-xb",
+ "colab_type": "text"
+ },
+ "source": [
+ "## Map -- *one to one*\n",
+ "\n",
+ "Let's say we have some elements and we want to do something with each element.\n",
+ "\n",
+ "We want to `map` a function to each element of the collection.\n",
+ "\n",
+ "`map` takes a _function_ that transforms a single input `a` into a single output `b`.\n",
+ "\n",
+ "> 🐌 -- For example, we want to multiply each element by 2."
Review comment:
What does the 🐌 indicate? I got the sense that "this might be a slow step" from seeing that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on a change in pull request #12892: [BEAM-10937] Add first introductory notebook
Posted by GitBox <gi...@apache.org>.
aaltay commented on a change in pull request #12892:
URL: https://github.com/apache/beam/pull/12892#discussion_r496872298
##########
File path: examples/notebooks/tour-of-beam/getting-started.ipynb
##########
@@ -0,0 +1,1016 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Getting started -- Tour of Beam",
+ "provenance": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python3",
+ "display_name": "Python 3"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/tour-of-beam/getting-started.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "SSKEd7tP-b2k",
+ "cellView": "form"
+ },
+ "source": [
+ "#@title Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
Review comment:
This is showing as a large cell at the top? Do we need this as a large title there?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rosetn commented on pull request #12892: [BEAM-10937] Add first introductory notebook
Posted by GitBox <gi...@apache.org>.
rosetn commented on pull request #12892:
URL: https://github.com/apache/beam/pull/12892#issuecomment-697683282
SGTM. I'm in favor of placements that lets users know where they can find all of the interactive notebooks at try them out. Querying "apache beam notebooks" on the Google search engine or DuckDuckGo, the Try Beam page is in the top 10 results for me: https://beam.apache.org/get-started/try-apache-beam/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidcavazos commented on pull request #12892: [BEAM-10937] Add first introductory notebook
Posted by GitBox <gi...@apache.org>.
davidcavazos commented on pull request #12892:
URL: https://github.com/apache/beam/pull/12892#issuecomment-700773916
Just updated it, forgot to update it yesterday.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rosetn commented on pull request #12892: [BEAM-10937] Add first introductory notebook
Posted by GitBox <gi...@apache.org>.
rosetn commented on pull request #12892:
URL: https://github.com/apache/beam/pull/12892#issuecomment-697653820
Where were we planning on hosting this? On the main landing page https://beam.apache.org/ or Documentation landing page https://beam.apache.org/documentation/ ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidcavazos commented on a change in pull request #12892: [BEAM-10937] Add first introductory notebook
Posted by GitBox <gi...@apache.org>.
davidcavazos commented on a change in pull request #12892:
URL: https://github.com/apache/beam/pull/12892#discussion_r500400418
##########
File path: examples/notebooks/tour-of-beam/getting-started.ipynb
##########
@@ -0,0 +1,1016 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Getting started -- Tour of Beam",
+ "provenance": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python3",
+ "display_name": "Python 3"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/tour-of-beam/getting-started.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "SSKEd7tP-b2k",
+ "cellView": "form"
+ },
+ "source": [
+ "#@title Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
Review comment:
Apparently we can make this an H6 and it appears smaller, I didn't know a `#@title` line was also Markdown.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidcavazos commented on a change in pull request #12892: [BEAM-10937] Add first introductory notebook
Posted by GitBox <gi...@apache.org>.
davidcavazos commented on a change in pull request #12892:
URL: https://github.com/apache/beam/pull/12892#discussion_r494480642
##########
File path: examples/notebooks/tour-of-beam/getting-started.ipynb
##########
@@ -0,0 +1,1021 @@
+{
+ "license": [
+ "Licensed to the Apache Software Foundation (ASF) under one",
+ "or more contributor license agreements. See the NOTICE file",
+ "distributed with this work for additional information",
+ "regarding copyright ownership. The ASF licenses this file",
+ "to you under the Apache License, Version 2.0 (the",
+ "\"License\"); you may not use this file except in compliance",
+ "with the License. You may obtain a copy of the License at",
+ "",
+ " http://www.apache.org/licenses/LICENSE-2.0",
+ "",
+ "Unless required by applicable law or agreed to in writing,",
+ "software distributed under the License is distributed on an",
+ "\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY",
+ "KIND, either express or implied. See the License for the",
+ "specific language governing permissions and limitations",
+ "under the License."
+ ],
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Getting started -- Tour of Beam",
+ "provenance": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python3",
+ "display_name": "Python 3"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/tour-of-beam/getting-started.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5UC_aGanx6oE",
+ "colab_type": "text"
+ },
+ "source": [
+ "# Getting started -- *Tour of Beam*\n",
+ "\n",
+ "[Apache Beam](https://beam.apache.org/get-started/beam-overview/)\n",
+ "is a library for parallel data processing.\n",
+ "\n",
+ "Beam is commonly used for\n",
+ "[Extract-Transform-Load (ETL)](https://en.wikipedia.org/wiki/Extract,_transform,_load)\n",
+ "jobs, where we _extract_ data from a data source, _transform_ that data, and _load_ it into a data sink like a database.\n",
+ "It does particularly well with large amounts of data since it can use mutliple machines to process everything at the same time.\n",
+ "\n",
+ "Let's begin by installing the `apache-beam` package with `pip`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "R_Yhoc6N_Flg",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "source": [
+ "# Install apache-beam with pip.\n",
+ "!pip install --quiet apache-beam"
+ ],
+ "execution_count": 25,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "WwxLB5EiVKP_",
+ "colab_type": "text"
+ },
+ "source": [
+ "You can express a _data processing pipeline_, and then run it on the\n",
+ "[_runner_ of your choice](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "For now, we use the `DirectRunner` which runs locally for simplicity."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "8rfQeLk2y9lx",
+ "colab_type": "text"
+ },
+ "source": [
+ "# What is a _pipeline_?\n",
+ "\n",
+ "A __pipeline__ is a __sequence of data transformations__.\n",
+ "You can think of it like a production line,\n",
+ "data comes in from one end,\n",
+ "it gets transformed by each step.\n",
+ "The outputs from one step are passed as inputs to the next step.\n",
+ "\n",
+ "In Beam, your data lives in a __`PCollection`__,\n",
+ "which stands for _Parallel Collection_.\n",
+ "A `PCollection` is like a __list of elements__,\n",
+ "but without any order guarantees.\n",
+ "This allows Beam to easily parallelize and distribute\n",
+ "the `PCollection`'s elements."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "81VNY478gep_",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "LaljqKvs4UU5",
+ "colab_type": "text"
+ },
+ "source": [
+ "Once you have your data, the next step is to transform it.\n",
+ "In Beam, you transform data using **`PTransform`**s,\n",
+ "which stands for _Parallel Transform_.\n",
+ "A `PTransform` is like a __function__,\n",
+ "they take some inputs, transform them and create some outputs."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Me_hvd6XgtPC",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "SVk4BsuKVlLj",
+ "colab_type": "text"
+ },
+ "source": [
+ "Now let's dive into creating our first pipeline.\n",
+ "\n",
+ "For this first pipeline, let's just feed it some data from a Python list and print the results.\n",
+ "\n",
+ "Each _step_ in the pipeline is delimited by the _pipe operator_ `|`.\n",
+ "The outputs of each transform are passed to the next transform as inputs.\n",
+ "And we can save the final results into a `PCollection` variable.\n",
+ "\n",
+ "```py\n",
+ "# We pass the elements from step1 through step3 and save the results into `outputs`.\n",
+ "outputs = pipeline | step1 | step2 | step3\n",
+ "```\n",
+ "\n",
+ "Pipelines can quickly grow long, so it's sometimes easier to read if we surround them with parentheses and break them into multiple lines.\n",
+ "\n",
+ "```py\n",
+ "# This is equivalent to the example above.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | step1\n",
+ " | step2\n",
+ " | step3\n",
+ ")\n",
+ "```\n",
+ "\n",
+ "Also, Beam expects each transform, or step, to have a unique _label_, or description.\n",
+ "This makes it a lot easier to debug, and it's in general a good practice to start.\n",
+ "You can use the _right shift operator_ `>>` to add a label to your transforms, like `'My description' >> MyTransform`.\n",
+ "\n",
+ "```py\n",
+ "# Try to give short but descriptive labels.\n",
+ "# These serve both as comments and help debug later on.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | 'First step' >> step1\n",
+ " | 'Second step' >> step2\n",
+ " | 'Third step' >> step3\n",
+ ")\n",
+ "```\n",
+ "\n",
+ "> ℹ️ The syntax might seem a little strange at first, but you'll get used to it.\n",
+ "> Once you're familiar with it, you'll see it's actually easy to read and write.\n",
Review comment:
Thank you for the suggestion!
##########
File path: examples/notebooks/tour-of-beam/getting-started.ipynb
##########
@@ -0,0 +1,1021 @@
+{
+ "license": [
+ "Licensed to the Apache Software Foundation (ASF) under one",
+ "or more contributor license agreements. See the NOTICE file",
+ "distributed with this work for additional information",
+ "regarding copyright ownership. The ASF licenses this file",
+ "to you under the Apache License, Version 2.0 (the",
+ "\"License\"); you may not use this file except in compliance",
+ "with the License. You may obtain a copy of the License at",
+ "",
+ " http://www.apache.org/licenses/LICENSE-2.0",
+ "",
+ "Unless required by applicable law or agreed to in writing,",
+ "software distributed under the License is distributed on an",
+ "\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY",
+ "KIND, either express or implied. See the License for the",
+ "specific language governing permissions and limitations",
+ "under the License."
+ ],
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Getting started -- Tour of Beam",
+ "provenance": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python3",
+ "display_name": "Python 3"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/tour-of-beam/getting-started.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5UC_aGanx6oE",
+ "colab_type": "text"
+ },
+ "source": [
+ "# Getting started -- *Tour of Beam*\n",
+ "\n",
+ "[Apache Beam](https://beam.apache.org/get-started/beam-overview/)\n",
+ "is a library for parallel data processing.\n",
+ "\n",
+ "Beam is commonly used for\n",
+ "[Extract-Transform-Load (ETL)](https://en.wikipedia.org/wiki/Extract,_transform,_load)\n",
+ "jobs, where we _extract_ data from a data source, _transform_ that data, and _load_ it into a data sink like a database.\n",
+ "It does particularly well with large amounts of data since it can use mutliple machines to process everything at the same time.\n",
+ "\n",
+ "Let's begin by installing the `apache-beam` package with `pip`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "R_Yhoc6N_Flg",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "source": [
+ "# Install apache-beam with pip.\n",
+ "!pip install --quiet apache-beam"
+ ],
+ "execution_count": 25,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "WwxLB5EiVKP_",
+ "colab_type": "text"
+ },
+ "source": [
+ "You can express a _data processing pipeline_, and then run it on the\n",
+ "[_runner_ of your choice](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "For now, we use the `DirectRunner` which runs locally for simplicity."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "8rfQeLk2y9lx",
+ "colab_type": "text"
+ },
+ "source": [
+ "# What is a _pipeline_?\n",
+ "\n",
+ "A __pipeline__ is a __sequence of data transformations__.\n",
+ "You can think of it like a production line,\n",
+ "data comes in from one end,\n",
+ "it gets transformed by each step.\n",
+ "The outputs from one step are passed as inputs to the next step.\n",
+ "\n",
+ "In Beam, your data lives in a __`PCollection`__,\n",
+ "which stands for _Parallel Collection_.\n",
+ "A `PCollection` is like a __list of elements__,\n",
+ "but without any order guarantees.\n",
+ "This allows Beam to easily parallelize and distribute\n",
+ "the `PCollection`'s elements."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "81VNY478gep_",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "LaljqKvs4UU5",
+ "colab_type": "text"
+ },
+ "source": [
+ "Once you have your data, the next step is to transform it.\n",
+ "In Beam, you transform data using **`PTransform`**s,\n",
+ "which stands for _Parallel Transform_.\n",
+ "A `PTransform` is like a __function__,\n",
+ "they take some inputs, transform them and create some outputs."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Me_hvd6XgtPC",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "SVk4BsuKVlLj",
+ "colab_type": "text"
+ },
+ "source": [
+ "Now let's dive into creating our first pipeline.\n",
+ "\n",
+ "For this first pipeline, let's just feed it some data from a Python list and print the results.\n",
+ "\n",
+ "Each _step_ in the pipeline is delimited by the _pipe operator_ `|`.\n",
+ "The outputs of each transform are passed to the next transform as inputs.\n",
+ "And we can save the final results into a `PCollection` variable.\n",
+ "\n",
+ "```py\n",
+ "# We pass the elements from step1 through step3 and save the results into `outputs`.\n",
+ "outputs = pipeline | step1 | step2 | step3\n",
+ "```\n",
+ "\n",
+ "Pipelines can quickly grow long, so it's sometimes easier to read if we surround them with parentheses and break them into multiple lines.\n",
+ "\n",
+ "```py\n",
+ "# This is equivalent to the example above.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | step1\n",
+ " | step2\n",
+ " | step3\n",
+ ")\n",
+ "```\n",
+ "\n",
+ "Also, Beam expects each transform, or step, to have a unique _label_, or description.\n",
+ "This makes it a lot easier to debug, and it's in general a good practice to start.\n",
+ "You can use the _right shift operator_ `>>` to add a label to your transforms, like `'My description' >> MyTransform`.\n",
+ "\n",
+ "```py\n",
+ "# Try to give short but descriptive labels.\n",
+ "# These serve both as comments and help debug later on.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | 'First step' >> step1\n",
+ " | 'Second step' >> step2\n",
+ " | 'Third step' >> step3\n",
+ ")\n",
+ "```\n",
+ "\n",
+ "> ℹ️ The syntax might seem a little strange at first, but you'll get used to it.\n",
+ "> Once you're familiar with it, you'll see it's actually easy to read and write.\n",
+ "\n",
+ "We use the `Create` transform to feed the pipeline with an\n",
+ "[`iterable`](https://docs.python.org/3/glossary.html#term-iterable)\n",
+ "of elements, like a `list`.\n",
+ "\n",
+ "> ℹ️ _Note:_ in Beam, you can __NOT__ access the elements from a `PCollection` directly like a Python list.\n",
+ "> This means, we can't simply `print` the output `PCollection` to see the elements.\n",
+ ">\n",
+ "> This is because, depending on the runner,\n",
+ "> the `PCollection` elements might live in multiple worker machines.\n",
+ "\n",
+ "We'll do a little trick to print the elements, but we'll explain it shortly."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "XAke4d5lV5f5",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "outputId": "8e032211-c70a-4a89-9904-58d6fa2c8cb7"
+ },
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "inputs = [0, 1, 2, 3]\n",
+ "\n",
+ "# Create a pipeline.\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " # Feed it some input elements with `Create`.\n",
+ " outputs = (\n",
+ " pipeline\n",
+ " | 'Create initial values' >> beam.Create(inputs)\n",
+ " )\n",
+ "\n",
+ " # `outputs` is a PCollection with our input elements.\n",
+ " # But printing it directly won't show us its contents :(\n",
+ " print(f\"printing won't show us the elements: {outputs}\\n-----\\n\")\n",
+ "\n",
+ " # We can only access the elements through another transform.\n",
+ " # Don't worry if you don't know what's happening here,\n",
+ " # we'll get to it just next :)\n",
+ " outputs | beam.Map(print)"
+ ],
+ "execution_count": 26,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "printing won't show us the elements: PCollection[[26]: Create initial values/Map(decode).None]\n",
+ "-----\n",
+ "\n",
+ "0\n",
+ "1\n",
+ "2\n",
+ "3\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "FY9TP3Tw5eZt",
+ "colab_type": "text"
+ },
+ "source": [
+ "# Transforming data\n",
+ "\n",
+ "Apache Beam is designed with a [functional paradigm](https://en.wikipedia.org/wiki/Functional_programming).\n",
+ "This means that, instead of _loops_, it uses `PTransform`s alongside with _functions_ to process each element in a `PCollection`.\n",
+ "\n",
+ "Let's go through some of the most common and basic data transforms in Beam."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "UMKgi9LD6-xb",
+ "colab_type": "text"
+ },
+ "source": [
+ "## Map -- *one to one*\n",
+ "\n",
+ "Let's say we have some elements and we want to do something with each element.\n",
+ "\n",
+ "We want to `map` a function to each element of the collection.\n",
+ "\n",
+ "`map` takes a _function_ that transforms a single input `a` into a single output `b`.\n",
+ "\n",
+ "> 🐌 -- For example, we want to multiply each element by 2."
Review comment:
It was just a cute emoji, but that makes sense. I'm replacing them with ▶️
##########
File path: examples/notebooks/tour-of-beam/getting-started.ipynb
##########
@@ -0,0 +1,1021 @@
+{
+ "license": [
+ "Licensed to the Apache Software Foundation (ASF) under one",
+ "or more contributor license agreements. See the NOTICE file",
+ "distributed with this work for additional information",
+ "regarding copyright ownership. The ASF licenses this file",
+ "to you under the Apache License, Version 2.0 (the",
+ "\"License\"); you may not use this file except in compliance",
+ "with the License. You may obtain a copy of the License at",
+ "",
+ " http://www.apache.org/licenses/LICENSE-2.0",
+ "",
+ "Unless required by applicable law or agreed to in writing,",
+ "software distributed under the License is distributed on an",
+ "\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY",
+ "KIND, either express or implied. See the License for the",
+ "specific language governing permissions and limitations",
+ "under the License."
+ ],
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Getting started -- Tour of Beam",
+ "provenance": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python3",
+ "display_name": "Python 3"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/tour-of-beam/getting-started.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5UC_aGanx6oE",
+ "colab_type": "text"
+ },
+ "source": [
+ "# Getting started -- *Tour of Beam*\n",
+ "\n",
+ "[Apache Beam](https://beam.apache.org/get-started/beam-overview/)\n",
+ "is a library for parallel data processing.\n",
+ "\n",
+ "Beam is commonly used for\n",
+ "[Extract-Transform-Load (ETL)](https://en.wikipedia.org/wiki/Extract,_transform,_load)\n",
+ "jobs, where we _extract_ data from a data source, _transform_ that data, and _load_ it into a data sink like a database.\n",
+ "It does particularly well with large amounts of data since it can use mutliple machines to process everything at the same time.\n",
+ "\n",
+ "Let's begin by installing the `apache-beam` package with `pip`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "R_Yhoc6N_Flg",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "source": [
+ "# Install apache-beam with pip.\n",
+ "!pip install --quiet apache-beam"
+ ],
+ "execution_count": 25,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "WwxLB5EiVKP_",
+ "colab_type": "text"
+ },
+ "source": [
+ "You can express a _data processing pipeline_, and then run it on the\n",
+ "[_runner_ of your choice](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "For now, we use the `DirectRunner` which runs locally for simplicity."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "8rfQeLk2y9lx",
+ "colab_type": "text"
+ },
+ "source": [
+ "# What is a _pipeline_?\n",
+ "\n",
+ "A __pipeline__ is a __sequence of data transformations__.\n",
+ "You can think of it like a production line,\n",
+ "data comes in from one end,\n",
+ "it gets transformed by each step.\n",
+ "The outputs from one step are passed as inputs to the next step.\n",
+ "\n",
+ "In Beam, your data lives in a __`PCollection`__,\n",
+ "which stands for _Parallel Collection_.\n",
+ "A `PCollection` is like a __list of elements__,\n",
+ "but without any order guarantees.\n",
+ "This allows Beam to easily parallelize and distribute\n",
+ "the `PCollection`'s elements."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "81VNY478gep_",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "LaljqKvs4UU5",
+ "colab_type": "text"
+ },
+ "source": [
+ "Once you have your data, the next step is to transform it.\n",
+ "In Beam, you transform data using **`PTransform`**s,\n",
+ "which stands for _Parallel Transform_.\n",
+ "A `PTransform` is like a __function__,\n",
+ "they take some inputs, transform them and create some outputs."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Me_hvd6XgtPC",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "SVk4BsuKVlLj",
+ "colab_type": "text"
+ },
+ "source": [
+ "Now let's dive into creating our first pipeline.\n",
+ "\n",
+ "For this first pipeline, let's just feed it some data from a Python list and print the results.\n",
+ "\n",
+ "Each _step_ in the pipeline is delimited by the _pipe operator_ `|`.\n",
+ "The outputs of each transform are passed to the next transform as inputs.\n",
+ "And we can save the final results into a `PCollection` variable.\n",
+ "\n",
+ "```py\n",
+ "# We pass the elements from step1 through step3 and save the results into `outputs`.\n",
+ "outputs = pipeline | step1 | step2 | step3\n",
+ "```\n",
+ "\n",
+ "Pipelines can quickly grow long, so it's sometimes easier to read if we surround them with parentheses and break them into multiple lines.\n",
+ "\n",
+ "```py\n",
+ "# This is equivalent to the example above.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | step1\n",
+ " | step2\n",
+ " | step3\n",
+ ")\n",
+ "```\n",
+ "\n",
+ "Also, Beam expects each transform, or step, to have a unique _label_, or description.\n",
+ "This makes it a lot easier to debug, and it's in general a good practice to start.\n",
+ "You can use the _right shift operator_ `>>` to add a label to your transforms, like `'My description' >> MyTransform`.\n",
+ "\n",
+ "```py\n",
+ "# Try to give short but descriptive labels.\n",
+ "# These serve both as comments and help debug later on.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | 'First step' >> step1\n",
+ " | 'Second step' >> step2\n",
+ " | 'Third step' >> step3\n",
+ ")\n",
+ "```\n",
+ "\n",
+ "> ℹ️ The syntax might seem a little strange at first, but you'll get used to it.\n",
+ "> Once you're familiar with it, you'll see it's actually easy to read and write.\n",
+ "\n",
+ "We use the `Create` transform to feed the pipeline with an\n",
+ "[`iterable`](https://docs.python.org/3/glossary.html#term-iterable)\n",
+ "of elements, like a `list`.\n",
+ "\n",
+ "> ℹ️ _Note:_ in Beam, you can __NOT__ access the elements from a `PCollection` directly like a Python list.\n",
+ "> This means, we can't simply `print` the output `PCollection` to see the elements.\n",
+ ">\n",
+ "> This is because, depending on the runner,\n",
+ "> the `PCollection` elements might live in multiple worker machines.\n",
+ "\n",
+ "We'll do a little trick to print the elements, but we'll explain it shortly."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "XAke4d5lV5f5",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "outputId": "8e032211-c70a-4a89-9904-58d6fa2c8cb7"
+ },
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "inputs = [0, 1, 2, 3]\n",
+ "\n",
+ "# Create a pipeline.\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " # Feed it some input elements with `Create`.\n",
+ " outputs = (\n",
+ " pipeline\n",
+ " | 'Create initial values' >> beam.Create(inputs)\n",
+ " )\n",
+ "\n",
+ " # `outputs` is a PCollection with our input elements.\n",
+ " # But printing it directly won't show us its contents :(\n",
+ " print(f\"printing won't show us the elements: {outputs}\\n-----\\n\")\n",
+ "\n",
+ " # We can only access the elements through another transform.\n",
+ " # Don't worry if you don't know what's happening here,\n",
+ " # we'll get to it just next :)\n",
+ " outputs | beam.Map(print)"
+ ],
+ "execution_count": 26,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "printing won't show us the elements: PCollection[[26]: Create initial values/Map(decode).None]\n",
+ "-----\n",
+ "\n",
+ "0\n",
+ "1\n",
+ "2\n",
+ "3\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "FY9TP3Tw5eZt",
+ "colab_type": "text"
+ },
+ "source": [
+ "# Transforming data\n",
+ "\n",
+ "Apache Beam is designed with a [functional paradigm](https://en.wikipedia.org/wiki/Functional_programming).\n",
+ "This means that, instead of _loops_, it uses `PTransform`s alongside with _functions_ to process each element in a `PCollection`.\n",
+ "\n",
+ "Let's go through some of the most common and basic data transforms in Beam."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "UMKgi9LD6-xb",
+ "colab_type": "text"
+ },
+ "source": [
+ "## Map -- *one to one*\n",
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidcavazos commented on pull request #12892: [BEAM-10937] Add first introductory notebook
Posted by GitBox <gi...@apache.org>.
davidcavazos commented on pull request #12892:
URL: https://github.com/apache/beam/pull/12892#issuecomment-704416036
@aaltay -- comments addressed, and staged link updated. This should be ready for review
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidcavazos commented on pull request #12892: [BEAM-10937] Add first introductory notebook
Posted by GitBox <gi...@apache.org>.
davidcavazos commented on pull request #12892:
URL: https://github.com/apache/beam/pull/12892#issuecomment-697663116
I was thinking on the "Get Started" section, and probably replace the "Try Beam" button in the main landing page with the "Tour of Beam" page once it's done. The "Try Beam" notebook would also be referenced in the "Tour of Beam". What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rosetn commented on pull request #12892: [BEAM-10937] Add first introductory notebook
Posted by GitBox <gi...@apache.org>.
rosetn commented on pull request #12892:
URL: https://github.com/apache/beam/pull/12892#issuecomment-700861407
I forgot to note this--can you add alt text to these images?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidcavazos commented on a change in pull request #12892: [BEAM-10937] Add first introductory notebook
Posted by GitBox <gi...@apache.org>.
davidcavazos commented on a change in pull request #12892:
URL: https://github.com/apache/beam/pull/12892#discussion_r496143507
##########
File path: examples/notebooks/tour-of-beam/getting-started.ipynb
##########
@@ -0,0 +1,1020 @@
+{
+ "license": [
+ "Licensed to the Apache Software Foundation (ASF) under one",
+ "or more contributor license agreements. See the NOTICE file",
+ "distributed with this work for additional information",
+ "regarding copyright ownership. The ASF licenses this file",
+ "to you under the Apache License, Version 2.0 (the",
+ "\"License\"); you may not use this file except in compliance",
+ "with the License. You may obtain a copy of the License at",
+ "",
+ " http://www.apache.org/licenses/LICENSE-2.0",
+ "",
+ "Unless required by applicable law or agreed to in writing,",
+ "software distributed under the License is distributed on an",
+ "\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY",
+ "KIND, either express or implied. See the License for the",
+ "specific language governing permissions and limitations",
+ "under the License."
+ ],
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Getting started -- Tour of Beam",
Review comment:
Thank for letting me know, I've updated them.
##########
File path: examples/notebooks/tour-of-beam/getting-started.ipynb
##########
@@ -0,0 +1,1020 @@
+{
+ "license": [
+ "Licensed to the Apache Software Foundation (ASF) under one",
+ "or more contributor license agreements. See the NOTICE file",
+ "distributed with this work for additional information",
+ "regarding copyright ownership. The ASF licenses this file",
+ "to you under the Apache License, Version 2.0 (the",
+ "\"License\"); you may not use this file except in compliance",
+ "with the License. You may obtain a copy of the License at",
+ "",
+ " http://www.apache.org/licenses/LICENSE-2.0",
+ "",
+ "Unless required by applicable law or agreed to in writing,",
+ "software distributed under the License is distributed on an",
+ "\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY",
+ "KIND, either express or implied. See the License for the",
+ "specific language governing permissions and limitations",
+ "under the License."
+ ],
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Getting started -- Tour of Beam",
+ "provenance": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python3",
+ "display_name": "Python 3"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/tour-of-beam/getting-started.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5UC_aGanx6oE",
+ "colab_type": "text"
+ },
+ "source": [
+ "# Getting started -- *Tour of Beam*\n",
+ "\n",
+ "[Apache Beam](https://beam.apache.org/get-started/beam-overview/)\n",
+ "is a library for parallel data processing.\n",
+ "\n",
+ "Beam is commonly used for\n",
+ "[Extract-Transform-Load (ETL)](https://en.wikipedia.org/wiki/Extract,_transform,_load)\n",
+ "jobs, where we _extract_ data from a data source, _transform_ that data, and _load_ it into a data sink like a database.\n",
+ "It does particularly well with large amounts of data since it can use mutliple machines to process everything at the same time.\n",
+ "\n",
+ "Let's begin by installing the `apache-beam` package with `pip`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "R_Yhoc6N_Flg",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "source": [
+ "# Install apache-beam with pip.\n",
+ "!pip install --quiet apache-beam"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "WwxLB5EiVKP_",
+ "colab_type": "text"
+ },
+ "source": [
+ "You can express a _data processing pipeline_, and then run it on the\n",
+ "[_runner_ of your choice](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "For now, we use the `DirectRunner` which runs locally for simplicity."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "8rfQeLk2y9lx",
+ "colab_type": "text"
+ },
+ "source": [
+ "# What is a _pipeline_?\n",
+ "\n",
+ "A __pipeline__ is a __sequence of data transformations__.\n",
+ "You can think of it like a production line,\n",
+ "data comes in from one end,\n",
+ "it gets transformed by each step.\n",
+ "The outputs from one step are passed as inputs to the next step.\n",
+ "\n",
+ "In Beam, your data lives in a __`PCollection`__,\n",
+ "which stands for _Parallel Collection_.\n",
+ "A `PCollection` is like a __list of elements__,\n",
+ "but without any order guarantees.\n",
+ "This allows Beam to easily parallelize and distribute\n",
+ "the `PCollection`'s elements."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "81VNY478gep_",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "LaljqKvs4UU5",
+ "colab_type": "text"
+ },
+ "source": [
+ "Once you have your data, the next step is to transform it.\n",
+ "In Beam, you transform data using **`PTransform`**s,\n",
+ "which stands for _Parallel Transform_.\n",
+ "A `PTransform` is like a __function__,\n",
+ "they take some inputs, transform them and create some outputs."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Me_hvd6XgtPC",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "SVk4BsuKVlLj",
+ "colab_type": "text"
+ },
+ "source": [
+ "Now let's dive into creating our first pipeline.\n",
+ "\n",
+ "For this first pipeline, let's just feed it some data from a Python list and print the results.\n",
+ "\n",
+ "Each _step_ in the pipeline is delimited by the _pipe operator_ `|`.\n",
+ "The outputs of each transform are passed to the next transform as inputs.\n",
+ "And we can save the final results into a `PCollection` variable.\n",
+ "\n",
+ "```py\n",
+ "# We pass the elements from step1 through step3 and save the results into `outputs`.\n",
+ "outputs = pipeline | step1 | step2 | step3\n",
+ "```\n",
+ "\n",
+ "Pipelines can quickly grow long, so it's sometimes easier to read if we surround them with parentheses and break them into multiple lines.\n",
+ "\n",
+ "```py\n",
+ "# This is equivalent to the example above.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | step1\n",
+ " | step2\n",
+ " | step3\n",
+ ")\n",
+ "```\n",
+ "\n",
+ "Also, Beam expects each transform, or step, to have a unique _label_, or description.\n",
+ "This makes it a lot easier to debug, and it's in general a good practice to start.\n",
+ "You can use the _right shift operator_ `>>` to add a label to your transforms, like `'My description' >> MyTransform`.\n",
+ "\n",
+ "```py\n",
+ "# Try to give short but descriptive labels.\n",
+ "# These serve both as comments and help debug later on.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | 'First step' >> step1\n",
+ " | 'Second step' >> step2\n",
+ " | 'Third step' >> step3\n",
+ ")\n",
+ "```\n",
+ "\n",
+ "> ℹ️ The syntax might seem a little strange at first, but you'll become familiar with it.\n",
+ "\n",
+ "We use the `Create` transform to feed the pipeline with an\n",
+ "[`iterable`](https://docs.python.org/3/glossary.html#term-iterable)\n",
+ "of elements, like a `list`.\n",
+ "\n",
+ "> ℹ️ In Beam, you can __NOT__ access the elements from a `PCollection` directly like a Python list.\n",
Review comment:
:)
##########
File path: examples/notebooks/tour-of-beam/getting-started.ipynb
##########
@@ -0,0 +1,1020 @@
+{
+ "license": [
+ "Licensed to the Apache Software Foundation (ASF) under one",
+ "or more contributor license agreements. See the NOTICE file",
+ "distributed with this work for additional information",
+ "regarding copyright ownership. The ASF licenses this file",
+ "to you under the Apache License, Version 2.0 (the",
+ "\"License\"); you may not use this file except in compliance",
+ "with the License. You may obtain a copy of the License at",
+ "",
+ " http://www.apache.org/licenses/LICENSE-2.0",
+ "",
+ "Unless required by applicable law or agreed to in writing,",
+ "software distributed under the License is distributed on an",
+ "\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY",
+ "KIND, either express or implied. See the License for the",
+ "specific language governing permissions and limitations",
+ "under the License."
+ ],
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Getting started -- Tour of Beam",
+ "provenance": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python3",
+ "display_name": "Python 3"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/tour-of-beam/getting-started.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5UC_aGanx6oE",
+ "colab_type": "text"
+ },
+ "source": [
+ "# Getting started -- *Tour of Beam*\n",
+ "\n",
+ "[Apache Beam](https://beam.apache.org/get-started/beam-overview/)\n",
+ "is a library for parallel data processing.\n",
+ "\n",
+ "Beam is commonly used for\n",
+ "[Extract-Transform-Load (ETL)](https://en.wikipedia.org/wiki/Extract,_transform,_load)\n",
+ "jobs, where we _extract_ data from a data source, _transform_ that data, and _load_ it into a data sink like a database.\n",
+ "It does particularly well with large amounts of data since it can use mutliple machines to process everything at the same time.\n",
+ "\n",
+ "Let's begin by installing the `apache-beam` package with `pip`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "R_Yhoc6N_Flg",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "source": [
+ "# Install apache-beam with pip.\n",
+ "!pip install --quiet apache-beam"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "WwxLB5EiVKP_",
+ "colab_type": "text"
+ },
+ "source": [
+ "You can express a _data processing pipeline_, and then run it on the\n",
+ "[_runner_ of your choice](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "For now, we use the `DirectRunner` which runs locally for simplicity."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "8rfQeLk2y9lx",
+ "colab_type": "text"
+ },
+ "source": [
+ "# What is a _pipeline_?\n",
+ "\n",
+ "A __pipeline__ is a __sequence of data transformations__.\n",
+ "You can think of it like a production line,\n",
+ "data comes in from one end,\n",
+ "it gets transformed by each step.\n",
+ "The outputs from one step are passed as inputs to the next step.\n",
+ "\n",
+ "In Beam, your data lives in a __`PCollection`__,\n",
+ "which stands for _Parallel Collection_.\n",
+ "A `PCollection` is like a __list of elements__,\n",
+ "but without any order guarantees.\n",
+ "This allows Beam to easily parallelize and distribute\n",
+ "the `PCollection`'s elements."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "81VNY478gep_",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "LaljqKvs4UU5",
+ "colab_type": "text"
+ },
+ "source": [
+ "Once you have your data, the next step is to transform it.\n",
+ "In Beam, you transform data using **`PTransform`**s,\n",
+ "which stands for _Parallel Transform_.\n",
+ "A `PTransform` is like a __function__,\n",
+ "they take some inputs, transform them and create some outputs."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Me_hvd6XgtPC",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "SVk4BsuKVlLj",
+ "colab_type": "text"
+ },
+ "source": [
+ "Now let's dive into creating our first pipeline.\n",
+ "\n",
+ "For this first pipeline, let's just feed it some data from a Python list and print the results.\n",
+ "\n",
+ "Each _step_ in the pipeline is delimited by the _pipe operator_ `|`.\n",
+ "The outputs of each transform are passed to the next transform as inputs.\n",
+ "And we can save the final results into a `PCollection` variable.\n",
+ "\n",
+ "```py\n",
+ "# We pass the elements from step1 through step3 and save the results into `outputs`.\n",
+ "outputs = pipeline | step1 | step2 | step3\n",
+ "```\n",
+ "\n",
+ "Pipelines can quickly grow long, so it's sometimes easier to read if we surround them with parentheses and break them into multiple lines.\n",
+ "\n",
+ "```py\n",
+ "# This is equivalent to the example above.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | step1\n",
+ " | step2\n",
+ " | step3\n",
+ ")\n",
+ "```\n",
+ "\n",
+ "Also, Beam expects each transform, or step, to have a unique _label_, or description.\n",
+ "This makes it a lot easier to debug, and it's in general a good practice to start.\n",
+ "You can use the _right shift operator_ `>>` to add a label to your transforms, like `'My description' >> MyTransform`.\n",
+ "\n",
+ "```py\n",
+ "# Try to give short but descriptive labels.\n",
+ "# These serve both as comments and help debug later on.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | 'First step' >> step1\n",
+ " | 'Second step' >> step2\n",
+ " | 'Third step' >> step3\n",
+ ")\n",
+ "```\n",
+ "\n",
+ "> ℹ️ The syntax might seem a little strange at first, but you'll become familiar with it.\n",
Review comment:
How about "a little different"?
We've heard that the use of custom operators is confusing at first for Python users since those operators are not commonly used, and I wanted to empathize with those users. If you have a better suggestion please let me know, I wasn't too sure on using "strange" to begin with :)
##########
File path: examples/notebooks/tour-of-beam/getting-started.ipynb
##########
@@ -0,0 +1,1020 @@
+{
+ "license": [
+ "Licensed to the Apache Software Foundation (ASF) under one",
+ "or more contributor license agreements. See the NOTICE file",
+ "distributed with this work for additional information",
+ "regarding copyright ownership. The ASF licenses this file",
+ "to you under the Apache License, Version 2.0 (the",
+ "\"License\"); you may not use this file except in compliance",
+ "with the License. You may obtain a copy of the License at",
+ "",
+ " http://www.apache.org/licenses/LICENSE-2.0",
+ "",
+ "Unless required by applicable law or agreed to in writing,",
+ "software distributed under the License is distributed on an",
+ "\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY",
+ "KIND, either express or implied. See the License for the",
+ "specific language governing permissions and limitations",
+ "under the License."
+ ],
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Getting started -- Tour of Beam",
+ "provenance": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python3",
+ "display_name": "Python 3"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/tour-of-beam/getting-started.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5UC_aGanx6oE",
+ "colab_type": "text"
+ },
+ "source": [
+ "# Getting started -- *Tour of Beam*\n",
+ "\n",
+ "[Apache Beam](https://beam.apache.org/get-started/beam-overview/)\n",
+ "is a library for parallel data processing.\n",
+ "\n",
+ "Beam is commonly used for\n",
+ "[Extract-Transform-Load (ETL)](https://en.wikipedia.org/wiki/Extract,_transform,_load)\n",
+ "jobs, where we _extract_ data from a data source, _transform_ that data, and _load_ it into a data sink like a database.\n",
+ "It does particularly well with large amounts of data since it can use mutliple machines to process everything at the same time.\n",
+ "\n",
+ "Let's begin by installing the `apache-beam` package with `pip`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "R_Yhoc6N_Flg",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "source": [
+ "# Install apache-beam with pip.\n",
+ "!pip install --quiet apache-beam"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "WwxLB5EiVKP_",
+ "colab_type": "text"
+ },
+ "source": [
+ "You can express a _data processing pipeline_, and then run it on the\n",
+ "[_runner_ of your choice](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "For now, we use the `DirectRunner` which runs locally for simplicity."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "8rfQeLk2y9lx",
+ "colab_type": "text"
+ },
+ "source": [
+ "# What is a _pipeline_?\n",
+ "\n",
+ "A __pipeline__ is a __sequence of data transformations__.\n",
+ "You can think of it like a production line,\n",
+ "data comes in from one end,\n",
+ "it gets transformed by each step.\n",
+ "The outputs from one step are passed as inputs to the next step.\n",
+ "\n",
+ "In Beam, your data lives in a __`PCollection`__,\n",
+ "which stands for _Parallel Collection_.\n",
+ "A `PCollection` is like a __list of elements__,\n",
+ "but without any order guarantees.\n",
+ "This allows Beam to easily parallelize and distribute\n",
+ "the `PCollection`'s elements."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "81VNY478gep_",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "LaljqKvs4UU5",
+ "colab_type": "text"
+ },
+ "source": [
+ "Once you have your data, the next step is to transform it.\n",
+ "In Beam, you transform data using **`PTransform`**s,\n",
+ "which stands for _Parallel Transform_.\n",
+ "A `PTransform` is like a __function__,\n",
+ "they take some inputs, transform them and create some outputs."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Me_hvd6XgtPC",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "SVk4BsuKVlLj",
+ "colab_type": "text"
+ },
+ "source": [
+ "Now let's dive into creating our first pipeline.\n",
+ "\n",
+ "For this first pipeline, let's just feed it some data from a Python list and print the results.\n",
+ "\n",
+ "Each _step_ in the pipeline is delimited by the _pipe operator_ `|`.\n",
+ "The outputs of each transform are passed to the next transform as inputs.\n",
+ "And we can save the final results into a `PCollection` variable.\n",
+ "\n",
+ "```py\n",
+ "# We pass the elements from step1 through step3 and save the results into `outputs`.\n",
+ "outputs = pipeline | step1 | step2 | step3\n",
+ "```\n",
+ "\n",
+ "Pipelines can quickly grow long, so it's sometimes easier to read if we surround them with parentheses and break them into multiple lines.\n",
+ "\n",
+ "```py\n",
+ "# This is equivalent to the example above.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | step1\n",
+ " | step2\n",
+ " | step3\n",
+ ")\n",
+ "```\n",
+ "\n",
+ "Also, Beam expects each transform, or step, to have a unique _label_, or description.\n",
+ "This makes it a lot easier to debug, and it's in general a good practice to start.\n",
+ "You can use the _right shift operator_ `>>` to add a label to your transforms, like `'My description' >> MyTransform`.\n",
+ "\n",
+ "```py\n",
+ "# Try to give short but descriptive labels.\n",
+ "# These serve both as comments and help debug later on.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | 'First step' >> step1\n",
+ " | 'Second step' >> step2\n",
+ " | 'Third step' >> step3\n",
+ ")\n",
+ "```\n",
+ "\n",
+ "> ℹ️ The syntax might seem a little strange at first, but you'll become familiar with it.\n",
+ "\n",
+ "We use the `Create` transform to feed the pipeline with an\n",
+ "[`iterable`](https://docs.python.org/3/glossary.html#term-iterable)\n",
+ "of elements, like a `list`.\n",
+ "\n",
+ "> ℹ️ In Beam, you can __NOT__ access the elements from a `PCollection` directly like a Python list.\n",
+ "> This means, we can't simply `print` the output `PCollection` to see the elements.\n",
+ ">\n",
+ "> This is because, depending on the runner,\n",
+ "> the `PCollection` elements might live in multiple worker machines.\n",
+ "\n",
+ "We'll do a little trick to print the elements, but we'll explain it shortly."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "XAke4d5lV5f5",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "outputId": "8e032211-c70a-4a89-9904-58d6fa2c8cb7"
+ },
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "inputs = [0, 1, 2, 3]\n",
+ "\n",
+ "# Create a pipeline.\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " # Feed it some input elements with `Create`.\n",
+ " outputs = (\n",
+ " pipeline\n",
+ " | 'Create initial values' >> beam.Create(inputs)\n",
+ " )\n",
+ "\n",
+ " # `outputs` is a PCollection with our input elements.\n",
+ " # But printing it directly won't show us its contents :(\n",
+ " print(f\"printing won't show us the elements: {outputs}\\n-----\\n\")\n",
+ "\n",
+ " # We can only access the elements through another transform.\n",
+ " # Don't worry if you don't know what's happening here,\n",
+ " # we'll get to it just next :)\n",
+ " outputs | beam.Map(print)"
+ ],
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "printing won't show us the elements: PCollection[[26]: Create initial values/Map(decode).None]\n",
Review comment:
You're right, thanks for spotting this. I've separated it into two code cells.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidcavazos commented on a change in pull request #12892: [BEAM-10937] Add first introductory notebook
Posted by GitBox <gi...@apache.org>.
davidcavazos commented on a change in pull request #12892:
URL: https://github.com/apache/beam/pull/12892#discussion_r494480642
##########
File path: examples/notebooks/tour-of-beam/getting-started.ipynb
##########
@@ -0,0 +1,1021 @@
+{
+ "license": [
+ "Licensed to the Apache Software Foundation (ASF) under one",
+ "or more contributor license agreements. See the NOTICE file",
+ "distributed with this work for additional information",
+ "regarding copyright ownership. The ASF licenses this file",
+ "to you under the Apache License, Version 2.0 (the",
+ "\"License\"); you may not use this file except in compliance",
+ "with the License. You may obtain a copy of the License at",
+ "",
+ " http://www.apache.org/licenses/LICENSE-2.0",
+ "",
+ "Unless required by applicable law or agreed to in writing,",
+ "software distributed under the License is distributed on an",
+ "\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY",
+ "KIND, either express or implied. See the License for the",
+ "specific language governing permissions and limitations",
+ "under the License."
+ ],
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Getting started -- Tour of Beam",
+ "provenance": [],
+ "toc_visible": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "name": "python3",
+ "display_name": "Python 3"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/tour-of-beam/getting-started.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5UC_aGanx6oE",
+ "colab_type": "text"
+ },
+ "source": [
+ "# Getting started -- *Tour of Beam*\n",
+ "\n",
+ "[Apache Beam](https://beam.apache.org/get-started/beam-overview/)\n",
+ "is a library for parallel data processing.\n",
+ "\n",
+ "Beam is commonly used for\n",
+ "[Extract-Transform-Load (ETL)](https://en.wikipedia.org/wiki/Extract,_transform,_load)\n",
+ "jobs, where we _extract_ data from a data source, _transform_ that data, and _load_ it into a data sink like a database.\n",
+ "It does particularly well with large amounts of data since it can use mutliple machines to process everything at the same time.\n",
+ "\n",
+ "Let's begin by installing the `apache-beam` package with `pip`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "R_Yhoc6N_Flg",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "source": [
+ "# Install apache-beam with pip.\n",
+ "!pip install --quiet apache-beam"
+ ],
+ "execution_count": 25,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "WwxLB5EiVKP_",
+ "colab_type": "text"
+ },
+ "source": [
+ "You can express a _data processing pipeline_, and then run it on the\n",
+ "[_runner_ of your choice](https://beam.apache.org/documentation/runners/capability-matrix/).\n",
+ "For now, we use the `DirectRunner` which runs locally for simplicity."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "8rfQeLk2y9lx",
+ "colab_type": "text"
+ },
+ "source": [
+ "# What is a _pipeline_?\n",
+ "\n",
+ "A __pipeline__ is a __sequence of data transformations__.\n",
+ "You can think of it like a production line,\n",
+ "data comes in from one end,\n",
+ "it gets transformed by each step.\n",
+ "The outputs from one step are passed as inputs to the next step.\n",
+ "\n",
+ "In Beam, your data lives in a __`PCollection`__,\n",
+ "which stands for _Parallel Collection_.\n",
+ "A `PCollection` is like a __list of elements__,\n",
+ "but without any order guarantees.\n",
+ "This allows Beam to easily parallelize and distribute\n",
+ "the `PCollection`'s elements."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "81VNY478gep_",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "LaljqKvs4UU5",
+ "colab_type": "text"
+ },
+ "source": [
+ "Once you have your data, the next step is to transform it.\n",
+ "In Beam, you transform data using **`PTransform`**s,\n",
+ "which stands for _Parallel Transform_.\n",
+ "A `PTransform` is like a __function__,\n",
+ "they take some inputs, transform them and create some outputs."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Me_hvd6XgtPC",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "SVk4BsuKVlLj",
+ "colab_type": "text"
+ },
+ "source": [
+ "Now let's dive into creating our first pipeline.\n",
+ "\n",
+ "For this first pipeline, let's just feed it some data from a Python list and print the results.\n",
+ "\n",
+ "Each _step_ in the pipeline is delimited by the _pipe operator_ `|`.\n",
+ "The outputs of each transform are passed to the next transform as inputs.\n",
+ "And we can save the final results into a `PCollection` variable.\n",
+ "\n",
+ "```py\n",
+ "# We pass the elements from step1 through step3 and save the results into `outputs`.\n",
+ "outputs = pipeline | step1 | step2 | step3\n",
+ "```\n",
+ "\n",
+ "Pipelines can quickly grow long, so it's sometimes easier to read if we surround them with parentheses and break them into multiple lines.\n",
+ "\n",
+ "```py\n",
+ "# This is equivalent to the example above.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | step1\n",
+ " | step2\n",
+ " | step3\n",
+ ")\n",
+ "```\n",
+ "\n",
+ "Also, Beam expects each transform, or step, to have a unique _label_, or description.\n",
+ "This makes it a lot easier to debug, and it's in general a good practice to start.\n",
+ "You can use the _right shift operator_ `>>` to add a label to your transforms, like `'My description' >> MyTransform`.\n",
+ "\n",
+ "```py\n",
+ "# Try to give short but descriptive labels.\n",
+ "# These serve both as comments and help debug later on.\n",
+ "outputs = (\n",
+ " pipeline\n",
+ " | 'First step' >> step1\n",
+ " | 'Second step' >> step2\n",
+ " | 'Third step' >> step3\n",
+ ")\n",
+ "```\n",
+ "\n",
+ "> ℹ️ The syntax might seem a little strange at first, but you'll get used to it.\n",
+ "> Once you're familiar with it, you'll see it's actually easy to read and write.\n",
Review comment:
Thank you for the suggestion!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #12892: [BEAM-10937] Add first introductory notebook
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #12892:
URL: https://github.com/apache/beam/pull/12892#issuecomment-696425242
/cc @KevinGG
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org